Watch, Follow, &
Connect with Us

Please visit our new home
community.embarcadero.com.


Welcome, Guest
Guest Settings
Help

Thread: TMongoCollection insert out of memory error, thousends records iterated


This question is answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 2 - Last Post: Sep 2, 2016 3:25 AM Last Post By: Gaetano Mantovani
Gaetano Mantovani

Posts: 19
Registered: 8/30/14
TMongoCollection insert out of memory error, thousends records iterated  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 30, 2016 3:23 AM
Hi all,
I followed the wiki here
http://docwiki.embarcadero.com/Libraries/Seattle/en/FireDAC.Phys.MongoDBWrapper.TMongoCollection.Insert
and the example contained in the folder CPP Embarcadero\Studio\18.0\Samples\CPP\Database\FireDAC\Samples\DBMS Specific\MongoDB\General

My Goal is to import tables stored in a MySQL Database in a new MongoDB database but the final structure will be not exactly the same. So I have to create the inserting manually. The little tables have no problems but if I iterated more 3600~3700 records (also in different queries) I faced an exception 'out of memory'

To be more detailed this is the actual issue:

I have a query in MySQL that select all the records from a table and I iterate the results with a while cicle, during the iterate I create a new structure in a MongoDB collection and insert the new records in the collection.
When I reach approximately 3700 items the compiler raised the exeption 'out of memory.'.This table has 10.000 records but I have other tables with 500.000 !

I tried to divide the insert in cicles of 3000 items but if I run for example 2 functions that insert first 3000 items and second the 3001 to 6000 when I reach the quantity 3700~ the out of memory error appears. If I run the first 3000 than I terminate the program then run only the second function inserted without clean the MongoDB collection all seems work.

see below my sample code to be clearly:

void __fastcall TMainDFM::AddPeople(int limit1,int limit2)
{
TMongoCollection * oCol = FConR->GetCollection("MyDB", "People");
if (limit1==0) {
oCol->RemoveAll();//
}

Dm->FDQuery1->Close();
Dm->FDQuery1->SQL->Clear();
Dm->FDQuery1->SQL->Add("select * from People limit "IntToStr(limit1)","+IntToStr(limit2));
Dm->FDQuery1->Open();
Dm->FDQuery1->First();

while (!Dm->FDQuery1->Eof) {
// std::unique_ptr<TMongoDocument> oDoc(FEnvR->NewDoc()); //error raised with unique_ptr too
TMongoDocument* oDoc = FEnvR->NewDoc();
oDoc
->Add("Surname", Dm->FDQuery1->FieldByName("SURNAME")->AsString)
->Add("Name", Dm->FDQuery1->FieldByName("NAME")->AsString)
//other Adds with objectS, arrays, fields etc etc (complex structure)
.
.
.
oCol->Insert(oDoc);//
delete oDoc;
Dm->FDQuery1->Next();
}
Memo1->Lines->Add("Record count " + IntToStr((__int64)oCol->Count()->Value())); //
delete oCol;
Dm->FDQuery1->Close();
}

I tried with BeginBulk/EndBulk, with unique pointer without ... etc etc no solutions, the only way is run the inserting only 3600~ record at time
Thanks
Gaetano Mantovani

Posts: 19
Registered: 8/30/14
Re: TMongoCollection insert out of memory error, thousends records iterated  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 31, 2016 5:25 AM   in response to: Gaetano Mantovani in response to: Gaetano Mantovani
Only to add more information the number of the records (documents) inserted depend to the compexity of the document's structure.
If I remove objects or arrays in the main document's structure the function can insert more documents.
I tried to change the dimension of the virtual memory of my pc but the behavior is the same or rather number of inserting was changed but not so much.

Thanks again for the replies
Gaetano Mantovani

Posts: 19
Registered: 8/30/14
Re: TMongoCollection insert out of memory error, thousends records iterated  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Sep 2, 2016 3:25 AM   in response to: Gaetano Mantovani in response to: Gaetano Mantovani
Issue seems solved, my fault :P
the problem was caused from a function that I use to retrieve the Object Id from a 'master collection', before the return of the value I didn't release the cursor and delete the collection.
I think that for this reason each cicle of the iterate creates a new cursor without release it.
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02