Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: Fastest Text File Load


This question is answered.


Permlink Replies: 5 - Last Post: Mar 15, 2017 5:17 AM Last Post By: Robert Triest
lib z

Posts: 9
Registered: 12/5/99
Fastest Text File Load  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 24, 2017 6:15 AM
Now in 2017, what is the fastest way to load a large text fie to a database? Typically I would use the kind of load below but know I am years out of touch. Would loading it into an array be faster? or something else?
var
F: system.TextFile;
fbuf: array [word] of byte;
....
Field5 := datamodule1.IBCMASTER.FieldByName('UNIQUE_ID');
assignfile(F,myfilename);
reset(F);
System.SetTextBuf(F, fbuf);
readln(F,mytext);
while not eof(F) do
begin
readln(F,mytext);
datamodule1.IBCMASTER.append;
Field1.asString := Copy(mytext,1,5);
Field2.asString := Copy(mytext,7,30);
Field3.asString := Copy(mytext,38,7);
Field4.asString := Copy(mytext,46,5);
Field5.asinteger := strtoint(Copy(mytext,77,8));
datamodule1.IBCMASTER.post;
etc
Robert Triest

Posts: 687
Registered: 3/24/05
Re: Fastest Text File Load
Correct
Click to report abuse...   Click to reply to this thread Reply
  Posted: Feb 28, 2017 5:40 AM   in response to: lib z in response to: lib z
Are you using FireDac?
lib z

Posts: 9
Registered: 12/5/99
Re: Fastest Text File Load  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Mar 14, 2017 12:46 PM   in response to: Robert Triest in response to: Robert Triest
Robert Triest wrote:
Are you using FireDac?
seems to be fastest to use Array DML technique.
Thanks.
Jeff Overcash (...

Posts: 1,529
Registered: 9/23/99
Re: Fastest Text File Load  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Mar 14, 2017 1:15 PM   in response to: lib z in response to: lib z
lib z wrote:
Robert Triest wrote:
Are you using FireDac?
seems to be fastest to use Array DML technique.
Thanks.

If this is against InterBase you might want to look at the IBX batch components.
They can read the file for you (delimited or raw) and use the Array API, which
FireDAC does not so should be significantly faster. It also allows you to read
from one database and use the dataset as the source for the pumping.

--
Jeff Overcash (TeamB)
(Please do not email me directly unless asked. Thank You)
Learning is finding out what you already know. Doing is demonstrating that you
know it. Teaching is reminding others that they know it as well as you. We are
all leaners, doers, teachers. (R Bach)
Robert Triest

Posts: 687
Registered: 3/24/05
Re: Fastest Text File Load  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Mar 15, 2017 12:47 AM   in response to: lib z in response to: lib z
Ian Murphy

Posts: 20
Registered: 2/25/00
Re: Fastest Text File Load
Helpful
Click to report abuse...   Click to reply to this thread Reply
  Posted: Mar 6, 2017 5:19 AM   in response to: lib z in response to: lib z
On Fri, 24 Feb 2017 06:15:03 -0800, lib z <> wrote:

Now in 2017, what is the fastest way to load a large text fie to a database? Typically I would use the kind of load below but know I am years out of touch. Would loading it into an array be faster? or something else?
var
F: system.TextFile;
fbuf: array [word] of byte;
....
Field5 := datamodule1.IBCMASTER.FieldByName('UNIQUE_ID');
assignfile(F,myfilename);
reset(F);
System.SetTextBuf(F, fbuf);
readln(F,mytext);
while not eof(F) do
begin
readln(F,mytext);
datamodule1.IBCMASTER.append;
Field1.asString := Copy(mytext,1,5);
Field2.asString := Copy(mytext,7,30);
Field3.asString := Copy(mytext,38,7);
Field4.asString := Copy(mytext,46,5);
Field5.asinteger := strtoint(Copy(mytext,77,8));
datamodule1.IBCMASTER.post;
etc
Use an inmemory datatable (eg: CDS), Pre define the fields, and just
do load from file (my fav if kbMemTable.

In fact if you have it set right , you set table to active, and it
autoloads by itself from the on disc store.

Regards

Smurf
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02