Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: Puzzle on TClientDataSet "Post" speed : much slower on later passes


This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 1 - Last Post: Apr 15, 2015 8:11 AM Last Post By: Kevin Killion
Kevin Killion

Posts: 19
Registered: 2/3/03
Puzzle on TClientDataSet "Post" speed : much slower on later passes  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 10, 2015 10:21 AM
I am going through a set of 4 weekly files, each with info on some 80,000 people. Most of these people are in all of the weeks, but not all, so I'm using a TClientDataSet to sum things up. This ClientDataSet is sorted as appropriate and LogChanges is set to false.

In the first week, as I pick up data for some 80,000 entries, I create a record for each in the ClientDataSet and initialize various fields.

In weeks 2 through 4, as we pick up the data, we find the appropriate entry (or create it if we haven't seen that person in earlier weeks) and sum it into the fields.

But the puzzle is in the Post! I did some timing checks, and this is what I found:
Week 1: 155 ticks for 81495 calls
Week 2: 13849 ticks for 81210 calls
Week 3: 14062 ticks for 81631 calls
Week 4: 14372 ticks for 81126 calls

Why are the calls to Post taking 100 times as long in weeks 2, 3, 4 than they do in week 1 ? Any suggestions on what I can do to speed things up?

Code excerpts follow.

MANY THANKS,
Kevin Killion

for weekx := 1 to 4 do  {we go through 4 weeks}
    for hx := 1 to MaxHouseholdsInThisWeek do {go thru households in this week}
       householdID := householdCodes(hx); {we retrieve a special code for the household} 
	for px := 1 to MaxPersonsInHH do
	    begin	      
	      if PersonDS.FindKey([householdID, px]) then {record already exists, created for a preceding week}
	        begin
	          PersonDS.Edit;
	        end
	      else {record needs to be created.  Either this is the first week, or this household wasn't in preceding weeks} 
	        begin
	          PersonDS.Append;
	          f_HH.AsInteger := householdID;
	          f_P.AsInteger := px;
 
	          {initialize fields that we are going to be summing up across weeks}
	          f_NWeeks.AsInteger := 0;
	          f_TVB.AsInteger := 0;
	          f_TVC.AsInteger := 0;
	          f_NetHome.AsInteger := 0;
	          f_NetWork.AsInteger := 0;
	        end;
 
 
	      {put this week's XPD personal demo data into PersonDS}
	      << retrieve xpdRecord from a in-memory array>>
	      f_Xpd.AsInteger := xpdRecord.xpdi; {note that this replaces anything picked up in earlier weeks}
 
	      {But other fields are accumulated across the weeks rather than replaced}
	      f_NWeeks.AsInteger := f_NWeeks.AsInteger + 1;
	      f_TVB.AsInteger := f_TVB.AsInteger + xpdRecord.tvb;
	      f_TVC.AsInteger := f_TVC.AsInteger + xpdRecord.tvc;
	      f_NetHome.AsInteger := f_NetHome.AsInteger + xpdRecord.netUse[1];
	      f_NetWork.AsInteger := f_NetWork.AsInteger + xpdRecord.netUse[2];
 
              {THIS is the problem line!!!}
	      PersonDS.Post;
	      
	          (*==============================================
	          TIMING FOR THAT "POST" LINE:
	              I did some timing checks on that single "PersonDS.Post" line, and here are the results
		  for each of the 4 weeks:
	          
	          Week 1:     155 ticks for 81495 calls
	          Week 2:     13849 ticks for 81210 calls
	          Week 3:     14062 ticks for 81631 calls
	          Week 4:     14372 ticks for 81126 calls
	          
	          So, why do weeks 2, 3, and 4 take 100 times as long as week 1 ????
	          ===============================================*)
 
 
	    end;
 
Kevin Killion

Posts: 19
Registered: 2/3/03
Re: Puzzle on TClientDataSet "Post" speed : much slower on later passes  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Apr 15, 2015 8:11 AM   in response to: Kevin Killion in response to: Kevin Killion
I received suggestions from other sources to replace my TClientDataSet with a TFDMemTable. This cut the time for the "Post" calls after an Edit IN HALF!!! So, switching to TFDMemTable is a win.

However, more than 90% of the entire processing time is still spent on the one call, Post.

I'm curious: why is Post after Append fast, but Post after Edit so slow, even for TFDMemTable?
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02