Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: TStringGrid with Unicode/UTF-8


This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 7 - Last Post: Oct 14, 2016 10:09 AM Last Post By: Remy Lebeau (Te... Threads: [ Previous | Next ]
Claude ARDILLER

Posts: 4
Registered: 9/26/13
TStringGrid with Unicode/UTF-8  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 13, 2016 11:14 AM
Hello to all.

I'm currently trying to use a TStringGrid with Unicode/UTF-8 characters (Chinese characters).
I have no problem filling the grid:
procedure TForm1.LOAD(N: TFileName);
begin
Reader := TStreamReader.Create(N, TEncoding.UTF8);
L := TStringList.Create;
L.Text := Reader.ReadToEnd();
Reader.Free();
with StringGrid1 do
  begin // get the number of Cols
  for I := 0 to L.Count -1 do if Length(L[I]) > N then N := Length(L[I]);
  ColCount := N +1;
  RowCount := L.Count +1; // get the number of Rows
  for I := 0 to L.Count -1 do
    begin
    S := L[I];
    for J := 0 to Length(S) -1 do Cells[J, I] := S[J +1];
    end;
  end;
L.Free;
end;

But when I try to retrieve the modified datas of the grid like this:
procedure TForm1.SAVE(N: TFileName);
var I, J, N : Integer;
    S : string;
    L: TStringList;
    Writer: TStreamWriter;
begin
L:= TStringList.Create;
with StringGrid1 do
  begin
  for I := 0 to RowCount -1 do
    begin
    S := '';
    for J := 0 to ColCount -1 do S := S + Cells[J, I];
    L.Add(S);
    end;
  end;
Writer := TStreamWriter.Create(Used, False, TEncoding.UTF8);
Writer.Write(L.Text);
Writer.Free();
end;

It is extremely long to process (about 1 or 2 second for each Row), when it doesn't freeze.

Where could be the issue ?

Any help or advice would be apreciated..
Thanks for reading me.

Friendly yours,
Claude.

PS: I'm using Delphi 10 at a friend's home.
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TStringGrid with Unicode/UTF-8  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 13, 2016 1:08 PM   in response to: Claude ARDILLER in response to: Claude ARDILLER
Claude wrote:

Reader := TStreamReader.Create(N, TEncoding.UTF8);
L := TStringList.Create;
L.Text := Reader.ReadToEnd();
Reader.Free();

That is kind of a waste of a TStreamReader. You could just use System.IOUtils.TFile.ReadAllText()
instead:

L.Text := TFile.ReadAllText(N, TEncoding.UTF8);


Or even use TStringList.LoadFromFile():

L.LoadFromFile(N, TEncoding.UTF8);


However, both approaches are not not very memory-efficient when parsing large
files.

If you continue using TStreamReader, you should at least use its ReadLine()
method instead, so you get a balance between memory usage and parsing overhead:

Reader := TStreamReader.Create(N, TEncoding.UTF8);
try
  while not Reader.EndOfStream do
    L.Add(Reader.ReadLine);
finally
  Reader.Free;
end;


That being said, your Load() method is not freeing the TStringList if an
exception is raised, you need a try/finally for it. And the code you showed
shouldn't even compile, since you are missing variable declarations, and
trying to treat the N variable as an Integer even though it is actually a
String. Try something more like this instead:

procedure TForm1.LOAD(N: TFileName);
var
  Cols: Integer;
  S: String;
  I, J: Integer;
begin
  L := TStringList.Create;
  try
    // load N into L as needed...
 
    // L.LoadFromFile(N, TEncoding.UTF8);
 
    // L.Text := TFile.ReadAllText(N, TEncoding.UTF8);
 
    {
    with TStreamReader.Create(N, TEncoding.UTF8) do
    try
      while not EndOfStream do
        L.Add(ReadLine);
    finally
      Free;
    end;
    }
 
    // get the number of Cols
    Cols := 0;
    for I := 0 to L.Count -1 do
    begin
      if Length(L[I]) > Cols then
        Cols := Length(L[I]);
    end;
 
    StringGrid1.ColCount := Cols+1;
    StringGrid1.RowCount := L.Count+1; // get the number of Rows
 
    for I := 0 to L.Count -1 do
    begin
      S := L[I];
      for J := 0 to Length(S) -1 do
      begin
        StringGrid1.Rows[I].BeginUpdate;
        try
          StringGrid1.Cells[J, I] := S[J +1];
        finally
          StringGrid1.Rows[I].EndUpdate;
        end;
      end;
    end;
  finally
    L.Free;
  end;
end;


Now, that being said, there are similar memory/parformance issues with your
Save() method. Try something more like this instead:

procedure TForm1.SAVE(N: TFileName);
var
  I, J, Len : Integer;
  S, T : string;
  L: TStringList;
  P: PChar;
begin
  L := TStringList.Create;
  try
    for I := 0 to StringGrid1.RowCount -1 do
    begin
      Len := 0;
      for J := 0 to StringGrid1.ColCount -1 do
        Inc(Len, Length(StringGrid1.Cells[J, I]));
 
      SetLength(S, Len);
      P := PChar(S);
 
      for J := 0 to StringGrid1.ColCount -1 do
      begin
        T := StringGrid1.Cells[J, I];
        Move(PChar(T)^, P, Length(T) * SizeOf(Char));
        Inc(P, Length(T));
      end;
 
      L.Add(S);
    end;
 
    // save L to N as needed...
 
    // L.SaveToFile(N, TEncoding.UTF8);
 
    // TFile.WriteAllText(N, L.Text, TEncoding.UTF8);
 
    {
    with TStreamWriter.Create(N, False, TEncoding.UTF8) do
    try
      for I := 0 to L.Count-1 do
        WriteLine(L[I]);
    finally
      Free;
    end;
    }
 
  finally
    L.Free;
  end;
end;


Alternativley, don't even use a TStringList at all, just write directly to
the file while you are looking through the Grid:

procedure TForm1.SAVE(N: TFileName);
var
  I, J, Len : Integer;
  S, T : string;
  P: PChar;
  Writer: TStreamWriter;
begin
  Writer := TStreamWriter.Create(N, False, TEncoding.UTF8);
  try
    for I := 0 to StringGrid1.RowCount -1 do
    begin
      Len := 0;
      for J := 0 to StringGrid1.ColCount -1 do
        Inc(Len, Length(StringGrid1.Cells[J, I]));
 
      SetLength(S, Len);
      P := PChar(S);
 
      for J := 0 to StringGrid1.ColCount -1 do
      begin
        T := StringGrid1.Cells[J, I];
        Move(PChar(T)^, P, Length(T) * SizeOf(Char));
        Inc(P, Length(T));
      end;
 
      Writer.WriteLine(S);
    end;
  finally
    Writer.Free;
  end;
end;


Or even:

procedure TForm1.SAVE(N: TFileName);
var
  I, J, Len : Integer;
  S : string;
  Writer: TStreamWriter;
  SB: TStringBuilder;
begin
  Writer := TStreamWriter.Create(N, False, TEncoding.UTF8);
  try
    SB := TStringBuilder.Create;
    try
      for I := 0 to StringGrid1.RowCount -1 do
      begin
        SB.Length := 0;
        SB.Capacity := StringGrid1.ColCount;
        for J := 0 to StringGrid1.ColCount -1 do
          SB.Append(StringGrid1.Cells[J, I]);
        Writer.WriteLine(SB.ToString);
      end;
    finally
      SB.Free;
    end;
  finally
    Writer.Free;
  end;
end;


--
Remy Lebeau (TeamB)
Claude ARDILLER

Posts: 4
Registered: 9/26/13
Re: TStringGrid with Unicode/UTF-8  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 13, 2016 1:53 PM   in response to: Claude ARDILLER in response to: Claude ARDILLER
Thank you for your numerous advices, Remi !
I tried all your saving methods:
procedure TForm1.SAVE(N: TFileName);
var
  I, J, Len : Integer;
  S, T : string;
  L: TStringList;
  P: PChar;
begin
  L := TStringList.Create;
  try
    for I := 0 to StringGrid1.RowCount -1 do
    begin
      Len := 0;
      for J := 0 to StringGrid1.ColCount -1 do
        Inc(Len, Length(StringGrid1.Cells[J, I]));
 
      SetLength(S, Len);
      P := PChar(S);
 
      for J := 0 to StringGrid1.ColCount -1 do
      begin
        T := StringGrid1.Cells[J, I];
        Move(PChar(T)^, P, Length(T) * SizeOf(Char));
        Inc(P, Length(T));
      end;
 
      L.Add(S);
    end;
 
    // save L to N as needed...
 
    // L.SaveToFile(N, TEncoding.UTF8);
 
    // TFile.WriteAllText(N, L.Text, TEncoding.UTF8);
 
    {
    with TStreamWriter.Create(N, False, TEncoding.UTF8) do
    try
      for I := 0 to L.Count-1 do
        WriteLine(L[I]);
    finally
      Free;
    end;
    }
 
  finally
    L.Free;
  end;
end;

26-28 sec to save a file of 3 KO..
procedure TForm1.SAVE2(N: TFileName);
var
  I, J, Len : Integer;
  S, T : string;
  P: PChar;
  Writer: TStreamWriter;
begin
  Writer := TStreamWriter.Create(N, False, TEncoding.UTF8);
  try
    for I := 0 to StringGrid1.RowCount -1 do
    begin
      Len := 0;
      for J := 0 to StringGrid1.ColCount -1 do
        Inc(Len, Length(StringGrid1.Cells[J, I]));
 
      SetLength(S, Len);
      P := PChar(S);
 
      for J := 0 to StringGrid1.ColCount -1 do
      begin
        T := StringGrid1.Cells[J, I];
        Move(PChar(T)^, P, Length(T) * SizeOf(Char));
        Inc(P, Length(T));
      end;
 
      Writer.WriteLine(S);
    end;
  finally
    Writer.Free;
  end;
end;

26-28 sec to save a BLANK file..
 
procedure TForm1.SAVE(N: TFileName);
var
  I, J, Len : Integer;
  S : string;
  Writer: TStreamWriter;
  SB: TStringBuilder;
begin
  Writer := TStreamWriter.Create(N, False, TEncoding.UTF8);
  try
    SB := TStringBuilder.Create;
    try
      for I := 0 to StringGrid1.RowCount -1 do
      begin
        SB.Length := 0;
        SB.Capacity := StringGrid1.ColCount;
        for J := 0 to StringGrid1.ColCount -1 do
          SB.Append(StringGrid1.Cells[J, I]);
        Writer.WriteLine(SB.ToString);
      end;
    finally
      SB.Free;
    end;
  finally
    Writer.Free;
  end;
end;

14-16 sec to save a file of 3 KO.. about the same duration than the method I originally posted.

I'm not trying at a big file here, but it still takes forever to get those Unicode strings from the StringGrid, while the saving itself is fast enough.
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TStringGrid with Unicode/UTF-8  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 13, 2016 2:14 PM   in response to: Claude ARDILLER in response to: Claude ARDILLER
Claude wrote:

26-28 sec to save a file of 3 KO..

What does "3 KO" mean? 3 rows? 3000 rows?

That first code you showed doesn't even touch the output file, so that time
would have to be spent simply looping through the StringGrid. Which is not
wholly unexpected, because TStringGrid is not a very efficient component
in general. Its internal management of colums/rows and their strings is
horrible.

I suggest you use a profiler, such as AQTime, to figure out where your code
is actually spending its time, and then you can adjust the code accordingly.

Personally, I wouldn't ue TStringGrid at all. I would use a virtual TListView
in report mode instead. Or even a TVirtualTreeView in grid mode. Or even
a TDrawGrid with custom string handling would be better than TStringGrid.

--
Remy Lebeau (TeamB)
Claude ARDILLER

Posts: 4
Registered: 9/26/13
Re: TStringGrid with Unicode/UTF-8  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 13, 2016 2:35 PM   in response to: Claude ARDILLER in response to: Claude ARDILLER
3 KO -> 3 Kilo Octets (very small file).
Even if it is not an efficient component, it should not take 15 seconds to get the unicode single-chararcter strings from a 20 rows/20 cols TStringGrid..
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TStringGrid with Unicode/UTF-8  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 13, 2016 3:31 PM   in response to: Claude ARDILLER in response to: Claude ARDILLER
Claude wrote:

3 KO -> 3 Kilo Octets (very small file).
Even if it is not an efficient component, it should not take 15
seconds to get the unicode single-chararcter strings from a 20
rows/20 cols TStringGrid..

Strings are reference-counted and copy-on-write, so simply looping through
the grid and reading its strings should be fairly fast. I suspect that the
majority of the time is being spent on temporary memory allocations. Using
the TStringBuilder helps reduce the time spent by 12 seconds because a single
memory buffer is being used to gather the column strings and not allocate
on each append. But the TStringBuilder.ToString() and TStreamWriter.WriteLine()
calls are still allocating and freeing memory every time. You might try
reducing the memory allocations to see if it helps speed up the code:

type
  TStringBuilderAccess = class(TStringBuilder)
  end;
 
procedure TForm1.SAVE(N: TFileName);
const
  uLineBreak: UTF8String = sLineBreak;
var
  I, J, ULen : Integer;
  FS: TStream;
  SB: TStringBuilder;
  Bytes: TBytes;
  Enc: TEncoding:
begin
  FS := TFileStream.Create(N, fmCreate); // or even TBufferedFileStream
  try
    SB := TStringBuilder.Create(StringGrid1.ColCount);
    try
      Enc := TEncoding.UTF8;
 
      for I := 0 to StringGrid1.RowCount -1 do
      begin
        SB.Length := 0;
        for J := 0 to StringGrid1.ColCount -1 do
          SB.Append(StringGrid1.Cells[J, I]);
 
        if SB.Length > 0 then
        begin
          ULen := Enc.GetMaxByteCount(SB.Length);
          if Length(Bytes) < ULen then SetLength(Bytes, ULen);
          FS.WriteBuffer(PByte(Bytes)^, Enc.GetBytes(TStringBuilderAccess(SB).FData, 
0, SB.Length, Bytes, 0));
        end;
 
        FS.WriteBuffer(PAnsiChar(uLineBreak)^, Length(uLineBreak));
      end;
    finally
      SB.Free;
    end;
  finally
    FS.Free;
  end;
end;


--
Remy Lebeau (TeamB)
Claude ARDILLER

Posts: 4
Registered: 9/26/13
Re: TStringGrid with Unicode/UTF-8  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 13, 2016 9:32 PM   in response to: Claude ARDILLER in response to: Claude ARDILLER
Thank you, Remi.
Tried it a few dfferent time: 16-18 sec, still longer than the 14-16 sec.
Considering the size of the grid (400 cells), it should not take longer than a couple of seconds to get 400 UTF-8 chinese characters..
I'll try a different way, maybe an iconless TListview..

Thanks anyway for your input.
Friendly yours,
Claude.
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: TStringGrid with Unicode/UTF-8  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Oct 14, 2016 10:09 AM   in response to: Claude ARDILLER in response to: Claude ARDILLER
Claude wrote:

Tried it a few dfferent time: 16-18 sec, still longer than
the 14-16 sec.

Then you really need to profile the code to find where the time is being
spent. Stop guessing and let a profiler tell you.

--
Remy Lebeau (TeamB)
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02