Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: A dataset written directly to Indy socket as fast as hell!


This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 10 - Last Post: Aug 27, 2016 10:24 AM Last Post By: Ahmed Sayed
Ahmed Sayed

Posts: 173
Registered: 8/9/07
A dataset written directly to Indy socket as fast as hell!  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 26, 2016 1:18 PM
Hello everyone,

I am trying to make something crazy here and i hope it really
works as it well make a big difference in my http server app
performance and memory consumption.

The is to achieve something like this but for a TDataSet descendant.
I am test with a local access database and a table the contains around
50,000 records to make a stress test.

void __fastcall TForm1::IdHTTPServer1CommandGet(TIdContext *AContext, 
TIdHTTPRequestInfo *ARequestInfo, 
TIdHTTPResponseInfo *AResponseInfo)
{
AResponseInfo->ContentStream = new TFileStream("test.jpg", fmOpenRead 
| fmShareDenyWrite);
AResponseInfo->ContentType = "image/jpeg";
AResponseInfo->ContentLength = AResponseInfo->ContentStream->Size;
} 


As all of you can see that the file stream is written directly to
the socket and is not loaded into memory at all.

Now i tried to do the same thing with FireDAC and using
the same mechanism but it was taking so long.

I used "TIdTCPStream" to write directly to socket like this:

IdTCPClient1->Connect();
 
 
if (IdTCPClient1->Connected())
	{
	TIdTCPStream *Stream = new TIdTCPStream(IdTCPClient1, 0);
 
	try
		{
		FDQuery1->Close();
		FDQuery1->SQL->Text = Memo1->Lines->Text.Trim();
		TimingStart();
 
		FDQuery1->Open();
		FDQuery1->SaveToStream(Stream, sfJSON);
 
		Label1->Caption = "Timing: " + String(TimingSeconds());
		}
	__finally
		{
		Stream->Free();
        IdTCPClient1->Disconnect();
		}
	}


That took "80" seconds to finish, on the other hand
saving data to a TMemoryStream first then copying it
to TIdTCPStream using CopyFrom was a lot faster:

IdTCPClient1->Connect();
 
if (IdTCPClient1->Connected())
	{
	TIdTCPStream *Stream = new TIdTCPStream(IdTCPClient1, 0);
	TMemoryStream *str = new TMemoryStream;
 
	try
		{
		FDQuery1->Close();
		FDQuery1->SQL->Text = Memo1->Lines->Text.Trim();
		TimingStart();
 
		FDQuery1->Open();
		FDQuery1->SaveToStream(str, sfJSON);
		str->Position = 0;
		Stream->CopyFrom(str, str->Size);
 
		Label1->Caption = "Timing: " + String(TimingSeconds());
		}
	__finally
		{
		str->Free();
		Stream->Free();
                IdTCPClient1->Disconnect();
		}
	}


That took only one second to finish but it used a
lot amount of memory shown in task manager like (250 - 350 MB)

I know the test is using a TCP client but this will actually be used
on an http server.

I mentioned the TFileStream before because i want to achieve
something similar.

1- After query is executed no data should be fetched in memory.

2- Data must be written very fast to either (response stream or
TIdTCPStream).

3- Data must be in a JSON format by using FireDAC format or
as json array.

Any ideas or help will be really and very appreciated.

--
The limits of my language mean the limits of my world
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: A dataset written directly to Indy socket as fast as hell!  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 26, 2016 2:58 PM   in response to: Ahmed Sayed in response to: Ahmed Sayed
Ahmed wrote:

AResponseInfo->ContentLength = AResponseInfo->ContentStream->Size;

FYI, you don't need to set the ContentLength manually, it will be calculated
automatically if not provided.

TIdTCPStream *Stream = new TIdTCPStream(IdTCPClient1, 0);

You are setting the AWriteThreshold parameter to 0, which means write buffering
will not be used. Every individual write to the stream, no matter its size,
will be written as-is directly to the socket. And also keep in mind that
a socket has the Nagle algorithm enabled by default, so the socket will buffer
small amounts of data internally and not transmit it in real-time. I don't
know how much data TFDQuery::SaveToStream() writes to the target stream on
each Write() call, but you might try either 1) disabling Nagle on the TIdTCPClient,
and/or 2) enabling write buffering on the TIdStream so the DB output data
gets cached in memory first and periodically written to the socket in batches
that would have a chance to counter Nagle.

Stream->Free();

Don't call Free() in C++. Use the 'delete' operator instead:

delete Stream;


That took "80" seconds to finish, on the other hand saving data to a
TMemoryStream first then copying it to TIdTCPStream using CopyFrom was
a lot faster:

Writing to a TMemoryStream is relatively fast, depending on how many reallocations
it has to perform during writing (I'm assuming TFDQuery::SaveToStream() does
not preallocate a stream's Size, which TIdTCPStream does not support). Also,
TStream::CopyFrom() copies data from one stream to the other in 60K chunks,
so writing data to the socket in 6K chunks will be more efficient then, say,
256B at a time.

1- After query is executed no data should be fetched in memory.

2- Data must be written very fast to either (response stream or
TIdTCPStream).

Since the data is dynamic in nature, you would not be able to use the AResponseInfo->ContentStream
unless you save the entire DB data to memory/disk first. Streaming dynamic
data to the client in real-time requires HTTP chunking, which TIdHTTPServer
does not currently support, but you can implement it manually (provided the
client supports HTTP 1.1, since chunking is not available in HTTP 1.0).
For example:

#include <memory>
 
void __fastcall TForm1::WriteChunk(const TIdBytes ABuffer, int AOffset, int 
ACount, int &VResult)
{
    TIdIOHandler *IO = (TIdIOHandler*) this;
    IO->WriteLn(IntToHex(ACount, 1));
    IO->Write(ABuffer, ACount, AOffset);
    IO->WriteLn();
    VResult = ACount;
}
 
void __fastcall TForm1::IdHTTPServer1CommandGet(TIdContext *AContext, TIdHTTPRequestInfo 
*ARequestInfo, TIdHTTPResponseInfo *AResponseInfo)
{
    ...
 
    FDQuery->Close();
    FDQuery->SQL->Text = ...;
    FDQuery1->Open();
 
    AResponseInfo->ResponseNo = 200;
    AResponseInfo->ContentType = "application/json";
 
    if (ARequestInfo->IsVersionAtLeast(1, 1))
    {
        AResponseInfo->TransferEncoding = "chunked";
 
        TIdIOHandler *IO = AContext->Connection->IOHandler;
 
        TIdStreamWriteEvent WriteHandler = &WriteChunk;
        reinterpret_cast<TMethod&>(WriteHandler).Data = IO; // trick to pass 
the IOHandler as the 'this' pointer of WriteChunk()...
 
        std::auto_ptr<TIdEventStream> Stream(new TIdEventStream);
        Stream->OnWrite = WriteHandler;
 
        IO->WriteBufferOpen(1024*64);
        try
        {
            AResponseInfo->WriteHeader();
            FDQuery->SaveToStream(Stream.get(), sfJSON);
            IO->WriteLn("0");
            IO->WriteLn();
            IO->WriteBufferClose();
        }
        catch (const Exception &)
        {
            IO->WriteBufferCancel();
            throw;
        }
    }
    else
    {
        AResponseInfo->ContentStream = new TMemoryStream;
        FDQuery->SaveToStream(AResponseInfo->ContentStream, sfJSON);
        AResponseInfo->ContentStream->Position = 0;
    }
 
    ...
} 


Alternatively:

class TChunkStream : public TIdBaseStream
{
private:
    TIdIOHandler *IO;
    TIdBytes SendBuf;
    DWORD BufPos;
 
protected:
    int __fastcall IdRead(TIdBytes &VBuffer, int AOffset, int ACount)
    {
        return 0;
    }
 
    int __fastcall IdWrite(const TIdBytes ABuffer, int AOffset, int ACount)
    {
        while (ACount > 0)
        {
            if (BufPos >= SendBuf.Length)
            {
                IO->WriteLn(IntToHex(SendBuf.Length, 1));
                IO->Write(SendBuf);
                IO->WriteLn();
                BufPos = 0;
            }
 
            int NumBytes = IndyMin(SendBuf.Length - BufPos, ACount);
            System::Move(&ABuffer[AOffset], &SendBuf[BufPos], NumBytes);
            BufPos += NumBytes;
            AOffset += NumBytes;
            ACount -= NumBytes;
        }
    }
 
    __int64 __fastcall IdSeek(const __int64 AOffset, TSeekOrigin AOrigin)
    {
        return 0;
    }
 
    void __fastcall IdSetSize(__int64 ASize) {}
 
public:
    __fastcall TChunkStream(TIdIOHandler *AIOHandler, DWORD ABufSize)
        : TIdBaseStream(), IO(AIOHandler), BufPos(0)
    {
        SendBuf.Length = ABufSize;
    }
 
    void Finish()
    {
        if (BufPos > 0)
        {
            IO->WriteLn(IntToHex(BufPos, 1));
            IO->Write(SendBuf, BufPos);
            IO->WriteLn();
        }
        IO->WriteLn("0");
        IO->WriteLn();
    }
};
 
#include <memory>
 
void __fastcall TForm1::IdHTTPServer1CommandGet(TIdContext *AContext, TIdHTTPRequestInfo 
*ARequestInfo, TIdHTTPResponseInfo *AResponseInfo)
{
    ...
 
    FDQuery->Close();
    FDQuery->SQL->Text = ...;
    FDQuery1->Open();
 
    AResponseInfo->ResponseNo = 200;
    AResponseInfo->ContentType = "application/json";
 
    if (ARequestInfo->IsVersionAtLeast(1, 1))
    {
        AResponseInfo->TransferEncoding = "chunked";
 
        std::auto_ptr<TChunkStream> Stream(new TChunkStream(AContext->Connection->IOHandler, 
1024*64));
 
        AResponseInfo->WriteHeader();
        FDQuery->SaveToStream(Stream.get(), sfJSON);
        Stream->Finish();
    }
    else
    {
        AResponseInfo->ContentStream = new TMemoryStream;
        FDQuery->SaveToStream(AResponseInfo->ContentStream, sfJSON);
        AResponseInfo->ContentStream->Position = 0;
    }
 
    ...
} 


--
Remy Lebeau (TeamB)
Ahmed Sayed

Posts: 173
Registered: 8/9/07
Re: A dataset written directly to Indy socket as fast as hell!  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 26, 2016 4:08 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Thanks for the fast response.

But your technique will still open the whole table into
memory a "50,000" records.

Also, Which of the two techniques will be faster on for
the whole process? The chunked or using ContentStream?

I tried your first method but nothing is returned in the response!!
When i tried it with rest debugger it gave me this error:

REST request failed: Error querying headers (12019) the handle is
in the wrong state for the request operation.

--
The limits of my language mean the limits of my world
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: A dataset written directly to Indy socket as fast as hell!  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 26, 2016 4:50 PM   in response to: Ahmed Sayed in response to: Ahmed Sayed
Ahmed wrote:

But your technique will still open the whole table into memory a
"50,000" records.

Only in the specific case where the client sends an HTTP 1.0 request, since
chunking is not possible in HTTP 1.0. In HTTP 1.1, the record data is streamed
to the socket in a chunked format as it is being generated by SaveToStream().

You can stream the response body to an HTTP 1.0 client, but not in a chunked
format, and the only way to signal the end of the stream is to close the
connection:

#include <memory>
 
void __fastcall TForm1::IdHTTPServer1CommandGet(TIdContext *AContext, TIdHTTPRequestInfo 
*ARequestInfo, TIdHTTPResponseInfo *AResponseInfo)
{
    ...
 
    FDQuery->Close();
    FDQuery->SQL->Text = ...;
    FDQuery1->Open();
 
    AResponseInfo->ResponseNo = 200;
    AResponseInfo->ContentType = "application/json";
 
    if (ARequestInfo->IsVersionAtLeast(1, 1))
    {
        // HTTP 1.1 chunking logic here ...
    }
    else
    {
        // HTTP 1.0 logic here ...
 
        // note: NOT a blank string! Otherwise WriteHeader() will change
        // ContentLength to 0, which we don't want to happen....
        AResponseInfo->TransferEncoding = " ";
 
        AResponseInfo->ContentLength = -1;
        AResponseInfo->CloseConnection = true;
 
        std::auto_ptr<TIdTCPStream> Stream(new TIdTCPStream(AContext->Connection, 
1024*64));
 
        AResponseInfo->WriteHeader();
        FDQuery->SaveToStream(Stream.get(), sfJSON);
    }
 
    ...
} 

Also, Which of the two techniques will be faster on for the
whole process?

You will have to profile the code and find out for yourself.

The chunked or using ContentStream?

Chunking sends the data over the socket in real-time as it is being generated.
Using ContentStream requires buffering the entire data before then sending
it.

I tried your first method but nothing is returned in the response!!

I did not test the code before posting it. I'm just giving you a general
overview of what is needed. You will have to debug the code yourself if
you are having problems with it.

When i tried it with rest debugger it gave me this error:

REST request failed: Error querying headers (12019) the
handle is in the wrong state for the request operation.

That is not an Indy error message. Maybe it is a bug in the REST debugger
itself. You will have to figure out what the REST debugger is expecting,
and then make sure your server is actually sending it, and not crashing along
the way.

--
Remy Lebeau (TeamB)
Ahmed Sayed

Posts: 173
Registered: 8/9/07
Re: A dataset written directly to Indy socket as fast as hell!  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 26, 2016 6:26 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
I added this line:

IO->WriteLn();


And now it works but it is still taking so long like "55" seconds
to fetch data.

I know that the first technique for version 1.1 is not saving data
into memory stream but FireDAC does. As i mentioned before
from 250 to 350 MB.

That's why i mentioned the TFileStream example before as i don't
want any kind of data to be stored first in memory then to be send
to socket. As in file stream case it is like from Hard disk to socket.
I want to make the same thing here from rdbms to socket but in json
format. Whether using FireDAC/ADO/UniDAC with streams or non-streams
using TJSONObject directly or another framewor. But it has to be fast
and do not consume a lot of memory.

I think Embarcadero should create these objects as they will be very
handy for RAD Server or web services in general:

TCompressedStream //But without the need for another stream in constractor
//I mean to be able to write compressed data directly to it.
 
TJSONStream // With functions to StartObject, StartArray.
//I know that there is TJSONBuilder, TJSONWriter. But having a json stream
//will save a lot of time.
 
//Change FastMM for a better memory manager because it sucks
//I know FastMM is doing good job but lets face it. It is not the best
 
//Add a function to TDataSet to save records as json array very fast as it is 
//required now by all web services.

--
The limits of my language mean the limits of my world

Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: A dataset written directly to Indy socket as fast as hell!  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 26, 2016 6:49 PM   in response to: Ahmed Sayed in response to: Ahmed Sayed
Ahmed wrote:

I added this line:

IO->WriteLn();


And now it works

You are right, I missed that after the WriteLn("0") calls. I have added
it in to my earlier examples now.

but it is still taking so long like "55" seconds to fetch data.

Did you profile the code to find out exactly where the time is actually being
spent?

I know that the first technique for version 1.1 is not saving data
into memory stream but FireDAC does. As i mentioned before from
250 to 350 MB.

Your earlier statement about that issue was in relation to your example that
saved the DB data to your own TMemoryStream and then sent that TMemoryStream
with Indy. I could easily see 50000 saved records taking up that much memory
in an output stream (5.12K - 7.168K per record). But if FireDAC itself is
allocating that much memory **internally**, regardless of what kind of output
stream you use, then 1) I would consider that a major FireDAC bug, and 2)
you would need to find a different solution to your DB access. Either way,
that is not an Indy issue.

TJSONStream With functions to StartObject, StartArray.
I know that there is TJSONBuilder, TJSONWriter. But having a
json stream will save a lot of time.

TJSONWriter writes its output to a designated TTextWriter. There is already
a TStreamWriter class, which is a TTextWriter descendant that writes to a
designated TStream.

--
Remy Lebeau (TeamB)
Ahmed Sayed

Posts: 173
Registered: 8/9/07
Re: A dataset written directly to Indy socket as fast as hell!  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 27, 2016 6:24 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Well, My statistics shows the following:

Fetching 50,000 records with 22 Fields all of type string:

FireDAC UniDAC
TFDQuery TUniQuery

Speed 777 MS 675 MS

Memory 450 MB 65 MB

Fast JSON True False

So, they both have approximately the same speed but
FireDAC consumes hell a lot of memory. WHY WHY WHY????

While UniDAC will be very good choice for an http server
but it lacks saving its data to JSON Stream. And that is very
annoying.

Now, When i looked at how FireDAC saves data to streams
it seemed to me that it is writing directly chars to a TStream
object. I tried to do the same thing in c++ but it wasn't fast.

Do Delphi code execute faster then C++ in RAD Studio?

Also. i noticed that saving FireDAC data as binary is faster
than JSON how can i save binary data from TDataSet to TStream
like they do?

--
The limits of my language mean the limits of my world

Ahmed Sayed

Posts: 173
Registered: 8/9/07
Re: A dataset written directly to Indy socket as fast as hell!  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 27, 2016 7:34 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
I found that FireDAC uses these functions while writing to
stream.

const
  C_Class: String = 'class';
  C_Bools: array[Boolean] of String = ('false', 'true');
  C_PairSep: Byte = Ord(',');
  C_ValSep: Byte = Ord(':');
  C_ObjBegin: Byte = Ord('{');
  C_ObjEnd: Byte = Ord('}');
  C_ArrBegin: Byte = Ord('[');
  C_ArrEnd: Byte = Ord(']');
  C_Indent: array[0 .. 1] of Byte = (Ord(' '), Ord(' '));
var
  C_EOL: array of Byte;
 
{-------------------------------------------------------------------------------}
procedure TFDJSONStorage.WriteStringBase(ApStr: PChar; ALen: Integer; AQuote: Boolean);
const
  C_Hex: array [0 .. 15] of Byte = (Ord('0'), Ord('1'), Ord('2'), Ord('3'),
    Ord('4'), Ord('5'), Ord('6'), Ord('7'), Ord('8'), Ord('9'), Ord('A'),
    Ord('B'), Ord('C'), Ord('D'), Ord('E'), Ord('F'));
var
  oMS: TMemoryStream;
  pRes: PByte;
  c: Char;
  iLen: Integer;
begin
  oMS := CheckBuffer(ALen * 6 + 2);
  pRes := oMS.Memory;
  if AQuote then begin
    pRes^ := Ord('"');
    Inc(pRes);
  end;
  while ALen > 0 do begin
    c := ApStr^;
    Inc(ApStr);
    Dec(ALen);
    case Word(Ord(c)) of
    $0:
      Break;
    Ord('"'):
      begin
        pRes^ := Ord('\');
        (pRes + 1)^ := Ord('"');
        Inc(pRes, 2);
      end;
    Ord('\'):
      begin
        pRes^ := Ord('\');
        (pRes + 1)^ := Ord('\');
        Inc(pRes, 2);
      end;
    $8:
      begin
        pRes^ := Ord('\');
        (pRes + 1)^ := Ord('b');
        Inc(pRes, 2);
      end;
    $9:
      begin
        pRes^ := Ord('\');
        (pRes + 1)^ := Ord('t');
        Inc(pRes, 2);
      end;
    $a:
      begin
        pRes^ := Ord('\');
        (pRes + 1)^ := Ord('n');
        Inc(pRes, 2);
      end;
    $c:
      begin
        pRes^ := Ord('\');
        (pRes + 1)^ := Ord('f');
        Inc(pRes, 2);
      end;
    $d:
      begin
        pRes^ := Ord('\');
        (pRes + 1)^ := Ord('r');
        Inc(pRes, 2);
      end;
    $1 .. $7,
    $b, $e .. $1f:
      begin
        pRes^ := Ord('\');
        (pRes + 1)^ := Ord('u');
        (pRes + 2)^ := C_Hex[(Ord(c) and 61440) shr 12];
        (pRes + 3)^ := C_Hex[(Ord(c) and 3840) shr 8];
        (pRes + 4)^ := C_Hex[(Ord(c) and 240) shr 4];
        (pRes + 5)^ := C_Hex[(Ord(c) and 15)];
        Inc(pRes, 6);
      end;
    $20, $21, $23 .. $5b, $5d .. $7f:
      begin
        pRes^ := Ord(c);
        Inc(pRes);
      end;
    $0080 .. $07FF:
      begin
        pRes^ := $C0 or (Ord(c) shr 6);
        (pRes + 1)^ := $80 or (Ord(c) and $3F);
        Inc(pRes, 2);
      end;
    $0800 .. $ffff:
      begin
        pRes^ := $E0 or (Ord(c) shr 12);
        (pRes + 1)^ := $80 or ((Ord(c) shr 6) and $3F);
        (pRes + 2)^ := $80 or (Ord(c) and $3F);
        Inc(pRes, 3);
      end;
    end;
  end;
  if AQuote then begin
    pRes^ := Ord('"');
    iLen := NativeUInt(pRes) - NativeUInt(oMS.Memory) + 1;
  end
  else
    iLen := NativeUInt(pRes) - NativeUInt(oMS.Memory);
  FStream.Write(oMS.Memory^, iLen);
end;
 
{-------------------------------------------------------------------------------}
procedure TFDJSONStorage.WriteObjectBegin(const AObjectName: String; AStyle: TFDStorageObjectStyle);
var
  eParent: TFDStorageObjectStyle;
  iMode: Integer;
begin
(*
  1  , {
  2  , { class : ObjName
  3  , ObjName : {
  4  , ObjName : [
  5  , [
 
  p a  use
  - -  ---
  O O  3
  O F  4
  O T  4
  F O  1
  F F  5
  F T  5
  T O  2
  T F  5
  T T  5
*)
  WritePairSep;
  if FStackIndex = -1 then
    eParent := osObject
  else
    eParent := FStack[FStackIndex].FStyle;
  iMode := 3;
  case eParent of
  osObject:
    if AStyle = osObject then
      iMode := 3
    else
      iMode := 4;
  osFlatArray:
    if AStyle = osObject then
      iMode := 1
    else
      iMode := 5;
  osTypedArray:
    if AStyle = osObject then
      iMode := 2
    else
      iMode := 5;
  end;
  case iMode of
  1:
    FStream.Write(C_ObjBegin, 1);
  2:
    begin
      FStream.Write(C_ObjBegin, 1);
      WriteStringBase(PChar(C_Class), Length(C_Class), True);
      WriteValSep;
      WriteStringBase(PChar(AObjectName), Length(AObjectName), True);
    end;
  3:
    begin
      WriteStringBase(PChar(AObjectName), Length(AObjectName), True);
      WriteValSep;
      FStream.Write(C_ObjBegin, 1);
    end;
  4:
    begin
      WriteStringBase(PChar(AObjectName), Length(AObjectName), True);
      WriteValSep;
      FStream.Write(C_ArrBegin, 1);
    end;
  5:
    FStream.Write(C_ArrBegin, 1);
  end;
  AddToStack(iMode <> 2, AStyle);
end;
 
{-------------------------------------------------------------------------------}
procedure TFDJSONStorage.WriteObjectEnd(const AObjectName: String; AStyle: TFDStorageObjectStyle);
begin
  if AStyle = osObject then
    FStream.Write(C_ObjEnd, 1)
  else
    FStream.Write(C_ArrEnd, 1);
  RemFromStack;
end;
 
{-------------------------------------------------------------------------------}


Why does it depend so much on "Ord" function for any written
data to the stream? and if they wanted to convert bytes to hex why
didn't they used "IntToHex" instead of converting from scratch?

I have another question does using "Ord" increases performance
instead of using the actual string?

--
The limits of my language mean the limits of my world
Ahmed Sayed

Posts: 173
Registered: 8/9/07
Re: A dataset written directly to Indy socket as fast as hell!  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 27, 2016 9:09 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
I tried to achieve something similar to FireDAC with these:

//Consts
const Byte ObjBgn = Byte('{');
const Byte ObjEnd = Byte('}');
const Byte ArrBgn = Byte('[');
const Byte ArrEnd = Byte(']');
const Byte ValSep = Byte(':');
const Byte PairSep = Byte(',');
const Byte Qoute = Byte('"');
//---------------------------------------------------------------------------
void WritePair(TStreamWriter *Writer, UnicodeString Name, UnicodeString Value, bool Qoutes)
{
Writer->Write(Qoute);
Writer->Write(Name);
Writer->Write(Qoute);
 
Writer->Write(ValSep);
 
if (Qoutes)
	Writer->Write(Qoute);
 
Writer->Write(Value);
 
if (Qoutes)
	Writer->Write(Qoute);
}
//---------------------------------------------------------------------------
void WriteFieldsDefs(TStreamWriter *Writer, TFields *AFields)
{
if (AFields->Count > 0)
	{
	Writer->Write(Qoute);
	Writer->Write("Fields");
	Writer->Write(Qoute);
	Writer->Write(ValSep);
	Writer->Write(ArrBgn);
	TField *Field;
	for (int i = 0; i < AFields->Count; i++)
		{
		Field = AFields->Fields[i];
		Writer->Write(ObjBgn);
 
		WritePair(Writer, "FieldName", Field->FieldName);
		Writer->Write(PairSep);
		WritePair(Writer, "Size", Field->Size);
		Writer->Write(PairSep);
		WritePair(Writer, "Type", GetEnumName(__delphirtti(TFieldType),Field->DataType));
		Writer->Write(ObjEnd);
 
		if (i < AFields->Count)
			Writer->Write(PairSep);
 
		}
	Writer->Write(ArrEnd);
	Writer->Write(PairSep);
	}
}
//---------------------------------------------------------------------------
void WriteDataSetToStream(TStream *AStream, TDataSet *DataSet)
{
if (!DataSet->Active || DataSet->RecordCount == 0)
	return;
 
DataSet->DisableControls();
 
TStreamWriter *WR = new TStreamWriter(AStream);
 
try
	{
	WriteFieldsDefs(WR, DataSet->Fields);
	}
__finally
	{
	AStream->Position = 0;
    delete WR;
    }
 
}
//---------------------------------------------------------------------------


And When i loaded that resulting stream into a Memo i got this:

34True34589112334FieldName345834OrderID344434Size3458340344434Type345834ftInteger341254412334FieldName345834Order Name344434Size345834255344434Type345834ftWideString34125449344


Why doesn't it shows JSON format? Why there are numbers
instead of these ( '{', '}', '[', ']', ':', ',') ?

--
The limits of my language mean the limits of my world
Antonio Estevez

Posts: 665
Registered: 4/12/00
Re: A dataset written directly to Indy socket as fast as hell!  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 27, 2016 9:46 AM   in response to: Ahmed Sayed in response to: Ahmed Sayed
El 27/08/2016 a las 18:09, Ahmed Sayed escribió:
I tried to achieve something similar to FireDAC with these:

{code}
//Consts
const Byte ObjBgn = Byte('{');
const Byte ObjEnd = Byte('}');
const Byte ArrBgn = Byte('[');
const Byte ArrEnd = Byte(']');
const Byte ValSep = Byte(':');
const Byte PairSep = Byte(',');
const Byte Qoute = Byte('"');
//---------------------------------------------------------------------------

const Char ObjBgn = '{';
const Char ObjEnd = '}';
const Char ArrBgn = '[';
const Char ArrEnd = ']';
const Char ValSep = ':';
const Char PairSep = ',';
const Char Qoute = '"';


void WritePair(TStreamWriter *Writer, UnicodeString Name, UnicodeString Value, bool Qoutes)
{
Writer->Write(Qoute);
Writer->Write(Name);
Writer->Write(Qoute);

Writer->Write(ValSep);

if (Qoutes)
Writer->Write(Qoute);

Writer->Write(Value);

if (Qoutes)
Writer->Write(Qoute);
}
//---------------------------------------------------------------------------
void WriteFieldsDefs(TStreamWriter *Writer, TFields *AFields)
{
if (AFields->Count > 0)
{
Writer->Write(Qoute);
Writer->Write("Fields");

Writer->Write(String("Fields"));

Writer->Write(Qoute);
Writer->Write(ValSep);
Writer->Write(ArrBgn);
TField *Field;
for (int i = 0; i < AFields->Count; i++)
{
Field = AFields->Fields[i];
Writer->Write(ObjBgn);

WritePair(Writer, "FieldName", Field->FieldName);
Writer->Write(PairSep);
WritePair(Writer, "Size", Field->Size);
Writer->Write(PairSep);
WritePair(Writer, "Type", GetEnumName(__delphirtti(TFieldType),Field->DataType));
Writer->Write(ObjEnd);

if (i < AFields->Count)
Writer->Write(PairSep);

}
Writer->Write(ArrEnd);
Writer->Write(PairSep);
}
}
//---------------------------------------------------------------------------
Ahmed Sayed

Posts: 173
Registered: 8/9/07
Re: A dataset written directly to Indy socket as fast as hell!  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Aug 27, 2016 10:24 AM   in response to: Antonio Estevez in response to: Antonio Estevez
Thanks.

But it is still not fast enough !!!!

--
The limits of my language mean the limits of my world
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02