Watch, Follow, &
Connect with Us

Please visit our new home
community.embarcadero.com.


Welcome, Guest
Guest Settings
Help

Thread: idtcpclient sending mp3 file in memorystream


This question is answered.


Permlink Replies: 4 - Last Post: Jan 22, 2018 3:54 PM Last Post By: Remy Lebeau (Te...
madammar ellias

Posts: 111
Registered: 8/17/17
idtcpclient sending mp3 file in memorystream  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 18, 2018 11:55 PM
i am trying to send an mp3 file with size 1 mega into a memorystream

thats what i did


var
mpaud: TMemoryStream;
begin
mpaud := TMemoryStream.Create;
try
mpaud.LoadFromFile ('mozart.mp3');
mpaud.Position := 0;
 
if mpaud.Size <= 1024000  then
begin
ClientThread.SendBuffer(mpaud);
end;
 
finally
mpaud.Free;
end;
end;
 
procedure TClientThread.SendBuffer(var MS: TMemoryStream);
Var
Scmd : String;
Strm: TMemoryStream;
begin
 
Scmd := 'mp3file';
 
Strm := TMemoryStream.Create;
 
 
try
Strm.LoadFromStream(MS);
Strm.Position := 0;
 
 
OutboundCache.AddObject(Scmd, Strm);
except
Strm.Free;
raise;
end;
 
 
end;
 


and here how i send

var
Cache, OutboundCmds: TStringList;
I : integer;
FMSFROMCALL : TMemorystream;
begin
 
 
OutboundCmds := nil;
try
Cache := OutboundCache.Lock;
try
if Cache.Count > 0 then
begin
OutboundCmds := TStringList.Create;
OutboundCmds.Assign(Cache);
Cache.Clear;
end;
finally
OutboundCache.Unlock;
end;
 
if OutboundCmds <> nil then
begin
for I := 0 to OutboundCmds.Count - 1 do
begin
FTCP.IOHandler.Writeln(OutboundCmds.Strings[I], IndyTextEncoding_UTF8);
FMSFROMCALL := TMemoryStream(OutboundCmds.Objects[I]);
if (FMSFROMCALL <> nil) then
begin
FTCP.IOHandler.LargeStream := True;
FTCP.IOHandler.Write(FMSFROMCALL, 0, true);
end;
end;
 
end;
 
finally
if OutboundCmds <> nil then
begin
for I := 0 to OutboundCmds.Count - 1 do
begin
OutboundCmds.Objects[I].Free;
end;
end;
OutboundCmds.Free;
end;
 
 
 
if FTCP.IOHandler.InputBufferIsEmpty then
begin
FTCP.IOHandler.CheckForDataOnSource(100);
FTCP.IOHandler.CheckForDisconnect;
if FTCP.IOHandler.InputBufferIsEmpty then
begin
exit;
end;
end;


i got out of memory exception is the size of memorystream through tcp is limited ?

First chance exception at $75025B68. Exception class EStreamError with message 'Out of memory while expanding memory stream'

this exception raised in IdIOhandler.pas unit at the following line of procedure

// TODO: move this into IdGlobal.pas
procedure AdjustStreamSize(const AStream: TStream; const ASize: TIdStreamSize);
var
  LStreamPos: TIdStreamSize;
begin
  LStreamPos := AStream.Position;
  AStream.Size := ASize;
  // Must reset to original value in cases where size changes position
  if AStream.Position <> LStreamPos then begin
    AStream.Position := LStreamPos;
  end;
end;
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: idtcpclient sending mp3 file in memorystream [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 19, 2018 6:03 PM   in response to: madammar ellias in response to: madammar ellias
madammar ellias wrote:

i am trying to send an mp3 file with size 1 mega into a memorystream

Why limited to 1MB? What is an MP3 is larger? TCP can handle large
data.

procedure TClientThread.SendBuffer(var MS: TMemoryStream);

You are loading the MP3 into memory, then you are making a 2nd copy in
memory for your outbound queue. You might want to consider changing
your SendBuffer() design to either:

1. take ownership of the input TMemoryStream object and set the
caller's pointer to nil (since you are passing it by 'var' anyway).
That would avoid needing a 2nd copy.

2. accept any TStream as input, take ownership of it, put it in the
queue, and have the sending thread free it after sending it. Then you
wouldn't even need to load the MP3 into memory at all, you could open
it with TFileStream and let the sending thread send the file as-is.

i got out of memory exception is the size of memorystream through tcp
is limited ?

No, TCP is not limited.

First chance exception at $75025B68. Exception class EStreamError
with message 'Out of memory while expanding memory stream'

There is only one place that error is raised from - by TMemoryStream
itself, inside its internal Realloc() method, which is called when the
Capacity property is changing value (either because Write/Buffer() is
writing more data than is currently allocated, or the Size property is
being changed directly). If the RTL's memory manager fails to
(re)alloc the stream's Memory block to the new Capacity, the error is
raised.

So, you need to double-check your uses of TMemoryStream, because
somewhere you are likely trying to (re)allocate a LOT of memory, more
than the RTL can handle.

this exception raised in IdIOhandler.pas unit at the following line
of procedure

That is not called in the code you showed. That is called by
TIdIOHandler.ReadStream() on the receiving end. If ReadStream() is
able to determine the size of the stream data being received (either
because the AByteCount parameter is a positive value, or ReadStream()
reads the size from the TCP connection) then the output TStream is
pre-allocated to that size before the actual stream data is then read.

The error you are seeing is an indication that you have most likely
corrupted your communications, causing ReadStream() to infer an invalid
stream size that happens to be way more than the 1MB you are actually
sending. That could happen, for instance, if your TIdIOHandler.Write()
and TIdIOHandler.ReadStream() parameters are not matched up correctly.

But you didn't show your reading code for the 'mp3file' command.
Though it shouldn't be any different than the other commands you have
shown before. So what are you doing (or not doing) differently for
this command vs your other commands?

--
Remy Lebeau (TeamB)
madammar ellias

Posts: 111
Registered: 8/17/17
Re: idtcpclient sending mp3 file in memorystream [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 19, 2018 7:00 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:

Why limited to 1MB? What is an MP3 is larger? TCP can handle large
data.

procedure TClientThread.SendBuffer(var MS: TMemoryStream);

You are loading the MP3 into memory, then you are making a 2nd copy in
memory for your outbound queue. You might want to consider changing
your SendBuffer() design to either:

1. take ownership of the input TMemoryStream object and set the
caller's pointer to nil (since you are passing it by 'var' anyway).
That would avoid needing a 2nd copy.

2. accept any TStream as input, take ownership of it, put it in the
queue, and have the sending thread free it after sending it. Then you
wouldn't even need to load the MP3 into memory at all, you could open
it with TFileStream and let the sending thread send the file as-is.

i got out of memory exception is the size of memorystream through tcp
is limited ?

No, TCP is not limited.

First chance exception at $75025B68. Exception class EStreamError
with message 'Out of memory while expanding memory stream'

There is only one place that error is raised from - by TMemoryStream
itself, inside its internal Realloc() method, which is called when the
Capacity property is changing value (either because Write/Buffer() is
writing more data than is currently allocated, or the Size property is
being changed directly). If the RTL's memory manager fails to
(re)alloc the stream's Memory block to the new Capacity, the error is
raised.

So, you need to double-check your uses of TMemoryStream, because
somewhere you are likely trying to (re)allocate a LOT of memory, more
than the RTL can handle.

this exception raised in IdIOhandler.pas unit at the following line
of procedure

That is not called in the code you showed. That is called by
TIdIOHandler.ReadStream() on the receiving end. If ReadStream() is
able to determine the size of the stream data being received (either
because the AByteCount parameter is a positive value, or ReadStream()
reads the size from the TCP connection) then the output TStream is
pre-allocated to that size before the actual stream data is then read.

The error you are seeing is an indication that you have most likely
corrupted your communications, causing ReadStream() to infer an invalid
stream size that happens to be way more than the 1MB you are actually
sending. That could happen, for instance, if your TIdIOHandler.Write()
and TIdIOHandler.ReadStream() parameters are not matched up correctly.

But you didn't show your reading code for the 'mp3file' command.
Though it shouldn't be any different than the other commands you have
shown before. So what are you doing (or not doing) differently for
this command vs your other commands?

--
Remy Lebeau (TeamB)

this is how i read on server side it has the same protocol on client side



 
Connection := AContext as TConnection;
 
  // check for pending outbound commands...
  OutboundCmds := nil;
  try
    Cache := Connection.OutboundCache.Lock;
    try
      if Cache.Count > 0 then
      begin
        OutboundCmds := TStringList.Create;
        OutboundCmds.Assign(Cache);
        Cache.Clear;
      end;
    finally
      Connection.OutboundCache.Unlock;
    end;
 
    if OutboundCmds <> nil then
    begin
      for I := 0 to OutboundCmds.Count - 1 do
      begin
 
        AContext.Connection.IOHandler.Writeln(OutboundCmds.Strings[I],
          IndyTextEncoding_UTF8);
 
 
        MS := TMemoryStream(OutboundCmds.Objects[I]);
        if (MS <> nil) then
        begin
          AContext.Connection.IOHandler.LargeStream := true;
          AContext.Connection.IOHandler.Write(MS, 0, true);
        end;
      end;
 
    end;
 
  finally
    if OutboundCmds <> nil then
    begin
      for I := 0 to OutboundCmds.Count - 1 do
      begin
        OutboundCmds.Objects[I].Free;
      end;
    end;
    OutboundCmds.Free;
  end;
 
  // check for a pending inbound command...
 if AContext.Connection.IOHandler.InputBufferIsEmpty then
  begin
    AContext.Connection.IOHandler.CheckForDataOnSource(100);
    AContext.Connection.IOHandler.CheckForDisconnect;
    if AContext.Connection.IOHandler.InputBufferIsEmpty then
    begin
      Exit;
    end;
 end;
 
command  := AContext.Connection.IOHandler.ReadLn(IndyTextEncoding_UTF8);
 
  MS := nil;
  try
    if command = 'mp3file' then 
    begin
 
      MS := TMemoryStream.Create;
      AContext.Connection.IOHandler.LargeStream := true;
      AContext.Connection.IOHandler.ReadStream(MS, -1, False);
      MS.Position := 0;
 
    end;
 
finally
Ms.free
end;
madammar ellias

Posts: 111
Registered: 8/17/17
Re: idtcpclient sending mp3 file in memorystream [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 22, 2018 2:14 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy Lebeau (TeamB) wrote:
madammar ellias wrote:

memory for your outbound queue. You might want to consider changing
your SendBuffer() design to either:

1. take ownership of the input TMemoryStream object and set the
caller's pointer to nil (since you are passing it by 'var' anyway).
That would avoid needing a 2nd copy.

--
Remy Lebeau (TeamB)


i have tried to do this with sendbuffer so i get ride of the var that passed and load it directly when its needed to be sent, i got exactly the same exception

procedure TClientThread.SendBuffer;
Var
Scmd : String;
Strm: TMemoryStream;
begin
 
Scmd := 'mp3file';
 
Strm := TMemoryStream.Create;
 
try
 
Strm .LoadFromFile ('mozart.mp3');
Strm .Position := 0;
 
if Strm .Size <= 1024000  then
begin
OutboundCache.AddObject(Scmd, Strm);
end;
 
except
Strm.Free;
raise;
end;
 
 
end;
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: idtcpclient sending mp3 file in memorystream [Edit]
Correct
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 22, 2018 3:54 PM   in response to: madammar ellias in response to: madammar ellias
madammar ellias wrote:

i have tried to do this with sendbuffer so i get ride of the var that
passed and load it directly when its needed to be sent, i got exactly
the same exception

I did not say to "get rid" of the 'var' parameter. I suggested you
utilize it better, eg:

var
  mpaud: TMemoryStream;
begin
  mpaud := TMemoryStream.Create;
  try
    mpaud.LoadFromFile ('mozart.mp3');
    mpaud.Position := 0;
 
    if mpaud.Size <= 1024000  then
    begin
      ClientThread.SendBuffer(mpaud);
    end;
  finally
    mpaud.Free;
  end;
end;
 
procedure TClientThread.SendBuffer(var MS: TMemoryStream);
begin
  OutboundCache.AddObject('mp3file', Strm);
  Strm := nil;
end;


Alternatively:

var
  mpaud: TStream;
begin
  mpaud := TFileStream.Create('mozart.mp3', fmOpenRead or
fmShareDenyWrite);
  try
    if mpaud.Size <= 1024000  then
    begin
      ClientThread.SendBuffer(mpaud);
    end;
  finally
    mpaud.Free;
  end;
end;
 
procedure TClientThread.SendBuffer(var Strm: TStream);
begin
  OutboundCache.AddObject('mp3file', Strm);
end;
 
...
 
var
  ...
  FStrmFromCall : TStream;
begin
  ...
  if OutboundCmds <> nil then
  begin
    for I := 0 to OutboundCmds.Count - 1 do
    begin
      FTCP.IOHandler.Writeln(OutboundCmds.Strings[I],
IndyTextEncoding_UTF8);
      FStrmFromCall := TStream(OutboundCmds.Objects[I]);
      if (FStrmFromCall <> nil) then
      begin
        FTCP.IOHandler.LargeStream := True;
        FTCP.IOHandler.Write(FStrmFromCall, 0, true);
      end;
    end;
  end;
  ...
end;


Either way, that doesn't solve the memory error, but it does reduce the
amount of memory you are actually using, which might be contributing to
the error.

--
Remy Lebeau (TeamB)
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02