Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: DataSnap Consumption Memory Server Side...



Permlink Replies: 10 - Last Post: Jan 18, 2018 6:11 AM Last Post By: Maria Testa
Thiago Silva


Posts: 37
Registered: 12/20/04
DataSnap Consumption Memory Server Side...
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 20, 2016 4:18 AM
Hello Everyone,

We have a 32 bits DataSnap/DBExpress app server using 5 DSServerModule and we identified some memory leak called by client side (VCL 32 Bits) and when the server process in the Windows Server 2008 R2 reaches near of 1.5 GB RAM the following message error is showing up: "Remote Error: Out Of Memory". We know that memory leak is a big fault but why 1.5 gb consumption ram memory to occours the remote error? Consider that we have 300 users connected to the server without memory leak maybe reach the same memory consumption and what to do to deal with?

We are making mistakes? Please help us! Below the code to figure the memory leak purposeful to undesrtand the case:

var cds: TClientDataSet;
dsp: TDSProviderConnection;
I: Integer;
begin
inherited;
for I := 1 to 10000 do
begin
dsp := TDSProviderConnection.Create(Self);
dsp.SQLConnection := DMMain.DSServer;
cds := TClientDataSet.Create(Self);
dsp.ServerClassNAme := 'TDSSMCad';
cds.ProviderName := 'dspProdutos';
cds.CommandText := 'SELECT * FROM PRODUTOS';
cds.RemoteServer := dsp;
cds.Open;
end;

Best regards,

Thiago.

Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: DataSnap Consumption Memory Server Side... [Edit]
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 20, 2016 10:11 AM   in response to: Thiago Silva in response to: Thiago Silva
Thiago wrote:

We have a 32 bits DataSnap/DBExpress app server using 5 DSServerModule
and we identified some memory leak called by client side (VCL 32 Bits)
and when the server process in the Windows Server 2008 R2 reaches
near of 1.5 GB RAM the following message error is showing up: "Remote
Error: Out Of Memory". We know that memory leak is a big fault but
why 1.5 gb consumption ram memory to occours the remote error?

Have you tried enabling ReportMemoryLeakOnShutdown and then closing the server
app? Does it report what is actually leaking? If not, have you tried using
any tools to track your app's memory usage to see where all the memory is
actually going? The full version of FastMM has live memory reporting capabilities.

We are making mistakes? Please help us! Below the code to figure the
memory leak purposeful to undesrtand the case:

Why are you creating 10000 TDSProviderConnection objects and 10000 TClientDataSet
objects? Are you ever freeing them? You have assigned an Owner to them,
but what is that Owner exactly? Is that Owner ever freed?

P.S. please stop wrapping your code snippets in '+' characters. Surround
them in {code} tags instead.

--
Remy Lebeau (TeamB)
Thiago Silva


Posts: 37
Registered: 12/20/04
Re: DataSnap Consumption Memory Server Side... [Edit]
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 20, 2016 11:30 AM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Why are you creating 10000 TDSProviderConnection objects and 10000 TClientDataSet
objects? Are you ever freeing them? You have assigned an Owner to them,
but what is that Owner exactly? Is that Owner ever freed?

P.S. please stop wrapping your code snippets in '+' characters. Surround
them in {code} tags instead.

Thank you to tips,

The question is: We have customers that get to customize the ERP and some of them build app with memory leak. My big question is: a big customer built some apps inside the ERP and when the datasnap server app reach 1.5 GB RAM show up the error message. In fact we cannot garantee that the customer will free the resources and now they want a your solution to stop crash.

We know that must free objects along the solution but we would like to know how many memory the datasnap server can handle without crash because of memory leak? Consider the code was sent, we can get crash the datasnap server in 2 minutes, but it is a test to discover the limits of your solution. Windows 32 bits can handle more (if you enable) 2 GB RAM buf why the server crashes when reach 1.5 gb consider the all memory leak test?

Undestand?

Best Regards,

Thiago.
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: DataSnap Consumption Memory Server Side... [Edit] [Edit]
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 20, 2016 12:35 PM   in response to: Thiago Silva in response to: Thiago Silva
Thiago wrote:

The question is: We have customers that get to customize the ERP and
some of them build app with memory leak. My big question is: a big
customer built some apps inside the ERP and when the datasnap server
app reach 1.5 GB RAM show up the error message. In fact we cannot
garantee that the customer will free the resources and now they want
a your solution to stop crash.

My issue is that you should not be creating that many DataSnap connections
to begin with. It is too much overhead. At most, you should create a handful
of connections and then pool and reuse them on an as-needed basis.

We know that must free objects along the solution but we would like
to know how many memory the datasnap server can handle without crash
because of memory leak?

First off, you haven't determined that you actually have a leak to begin
with. If you are allocating a lot of objects and not freeing them, that
is obviously going to use a lot of memory potentially.

Consider the code was sent, we can get crash the datasnap server in 2 minutes,
but it is a test to discover the limits of your solution. Windows 32 bits
can handle
more (if you enable) 2 GB RAM buf why the server crashes when reach 1.5 gb
consider the all memory leak test?

On 32bit, Windows might allow you app to use up to 3GB of memory, if you
mark your executable as LargeAddressAware in the PE header. But the VCL
certainly cannot handle that much memory. The memory manager is limited
to 2GB max, and a portion of that is going to be used for bookkeeping and
such. So that could easily account for your 1.5GB limit. That is not necessary
caused by actual leaks, just bad memory management on your part if you are
allocating a lot of objects and not freeing them over time. And even if
you were freeing them, also remember that the memory manager caches and reuses
allocated memory blocks (up to a few MBs, IIRC), they are not returned to
the OS when "freed". So that could also be accounting for your high memory
usage.

On 64bit, those limits are much higher and less likely to be a problem.

--
Remy Lebeau (TeamB)
Thiago Silva


Posts: 37
Registered: 12/20/04
Re: DataSnap Consumption Memory Server Side... [Edit] [Edit]
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 20, 2016 2:30 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
My issue is that you should not be creating that many DataSnap connections
to begin with. It is too much overhead. At most, you should create a handful
of connections and then pool and reuse them on an as-needed basis.
I do it already the problem is some connections consume memory because the client side does not freeing some objects (the programmer made a mistake for example). During the day the app crashes near 1.5 RAM. I am not sure how Datasnap server using TDSServerModule (many sqldatasets and providers) handle the many SQL statements executed by client that does not freeing the objects used to get info of a sql table.

First off, you haven't determined that you actually have a leak to begin
with. If you are allocating a lot of objects and not freeing them, that
is obviously going to use a lot of memory potentially.
Great but the result is crashes right? But why 1.5 GB RAM in my case? Who determine the RAM limit why not 1.8 GB e.g?

On 32bit, Windows might allow you app to use up to 3GB of memory, if you
mark your executable as LargeAddressAware in the PE header. But the VCL
certainly cannot handle that much memory. The memory manager is limited
to 2GB max, and a portion of that is going to be used for bookkeeping and
such. So that could easily account for your 1.5GB limit. That is not necessary
caused by actual leaks, just bad memory management on your part if you are
allocating a lot of objects and not freeing them over time. And even if
you were freeing them, also remember that the memory manager caches and reuses
allocated memory blocks (up to a few MBs, IIRC), they are not returned to
the OS when "freed". So that could also be accounting for your high memory
usage.

I see but there is a way to clear the allocated memory block to avoid the crashes? the weird is 1.5 GB is too high for a DS Serveror another app ?

Sorry to insist in it but I need to understand why the app crashes in 1.5 GB.

Thiago

Edited by: Thiago Silva on Jan 20, 2016 2:30 PM

Edited by: Thiago Silva on Jan 20, 2016 2:32 PM
Remy Lebeau (Te...


Posts: 9,447
Registered: 12/23/01
Re: DataSnap Consumption Memory Server Side... [Edit] [Edit] [Edit]
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 20, 2016 3:15 PM   in response to: Thiago Silva in response to: Thiago Silva
Thiago wrote:

I do it already the problem is some connections consume memory because
the client side does not freeing some objects (the programmer made a
mistake for example).

What do you expect the server to do about that? The client asked for some
objects and did not release them. The server doesn't know if the client
is still using them or not. Fix the client.

Great but the result is crashes right?

If you exceed the maximum amount of memory that the memory manager can handle,
yes. Just like anything else in Delphi. Any memory allocation operation
has the potential of throwing an "Out of memory" error.

But why 1.5 GB RAM in my case?
Who determine the RAM limit why not 1.8 GB e.g?

I already addressed that in my last reply. The memory manager has its own
overhead that it needs to keep track of the memory it allocates. If you
have a lot of objects allocated, and especially if they internally allocate
data dynamically, that is a lot of tracking data, and that could easily account
for 0.5 GB of memory, depending on how many individual allocations have been
made and how large they each are.

I see but there is a way to clear the allocated memory block to avoid
the crashes?

No. Stop allocating so many objects to begin with. If you see your DataSnap
server is starting to use a lot of memory/objects, can you simply stop accepting
clients requests? I don't know how DataSnap works.

Sorry to insist in it but I need to understand why the app crashes
in 1.5 GB.

Have you tried simply DEBUGGING your app to find out where all the memory
is going?

--
Remy Lebeau (TeamB)
Thiago Silva


Posts: 37
Registered: 12/20/04
Re: DataSnap Consumption Memory Server Side... [Edit] [Edit] [Edit]
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 20, 2016 6:13 PM   in response to: Remy Lebeau (Te... in response to: Remy Lebeau (Te...
Remy,

Thank you for your assistance in my problem! I am going To solv memory leak and see how does delphi deal with this.

Thiago
Markus Humm

Posts: 5,113
Registered: 11/9/03
Re: DataSnap Consumption Memory Server Side... [Edit] [Edit] [Edit]
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 21, 2016 10:26 AM   in response to: Thiago Silva in response to: Thiago Silva
Hello,

if possible you could implement some watchdog: if a connection sees no
activity for a certain time (say 30 min.) the connection is being closed
server side.

Greetings

Markus
Thiago Silva


Posts: 37
Registered: 12/20/04
Re: DataSnap Consumption Memory Server Side... [Edit] [Edit] [Edit]
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 24, 2016 3:52 PM   in response to: Markus Humm in response to: Markus Humm
Markus,

Thank you for the tip. Do you have some example to do this?

Regards,

Thiago.
Markus Humm

Posts: 5,113
Registered: 11/9/03
Re: DataSnap Consumption Memory Server Side... [Edit] [Edit] [Edit]
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 26, 2016 10:33 AM   in response to: Thiago Silva in response to: Thiago Silva
Am 25.01.2016 um 00:52 schrieb Thiago Silva:
Markus,

Thank you for the tip. Do you have some example to do this?

Regards,

Thiago.

Sorry no, I don't use DataSnap.
But a list with the last answertime for each connection on the server
side would be one thing (can be a sorted generic list of TDateTime
values) and the list would store the connection for each time entry as well.

You'd have a timer check periodically if now-timestamp is over the limit
and if yes lose the connection.

Now the thing left is to notice when activity has been seen on the
connection and to update the timestamp of that connection in the list.

Greetings

Markus

Maria Testa

Posts: 1
Registered: 2/18/18
Re: DataSnap Consumption Memory Server Side...
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jan 18, 2018 6:11 AM   in response to: Thiago Silva in response to: Thiago Silva
the problem be solved?
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02