Watch, Follow, &
Connect with Us

For forums, blogs and more please visit our
Developer Tools Community.


Welcome, Guest
Guest Settings
Help

Thread: Application runs out of memory, don't know what to do.


This question is not answered. Helpful answers available: 2. Correct answers available: 1.


Permlink Replies: 10 - Last Post: Jun 13, 2017 11:52 AM Last Post By: Dimitrij Kowalski
Dimitrij Kowalski

Posts: 92
Registered: 1/6/05
Application runs out of memory, don't know what to do.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 12, 2017 7:18 AM
Hi guys.

My situation is like this: application in our shop runs out of memory after running for a few days and I don't know how to fix it.
The problem is that there is no memory leak detected by memory manager at shut down. There are also no windows resource leaks.
Application is a big ball of mud and it is hard to support. I've started to write unit / functional tests for it, fixed
a lot of potential bugs, but the memory problem is still there. Part of the application is hard to test because it receives a live video stream from a camera
and I don't have such hardware at home. So, I've emulated the camera by substituting camera stream with a file stream.

My latest idea was to record all memory allocations for all threads just to have a vague idea in which thread memory consumption is highest, so I could look for the problem in this thread.
I've wrote an extension to memory manager, substituted all methods: GetMem, FreeMem, ReallocMem, AllocMem, with my own, which track for what thread memory is allocated.
Te problem is that it only tracks memory allocated in heap, am I right? So, when I've run test I see in Windows Task manager that applications consumes 110MB but in my log I see:
DateTime:12/06/2017 15:44:40;PointerCount:14;ThreadID:4364;Memory:1112;ThreadID:14944;Memory:124;
DateTime:12/06/2017 15:44:50;PointerCount:1678;ThreadID:4364;Memory:211573;ThreadID:2772;Memory:572;ThreadID:4048;Memory:230;ThreadID:8392;Memory:230;ThreadID:8776;Memory:230;ThreadID:14944;Memory:92;
DateTime:12/06/2017 15:45:00;PointerCount:3673;ThreadID:4364;Memory:472941;ThreadID:2772;Memory:572;ThreadID:14944;Memory:284;ThreadID:1212;Memory:234;ThreadID:12752;Memory:230;ThreadID:11932;Memory:230;ThreadID:4048;Memory:230;ThreadID:12536;Memory:230;ThreadID:8392;Memory:230;ThreadID:8776;Memory:230;
DateTime:12/06/2017 15:45:10;PointerCount:4029;ThreadID:4364;Memory:520008;ThreadID:2772;Memory:572;ThreadID:8280;Memory:234;ThreadID:1212;Memory:234;ThreadID:12752;Memory:230;ThreadID:4048;Memory:230;ThreadID:11932;Memory:230;ThreadID:8392;Memory:230;ThreadID:12536;Memory:230;ThreadID:8776;Memory:230;ThreadID:14944;Memory:156;
DateTime:12/06/2017 15:45:20;PointerCount:4085;ThreadID:4364;Memory:519859;ThreadID:4048;Memory:5518;ThreadID:2772;Memory:1850;ThreadID:14944;Memory:364;ThreadID:8280;Memory:234;ThreadID:1212;Memory:234;ThreadID:12752;Memory:230;ThreadID:11932;Memory:230;ThreadID:12536;Memory:230;ThreadID:8392;Memory:230;ThreadID:8776;Memory:230;
DateTime:12/06/2017 15:45:30;PointerCount:2331;ThreadID:4364;Memory:265947;ThreadID:8392;Memory:9156;ThreadID:4048;Memory:9024;ThreadID:12536;Memory:6360;ThreadID:8776;Memory:6360;ThreadID:11932;Memory:5552;ThreadID:2772;Memory:1606;ThreadID:14944;Memory:364;ThreadID:1212;Memory:234;ThreadID:8280;Memory:234;ThreadID:12752;Memory:230;ThreadID:13784;Memory:28;


So as you can see, the highest amout of memory was 520008 bytes for a one thread which is around of 0.5% of total memory reserved for application.
Do you have any suggestions for me? How can I track other memory areas(stack, data segment) allocated for my threads? Could you suggest any other techniques which would help me in finding where the problem is?

Best regards.
Igor Savkic

Posts: 27
Registered: 8/26/01
Re: Application runs out of memory, don't know what to do.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 12, 2017 8:17 AM   in response to: Dimitrij Kowalski in response to: Dimitrij Kowalski
My situation is like this: application in our shop runs out of memory after
running for a few days and I don't know how to fix it.

How does it manifests, app cannot allocate anything and you get Out of Memory?
Does it reach 2GB?

The problem is that there is no memory leak detected by memory manager at
shut down.

I only have few suggestions, perhaps you've alredy tried them.

Have you tried with FastMM in full debug mode? Or perhaps try with some other
memory manager (if you're using FastMM already). Do you use some third party
libraries/dlls for video striming, leaks might be there.

I've started to write unit / functional tests for it, fixed
a lot of potential bugs, but the memory problem is still there. Part of the
application is hard to test because it receives a live video stream from a
camera and I don't have such hardware at home

Try to make some test application, start it empty then add things to it (first
with video streaming), untill it begins crashing. Alternative is other way
around, take your application and remove functionality until it stops crashing.
Dimitrij Kowalski

Posts: 92
Registered: 1/6/05
Re: Application runs out of memory, don't know what to do.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 12, 2017 9:07 AM   in response to: Igor Savkic in response to: Igor Savkic
Igor Savkic wrote:
My situation is like this: application in our shop runs out of memory after
running for a few days and I don't know how to fix it.

How does it manifests, app cannot allocate anything and you get Out of Memory?
Does it reach 2GB?

It reaches around 1.2GB and starts to crash. Perhaps by memory fragmenation.


The problem is that there is no memory leak detected by memory manager at
shut down.

I only have few suggestions, perhaps you've alredy tried them.

Have you tried with FastMM in full debug mode? Or perhaps try with some other
memory manager (if you're using FastMM already).

You are asking me question to which you have answer in your quote of me..

Try to make some test application, start it empty then add things to it (first
with video streaming), untill it begins crashing. Alternative is other way
around, take your application and remove functionality until it stops crashing.

I said that I wrote tests for parts of the application and your suggestion is to make a test application :)

I've tested what I can test without a specific hardware which runs on site. The only thing which I can do now is to somehow profile running application, gather some statistics and it has to be done on client site...
Igor Savkic

Posts: 27
Registered: 8/26/01
Re: Application runs out of memory, don't know what to do.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 12, 2017 3:06 PM   in response to: Dimitrij Kowalski in response to: Dimitrij Kowalski
Have you tried with FastMM in full debug mode? Or perhaps try with some other
memory manager (if you're using FastMM already).
You are asking me question to which you have answer in your quote of me..

From your mail it really wasn't clear which Delphi version or memory manager
you are using. My best guess was that it wasn't FastMM since you mentioned
writing extension to it that records allocated memory and FastMM does have that
already.
FastMM does have some nice logging capabilities, most likely fragmentation is
the problem but you can test that with FastMM. You can have detailed memory
reported each 15mins or so and that should give enough clues to confirm it as an
issue. As for a solution, a different memory manager or Peter Below's suggestion
about allocating one big chunk enough for everything.

I said that I wrote tests for parts of the application and your suggestion is to
make a test application :)

I'm just trying to help ;) I did read that, but it really wasn't clear what kind
of test you wrote, what was it scope and have you distributed it on production
computer or only on your development machine. I forgot to write to put test
limited functionality app on production computer and log all results.
Dimitrij Kowalski

Posts: 92
Registered: 1/6/05
Re: Application runs out of memory, don't know what to do.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 12, 2017 4:36 PM   in response to: Igor Savkic in response to: Igor Savkic
Igor Savkic wrote:
I'm just trying to help ;) I did read that, but it really wasn't clear what kind
of test you wrote, what was it scope and have you distributed it on production
computer or only on your development machine. I forgot to write to put test
limited functionality app on production computer and log all results.

Thanks Igor, I appreciate it:) I am using Delphi XE5 with FastMM. Didn't want to sound harsh, I am just bit sensitive when someone is explaining me things that I already know and I hinted that in my post . Like Peter in his answer is doing exactly the same:P I wrote that the cause might be memory fragmentation and he explains me what memory fragmentation is, great :P

Is it possible to query FastMM every few minutes and got a detailed log about memory allocation? I thought it only reports at shut down. However, I dont think that it tracks memory allocated per thread, but if it can show me that problem is in memory fragmentation... that would be great.

Also, I don't understand how FastMM is allocating rest of memory. The part that I am unable to track. This memory is not allocated by GetMem so how FastMM knows how to allocate it?

Unfortunately, I can't turn off application on a production for a couple of days and replace it with other with limited funtionality. Because this application is making money for customer 24/7 (lucky one;) )

Thanks again.

Edited by: Dimitrij Kowalski on Jun 12, 2017 4:36 PM
Igor Savkic

Posts: 27
Registered: 8/26/01
Re: Application runs out of memory, don't know what to do. [Edit]  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 13, 2017 11:18 AM   in response to: Dimitrij Kowalski in response to: Dimitrij Kowalski
Is it possible to query FastMM every few minutes and got a detailed log about
memory allocation? I thought it only reports at shut down. However, I dont
think that it tracks memory allocated per thread, but if it can show me that
problem is in memory fragmentation... that would be great.

In fullDebugMode it tracks allocation by threads but it doesn't report it by
default. You could write some routine that calls WalkAllocatedBlocks each x mins
and writes down in some txt format details of allocation.
Also try using FastMMUsageTracker/ShowFastMMUsageTracker it gives nice visual
representation of allocated blocks.

You can also set FullDebugModeScanMemoryPoolBeforeEveryOperation and
FullDebugModeRegisterAllAllocsAsExpectedMemoryLeak but that would probably bring
app to unusable in production.

Also, I don't understand how FastMM is allocating rest of memory. The part
that I am unable to track. This memory is not allocated by GetMem so how
FastMM knows how to allocate it?

If it allocated directly from Windows with VirtuallAlloc then it won't track it,
but perhaps UsageTracker can report it as occupied.

Unfortunately, I can't turn off application on a production for a couple of
days and replace it with other with limited funtionality. Because this
application is making money for customer 24/7 (lucky one;) )

That's a real problem, do a debug but not affect app itself. I guess you cannot
run main app and test app in parallel either, best best is what you're already
doing, using some recorded stream.
Peter Below

Posts: 1,227
Registered: 12/16/99
Re: Application runs out of memory, don't know what to do.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 12, 2017 11:01 AM   in response to: Dimitrij Kowalski in response to: Dimitrij Kowalski
Dimitrij Kowalski wrote:

Hi guys.

My situation is like this: application in our shop runs out of memory
after running for a few days and I don't know how to fix it. The
problem is that there is no memory leak detected by memory manager at
shut down.

The problem is then likely to be memory fragmentation, not a plain lack
of free memory. That can hit you if your app has a cycle of
- allocate a huge chunk of memory (e.g. for a video)
- allocate some small blocks (e.g. strings etc.)
- free the huge chunk
repeat with huge chunks of different size

This way you can end up with a memory space that has a lot of small
blocks allocated across the address range, interspaced with larger free
gaps, of which none is large enough to accomodate the next huge chunk.

If you can control how the huge blocks are allocated (i. e. it is in
our code, not in some OS system code) you can ameliorate this situation
a bit by allocating one chunk up front that is large enough to serve
any huge chunk allocation and then reuse that instead of freeing and
re-allocating it. You only free it once the application is on its way
to the great bit bucket beyond.

There are also no windows resource leaks. Application is
a big ball of mud and it is hard to support. I've started to write
unit / functional tests for it, fixed a lot of potential bugs, but
the memory problem is still there. Part of the application is hard to
test because it receives a live video stream from a camera and I
don't have such hardware at home. So, I've emulated the camera by
substituting camera stream with a file stream.

--
Peter Below
TeamB

Alex Belo

Posts: 626
Registered: 10/8/06
Re: Application runs out of memory, don't know what to do.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 13, 2017 10:15 AM   in response to: Dimitrij Kowalski in response to: Dimitrij Kowalski
Dimitrij Kowalski wrote:

Hi guys.

My situation is like this: application in our shop runs out of memory
after running for a few days and I don't know how to fix it. The
problem is that there is no memory leak detected by memory manager at
shut down. There are also no windows resource leaks

As Peter alredy noted it looks as memory fragmentation issue.

So, when I've run test I see in
Windows Task manager that applications consumes 110MB but in my log I
see: ...
ThreadID:4364
ThreadID:14944
ThreadID:4364
ThreadID:2772
...

Another problem (but it is also kind of memory fragmentation) can be
default stack size of thread (2 Mb): if you use N threads
simultaneously they need additional N*2Mb of memory. Stacks can eat a
lot af memory and fragment memory too.

Unfortunately TThread has no parameter to control stack size. AFAIR
there was very old QC request to add such parameter but it is in
"ignore list" as many others.

I don't know if stack size parameter in project options controls
default stack size of threads in application.

--
Alex
Dimitrij Kowalski

Posts: 92
Registered: 1/6/05
Re: Application runs out of memory, don't know what to do.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 13, 2017 11:13 AM   in response to: Dimitrij Kowalski in response to: Dimitrij Kowalski
Take a look at this image: https://s1.postimg.org/t9ztfi99r/Fragmentation_1.png It does not seem that it is a fragmentation issue?

Currently taskmgr says that 1035MB is used but VMMap statistics are like this: https://s10.postimg.org/qos2d8wcp/Fragmentation_2.png

What is interesting that when memory usage grows, handle count shown by taskmanager also grows, it is currently at 1800 handles.
Alex Belo

Posts: 626
Registered: 10/8/06
Re: Application runs out of memory, don't know what to do.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 13, 2017 11:41 AM   in response to: Dimitrij Kowalski in response to: Dimitrij Kowalski
Dimitrij Kowalski wrote:

Take a look at this image:
https://s1.postimg.org/t9ztfi99r/Fragmentation_1.png It does not seem
that it is a fragmentation issue?

Looks exactly as fragmentation (AFAIUI without legend): (white) free
memory block is fragmented (yellow fragments).

What is interesting that when memory usage grows, handle count shown
by taskmanager also grows, it is currently at 1800 handles.

You can try madExcept's detector of resource leaks.

--
Alex
Dimitrij Kowalski

Posts: 92
Registered: 1/6/05
Re: Application runs out of memory, don't know what to do.  
Click to report abuse...   Click to reply to this thread Reply
  Posted: Jun 13, 2017 11:52 AM   in response to: Alex Belo in response to: Alex Belo
Alex Belo wrote:
Dimitrij Kowalski wrote:

Take a look at this image:
https://s1.postimg.org/t9ztfi99r/Fragmentation_1.png It does not seem
that it is a fragmentation issue?

Looks exactly as fragmentation (AFAIUI without legend): (white) free
memory block is fragmented (yellow fragments).
I don't think it is an issue, each program will be like that. You can't have continous blocks of memory if you haven't allocated it upfront.
Legend
Helpful Answer (5 pts)
Correct Answer (10 pts)

Server Response from: ETNAJIVE02