Hi All,
We are facing a strange issue in one of the customer production system. OS is RHEL 5.5. Application is written in c++. When Application starts it connects to Oracle DB and caches some data (few thousand rows) in to process memory. Usually this caching takes few minutes. Until the caching is completed the application doesn’t starts processing data.
2-3 times in last few months customer reported that application startup is taking huge time: close to few hours. While investigating it was found that the caching process is slow and due to that startup is delayed. Investigation was performed using GDB tracing as well as application debug log.
To immediate overcome this issue server restart was recommended and after server restart application started normally (within few minutes caching is completed and data is started processing).
Note that application is not demanding bulk memory at one go, it caches one DB row and when it goes for the next row (using a cursor) it demands the memory that would be required to cache that one row only.
There is a theory that might explain this behavior and I would like to get expert opinion if this theory sounds good.
Server has 128 GB of RAM and usually application consume around 30-40 GB memory. Initially we see huge free memory and very minimum OS cached memory (in top). Over a period of time OS cached memory increases and after some weeks most of the memory is found in OS cache.
At this moment when we try to restart the application it takes huge time giving an impression that application is hanging.
The theory is that when application is demanding the memory in order to cache the DB data at process memory it is requesting the same from the OS cache but de-allocation from OS cache and allocation to application is slow, this is resulting is delay. Does this sound possible cause?