I wanted to start blogging just becuase of the sheer number of issues i face on production systems and thought of sharing some real tough issues.
We have quite a lot Sun T4-2 boxes for our Oracle Databases running Oracle 11.2.0.3. This server is quite a powerful server with 2 Socket(16Core) with 120GB of Memory. Off late we started to encounter ora-04031 in our production systems.We saw rapid growth in the shared pool and subsequently hitting memory issues.
Initially when the system went live we allocated 20GB of Memory to the DB with AMM Enabled. After discussion we decided to increase the Oracle DB memory to 40GB . This is where all started to happen, When we tried to bounce to database it took More than usual timing to open the DB.The Server became really slow and the database was crawling.
prstat -t shows Oracle running at 23GB and OS as 2% but the overall system has 120GB of memory .After couple of hours after discussing with the management we decide to revert to 20GB. The system was back to normal but when it grew beyond including all virtual memories it begin to hang again....Since this was RAC system we decided to shutdown this Instance to investigate. We got a clue that this Server begins to crawl when the memory begins to go beyond 20GB....
Severity was raised with Oracle Solaris Team but they asked us to check with Oracle DB Team . But i was very clear it was something to do with some setup.
After a day of scratching my head it was found out that ZFS FILE DATA was consuming 80GB of Memory which is the maximum value(3/4 of Physcial RAM) that can be allocated to ZFS ARC SIZEwhich is the default value. Then we have decided to cap a limit for the ZFS FILE DATA with the parameter ZFS_ARC_MAX in the etc/system .The reason for why ZFS FILE DATA went to 75% is unknown :(
It was really a nightmare to see a FIle system cache occupying 80GB.
Prstat -t --will never accurately give the usage
vmstat 5 10 should give more accurate free memory usage
mdb -k then followed by ::memstat will give clear picture on the memory usage
NIce day.
No comments:
Post a Comment