SOLARIS and DISM

Recently i ran into lot of Database Performance issues so i though of writing it here so that anyone who faces slowness in Solaris can be benefitted.

Solaris : 10
Oracle:11.2.03.
16 Core, 2 socket and 128 vCPU
Oracle T4-2 Rac One Database:
120GB RAM

Size 800GB and growing at 5GB per day.

We were running the DB with 20GB Memory without any issues for the almost 4 months and suddenly the ORA-04031 started appearing on shared Pool...after some testing we decided to increase the memory as we had 120GB Memory. THe database was configured with AMM(Memory_Max and Memory_Max_Target) parameter. We increase the Value to 30GB and set a default value of 10GB for shared pool. Here is where everything happened. The database startup took 15 minutes to start the instance and was opened. By the time the DB was started the server begin to Crawl. The DB server was absolutely slow and we had to revert to 20GB again and bounced the DB(basically downtime and pressure). The DB started fine and it was running...
Here is where all troubleshooting started..After checking lot of documents in metalink and contacting Oracle Support , The database was running on DISM(Dynamic Intimate shared Memory) which gives the felixibility to the application to lock the memory rather than OS and in this DB will lock and release memory. DISM also lock the equivalent amount of RAM on the disk swap as well. THis is where all went wrong becuase we had 30 GB of diskswap. For eg. if you want to allocate 40GB of memory to DB you need to have 40GB of disk swap allocated anything less than the Memory assingned will lead to server hanging or slow to respond as it has exhausted the disk swap.

DISM:
Oracle 40GB Memory=40GM Memory +40GB Disk swap=Vitrual swap

so the recommendation was to change the DB behavior from DISM to ISM

Unset the Memory_Max and Memory_Max_Target.

set the SGA_MAX_SIZE and SGA_TARGET to a same value. When we set this value Shared memory is allocated and locks the Memory during the startup and it doesnt care about the disk swap and the DB starts in few seconds....

If you have DISM enable make sure you have disk swap equivalent to memory allocation

ps -ef | grep dism

will display if the DISM process running..

ps -ef | grep oracle


Get the process id and check if the process is using dism or ism

pmap -xs  
will display lot of things and look for
ism shmid.......

In solaris the virtual disk usage is really hard to interpret

If using DISM

swap -l will give physically allocated size

vmstat 5 10 will display the disk space usage especially the swap column.....

in ISM

VMSTAT 5 10

now the swap columns means disk swap free + memory free...

use echo ::memstat | mdb -k
to find the accurate usage of the Memory by oracle... which will be displayed as ANON.

ZFS FILE DATA and Performance issues in Solaris 10

I wanted to start blogging just becuase of the sheer number of issues i face on production systems and thought of sharing some real tough issues.

We have quite a lot Sun T4-2 boxes for our Oracle Databases running Oracle 11.2.0.3. This server is quite a powerful server with 2 Socket(16Core) with 120GB of Memory. Off late we started to encounter ora-04031 in our production systems.We saw rapid growth in the shared pool and subsequently hitting memory issues.

Initially when the system went live we allocated  20GB of Memory to the DB with AMM Enabled. After discussion we decided to increase the Oracle DB memory to 40GB . This is where all started to happen, When we tried to bounce to database it took More than usual timing to open the DB.The Server became really slow and the database was crawling.

prstat -t shows Oracle running at 23GB and OS as 2%  but the overall system has 120GB of memory .After couple of hours after discussing with the management we decide to revert to 20GB. The system was back to normal but when it grew beyond including all virtual memories it begin to hang again....Since this was RAC system we decided to shutdown this Instance to investigate. We got a clue that this Server begins to crawl when the memory begins to go beyond 20GB....

Severity was raised with Oracle Solaris Team but they asked us to check with Oracle DB Team . But i was very clear it was something to do with some setup.

After a day of scratching my head it was found out that ZFS FILE DATA was consuming 80GB of Memory which is the maximum value(3/4 of Physcial RAM) that can be allocated to ZFS ARC SIZEwhich is the default value. Then we have decided to cap a limit for the ZFS FILE DATA with the parameter ZFS_ARC_MAX in the etc/system .The reason for why ZFS FILE DATA went to 75% is unknown :(

It was really a nightmare to see a FIle system cache occupying 80GB.

Prstat -t --will never accurately give the usage

vmstat 5 10 should give more accurate free  memory usage

mdb -k then followed by ::memstat will give clear picture on the memory usage

NIce day.