High Memory Utilization

[samyamkrishna]

Hi,

I am just checking the health of our DS servers for a week now.
The reports consistantly show high memory utilization usually around 90%.
The memory on the servers is 128 GB

Is this something to be worried about? like bad design or need of performance tuning?

Regards,
Samyam

[PaulVL]

Well, you need to understand your environment.

Is is it a cluster environment where the data crunching happens on a compute node (thus freeing up memory on your head node).

Is is running a ton of job at the time of monitoring?

Do you have shell scripts doing non (pure) datastage work on the host?

What is the "top" or "topas" telling you?

Have you run nmon to get system stats to tell you who is chewing up your memory?


We can't answer those questions for you.

[samyamkrishna]

Thanks for the reply.
I will check all of this and see if I can come to some conclusion.

[chulett]

Out of curiousity, what O/S are you running?

[samyamkrishna]

Hi,

Its red hat linux 2.6.32

Regards,
Samyam

[samyamkrishna]

Hi All,

The top -m command gives the below result.

top - 14:02:20 up 40 days, 4:40, 5 users, load average: 3.91, 2.80, 2.71
Tasks: 1117 total, 1 running, 1106 sleeping, 0 stopped, 10 zombie
Cpu(s): 0.2%us, 0.5%sy, 0.0%ni, 99.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 98871628k total, 98052996k used, 818632k free, 1452992k buffers
Swap: 1048568k total, 0k used, 1048568k free, 90826696k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23244 root 25 0 1140m 192m 10m S 0.3 0.2 425:44.65 java
27380 dsadmin 16 0 911m 75m 16m S 0.0 0.1 17:55.93 osh
23209 root 25 0 743m 75m 7248 S 0.0 0.1 25:50.35 java
27289 dsadmin 16 0 901m 70m 16m S 0.0 0.1 17:36.29 osh
27381 dsadmin 16 0 912m 69m 16m S 0.0 0.1 17:21.65 osh
27372 dsadmin 16 0 901m 69m 16m S 0.0 0.1 20:04.11 osh
27377 dsadmin 18 0 901m 67m 16m S 0.0 0.1 17:53.93 osh
26326 dsadmin 19 0 248m 58m 7812 S 0.0 0.1 0:01.68 osh
26401 dsadmin 18 0 248m 58m 7812 S 0.0 0.1 0:01.46 osh
and more and more osh process say about a 100 more osh processes.

Is it normal to have so many osh running all the time?
Also there are 1106 processes sleeping is that normal too?..

Regards,
Samyam

[chulett]

How many jobs were running?

[ray.wurlod]

As a general rule every stage on every node creates an osh process, and there is one extra osh process (the "section leader") per node per job, and one extra osh process (the "conductor") per job. The more nodes you have, and the more stages you have, the more osh processes you will see.

DataStage is not always brilliant at cleaning up osh processes at job end, particularly where jobs abort, but they should be cleaned up eventually.

[samyamkrishna]

There are 6 ISD jobs which are always runnig thats it.

[qt_ky]

Yes, that's probably normal. Even a couple hundred osh processes always running from ISD jobs may be normal. You can verify normal by checking the same counts at various times.

If you're running a default setup on an SMP server, then by default DB2 will allocate 85 to 90% of the physical memory at startup whether it needs it or not. You can override that behavior if you want to and give it a hard upper limit based on a number 4K pages that you specify.

I probably wouldn't worry about it unless you're seeing jobs run unusually slow or jobs or other processes abort due to out of memory errors or the like.

[chulett]

I probably wouldn't worry about it unless you're seeing jobs run unusually slow or jobs or other processes abort due to out of memory errors or the like.

Agree. Why spend money on perfectly good memory and then not use it?

[samyamkrishna]

There are no performance issues at all.
was just a bit worried to see the memory usage high all the time, but not anymore.

Thanks for all your responses.

Regards,
Samyam