RAM Usage

asitagrawal · Post by **asitagrawal** » Fri Jun 15, 2007 4:58 pm

Hi,

The Hardware of the DataStage server has 12 GB RAM...
but the windows administrator is complaining that the processes are not using more than 2 GB of RAM... He says that the processes are not utilizing the availiable memory ... Why ?

What do I explain him now ? or What do I modify ?
Please suggest.

ArndW · Post by **ArndW** » Fri Jun 15, 2007 9:33 pm

The amount of memory used by programs is mostly dependant upon how you write your jobs. There are some possible configuration changes that can make DS use more memory but it is doubtful that they will speed up processing appreciably.

If you design your jobs to use more memory (i.e. loading lookups to memory) you will make your system administrators happy and will also speed up execution.

Perhaps you could explain what you are doing on your system to get speific recommendations.

chulett · Post by **chulett** » Fri Jun 15, 2007 10:40 pm

Funny, mine complain when we do utilize all the available memory.

ray.wurlod · Post by **ray.wurlod** » Fri Jun 15, 2007 10:58 pm

DataStage is a 32-bit application. The highest address you can access in any one memory space with a signed 32-bit integer is, curiously enough, 2GB.

To address more memory than that requires some fairly fancy coding.

But, then, why should it grab more than it needs?

asitagrawal · Post by **asitagrawal** » Mon Jun 18, 2007 10:47 am

ray.wurlod wrote:But, then, why should it grab more than it needs?

Yes, even I agree with this statement... but how to justify that there is no place in DataStage where I can configure a top limit if memeory usage...

The Win Admin wants to know where all in DataSatge, can the memeory usage be configured ??

asitagrawal · Post by **asitagrawal** » Mon Jun 18, 2007 10:53 am

Also to highlight the complaint of the Win Admin... he says, that the processess are not using all available memory and causing high page faults... Why ??

Please advice.

ray.wurlod · Post by **ray.wurlod** » Mon Jun 18, 2007 2:48 pm

Tell the Win Admin nowhere

DataStage is a 32-bit application. It uses malloc() to request only the memory it requires.

Why is that a problem?

If your Win Admin can't understand that, a job with Micro$oft is probably beckoning.

ArndW · Post by **ArndW** » Mon Jun 18, 2007 3:34 pm

asitagrawal wrote:...not using all available memory and causing high page faults...

A page fault occurs when a page requested by a program is not in memory and needs to be fetched from elsewhere. Often this is corrupted and misinterpreted to mean that the system is paging (particularly in Windows environments). Note that paging is very different from generating page faults. If your system is paging while there is still unused memory then you will need to consult a system administrator who understands such things on Windows.

IF a DS job only needs n-Mb to run, then trying to allocate it more memory won't magically make it run faster. As most DS job use files (as input, output or intermediate hashed files), if you have free system memory you will be better served in having that space used for file I/O buffering than by artificially inflating DataStage tunable values.