Is it possible to assign an amount of RAM for Datastage?

drostnu · Post by **drostnu** » Tue Jun 10, 2008 2:31 pm

Hello All,

I was wondering if its possible to assing a certain amount of ram to Datastage, since after the tomcat i have running, it takes all the ram and server goes slow, so does tomcat too.

Is there anyway to set this up on any file of datastage?

Thanks in advance to all

Diego.

lstsaur · Post by **lstsaur** » Tue Jun 10, 2008 2:40 pm

No, just throw in more RAM to your server box.

ArndW · Post by **ArndW** » Wed Jun 11, 2008 1:24 am

There are job design methods you can use to control memory (i.e. not cacheing hashed lookups will use less memory, but work slower) and a couple of the uvconfig settings also affect memory use. But you need to know that by doing so you are going to seriously slow down DS processing and it is much cheaper, simpler & more efficient to add memory.

ray.wurlod · Post by **ray.wurlod** » Wed Jun 11, 2008 1:33 am

Try starting DataStage services before you start tomcat.

DataStage only takes the memory it needs, using specialized malloc() and free() calls.

iDomz · Post by **iDomz** » Thu Jun 12, 2008 3:50 am

Based on which shell(and unix flavour) you are on, you can use limit or ulimit to restrict physical memory usage.

syntax for limit is 'limit memorysize 1024M' which limits all programs executed after this command to 1Gig of ram( programs start swapping after this limit is reached)

I believe syntax for ulimit is 'ulimit memory mem-in-kb', though I don't have a unix box to check now.

run ulimit command on your box - possibly it will be now showing a line as memory unlimited

Be pretty careful with this - Setting a wrong hard limit will cause big perf issues, esp for ram intensive tasks. Check with an experienced admin before you try (I could be wrong as I havent tried these for years)

HTH,
D

ArndW · Post by **ArndW** » Thu Jun 12, 2008 3:58 am

iDomz - that approach can work and is a good thought, but since PX jobs run in many processes and the ulimit setting is per process you will only achieve a very indirect control.

ag_ram · Post by **ag_ram** » Thu Jun 12, 2008 5:28 am

ray.wurlod wrote:DataStage only takes the memory it needs, using specialized malloc() and free() calls.

ray.wurlod - Could you let us know more of the statement you declared?(curious to know)

1. Is this only applicable to Server Job?

2. Is there any internal BASIC Subroutine/C program for memory management?

Please clarify.

ArndW · Post by **ArndW** » Thu Jun 12, 2008 5:57 am

the internal malloc() and free() statements are not documented.

No, this is not applicable to only server jobs.
No, there is no internal program for memory management.

Server strings are not contiguous, so Server tends to allocate smaller chunks of memory at a time and only when necessary - this allow strings to grow to the size of available process virtual memory; unlike other applications where the c-string constructs require contiguous blocks for string storage.

iDomz · Post by **iDomz** » Thu Jun 12, 2008 6:14 am

ArndW,
Regarding your earlier post, You are absolutely right - It is very difficult to control limits at user level, so the approach would not be practical for Px jobs.

That said, it is not impossible. There is a config file (actually, two) in most *nix where you can set limits at user level. All kernels have different approaches as to whether to honour these limits - vanilla linux kernels don't AFAIK.

99.99% cases, you really ought not be deciding memory requirements at app level.. you'll know if you have used mac os classic