DSXchange

Hello everyone,

I have a question on how Datastage manages memory at the service level and within the execution of jobs.

The environment:

Datastage EE 8.1 on a Windows Server 2003 Enterprise Edition, service pack 2.
(2 x quad core Intel Xeon CPUs, 8,00GB Ram with PAE enabled)

The consultant from IBM who works with us is strongly suggesting that we move to the 8.5 64-bit version when it becomes

available. The reason given is that DS on a 32-bit environment has "memory issues". I'm not ready to swallow this without

having my doubts cleared.

I understand that applications running on 32-bit windows will only be able to utilize up to 2GB of RAM. A good example is

SQL Server, whereby the service cannot consume more than 2GB of RAM. My question - is Datastage the same?

My understanding is that given unlimited RAM, the DS service itself will still limit itself to 2GB. However, each job that

is spawned will also be able to consume with another 2GB of RAM each. Is my understanding correct?

Additionally, where does the memory cost of xmeta and lookups go - the main DS service?

Thanks in advance for your input,
Rama.

This post should help you get your answer:

http://www.b-eye-network.com/blogs/lins ... es_vld.php

DataStage is basically the same, though there is a patch for Windows Server 2003 that allows as much as 3GB of memory to be accessed in a 32-bit environment.

I was thinking you were talking about PAE but that's actually for usage of more than 4GB of memory in a 32bit world. Not sure what patch you would be refering to... this would be a Windows patch or a DataStage patch?

ray.wurlod wrote:DataStage is basically the same, though there is a patch for Windows Server 2003 that allows as much as 3GB of memory to be accessed in a 32-bit environment. ...

Ray, Julio, Craig,

Thanks for replying so quickly. The link has been most helpful and I now know that the DS service will consume not more than 2GB of RAM.

The part I'm not clear is about the individual jobs that get executed. As I understand, jobs are spawned with layers of controller, section leaders and worker components working in tandem.

Since the jobs are seperate processes, will the jobs be able to take 2GB of RAM each, or are the jobs supposed to take what is left for the DS service?

Thanks and regards,
Rama

They will demand varying amounts, perhaps as much as 2GB, perhaps not. Use the Resource Estimation tool to get numbers. If the system can not supply all resources demanded, some waiting (for CPU) or paging (of memory) is likely to occur.

Thanks for so much to everyone who replied. Much appreciated.

During the weekend I carried out some simple tests to add to my understanding of the topic I posted.

I created a job with 2 Row Generators, 1 acting as an input and another as a reference link. The Lookup stage connected both the input and reference links and the matching and reject outputs were sent to 2 Copy stages respectively.

I configured the reference RG to produce 20000000 rows of numbers, just to make sure that the job ran long enough for me to make sense out of what I see in the Task Manager. I also wanted to make sure that the RAM spiked to a noticable level.

I ran the job and noticed that the dsservice.exe staying very stable at around 6MB the whole 3 minutes. I also noticed a whole bunch of osh.exe, uv*.exe and nut*.exe being spawned. These executables used RAM to varying degrees throughout the process, with one of the osh.exe instance spiking up to 300MB at the very end.

I guess the osh.exe set that got spawned took care of executing the stages in the job as seperate processes. It also looks like each one of the osh.exe will be able to go up to 2GB of RAM (providing infinite amount of RAM is made available) and but the dsservice.exe will stay out of the way and be happy with minimal RAM.

If this is correct, then I'm a happy man. I won't be pushing for a move to 64-bit, mainly because the jobs in my environment don't look to be candidates that can reach the 2GB limit per process. I'll just have to be carefull in controlling the number of jobs that get spawned together so that paging can be avoided.

Providing the above, I feel that DS is then also different from SQL Server. SQL Server spawns off 1 exe into the Task Manager and this is the whole db environment limited to 2GB of RAM. DS on the other hand, spawns multiple exe processes to get jobs executed, so the limit is within a process of a job.

Hope I am on the right track