DSXchange

Hi
One basic question,

I know it is better to use lookup stage when the amount of reference data is
is not huge, but how big is big, based on what the reference data is
considered as big or small, is it based on RAM memory availability or
something else

your response is appreciated

cheers
MJ

if your reference data is small enough to fit into RAM, job throughput will be more

The answer changes based on 32bit v. 64bit architecture. The former has a 2GB limit while the latter does not.

Also look for user limits defined for heap memory/data size for unix users

chulett wrote:The answer changes based on 32bit v. 64bit architecture. The former has a 2GB limit while the latter does not.

Was wondering, what if on a 32bit AIX machine has heap/data limit defined as unlimited, Will it override the 2GB constraint ?

Nope. It's a hard-wired limit.

I remember changing the AIX system to use large file support. I think Arnd posted what we need to change on AIX, probably ldedit?

two questions

1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we
can go for a lookup stage if the reference data is 1.00 GB?

2) So in a 64 Bit machine, where heap memory has no limit ( if i understand
correctly) , what is the safe bet for the limit of reference data if i want to
use lookup stage

cheers
Mj

Maximus_Jack wrote:1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we can go for a lookup stage if the reference data is 1.00 GB?

there are lot many processes running over the server and using up 50% of memory just for one stage is not wise.

Maximus_Jack wrote:2) So in a 64 Bit machine, where heap memory has no limit ( if i understand correctly) , what is the safe bet for the limit of reference data if i want to use lookup stage

These is no standard limit and usually few hundreds of MB should not pose a big problem but again depends on lot of factors.

thanks a lot guy.. very informative

zulfi123786 wrote:

Maximus_Jack wrote:1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we can go for a lookup stage if the reference data is 1.00 GB?

there are lot many processes running over the server and using up 50% of memory just for one stage is not wise.

Well assuming the server has only 2 GB memory is also not wise . The limit is 2 GB for a file in memory, not for entire system. Unless the processes are not eating up the entire memory it should be fine.

I do support the second answer though.

priyadarshikunal wrote:

zulfi123786 wrote:

Maximus_Jack wrote:1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we can go for a lookup stage if the reference data is 1.00 GB?

there are lot many processes running over the server and using up 50% of memory just for one stage is not wise.

Well assuming the server has only 2 GB memory is also not wise . The limit is 2 GB for a file in memory, not for entire system. Unless the processes are not eating up the entire memory it should be fine.

Right, The intention was relativity . The server that host our Datastage Engine has 32 GB primary memory