Lookup - How big a reference data is big?

Maximus_Jack · Post by **Maximus_Jack** » Wed Mar 06, 2013 8:20 pm

Hi
One basic question,

I know it is better to use lookup stage when the amount of reference data is
is not huge, but how big is big, based on what the reference data is
considered as big or small, is it based on RAM memory availability or
something else

your response is appreciated

cheers
MJ

singhald · Post by **singhald** » Thu Mar 07, 2013 12:27 am

if your reference data is small enough to fit into RAM, job throughput will be more

chulett · Post by **chulett** » Thu Mar 07, 2013 7:29 am

The answer changes based on 32bit v. 64bit architecture. The former has a 2GB limit while the latter does not.

zulfi123786 · Post by **zulfi123786** » Thu Mar 07, 2013 8:34 am

Also look for user limits defined for heap memory/data size for unix users

zulfi123786 · Post by **zulfi123786** » Thu Mar 07, 2013 8:35 am

chulett wrote:The answer changes based on 32bit v. 64bit architecture. The former has a 2GB limit while the latter does not.

Was wondering, what if on a 32bit AIX machine has heap/data limit defined as unlimited, Will it override the 2GB constraint ?

chulett · Post by **chulett** » Thu Mar 07, 2013 8:48 am

Nope. It's a hard-wired limit.

priyadarshikunal · Post by **priyadarshikunal** » Thu Mar 07, 2013 10:32 am

I remember changing the AIX system to use large file support. I think Arnd posted what we need to change on AIX, probably ldedit?

Maximus_Jack · Post by **Maximus_Jack** » Thu Mar 07, 2013 10:58 am

two questions

1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we
can go for a lookup stage if the reference data is 1.00 GB?

2) So in a 64 Bit machine, where heap memory has no limit ( if i understand
correctly) , what is the safe bet for the limit of reference data if i want to
use lookup stage

cheers
Mj

zulfi123786 · Post by **zulfi123786** » Thu Mar 07, 2013 12:49 pm

Maximus_Jack wrote:1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we can go for a lookup stage if the reference data is 1.00 GB?

there are lot many processes running over the server and using up 50% of memory just for one stage is not wise.

Maximus_Jack wrote:2) So in a 64 Bit machine, where heap memory has no limit ( if i understand correctly) , what is the safe bet for the limit of reference data if i want to use lookup stage

These is no standard limit and usually few hundreds of MB should not pose a big problem but again depends on lot of factors.

Maximus_Jack · Post by **Maximus_Jack** » Thu Mar 07, 2013 2:10 pm

thanks a lot guy.. very informative

priyadarshikunal · Post by **priyadarshikunal** » Thu Mar 07, 2013 3:06 pm

zulfi123786 wrote:
Maximus_Jack wrote:1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we can go for a lookup stage if the reference data is 1.00 GB?
there are lot many processes running over the server and using up 50% of memory just for one stage is not wise.

Well assuming the server has only 2 GB memory is also not wise

. The limit is 2 GB for a file in memory, not for entire system. Unless the processes are not eating up the entire memory it should be fine.

I do support the second answer though.

zulfi123786 · Post by **zulfi123786** » Thu Mar 07, 2013 11:41 pm

priyadarshikunal wrote:
zulfi123786 wrote:
Maximus_Jack wrote:1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we can go for a lookup stage if the reference data is 1.00 GB?
there are lot many processes running over the server and using up 50% of memory just for one stage is not wise.
Well assuming the server has only 2 GB memory is also not wise . The limit is 2 GB for a file in memory, not for entire system. Unless the processes are not eating up the entire memory it should be fine.

Right, The intention was relativity

. The server that host our Datastage Engine has 32 GB primary memory