Lookup - How big a reference data is big?
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 139
- Joined: Fri Apr 11, 2008 1:02 pm
Lookup - How big a reference data is big?
Hi
One basic question,
I know it is better to use lookup stage when the amount of reference data is
is not huge, but how big is big, based on what the reference data is
considered as big or small, is it based on RAM memory availability or
something else
your response is appreciated
cheers
MJ
One basic question,
I know it is better to use lookup stage when the amount of reference data is
is not huge, but how big is big, based on what the reference data is
considered as big or small, is it based on RAM memory availability or
something else
your response is appreciated
cheers
MJ
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
-
- Premium Member
- Posts: 139
- Joined: Fri Apr 11, 2008 1:02 pm
two questions
1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we
can go for a lookup stage if the reference data is 1.00 GB?
2) So in a 64 Bit machine, where heap memory has no limit ( if i understand
correctly) , what is the safe bet for the limit of reference data if i want to
use lookup stage
cheers
Mj
1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we
can go for a lookup stage if the reference data is 1.00 GB?
2) So in a 64 Bit machine, where heap memory has no limit ( if i understand
correctly) , what is the safe bet for the limit of reference data if i want to
use lookup stage
cheers
Mj
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
there are lot many processes running over the server and using up 50% of memory just for one stage is not wise.Maximus_Jack wrote:1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we can go for a lookup stage if the reference data is 1.00 GB?
These is no standard limit and usually few hundreds of MB should not pose a big problem but again depends on lot of factors.Maximus_Jack wrote:2) So in a 64 Bit machine, where heap memory has no limit ( if i understand correctly) , what is the safe bet for the limit of reference data if i want to use lookup stage
- Zulfi
-
- Premium Member
- Posts: 139
- Joined: Fri Apr 11, 2008 1:02 pm
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
Well assuming the server has only 2 GB memory is also not wise . The limit is 2 GB for a file in memory, not for entire system. Unless the processes are not eating up the entire memory it should be fine.zulfi123786 wrote:there are lot many processes running over the server and using up 50% of memory just for one stage is not wise.Maximus_Jack wrote:1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we can go for a lookup stage if the reference data is 1.00 GB?
I do support the second answer though.
Priyadarshi Kunal
Genius may have its limitations, but stupidity is not thus handicapped.
Genius may have its limitations, but stupidity is not thus handicapped.
-
- Premium Member
- Posts: 730
- Joined: Tue Nov 04, 2008 10:14 am
- Location: Bangalore
Right, The intention was relativity . The server that host our Datastage Engine has 32 GB primary memorypriyadarshikunal wrote:Well assuming the server has only 2 GB memory is also not wise . The limit is 2 GB for a file in memory, not for entire system. Unless the processes are not eating up the entire memory it should be fine.zulfi123786 wrote:there are lot many processes running over the server and using up 50% of memory just for one stage is not wise.Maximus_Jack wrote:1) So in a 32 bit machine with a heap memory of 2GB, Is it safe to say we can go for a lookup stage if the reference data is 1.00 GB?
- Zulfi