Job failing in lookup Stage

just4geeks · Post by **just4geeks** » Tue Mar 06, 2012 2:49 pm

Hello

We worked around above issue by replacing lookup by join stage but recently I encountered another issue with a similar job (with lookup) in same environment which confused me:

Earlier message was:

Could not map table file "/opt/IBM/InformationServer/Server/Datasets/lookuptable.20111207.hezfuxb (size 3938972368 bytes)": Cannot allocate memory

But I am also getting another message from lookup stage in similar job:

Requested memory for table file :"/opt/IBM/InformationServer/Server/Datasets/lookuptable.20120304.p3odhbc" (size 2292096328 bytes) exceeds limit. Please consider using join instead of lookup

-What is difference in these two messages?
- What is limit- is it 4 GB or 2 GB or anything else?
-is it a DataStage limitation or limit imposed by Linux OS?

As stated above , we are using InfoSphere 8.1 FP1 ( 32 bit) with Linux Suse (64 bit)

I am really confused. Any help/guidance will be highly appreciated.
Please let me know if you need more details.

Thanks in Advance.

kwwilliams · Post by **kwwilliams** » Wed Mar 07, 2012 11:38 am

The first says you didn't have available memory to allocate, the second says that you have exceeded the physical limitation of memory allowed.

just4geeks · Post by **just4geeks** » Wed Mar 07, 2012 4:17 pm

Thanks Keith!

Assuming what you are saying is true-In that case first error message should never occur because second condition ( physical limit) will be encountered first always since 2GB is less than 4 GB. Therefore second error message should always appear even before it satisfies the condition for second error. and thats what I am confused about.

Thanks for help!

qt_ky · Post by **qt_ky** » Wed Mar 07, 2012 6:32 pm

You have a 32-bit software version imposing its limit of 4 GB or roughly 3.66 GB and your OS can also impose any limit (see ulimit settings discussed earlier). You can upgrade your software to a 64-bit version and increase OS ulimit settings and/or you can follow the rule of thumb that says don't use lookups for everything, or more specifically, only use Lookup stages for reference data that will fit into the limited amount of physical memory available according to your software/OS combination. If you've started replacing very large/error-prone lookups with Join or Merge stages, then I think you're on the right track.