lookup stage partition method issue

[abyss]

Hi all
I created a job involve one lookup stage with 10 ref links. each lookup table have 200 - 80k rows. I hash partitioned each ref link by their unique key and sort&partitioned main input link by all the foreign key columns.

when i run the job it returns 'mapping file error'. i know by default partition method is 'entire' to ensure every node have all the lookup data.

if i change the partition method to auto or entire job returns different error messages: 'Node <node name> could not map table file <dataset degment dir> Not enough space, retrying mmap' and 'not enough space in temp dir'

my question is: what's the best partition method for this case? or it is just wrong to have 10 ref links on lookup stage, i should reduce number of ref links or use merge stage?

another question: i set the environment variable TMPDIR to F:/TEMP(it is a local drive) but why the log file shows me the temp file dir is still c:/temp?

thanks
abyss

[abyss]

just read this from IBM:
Fatal Error: APT_Communicator::shmemInitBuffers: createMapFile (/tmp) failed: Not enough space on node.

This message can also be caused by system-wide limit on the number of mmap'ed shared memory segments. Often this issue occurs when the value for APT_DEFAULT_TRANSPORT_BLOCK_SIZE is set too high. Please check the environmental variable APT_DEFAULT_TRANSPORT_BLOCK_SIZE. The default for this variable is 131072 or 128 KB. The maximum value should be is 1048576 or 1MB. Please see page 79 on the Parallel Job Advanced Developer Guide for additional information regarding this environmental variable.

Resolving the problem
To resolve this issue, ensure that there is adequate space in the assigned TMPDIR (temporary directory) or verify the settings on the environmental variable APT_DEFAULT_TRANSPORT_BLOCK_SIZE.

hmmmm...... maybe i should look into it.

[Mike]

and sort&partitioned main input link by all the foreign key columns.

It seems your understanding of how partitioning works is flawed.

Besides $TMPDIR, please check for the settings of $TMP and $TEMP in your job log. If present, they will be listed in the second log entry.

You can use 1 lookup stage with 10 links or 10 lookup stages with a single link each, but you do need to get the partitioning correct for correct results.

A good way to test your partitioning logic is to run the job twice: 1) with your default multi-node parallel configuration, and 2) with a 1-node parallel configuration. If you observe differences in link counts, your partitioning logic is bad.

Mike

[chulett]

Perhaps this is something you've already looked into but there are 48 other posts that mention lookup stage partitioning and that "could not map table file" error message and the things people have done to resolve it. For whatever it is worth, here are my adventures with the problem from back in the day.

[abyss]

Besides $TMPDIR, please check for the settings of $TMP and $TEMP in your job log. If present, they will be listed in the second log entry.

Mike, i can not find $TMP and $TEMP in the environment variables list. do you mean system $TEMP variable?

A good way to test your partitioning logic is to run the job twice...

ATM i can not run the job at all as it gives me error. I understand i should partition the values, however I am not sure how to partition the data. I believe I should hash partition the ref links by the lookup key column like join stage. how should I partition the main input then?

[abyss]

'Each physical process can address only up to 4 GB of memory because it is a 32-bit application. The Windows version of the InfoSphere DataStage Parallel Engine is available only with 32-bit pointers. '

ha, got it!