Lookup in a Grid environment

[vnspn]

Hi,

We are working in a DataStage Grid environment with multiple compute nodes. We came across a situation that worked unexpectedly.

We run a job using 2 compute nodes and 2 partitions each. The job works as expected. The lookup to an Oracle table fetches correct reference data.

When we run the same same job with 5 compute nodes, the lookup returns null value for all the incoming records.

The Lookup stage has both the input links Hash partitioned on the key column.

Could anyone suggest what could cause a lookup not to work when increasing the compute nodes.

Thanks.

[lstsaur]

Check your job's log to see whether all five lookup.0-4 have records imported successfully instead of records rejected.

[ray.wurlod]

What partitioning algorithm is being used on the reference link? What stage type supplies records to the reference link?

[vnspn]

lstsaur, I checked the log and it does not show this information. The reference table has 5 records and I see the same count that shows on the link using performance statistics. I'm sure the records are not being dropped.

ray, the partitioning algorithm on the reference link is Hash partition on the key column used for join. The stage that supplies records to the reference link is an Oracle Enterprise stage.

[lstsaur]

In job's log should show 10 entries for the lookup stage. You didn't see any?

[ray.wurlod]

What happens if you use Entire as the partitioning algorithm on the reference input link? I'm asking as a diagnostic question, not necessarily with a view to recommending this as the solution.

[vnspn]

lstsaur, no, the job log did not show entries from the link where reference data is fetched. Infact we are using a join stage for fetching the reference data and not a lookup stage.

ray, we did not try using Entire partition. Will check that and see what happens...