DSXchange

We are in the process of migrating the jobs from version 8.5 to version 11.3.


In version 8.5, the job runs with out warnings.
In version 11.3 , the same job runs with a warning.

warning:

"Ignoring duplicate entry at table record 003; no further warnings will be issued for this table "

Job design is explained below.

Job 1: Reads data from a sql server table and loads into dataset. The dataset is ENTIRE partitioned.
Job 2: Uses the dataset in a reference link for a lookup. The reference link is SAME partition and the main link is HASH partitioned.


As a fix for this warning: I did enable the multiple rows from reference link. After the fix, the job is producing duplicate records from the lookup.

Can anyone help me to fix the issue?

Is your source has got duplicates this time? You might not have those when you ran last time.

Insert a Remove Duplicates stage in the Reference input link to the Lookup stage.

Thanks Ray.

Currently the jobs are in production in 8.5. The client wants to migrate the code to 11.3 with no change. Installing patch will help to fix this issue? Need suggestions.


Current version is
DataStage Core/Client Rollup Patch V2
11.3.1.0

Since there's no more recent patch than R2, the answer to your patch question is currently "no". Version 11.3 is being more rigorous at detecting various things (such as duplicates in the reference data set) than version 8.5. You have been relying upon a deficiency in the older version.

You can use a message handler to demote the message to informational.