Issues in selecting the last occurance in the datastage job
Posted: Thu Apr 30, 2009 11:25 am
Hi,
I have a datastage job which uses a dataset as the source (this inturn is created by a file that comes from mainframes) and the job has to remove duplicates based a column and retain the last occurance.
In the mainframe file there are 3 occurances for the same column. Basically there are some other columns that are different in these occurances. Once the datastage job is complete the job loads the unique records to a dataset and then inserts to a table.
The issue here is - the last occurance record what I see in the mainframe file is diffferent than the one I see in the table.
Some times the job picks first occurance, some times the last occurance, the configuration file I use has 4 nodes.
Can some one please help me explain why the job is not picking the last occurance correctly?
thanks,
Vij
I have a datastage job which uses a dataset as the source (this inturn is created by a file that comes from mainframes) and the job has to remove duplicates based a column and retain the last occurance.
In the mainframe file there are 3 occurances for the same column. Basically there are some other columns that are different in these occurances. Once the datastage job is complete the job loads the unique records to a dataset and then inserts to a table.
The issue here is - the last occurance record what I see in the mainframe file is diffferent than the one I see in the table.
Some times the job picks first occurance, some times the last occurance, the configuration file I use has 4 nodes.
Can some one please help me explain why the job is not picking the last occurance correctly?
thanks,
Vij