how to identify older and new records from same file

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
sathyak098
Participant
Posts: 23
Joined: Tue May 14, 2013 6:34 am

how to identify older and new records from same file

Post by sathyak098 »

Hi,
I have a scenario in which I receive files on hourly basis.
Sometimes I may receive file with data of mutiple hours.(ie., not only data of that hour alone, but also data of its previous hours).
Data will be in ascending order ( I mean, data of old hours at first and data of latest hour at last ). I can expect records in such a way that, records which i need to insert new (will be on old hour) and same records may need to update(it will be in latest hour) will be in same file.

The approach that I'm following is
while reading the file, I have created row number in sequential file stage.
In the loading job, I'm using single node config file and sorting the data based on Row num and based on primary column, i'm deciding whether this record need go to insert or update

Please guide me if the above approach is not good
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

From your description the row number you generate is enough to give an order by time.
Post Reply