how do I ensure that, I am not reading the same record from the source?
Is there any specific mechanism?
thanks.
VPR
Extracting data from source
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Somewhere you need to record that you've processed it. If there's an incrementing key in the source, all you need to record is the maximum key value processed, and select any key larger. If the records in the source are timestamped, you can select on this basis. Store the primary keys when extracting; at worst you can use a NOT IN join when selecting the next batch. There are other ways too. Know your data and be creative in thinking about your ETL techniques!
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.