Page 1 of 1
Extracting data from source
Posted: Mon Jul 26, 2004 10:54 am
by waitnsee
how do I ensure that, I am not reading the same record from the source?
Is there any specific mechanism?
thanks.
VPR
Posted: Mon Jul 26, 2004 11:38 am
by kduke
I think that you need to store the source table's primary key in the target table or a permanent staging table.
Posted: Mon Jul 26, 2004 2:08 pm
by ray.wurlod
Somewhere you need to record that you've processed it. If there's an incrementing key in the source, all you need to record is the maximum key value processed, and select any key larger. If the records in the source are timestamped, you can select on this basis. Store the primary keys when extracting; at worst you can use a NOT IN join when selecting the next batch. There are other ways too. Know your data and be creative in thinking about your ETL techniques!