Extracting data from source

waitnsee · Post by **waitnsee** » Mon Jul 26, 2004 10:54 am

how do I ensure that, I am not reading the same record from the source?
Is there any specific mechanism?

thanks.

VPR

kduke · Post by **kduke** » Mon Jul 26, 2004 11:38 am

I think that you need to store the source table's primary key in the target table or a permanent staging table.

ray.wurlod · Post by **ray.wurlod** » Mon Jul 26, 2004 2:08 pm

Somewhere you need to record that you've processed it. If there's an incrementing key in the source, all you need to record is the maximum key value processed, and select any key larger. If the records in the source are timestamped, you can select on this basis. Store the primary keys when extracting; at worst you can use a NOT IN join when selecting the next batch. There are other ways too. Know your data and be creative in thinking about your ETL techniques!