Incremental Load with Multiple Sources

Havoc · Post by **Havoc** » Wed Jul 11, 2007 8:44 am

This is more of a solution question than a DataStage question. We have a requirement as follows:

There are two or more source tables each having different load frequencies(meaning these tables might get refreshed on different dates). Now, these two sources load the same target table.

The approach used when loading this target table is Incremental Load.

Is there a way we can load this target table ensuring that there is no data loss because data from one source might get loaded on one day while the next one is not. Is there a method to go about doing this?

There are no timestamps in the target table.

Thanks in advance

ray.wurlod · Post by **ray.wurlod** » Wed Jul 11, 2007 10:18 am

The best approach would be to write out a detailed specification of what needs to happen under various scenarios, and to design accordingly.

It would be really useful if the target included some way to recognize when it was most recently updated - either a timestamp or a unique run ID. Even though you state that it doesn't, your specification may be used to prove that it is necessary. ALTER TABLE can add a new column.