Change Capture & Change apply to a dataset

DSFreddie · Post by **DSFreddie** » Mon Aug 13, 2012 12:17 pm

Hi All,

I have a requirement where I need to first capture the changed records when compared with a full file & daily delta. (Both are datasets). Also, once the changed records are identified, these records need to be updated back to the Full file to keep it current & up-to-date.

Can you pls let me know the best way to accomplish this scenario. (All the files are datasets).

Thanks Much
Freddie.

ray.wurlod · Post by **ray.wurlod** » Mon Aug 13, 2012 3:06 pm

Have you tried using Change Capture and Change Apply stage types?

DSFreddie · Post by **DSFreddie** » Thu Aug 16, 2012 10:09 am

Thanks Ray for your inputs

The change capture/Apply seems to work. The new question that is arising in my mind is on the performance of these stages.

In the Full File, we have around 500 million records. The delta file contains 10 million (Daily). I am hearing that the change capture/change apply stages are not the right option since it takes more processing time.

Can you pls share your knowledge on this ? Also, it would be great if you have any other ideas of handling this.

Thanks Much
Freddie

BI-RMA · Post by **BI-RMA** » Fri Aug 17, 2012 12:59 am

You are hearing from whom? And it takes more processing time than what? Change Capture is a about as fast as the I/O throughput of your DataStage server allows.

Since you are reading files, costly options like IBM CDC or Oracle Golden Gate - utilising database changelogs to identify changed records in a database directly - are not an option.

Any other ideas? There are possibly a number of ways to use other stages as a workaround for this. But datastage provides a specialized solution for exactly the purpose you want: so use it!