How to implement Change Data Capture in PX job

vmcburney · Post by **vmcburney** » Mon Jun 20, 2011 5:39 pm

Do you have the InfoSphere CDC plugin stage for DataStage? I seem to recall it is installed by Fix Pack 1 for Information Server 8.5. You should find with that pack the documentation for configuring and using it. The InfoSphere CDC plugin stage for DataStage has gone through a few rapid changes in the last year so you really need the very latest version to make sure you have the full guaranteed data delivery functionality.

buzzylee · Post by **buzzylee** » Mon Jun 20, 2011 6:30 pm

It's worth to add that InfoSphere Change Data Capture for DataStage can also work without CDC plugin stage - there is an option to generate file-driven DataStage jobs directly from CDC Management Console. In this approach CDC creates files on the DataStage server side which are then picked up by prebuild DS jobs. You can also control size of the files (number of records) and frequency of load (time).

Regards
Buzz

Billyqing · Post by **Billyqing** » Tue Jun 21, 2011 1:58 pm

Hi all,

Thanks all for your inputs.

I have installed the CDC components combining with Datastage 8.5. The CDC Plugin Stage has come with Fix Pack 1 or Patch 1 for installation, which does not come with normal installation of IS 8.5.

I may have to use the CDC plugin Stage, or middle tables CDC replicates from source systems.

This is good experience about CDC components with DataStage.

I will continue this topic to gain or share the knowledge about CDC.

Thanks,

buzzylee · Post by **buzzylee** » Tue Jun 21, 2011 5:27 pm

Please keep in mind that when using middle tables you will need CDC licence (database-to-database) while usage of CDC Plugin Stage or file-driven approach that I mentioned might be implemented via CDD license (database-to-datastage).

If you posses adequate amount of CDC database licenses then your work-around approach is fine, otherwise I strongly recommend going for CDD (Change Data Delivery). The other thing is that it's more "friendly" from the point of view of paying for what you really use

Regards
Buzz

mhester · Post by **mhester** » Wed Jun 22, 2011 7:04 am

Bill,

If you would like a datastage job (template) that can be used for change data capture then please pm me and I would be happy to send it to you. There is nothing proprietary in how this is accomplished using the parallel construct. It is different than what we used to do in the server world, but once you have created your first one - the rest are simple. You can even implement RCP and make if somewhat generic and not have to create one for each table you want to detect change in.

You can buy all kinds of fancy products, but in datastage you can do it with less than 10 - 12 operators it is very fast.

The template I offer can be used in pre 8.x and any current version of datastage. This functionality has not changed at all.

Regards,