How to capture deletes incrementally
Posted: Wed Jun 11, 2008 1:44 pm
First of all, I apologize if this has been discussed here before. In looking through other topics regarding change data detection, I wasn't able to find an answer to what I'm looking for, but I may have missed it somewhere. If so, please point me in that direction. Most posts that I saw regarding deletes also mentioned Parallel version and/or a CDC stage that has to be purchased separately. Neither of those are currently an option.
I am wondering what the best way to go about capturing deletes from a source system is. Both source and target databases are Oracle 10g. Currently, we have a number of staging jobs that use CRC32 logic to load inserts and updates incrementally (Source stage -> Merge stage -> Transform to compare CRC rows -> Split stage -> Target stage). This takes care of capturing any new rows or any rows where a field has changed.
However, we currently have no process in place to handle deleted rows. There is no CDC on the source side. What are my options for taking care of this? I also need to take into account how to propagate the deleted rows into the fact and dimension tables.
Any information in regards to this would be helpful. Thanks!
I am wondering what the best way to go about capturing deletes from a source system is. Both source and target databases are Oracle 10g. Currently, we have a number of staging jobs that use CRC32 logic to load inserts and updates incrementally (Source stage -> Merge stage -> Transform to compare CRC rows -> Split stage -> Target stage). This takes care of capturing any new rows or any rows where a field has changed.
However, we currently have no process in place to handle deleted rows. There is no CDC on the source side. What are my options for taking care of this? I also need to take into account how to propagate the deleted rows into the fact and dimension tables.
Any information in regards to this would be helpful. Thanks!