Looking for some advise on best practise to capture 'changed data' for huge data volume. Assuming I have a Base table with huge data volume and I get a full dump of the source data in some Staging table. So, will the 'Change Capture' stage be good enough (performance wise) to find the changed data (was told it is not). If not, what is the other alternatives.
Thanks.
Change data capture
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Re: Change data capture
Last year i did the test especially in relates with performance and i can't find any notable difference between stages.mydsworld wrote:'Change Capture' stage be good enough (performance wise)
I tried with Change Capture, SCD and Full Outer Join. I passed 100,000 records for test and i was not able to find any big difference in performance.
NOTES: Consider Rays comment which is totally different.
Thanks
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The Change Capture stage takes advantage of the fact that the incoming data are sorted on the change key values, so that it only needs to be processing on change key value at a time in memory. Therefore there is no added detriment to the linear scaling you should see as you increase the number of rows. And even this can be reduced by adding nodes and correctly partitioning the incoming data.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.