DSXchange

Hi
I have around 20 million rows in the input table which is a dimension table. Every day there can be around 200-300 new rows coming in and around 500 update records where the value for some of the column changes.

I am trying to see how i can use the lookup stage to capture these updates. If i just use the key column for lookup, then all the records except the new records(inserts) will be considered as updates and will flow through the output link. But i want to capture onlly those 500 records. Is there a way to do that.

I dont want to use change capture as we have had performance issues before..

You need to compare the looked up values to the incoming values and only pass along a 'change' record when something has actually changed. Don't really see how this will be any more performant than the CDC stage, however.

I would take advantage of timestamps to identify records that have changed and select only those.