DSXchange

Note sure if this is asked before (could'nt find it in the search)
----
-SCD-II(slowly changind dim) change for customer

-I will be pulling from a DB which have multiple changes within a day
eg: customer data changes 5 times a day at the source

-I will be pulling data once End of Day(EOD), for all the changes happened within that day or depending on my Timestamp
eg: there were 5 different changes at the source and I pull 5 changes for the same Natural Key.

-Is there a way to handle these in the data flow with in the PX

My need is show in the target of 5 changes for the day (start and end date each change) and the last one will be active, as of end of Day
I need to capture the lineage of source data.

PS: I guess we can use stage variables to access previous values, which might be cumbersum!

Also, this can be implemented using Server with a Hash file doing update and lookup on the same file.

Any suggestion from all you experts.

I would imagine that SCD stage would handle multiple changes in the one stream.

For SCD, (with my limited understanding is that),
my reference link should have the latest updated row for the next row to see if its a new or old record.
I am not sure if it can update the target and do a lookup on the same
(Similar to the one we do in Server, using a transform and dynamic HASH file, which can be used to as target and the ref link)

If we are using any database as ref link, I will be concerned on the performance.

The SCD stage caches the dimension table (or at least those rows thus far processed) in memory, so it should be able to detect that the natural key has already been seen in the current run.

Yes, it did for what ever i tested - might need more rigourous testing.