This seems to be another simple question to you all. But a bit tough for me
.I have here with me a prototype of ETL jobs done. This now needs to go up against the database.
Now I need to come to conclusions as to how to maintain the history data. Because everytime job is executed, fresh data will be loaded into the target.
Can you give me few suggestions
a) regarding when to populate surrogate key, how to and from which stage (i.e. after surrogate key or before surrogate key) should the history data be fetch to the input.
b) how to decide on type 1, type 2, type 3.
c) Any suggestions as to what more columns (like flags) or anything else is needed to accomplish this.
The question seems blurry, but its blurry for me as well. Please do as well tell me what i need to take into consideration from the input data to proceed with the steps to maintain history data.
With Regards
thank you