Hey guys,
I'm loading a 3 300 000 rows table containing invoices.
I have two modes :
1) Full loading
2) Daily Loading
For the daily loading mode I have 2 fields containing the creation date and the update date in the source date. Thus I can filter only records created or updated after the last load.
However, if the job crashes and I do want reseume what is the best ?
1) Delete all rows loaded before it crashed and restart ?
2) Check against exsiting records to see which ones are new ? I like this one but I fear to have to lookup against a 3 million rows table.
What do you suggest ?
Thanks
Facts loading strategy
Moderators: chulett, rschirm, roy
Re: Facts loading strategy
Will try with the CDCalgfr wrote:Hey guys,
I'm loading a 3 300 000 rows table containing invoices.
I have two modes :
1) Full loading
2) Daily Loading
For the daily loading mode I have 2 fields containing the creation date and the update date in the source date. Thus I can filter only records created or updated after the last load.
However, if the job crashes and I do want reseume what is the best ?
1) Delete all rows loaded before it crashed and restart ?
2) Check against exsiting records to see which ones are new ? I like this one but I fear to have to lookup against a 3 million rows table.
What do you suggest ?
Thanks