Hello,
I hv to load data frm a source view in foll way:-
Compare the incoming record with existing ones in target, and if there is any difference, load the incoming record else dont load. There is no key column defined in the source, so we cant check any 1 column but need to check data in all the columns of a record
Based on this requirement, I thought of using a change capture stage. Is there any other way of achieving this? As of now I am getting the foll error and the job aborts after abt loading 11 million recs..
Fatal Error: Pipe read failed: short read and
Fatal Error: Tsort merger aborting: Scratch space full
The source is a view with arnd 230 million recds.
Thnks
Richa
How to load data by doing record by record comparision
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 24
- Joined: Mon Apr 24, 2006 3:34 am
- Location: USA
Re: How to load data by doing record by record comparision
I still do not understand why a source with 230 million rows doesnt have a key column. And if there is no key, how do you define delta rows?
Re: How to load data by doing record by record comparision
Not having source system keys can be a pain. Now, while there is a possibility that there may be duplicity, but computing a CRC value on the concatenated values of the row and then comparing it against the same for target may be the way to go.
gateleys
-
- Premium Member
- Posts: 783
- Joined: Mon Jan 16, 2006 10:17 pm
- Location: Sydney, Australia