Data Integrity
Posted: Mon Nov 20, 2006 2:40 am
Hi There,
We are loading data from sequential file into Oracle DB using OCI-Load,
If we use Upsert, it provides us facility of Reject file.
However, Load property does not provide that functionality.
Since the data is massive, we can not use Upsert.
Could any one please advice about the different scenarios for
1. To confirm the data integrity vis a vis sequential file and Database post load (If job runs successfully).
2. If the job crashes midway,
how to check if the data which is read by datastage has actually gone inside the database ie can there be a possibility that Datastage might have read 100 records but only 99 have been actually loaded.
3. What should be the restart/ recover mechanism on crash, so that we do not have rerun the whole batch process.
It is a 8-10 hours of batch window and 200 gig of sequential file load.
I look forward to have your expertise comments.
Thanks
MK
We are loading data from sequential file into Oracle DB using OCI-Load,
If we use Upsert, it provides us facility of Reject file.
However, Load property does not provide that functionality.
Since the data is massive, we can not use Upsert.
Could any one please advice about the different scenarios for
1. To confirm the data integrity vis a vis sequential file and Database post load (If job runs successfully).
2. If the job crashes midway,
how to check if the data which is read by datastage has actually gone inside the database ie can there be a possibility that Datastage might have read 100 records but only 99 have been actually loaded.
3. What should be the restart/ recover mechanism on crash, so that we do not have rerun the whole batch process.
It is a 8-10 hours of batch window and 200 gig of sequential file load.
I look forward to have your expertise comments.
Thanks
MK