Data Migration Issue
Posted: Mon Jul 01, 2002 9:13 am
We have designed data stage jobs and migrated small amount of test data. But now that we are ready to load the live data which is millions of records per table, we are having some issues.
Problem Scenario is as given below.
------------------------------------------------------------------------------------
Table Type Records Fields
------------------------------------------------------------------------------------
A ORAOCI8 836374 10 ( source table )
B ORAOCI8 836374 2 ( lookup table )
C ORAOCI8 - 11 ( target table )
R1 Sequenctial File - 2 ( reject file )
R2 Sequenctial File - 1 ( reject file )
R3 Sequenctial File - 10 ( reject file )
------------------------------------------------------------------------------------
Job Detail
==========
Field X from table A is linked to field Y from table B.
Table A is linked to transformer as Stream
Table B is linked to transformer as Reference
Table C is transformer output table.
R1 is the reject file for Field X value in table A not matching any of field Y value in table B.
R2 is the reject file for Field Z in table A with NULL values
R3 is the general reject file which captures all records not transfered to Target C , R1 and R2
Process Detail
==============
Process started on JUN 27 2002 at 4:19 PM and was aborted on JUN 29 2002 at 1:59 AM.
When process was aborted 41780 records were transferred to Target Table C.
Kindly note that the source table and the target table are accessed from 2 different oracle data sources
Problem
=======
First of all 2 days for migration of 836374 records is way beyond the performance we are looking forward for and even then the jobs are getting aborted before completion.Some of our tables have 25 million records to be migrated.
Do we need to relook at our jobs or is this some server installation issue...Looking forward for Reply.
Thanks.
Problem Scenario is as given below.
------------------------------------------------------------------------------------
Table Type Records Fields
------------------------------------------------------------------------------------
A ORAOCI8 836374 10 ( source table )
B ORAOCI8 836374 2 ( lookup table )
C ORAOCI8 - 11 ( target table )
R1 Sequenctial File - 2 ( reject file )
R2 Sequenctial File - 1 ( reject file )
R3 Sequenctial File - 10 ( reject file )
------------------------------------------------------------------------------------
Job Detail
==========
Field X from table A is linked to field Y from table B.
Table A is linked to transformer as Stream
Table B is linked to transformer as Reference
Table C is transformer output table.
R1 is the reject file for Field X value in table A not matching any of field Y value in table B.
R2 is the reject file for Field Z in table A with NULL values
R3 is the general reject file which captures all records not transfered to Target C , R1 and R2
Process Detail
==============
Process started on JUN 27 2002 at 4:19 PM and was aborted on JUN 29 2002 at 1:59 AM.
When process was aborted 41780 records were transferred to Target Table C.
Kindly note that the source table and the target table are accessed from 2 different oracle data sources
Problem
=======
First of all 2 days for migration of 836374 records is way beyond the performance we are looking forward for and even then the jobs are getting aborted before completion.Some of our tables have 25 million records to be migrated.
Do we need to relook at our jobs or is this some server installation issue...Looking forward for Reply.
Thanks.