Please let me know the following :
1. How can we perform a incremental data load in datastage
2. Can we restart a job sequence from a certain point on failure instead of running the whole sequence
DS questions
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 1255
- Joined: Wed Feb 02, 2005 11:54 am
- Location: United States of America
>1. How can we perform a incremental data load in datastage
this is more of an ETL question and not a DS question. One method is to maintain a copy of the target table as a dataset and check for the difference with your input file every time the job starts. That will get you an incremental data set.
A lot of times it is faster to do this using the database so that you do not need to send a lot of data over the network. You can maintain incremental changes with triggers or do the difference of two table dumps (previous and current) on the database server. It depends on your system setup.
>2. Can we restart a job sequence from a certain point on failure instead of running the whole sequence
In your sequence job, you have to go to job properties and check the restartable option. This can also be set in Administrator as a project wide setting. A job will only restart from a checkpoint if it fails. Otherwise it will think that the run was successful. So you can either set it to "fail after 1 warning" in director or make it fail on a warning in the sequence job itselft.
this is more of an ETL question and not a DS question. One method is to maintain a copy of the target table as a dataset and check for the difference with your input file every time the job starts. That will get you an incremental data set.
A lot of times it is faster to do this using the database so that you do not need to send a lot of data over the network. You can maintain incremental changes with triggers or do the difference of two table dumps (previous and current) on the database server. It depends on your system setup.
>2. Can we restart a job sequence from a certain point on failure instead of running the whole sequence
In your sequence job, you have to go to job properties and check the restartable option. This can also be set in Administrator as a project wide setting. A job will only restart from a checkpoint if it fails. Otherwise it will think that the run was successful. So you can either set it to "fail after 1 warning" in director or make it fail on a warning in the sequence job itselft.
Mike