Rerunning the job from fail point
Moderators: chulett, rschirm, roy
Rerunning the job from fail point
Hi,
I have a job which aborts after processing 1000 records. The input file has total of 10,000 records. How do I make sure that when the job runs next time, it doesn't process the already successfully processed records and starts from 1001th reord?
I have a job which aborts after processing 1000 records. The input file has total of 10,000 records. How do I make sure that when the job runs next time, it doesn't process the already successfully processed records and starts from 1001th reord?
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Depends.
What does "processing" involve?
Are you keeping track of which records have been processed?
What is the source? How are you extracting rows from that source?
What does "processing" involve?
Are you keeping track of which records have been processed?
What is the source? How are you extracting rows from that source?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Source is a db2 table.
Processing involves some simple transformations.
No, I am not keeping track of how many records have been processed and I don't know even how can this be done. So just wanted to know is there any way in data stage where in it can identify the last successfully processed record and in case of restart, can start from the first failed record?
Would check point serve this purpose? According to my understanding checkpoint can restart the whole failed job but not sure how to put it in at job record level
Processing involves some simple transformations.
No, I am not keeping track of how many records have been processed and I don't know even how can this be done. So just wanted to know is there any way in data stage where in it can identify the last successfully processed record and in case of restart, can start from the first failed record?
Would check point serve this purpose? According to my understanding checkpoint can restart the whole failed job but not sure how to put it in at job record level
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 12
- Joined: Wed Jun 13, 2007 2:12 am
This is what i have done whn i faced situation like you...
I have put in a lookup to same table (in my case it was table) where i check whether my already loaded 1000 rows are present if yes...leave it as it is and pass rest 9000 rows to reject link for rest of processing..
Not sure about if you using Seq. file or dataset..
This would add extra processing..but in case of table load ..it avoid abort due to duplicates...
I have put in a lookup to same table (in my case it was table) where i check whether my already loaded 1000 rows are present if yes...leave it as it is and pass rest 9000 rows to reject link for rest of processing..
Not sure about if you using Seq. file or dataset..
This would add extra processing..but in case of table load ..it avoid abort due to duplicates...
-
- Participant
- Posts: 40
- Joined: Tue Nov 11, 2008 5:49 am
If you want aviod by doing with lookup way, then i have one solutionagpt wrote:Hi All,
Just wanted to follow up on the same if any body can help me out on this?
Extract records which are loaded in the previous run , funnel the previous run records with new records and do the aggregation(find the count) then filter out the records which is having count is = 1, i hope this this will look like lengthy job, but am not sure if anyother way, if it is there i am eager to know!
![Surprised :o](./images/smilies/icon_surprised.gif)
Knowledge is Fair,execution is matter!