Hi,
I have 1,00,00,000 records of data.After loading 65,00,000 data the job has been aborted.
Is there any possibility to load data 65,00,001 onwards not starting onwards in datastage.
If it is possible Could you please suggest me??
How to Load after aborted data?
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 536
- Joined: Thu Oct 11, 2007 1:48 am
- Location: Bangalore
No such checkpoint is there in datastage But you can always design a job for your requirement.
Thanks
Prasoon
ETL Consultant
LinkedIn :- http://www.linkedin.com/profile/view?id ... ab_pro_top
Blog:- http://dsshar.blogspot.com/
Prasoon
ETL Consultant
LinkedIn :- http://www.linkedin.com/profile/view?id ... ab_pro_top
Blog:- http://dsshar.blogspot.com/
In other words, you need to design that capability into your jobs. How you do that depends on many factors such as the nature of both the load and the data being loaded. Primary question in my mind is - is your source data static? Will record #65000001 be the same on the rerun as it was on the first? Answering 'no' will obviously complicate things. Otherwise it could be as simple as a job parameter with a 'start record' counter that is the number of records to skip. Default to zero, override for a post-failure rerun.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
Re: How to Load after aborted data?
It seems an Interview question and one of the common Interview question in India. Entire reload vs Check the loaded one and then start loading for the rest, which one is less time consumption will decide how to restart the job.
Answer to your question is YES.
But there is no straight forward option (Eg: Check box / ON - OFF) ask Datastage to start again from the record where it stopped!!
As Ray said there are ways and you can do that!
Answer to your question is YES.
But there is no straight forward option (Eg: Check box / ON - OFF) ask Datastage to start again from the record where it stopped!!
As Ray said there are ways and you can do that!
Thanks
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
Ram
----------------------------------
Revealing your ignorance is fine, because you get a chance to learn.
It should be done at Design Level..
Let us assume that your primary Data having 1000 records and make a your target as a reference and by filtering the data based on any dummy value, then only records which are not already inserted will be loaded into target.
your job desing should be like this :
Reference(Target)
|
|
Join------>Filter----->Target
|
|
Source
suppose you had some key columns, write a User defined SQL
like Select 'Y' , KC1,KC2...etc., and Filter the Records in Filter stage Where COL='Y'.
Then the already inserted records will be Filtered out.
your job desing should be like this :
Reference(Target)
|
|
Join------>Filter----->Target
|
|
Source
suppose you had some key columns, write a User defined SQL
like Select 'Y' , KC1,KC2...etc., and Filter the Records in Filter stage Where COL='Y'.
Then the already inserted records will be Filtered out.
Jaypee