How to run a job efficiently?
Moderators: chulett, rschirm, roy
-
chandra.shekhar@tcs.com
- Premium Member
- Posts: 353
- Joined: Mon Jan 17, 2011 5:03 am
- Location: Mumbai, India
How to run a job efficiently?
I have a parallel job, its a plain one to one mapping. Source is Oracle(oracle connector) and the target is DB2(DB2 Bulk Load). My problem is the source has 2.36 billion records and I have to optimize the job so that it shud finish within 5-6 hours. Plz suggest.
Re: How to run a job efficiently?
My experience to such high volume loads is it takes way longer to execute. As we did similar exercise in the past where we preferred parallel writes to a flat file & import those flat iles into another database. This works way faster.chandra.shekhar@tcs.com wrote:I have a parallel job, its a plain one to one mapping. Source is Oracle(oracle connector) and the target is DB2(DB2 Bulk Load). My problem is the source has 2.36 billion records and I have to optimize the job so that it shud finish within 5-6 hours. Plz suggest.
Thanks with regards,
videsh.
videsh.
Hi Videsh,
1. Major columns or used where clause columns should be indexed.
2. Check before run this job - should not be any row / table lock.
3. Loading is may be based on how many based number of node you are using.
4. If you want to select one particluar date information then instead of put condition in the table you can create view or table for that day and use it in Datastage.
1. Major columns or used where clause columns should be indexed.
2. Check before run this job - should not be any row / table lock.
3. Loading is may be based on how many based number of node you are using.
4. If you want to select one particluar date information then instead of put condition in the table you can create view or table for that day and use it in Datastage.
Jothi
-
chandra.shekhar@tcs.com
- Premium Member
- Posts: 353
- Joined: Mon Jan 17, 2011 5:03 am
- Location: Mumbai, India
@Videsh
If I use a flat file then I have to make another job which will write the data from file to my target.
@Jothi
I dont have any where clause nor I have any date specific data, its a simple one to one mapping. I am using 32 nodes, I also tried DB2 connector as my target with Insert mode. Even then it is not giving satisfoctory results. I am using oracle connector which works in parallel mode. Still i dont find any good results.
If I use a flat file then I have to make another job which will write the data from file to my target.
@Jothi
I dont have any where clause nor I have any date specific data, its a simple one to one mapping. I am using 32 nodes, I also tried DB2 connector as my target with Insert mode. Even then it is not giving satisfoctory results. I am using oracle connector which works in parallel mode. Still i dont find any good results.