Performance Tuning needed in my Job
Posted: Fri Mar 13, 2009 10:01 am
Hi Guys
I did a parallel job which reads data from two tables residing in an Oracle 8 platform. I used the Dynamic RDBMS stage to read the two tables. Table 1 has around 2.5 Million records (15 columns) and table 2 has around 5.3 million records (10 columns). I use a lookup stage(Auto partition) to accomplish the lookup logic. I was not sure if a join would help improve some performance. After lookup I have a transformer stage to do some business transformation, this transformer runs sequentially in a parallel job since I have logic to generate sequential numbers like (1,2,3,4.....) for one of the column. and finally I write the data into a flat file. The job runs around 18 minutes to process 5.1 millions records from the two sources to one file.
Is there anyway I can tweak my job and reduce the runtime to less than 18 mins. I am hoping I find a way to bring it down to 10 mins at the most. Please advice.
Thanks
I did a parallel job which reads data from two tables residing in an Oracle 8 platform. I used the Dynamic RDBMS stage to read the two tables. Table 1 has around 2.5 Million records (15 columns) and table 2 has around 5.3 million records (10 columns). I use a lookup stage(Auto partition) to accomplish the lookup logic. I was not sure if a join would help improve some performance. After lookup I have a transformer stage to do some business transformation, this transformer runs sequentially in a parallel job since I have logic to generate sequential numbers like (1,2,3,4.....) for one of the column. and finally I write the data into a flat file. The job runs around 18 minutes to process 5.1 millions records from the two sources to one file.
![Rolling Eyes :roll:](./images/smilies/icon_rolleyes.gif)
Is there anyway I can tweak my job and reduce the runtime to less than 18 mins. I am hoping I find a way to bring it down to 10 mins at the most. Please advice.
![Confused :?](./images/smilies/icon_confused.gif)
Thanks