Hi
I am having two jobs. In the first job i am creating a dataset using Auto Partitioning. This dataset once created contains around 45k records, Its taking only 10 seconds in creating this dataset. In the 2nd job i am using this dataset as source to update an oracle table and the method i am using is upsert. Here also i am using the auto partitioning. But in this job the read from dataset is veru slow about 60 recs /sec. I am not sure where i am going wrong. Plz. help me in improving the read performance of dataset.
Read From Dataset is slow
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 46
- Joined: Tue Mar 20, 2007 3:30 am
- Location: India
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
It's almost certainly not the read from the Data Set that's slow.
To prove this create a copy of the job but replace the Oracle Enterprise stage with a Copy stage or a Peek stage. Run that job to get an estimate of the true read speed of a Data Set.
That means that the problem must be with upsert. Think about what upsert does - the potential exists that, on average, more than one SQL statements will need to be executed for each row processed. If all the rows are to be inserted consider using direct write rather than SQL.
To prove this create a copy of the job but replace the Oracle Enterprise stage with a Copy stage or a Peek stage. Run that job to get an estimate of the true read speed of a Data Set.
That means that the problem must be with upsert. Think about what upsert does - the potential exists that, on average, more than one SQL statements will need to be executed for each row processed. If all the rows are to be inserted consider using direct write rather than SQL.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
:D It's good to see others remembering!DSguru2B wrote:This is the phenomenon that Ken likes to call 'back pressure'. Your target stage can only handle the numbers you see.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: