Read From Dataset is slow

[bi_fujitsu]

Hi

I am having two jobs. In the first job i am creating a dataset using Auto Partitioning. This dataset once created contains around 45k records, Its taking only 10 seconds in creating this dataset. In the 2nd job i am using this dataset as source to update an oracle table and the method i am using is upsert. Here also i am using the auto partitioning. But in this job the read from dataset is veru slow about 60 recs /sec. I am not sure where i am going wrong. Plz. help me in improving the read performance of dataset.

[ray.wurlod]

It's almost certainly not the read from the Data Set that's slow.

To prove this create a copy of the job but replace the Oracle Enterprise stage with a Copy stage or a Peek stage. Run that job to get an estimate of the true read speed of a Data Set.

That means that the problem must be with upsert. Think about what upsert does - the potential exists that, on average, more than one SQL statements will need to be executed for each row processed. If all the rows are to be inserted consider using direct write rather than SQL.

[DSguru2B]

This is the phenomenon that Ken likes to call 'back pressure'. Your target stage can only handle the numbers you see.

[kcbland]

This is the phenomenon that Ken likes to call 'back pressure'. Your target stage can only handle the numbers you see.

:D It's good to see others remembering!

[chulett]

I think it's a great descriptive moniker for the phenomenon. :D

[DSguru2B]

:D It's good to see others remembering!

Always remeber your posts, almost word for word

[ray.wurlod]

It's what electrical engineers know as "back emf" (which I learned in my BE degree in the 1960s).