Update performance issue
Moderators: chulett, rschirm, roy
Update performance issue
Hello all,
When using the insert/update method I get ridiculous performance on volumes over 5000 records. I have a dimension over 150,000 records. Performing an update of 10,000 records using a SCD stage took 2 hours...
I have to use it because I have a SCD stage which cannot be used with a load/truncate method.
I checked if the join columns were indexed, they are...
Any idea of where to optimize ?
Thanks people
When using the insert/update method I get ridiculous performance on volumes over 5000 records. I have a dimension over 150,000 records. Performing an update of 10,000 records using a SCD stage took 2 hours...
I have to use it because I have a SCD stage which cannot be used with a load/truncate method.
I checked if the join columns were indexed, they are...
Any idea of where to optimize ?
Thanks people
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
get a trace from oracle, i suspect there are some other queries blocking the update query.
I performed few tests on oracle 10g with Oracle Enterprise Stage earlier with the similar number of records than you, below are the results in short:
2 oracle stages one with insert/load and one with update
if there are records in load with same key as update. It took 12 minutes where one was blocking another.
with few of the current load records updated
when I change the upsert query to identify the previous records and force it to update only previous records, it took only 30 seconds also populated correct data.
see if its the same case with you.
I performed few tests on oracle 10g with Oracle Enterprise Stage earlier with the similar number of records than you, below are the results in short:
2 oracle stages one with insert/load and one with update
if there are records in load with same key as update. It took 12 minutes where one was blocking another.
with few of the current load records updated
![Shocked :shock:](./images/smilies/icon_eek.gif)
when I change the upsert query to identify the previous records and force it to update only previous records, it took only 30 seconds also populated correct data.
see if its the same case with you.
Priyadarshi Kunal
Genius may have its limitations, but stupidity is not thus handicapped.![Wink :wink:](./images/smilies/icon_wink.gif)
Genius may have its limitations, but stupidity is not thus handicapped.
![Wink :wink:](./images/smilies/icon_wink.gif)
Thanks.priyadarshikunal wrote:get a trace from oracle, i suspect there are some other queries blocking the update query.
I performed few tests on oracle 10g with Oracle Enterprise Stage earlier with the similar number of records than you, below are the results in short:
2 oracle stages one with insert/load and one with update
if there are records in load with same key as update. It took 12 minutes where one was blocking another.
with few of the current load records updated![]()
when I change the upsert query to identify the previous records and force it to update only previous records, it took only 30 seconds also populated correct data.
see if its the same case with you.
Do you mean that the insert/update option should never be used ?
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
not in same job with 2 different Oracle enterprise stage. If you are bound to use that then make sure the where condition contains enough constraints to leave the current records.
upsert mode is OK to be used with SCD.
Before I can suggest any thing you need to answer the question asked by miwinter and explain the job design.
Also get trace file/ report from DBA to see if there was a lock.
upsert mode is OK to be used with SCD.
Before I can suggest any thing you need to answer the question asked by miwinter and explain the job design.
Also get trace file/ report from DBA to see if there was a lock.
Priyadarshi Kunal
Genius may have its limitations, but stupidity is not thus handicapped.![Wink :wink:](./images/smilies/icon_wink.gif)
Genius may have its limitations, but stupidity is not thus handicapped.
![Wink :wink:](./images/smilies/icon_wink.gif)
This looks like this (it update a Product dimension with 150,000 records)priyadarshikunal wrote:not in same job with 2 different Oracle enterprise stage. If you are bound to use that then make sure the where condition contains enough constraints to leave the current records.
upsert mode is OK to be used with SCD.
Before I can suggest any thing you need to answer the question asked by miwinter and explain the job design.
Also get trace file/ report from DBA to see if there was a lock.
part 1
LKP1 LKP2 LKP3
| | |
ODBC Stage => Transformer => Lookup constraints==>Dispatch rejects ==>
part 2
Old Product dim
|
==> SCD Stage ==> New Product dim (update/insert)
|
Dummy copy stage
If I load the table for the fisrt time, I assume it uses insert statements. This runs ok.
If I load it for the second time and rows haven't all be loaded in the first place, the update statements take forever.
I use this design with all my dim tables. It doesn't happne when tables are small.
When I check the SGA trace, I don't see any lock but I can see many inactive sessions.priyadarshikunal wrote:not in same job with 2 different Oracle enterprise stage. If you are bound to use that then make sure the where condition contains enough constraints to leave the current records.
upsert mode is OK to be used with SCD.
Before I can suggest any thing you need to answer the question asked by miwinter and explain the job design.
Also get trace file/ report from DBA to see if there was a lock.
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI