Oracle Enterprise stage - Load and Upsert
Moderators: chulett, rschirm, roy
Oracle Enterprise stage - Load and Upsert
Hi,
We have a process were we would need to load data in range of a million into an Oracle table.
We started using the Write Method as Load in the Oracle Enterprise stage as default, thinking that this should be the fastest as it uses Oracle loader concept.
But, just by chance we gave a try with Write Method as Upsert . We find that the Upsert method consistently is slightly faster than when using the Load method.
What could be the reason that Upsert is faster than Load when we always have only inserts to be done? Should Load be done with some special settings?
Thanks.
We have a process were we would need to load data in range of a million into an Oracle table.
We started using the Write Method as Load in the Oracle Enterprise stage as default, thinking that this should be the fastest as it uses Oracle loader concept.
But, just by chance we gave a try with Write Method as Upsert . We find that the Upsert method consistently is slightly faster than when using the Load method.
What could be the reason that Upsert is faster than Load when we always have only inserts to be done? Should Load be done with some special settings?
Thanks.
In Oracle if you're loading a partitioned table with global indexes, you can find that for smaller volumes CONVENTIONAL load is more efficient than DIRECT path load because of deferred index rebuilding/updating.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
-
- Participant
- Posts: 73
- Joined: Tue May 10, 2005 6:21 pm
- Location: Sydney
Re: Oracle Enterprise stage - Load and Upsert
what value did you assign to $APT_ORACLE_LOAD_OPTIONS?
Is that table partitioned? Are there any constraints/indexes on that table and how did you maintain the indexes/constraints during loading?
How many records are to be loaded and how many records exist in that table?
=============================================
Couldn't comment if I don't know these information. But I guess if the number of existing records are much more than the records to be loaded. it may be worthy to keep the constraints on instead of disabling them before loading and then renabling them back after loading.
Is that table partitioned? Are there any constraints/indexes on that table and how did you maintain the indexes/constraints during loading?
How many records are to be loaded and how many records exist in that table?
=============================================
Couldn't comment if I don't know these information. But I guess if the number of existing records are much more than the records to be loaded. it may be worthy to keep the constraints on instead of disabling them before loading and then renabling them back after loading.
vnspn wrote:Hi,
We have a process were we would need to load data in range of a million into an Oracle table.
We started using the Write Method as Load in the Oracle Enterprise stage as default, thinking that this should be the fastest as it uses Oracle loader concept.
But, just by chance we gave a try with Write Method as Upsert . We find that the Upsert method consistently is slightly faster than when using the Load method.
What could be the reason that Upsert is faster than Load when we always have only inserts to be done? Should Load be done with some special settings?
Thanks.
Edward Yuan
Re: Oracle Enterprise stage - Load and Upsert
Hi,Yuan_Edward wrote:what value did you assign to $APT_ORACLE_LOAD_OPTIONS?
Is that table partitioned? Are there any constraints/indexes on that table and how did you maintain the indexes/constraints during loading?
How many records are to be loaded and how many records exist in that table?
=============================================
Couldn't comment if I don't know these information. But I guess if the number of existing records are much more than the records to be loaded. it may be worthy to keep the constraints on instead of disabling them before loading and then renabling them back after loading.
Here are the answers for your questions,
No, we haven't added this environment variable APT_ORACLE_LOAD_OPTIONS.
No, the table is not partitioned. There are some indexes on the table and we used the option to Rebuild the index.
Initially the table is empty and we load through a set of around 5 jobs. Each load process may write around 1 million rows to the table.
Thanks.
-
- Participant
- Posts: 73
- Joined: Tue May 10, 2005 6:21 pm
- Location: Sydney
Re: Oracle Enterprise stage - Load and Upsert
I am assuming that there are no constraints (primary/foreign/unique keys) except some checking constraints (e.g. some columns are non-nullable).
In the Oracle Enterprise stage, did you specify the option DISABLE CONSTRAINTS to FALSE? If you set it to TRUE, Oracle will spend some time enabling the non-nullable constraints after loading.
This can be observed in the job log. Maybe this is one of the reasons.
=============================================
By the way, $APT_ORACLE_LOAD_OPTIONS is defaulted to empty at the project level, and it will be set to OPTIONS(DIRECT=TRUE, PARALLEL=TRUE, SKIP_INDEX_MAINTENANCE=YES) at runtime.
In the Oracle Enterprise stage, did you specify the option DISABLE CONSTRAINTS to FALSE? If you set it to TRUE, Oracle will spend some time enabling the non-nullable constraints after loading.
This can be observed in the job log. Maybe this is one of the reasons.
=============================================
By the way, $APT_ORACLE_LOAD_OPTIONS is defaulted to empty at the project level, and it will be set to OPTIONS(DIRECT=TRUE, PARALLEL=TRUE, SKIP_INDEX_MAINTENANCE=YES) at runtime.
Edward Yuan
vnspn-
What is the total number of records inserted. Per your earlier post, i understand that you will be processing about 5 million in 5 different loads.
And can you please let me know what is the total processing time taken for all these inserts? Cause, i am doing a performance tuning with all different scenerios..
What is the total number of records inserted. Per your earlier post, i understand that you will be processing about 5 million in 5 different loads.
And can you please let me know what is the total processing time taken for all these inserts? Cause, i am doing a performance tuning with all different scenerios..
Last edited by dspxlearn on Fri Apr 04, 2008 1:11 pm, edited 1 time in total.
Thanks and Regards!!
dspxlearn
dspxlearn
Thanks vnspn,
As of now i am using the Upsert method to load. We have two joins stages and aggregation functions(on these 1.5 million records) and filters before loading the data into the target database. For all this process it is taking 1hr and 45 mins.
I am waiting for my DBA's permission to get the SELECT previleges on the System tables before using the 'Load' method.
Thanks again.
As of now i am using the Upsert method to load. We have two joins stages and aggregation functions(on these 1.5 million records) and filters before loading the data into the target database. For all this process it is taking 1hr and 45 mins.
I am waiting for my DBA's permission to get the SELECT previleges on the System tables before using the 'Load' method.
Thanks again.
Thanks and Regards!!
dspxlearn
dspxlearn