Page 1 of 1

Issue with Parallel loads with logical nodes

Posted: Tue Jun 15, 2010 4:57 pm
by vskr72
We have a simple job that uses a direct table to table dump with a Copy stage in between. This is what we did:
- We created 2 logical nodes in the config file and used that config file in the current job
- Added the 2 nodes in the node map constraint field
- Rest of the options are defaults.

When I run this job, the number of records that load in the target is twice of what we have in the source table. If I increase it to 4, then the target has 4 X.

I am new to DS. Can any one tell me what do i need to configure in this case?

Satish

Posted: Tue Jun 15, 2010 5:00 pm
by mavrick21
"Added the 2 nodes in the node map constraint field " ->Do you have any particular reason for adding this?

Can you remove it from node map constraint field and check?

Posted: Tue Jun 15, 2010 5:21 pm
by vskr72
Even if I remove the nodes in the Nodemap constraint, it still is duplicating based on the number of nodes.

Posted: Tue Jun 15, 2010 5:33 pm
by chulett
Then you've got a partitioning issue. Can you describe your job design - stages, partitioning settings per stage, etc?

Posted: Tue Jun 15, 2010 5:51 pm
by vskr72
This is my job design:

Oracle Ent Stage --> Copy --> Seq.File

Oracle Stage:
- have a SQL override with conditions rownum < 50000
- Buffering is default
- Combinability - Auto
- Preserve partition default

Copy Stage:
- Preserve partition propagate
- Auto Partitioning

Seq File:
Default settings

Thanks

Posted: Tue Jun 15, 2010 5:56 pm
by vskr72
Also have this in the job properties for $APT_ORACLE_LOAD_OPTIONS:

OPTIONS(DIRECT=FALSE,PARALLEL=FALSE)

Posted: Tue Jun 15, 2010 9:47 pm
by chulett
Why have 'load options' when there is no loading going on? :?

If your Oracle stage's Execution Mode is parallel, try setting it to sequential.

Posted: Wed Jun 16, 2010 2:15 am
by udayk_2007
Remove the copy stage if it is not required and try running the job. Check if you get the same result

Posted: Wed Jun 16, 2010 2:49 am
by Sainath.Srinivasan
View the run statistics from the director with the rows per node mode enabled. That will give you an idea of which stage is increasing the volume.