Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett , rschirm , roy
santoshkumar
Charter Member
Posts: 35 Joined: Sun Jan 16, 2005 8:39 am
Location: US
Post
by santoshkumar » Tue Dec 05, 2006 11:50 am
Hi,
I am running a job in DS7.5.1A and it is taking for 0.6 million records 3 hours.
Design:
oracle source--->Xfm------>Pivot----------->DataSet.
From oracle stage to Xfm it is taking only 3 minutes.
Pivot stage is the overhead.
How can i improve the performance.
I am pivoting 16 columns to rows.
Thanks.
Santosh
thebird
Participant
Posts: 254 Joined: Thu Jan 06, 2005 12:11 am
Location: India
Contact:
Post
by thebird » Tue Dec 05, 2006 11:57 am
santoshkumar wrote:
Pivot stage is the overhead.
The pivot stage is known to give a performance overhead, especially when pivoting too many columns.
By default the Pivot stage runs in a Sequential mode - have you changed this to run in a parallel mode? This should improve the performance a little bit.
Aneesh
santoshkumar
Charter Member
Posts: 35 Joined: Sun Jan 16, 2005 8:39 am
Location: US
Post
by santoshkumar » Tue Dec 05, 2006 1:28 pm
Is there any other way i can replicate the pivot stage functionality and improve performance.
Santosh
thebird
Participant
Posts: 254 Joined: Thu Jan 06, 2005 12:11 am
Location: India
Contact:
Post
by thebird » Tue Dec 05, 2006 1:50 pm
santoshkumar wrote: Is there any other way i can replicate the pivot stage functionality and improve performance.
Did you try to run in a Parallel mode? Any significant difference?
santoshkumar
Charter Member
Posts: 35 Joined: Sun Jan 16, 2005 8:39 am
Location: US
Post
by santoshkumar » Tue Dec 05, 2006 4:20 pm
Not much of diff.
I am just thinking of having 8 output links form tranformer to pivot with each having pivot columns.
I think this should give me some improvement.
If anyone has any other thought just let me know..
Santosh
thebird
Participant
Posts: 254 Joined: Thu Jan 06, 2005 12:11 am
Location: India
Contact:
Post
by thebird » Tue Dec 05, 2006 4:22 pm
santoshkumar wrote: I am just thinking of having 8 output links form tranformer to pivot with each having pivot columns.
Why dont you try writing the Pivot Logic inside the Transformer and checkout the performance?
santoshkumar
Charter Member
Posts: 35 Joined: Sun Jan 16, 2005 8:39 am
Location: US
Post
by santoshkumar » Tue Dec 05, 2006 9:22 pm
how can i achieve in a transformer
Santosh