Tuning a Ds job involving Oracle stages

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
suneyes
Participant
Posts: 82
Joined: Mon Jul 21, 2008 8:42 am

Tuning a Ds job involving Oracle stages

Post by suneyes »

Hi,
I am dealing with tuning a DS job which mainly involves oracle stages.

job contains two oracle stages whose output is clubbed using a funnel stage,written into a sequential file.

the query in oracle stage-1 will resemble

select c1,c2,c3 from t1,t2,t3

the query in oracle stag-2 will resemble

select count(t4.c1),sum(t4.c2) from
(select c1,c2,c3 from t1,t2,t3
)t4


from the datastage logs each of these satges are taking around 1.2hrs.because of parallel execution the job is getting completed in 1.2hrs.

what I am thinking of is as the part of logic (select c1,c2,c3 from t1,t2,t3) is same in both the satges,I am planning of doing both these stages in a single stage by creating a temp table.

what I want to know is.

1)Is my aproach correct
2)can I create a temp table in datastage(if yes,please tell me,how?).

please help me find a solution.
sun
DSDexter
Participant
Posts: 94
Joined: Wed Jul 11, 2007 9:36 pm
Location: Pune,India

Post by DSDexter »

Please give us more information about the queries. Are you trying to create a Cross-Product looking at the queries that you posted that will surely fetch a lot of data, Since it will be a cartesian product. Give us clear requirements than we can help you.
Thanks
DSDexter
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

While the approach might be correct, I am not certain that your measurements are. Did you separate these stages into their own jobs with no other links and check the timing? If not, then your results are likely skewed since timings in jobs usually depend on other stages and often you just end up measuring the slowest stage in the chain but don't really know which one it was.
suneyes
Participant
Posts: 82
Joined: Mon Jul 21, 2008 8:42 am

Post by suneyes »

DSDexter wrote:Please give us more information about the queries. Are you trying to create a Cross-Product looking at the queries that you posted that will surely fetch a lot of data, Since it will be a cartesian product. Give us clear requirements than we can help you.
no in ora1 stage we get a set of records from a select statement(detail records).
in ora2 stage we will get an aggregated result(trailer record).
so in ora2 stage,we rewrite the same query written in ora1 stage as a subquery and take an aggregated sum.

we will get some 2 million records from ora1 stg and 1 record from ora2 stg.

my job looks similar to this

ora1---
|----funnel---sequentai_file
ora2---[/img]
sun
suneyes
Participant
Posts: 82
Joined: Mon Jul 21, 2008 8:42 am

Post by suneyes »

DSDexter wrote:Please give us more information about the queries. Are you trying to create a Cross-Product looking at the queries that you posted that will surely fetch a lot of data, Since it will be a cartesian product. Give us clear requirements than we can help you.
no in ora1 stage we get a set of records from a select statement(detail records).
in ora2 stage we will get an aggregated result(trailer record).
so in ora2 stage,we rewrite the same query written in ora1 stage as a subquery and take an aggregated sum.

we will get some 2 million records from ora1 stg and 1 record from ora2 stg.

my job looks similar to this

ora1---
|----funnel---sequentai_file
ora2---
sun
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Please wrap your job design in Code tags then use Preview to get it right so that we can understand which link connects to which stages.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply