How to improve performance..

Das · Post by **Das** » Thu Oct 08, 2009 11:34 pm

Hi All,
Plese advice ,Which is best way to use the stage to get the optmum performance.

1-Currently I am using remove duplicate stage to remove the duplicate records,but i think using of sort stage is better to do the same operation..
Please suggest..Can I achieve any performance improvement out of that..Please advice

2-Transformer and filter stage:Currently am using filter stage ,If i replace it with transformer (using DS8x) can i expect any performance imrovement?..Please advice

3-During my loading process am creating N nubber of datasets,The same datasets will get overwrite in next run.The questing here is Can I expect any prformance improvemnt,If I delete the Datasets using orchadmin delete command after each run,Insted of overwrite though job.?Please give me your comments/suggestions

3-In most of the jobs ,I am crating 10-20 datasets in a final transformer and also that is a reject out put after a look up,Now the process looks very sow..Please advise..?

gssr · Post by **gssr** » Thu Oct 08, 2009 11:53 pm

You can hit some tutorial white papers in your favorite search engine for this issue....

Das · Post by **Das** » Fri Oct 09, 2009 12:53 am

gssr wrote:You can hit some tutorial white papers in your favorite search engine for this issue....

Okey..But i want to know something about from ral datastge gurus or spcific issues...

kiran259 · Post by **kiran259** » Fri Oct 09, 2009 2:30 am

First,How do you define performance?Actually,it depends on various factors like hardware config,CPUs used,volume of source data,etc..Performance improvement is like an iteration process and get the best possible solution.
-Transformer would be better than Filter due to various reasons
Moreover,without knowing your requirement,conclusions cannot be perceived.

Do you have an interview soon?

Das · Post by **Das** » Sat Oct 10, 2009 5:46 pm

kiran259 wrote:First,How do you define performance?Actually,it depends on various factors like hardware config,CPUs used,volume of source data,etc..Performance improvement is like an iteration process and get the best possible solution.
-Transformer would be better than Filter due to various reasons
Moreover,without knowing your requirement,conclusions cannot be perceived.

Do you have an interview soon?

The things which i have mentioned here are some facts,About the jobs in my project...The issues mentioned here are the identified facts..
My enviroment is 4 node,MPP system...And souce data is around 50k record in every 30 mts.Now the entire process is taking 1.2 hrs,I need to reduce it to 30 mts...and os is SunOS 5.10..

If you can comment on this it woluld be better for you also..may it will help you in interviews...also..

chulett · Post by **chulett** » Sat Oct 10, 2009 8:02 pm

1. Try it, let us know.

2. No.

3. No.

4. Identify the bottleneck. Based on such limited information I suggest the lookup.

ray.wurlod · Post by **ray.wurlod** » Sun Oct 11, 2009 4:36 pm

Before you do or worry about anything else, define "performance" in an ETL context. To my mind, the metric of performance is unlikely to be any kind of rate (particularly not rows per second). The main driver of ETL effort should - nay, must - be the ability to perform extraction and load phases within the time windows available, with a margin for safety.