Page 1 of 1

Usage of IPC stage

Posted: Sat May 26, 2007 2:01 am
by pg_smh
If I use an IPC stage between a DRS stage and a hash file, will it improve the performance. I'm trying to load data from an Oracle DB table containing around 3 million records into a hash file and it took around 30 minutes, I want to improve upon this.

Posted: Sat May 26, 2007 3:24 am
by Minhajuddin
Do you use all the records you are loading into the hash file?
If not then doing a lookup without a hash file would improve your performance.
Please post your scenario, So that people can help you better.
There are a lot of ways to improve your current performance, if you can post the exact scenario.

Posted: Sat May 26, 2007 7:04 am
by chulett
First determine the bottleneck. If you replace the Hashed File stage with a Sequential File stage, who long does it take to run then?

Posted: Sat May 26, 2007 7:19 am
by pg_smh
This hash file acts as a shared lookup across different warehouses(In my case warehouses differ on the basis of line of business - Finances, HR, etc..). and all the rows are needed.

Posted: Sat May 26, 2007 9:02 am
by Minhajuddin
Are you using Static or Dynamic Hashed files?
Wrong settings in Static hashed files can reduce the performance drastically.

Posted: Sat May 26, 2007 11:01 am
by narasimha
Were you able to identify the bottleneck using Craig's suggestion.
You optimization route will depend on that.