Usage of IPC stage

pg_smh · Post by **pg_smh** » Sat May 26, 2007 2:01 am

If I use an IPC stage between a DRS stage and a hash file, will it improve the performance. I'm trying to load data from an Oracle DB table containing around 3 million records into a hash file and it took around 30 minutes, I want to improve upon this.

Minhajuddin · Post by **Minhajuddin** » Sat May 26, 2007 3:24 am

Do you use all the records you are loading into the hash file?
If not then doing a lookup without a hash file would improve your performance.
Please post your scenario, So that people can help you better.
There are a lot of ways to improve your current performance, if you can post the exact scenario.

chulett · Post by **chulett** » Sat May 26, 2007 7:04 am

First determine the bottleneck. If you replace the Hashed File stage with a Sequential File stage, who long does it take to run then?

pg_smh · Post by **pg_smh** » Sat May 26, 2007 7:19 am

This hash file acts as a shared lookup across different warehouses(In my case warehouses differ on the basis of line of business - Finances, HR, etc..). and all the rows are needed.

Minhajuddin · Post by **Minhajuddin** » Sat May 26, 2007 9:02 am

Are you using Static or Dynamic Hashed files?
Wrong settings in Static hashed files can reduce the performance drastically.

narasimha · Post by **narasimha** » Sat May 26, 2007 11:01 am

Were you able to identify the bottleneck using Craig's suggestion.
You optimization route will depend on that.