Performance while reading from hash file

Post questions here relative to DataStage Server Edition for such areas as Server job design, DS Basic, Routines, Job Sequences, etc.

Moderators: chulett, rschirm, roy

Post Reply
SonShe
Premium Member
Premium Member
Posts: 65
Joined: Mon Aug 09, 2004 1:48 pm

Performance while reading from hash file

Post by SonShe »

I have a job that has been designed as below:

Oracle Stage ---> Transformer ----> Transformer ----> Transformer ----> HashFile ------> Transformer ---> Transformer --- > Oracle Stage.

These various transformers have look-ups from hash files. When I run the job, data moves from the first Oracle stage through the transformers to the hash file. Until all the data are written to the hash file, the read process from the hash file to the final Oracle stage does not start.

This job has Inter-process buffer enabled at the job level. The unix server has 2 CPUs. My understanding is that at least 2 processes can run simultaneously with 2 CPUs. I am of the impression that the first OCI stage to the hash file is one process and from the hash file to the destination OCI stage is the second process. SO why are not these processes running at the same time?

I will appreciate any help in this regard.

Thanks.
kcbland
Participant
Posts: 5208
Joined: Wed Jan 15, 2003 8:56 am
Location: Lutz, FL
Contact:

Post by kcbland »

Because that's not how it works. What it does is allow multiple rows to be in transit across a string of active stages between two passive stages, as opposed to a single row having to make it all of the way across to the second passive stage before the next row can start.

Your job is working perfectly. Obviously, someone put the hash file there for a reason, most likely because they either wanted to reference it while loading it, or because they're using it as a cheap duplication reduction device. You should double check the jobs requirements before removing that hash file.

What is happening is that each transformer between the first OCI and hash file is operating as its own process and each process is allowed to use different cpus.
Kenneth Bland

Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Post Reply