Reading one Dataset multiple times in parallel job

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
sourabhverma
Participant
Posts: 74
Joined: Thu Jan 05, 2006 2:07 am

Reading one Dataset multiple times in parallel job

Post by sourabhverma »

Hi,

My requirment is to replicate the logic of oracle user defined function into datastage and my database is netezza which does not support these types of user defined functions. So for that i have prepared one parallel job for achieving the same thing with the help of DS job but because of the business logic i had to use one particular dataset 7 times (As lookup) in that single job.

So i just wanted to know whether referring one dataset in this way will make any performance impact ?
Thanks,
Sourabh Verma
sourabhverma
Participant
Posts: 74
Joined: Thu Jan 05, 2006 2:07 am

Post by sourabhverma »

I forgot to mention the number of records, the job is processing:

The input stream contains around 3-4 millions of data where as the lookup dataset contains around 0.1 million of data.
Thanks,
Sourabh Verma
nitkuar
Participant
Posts: 46
Joined: Mon Jun 23, 2008 3:09 am

Post by nitkuar »

use copy stage between reference source and lookup stage and have 7 output links from copy stage for lookup purpose.
sourabhverma
Participant
Posts: 74
Joined: Thu Jan 05, 2006 2:07 am

Post by sourabhverma »

Thanks for reminding me the COPY stage.
Thanks,
Sourabh Verma
Post Reply