multiple links to the same file in one job

supernova2005 · Post by **supernova2005** » Tue Aug 22, 2006 9:48 am

Good morning everybody,

I need to have multiple links writing to the same target, the target could be a sequential file, afileset or a dataset. is there anyway I can do it without using a funnel stage?

Krazykoolrohit · Post by **Krazykoolrohit** » Tue Aug 22, 2006 9:52 am

you will have to join all those multiple link using a merge/join stage before you write it to the target

kcbland · Post by **kcbland** » Tue Aug 22, 2006 10:29 am

If you intention is an UNION ALL type result of links, the Funnel is the stage you use. If you're joining, consider the Join, Merge, or Lookup stages.

bcarlson · Post by **bcarlson** » Tue Aug 22, 2006 2:58 pm

Just a quick caution. We have run into performance issues with the Funnel stage when dealing with large volumes.

Assume we have a single job that has multiple inputs into the funnel, then after the funnel it does some other work, even if it is just writing to a final dataset or export file. If the volume is too high, the job slows WAY down. If we just break up the program to write to multiple datasets in one job, then have a new job funnel the datasets together, the overall runtime drops significantly.

Not entirely sure why this is happening, and unfortunately we do not know at what point (volume wise, that is) the slowdown begins. I was asked recently to put together some test cases to pass along to IBM so they can fix the issue, but have not yet had time to do so.

Please don't interpret this as a reason not to use the Funnel. Like kcbland recommended, if you need the equivalent of a Union all, the funnel is the right way to go. Just know that if your volumes are high and your performance is slow, you may need to break up your job.

Brad.

kcbland · Post by **kcbland** » Tue Aug 22, 2006 3:26 pm

bcarlson wrote:you may need to break up your job.

Shocking! In some circles, you would be burned at the stake for such heresy. Don't worry, I'll be nearby to throw some water on you.

bcarlson · Post by **bcarlson** » Tue Aug 22, 2006 3:28 pm

I know, I burned some COBOL books as sacrifice....

ray.wurlod · Post by **ray.wurlod** » Wed Aug 23, 2006 2:58 am

The operating system will not permit multiple writers to a Sequential File.

Therefore the parallel Sequential File mandatorily executes in sequential mode and imposes a Collector on its input link.