How to limit the number of output files to a COPY stage

dsean · Post by **dsean** » Sun May 25, 2008 4:15 pm

Hi,
I am using the Enterprise edition for the first time and I want to know how to send dynamically the number of output files that we want to create or copy using the COPY stage in Enterprise edition. I did a search for this but did not find anything about this. Any help can be greatly appreciated.

Thanks in advance,
Sean

ray.wurlod · Post by **ray.wurlod** » Sun May 25, 2008 7:49 pm

Copy stage does not create output files. The Copy stage creates as many copies of its input stream as there are output links from the stage. This is done in memory - no files are involved.

dsean · Post by **dsean** » Sun May 25, 2008 7:57 pm

Ray,
Thanks for your reply. may be the question that I posted is not clear.

Lets say I have a Copy stage with one input link and 4 output links . Now without changing the job design , Can I limit the number of output copies to 3 during runtime from the COPY stage??

Once again, I appreciate your help.

Thanks in advance,
Sean

chulett · Post by **chulett** » Sun May 25, 2008 8:41 pm

Not that I am aware of.

ray.wurlod · Post by **ray.wurlod** » Mon May 26, 2008 12:17 am

More clearly, No.

You can do achieve what you describe using a Filter stage or a Transformer stage. You may be able to do it with a Switch stage (if the condition for output link selection is based on values in the data).

If a Copy stage has N output links it will always generate N streams of output (for N >= 0).

chulett · Post by **chulett** » Mon May 26, 2008 7:01 am

Thanks for the more clearly... was pretty sure but not 100% Wurlod proof sure.