Sequential processing of Files

pravin1581 · Post by **pravin1581** » Tue Apr 10, 2007 7:13 am

Hi ,

I have a requirement where i need to process the files in the job sequentially. By sequentially I mean that there will be an output file which in turn will be the input to another file.

File-----> Transformer-----> File------->Transformer------> File

I am unable to attach the third file after the transformer . The second file is giving a error like the file (3rd) cannot be attached as it is not allowed due to the current setting or lack of the reject mode property. Some settings need to be changed. Please provide some solution.

Thanx in advance.

DSguru2B · Post by **DSguru2B** » Tue Apr 10, 2007 7:22 am

The second file can only support a reject mode and not an input stream to your second transformer. What exactly are you trying to achieve. From your design and explanation it is clear that you are reading from a sequential file and writing to another which inturn is writing to another. Why do you need another link from your second file?

pravin1581 · Post by **pravin1581** » Tue Apr 10, 2007 7:40 am

DSguru2B wrote:The second file can only support a reject mode and not an input stream to your second transformer. What exactly are you trying to achieve. From your design and explanation it is clear that you are reading from a sequential file and writing to another which inturn is writing to another. Why do you need another link from your second file?

The reuirement is such that the 3rd file needs to be generated from the 2nd file in the same job or otherwise we have to design another job to generate the 3rd file which will increase the number of jobs tremendously.

DSguru2B · Post by **DSguru2B** » Tue Apr 10, 2007 7:49 am

It wont let you do that. The sequential file stage supports only a reject link as the second link. What transformations are involved? Cant you do all in just one transformer stage?

pravin1581 · Post by **pravin1581** » Tue Apr 10, 2007 8:06 am

DSguru2B wrote:It wont let you do that. The sequential file stage supports only a reject link as the second link. What transformations are involved? Cant you do all in just one transformer stage?

Both the transformations are doing different thing, and the second transformation stage needs to work on the file produced after the first transformation. Can you suggest some other way to achieve it. This is a great bottleneck in parallel job as the same can be done very easily in server job . Please suggest some other alternative.

DSguru2B · Post by **DSguru2B** » Tue Apr 10, 2007 8:13 am

Please advise the different transformations you are doing. Maybe both can be incorporated into one and the same shall be advised. Help us to help you.

pp · Post by pp » Tue Apr 10, 2007 8:16 am

Try using containers.

pravin1581 · Post by **pravin1581** » Tue Apr 10, 2007 8:25 am

DSguru2B wrote:Please advise the different transformations you are doing. Maybe both can be incorporated into one and the same shall be advised. Help us to help you.

The second transformation will work only on the second output file generated. The transformation is for hierarchy.

DSguru2B · Post by **DSguru2B** » Tue Apr 10, 2007 8:33 am

Two options:
1) Do it in a server job.
2) Do it in two seperate px jobs.

prnath78 · Post by **prnath78** » Tue Apr 10, 2007 9:04 am

how about this

File-----> Transformer-----> Copy--o/plink1----->Transformer------> File
--------------------------------Copy--o/plink2-----> File

Thanks & Regards,
Ranga

ray.wurlod · Post by **ray.wurlod** » Tue Apr 10, 2007 1:37 pm

Parallel jobs do not permit "passive" stages to have both input and output link (reject link is a special case).

You need two jobs, possibly run under control of a job sequence. It's the only way. Ranga's solution does not achieve the dependency of the third file upon the output from the second.

pravin1581 · Post by **pravin1581** » Wed Apr 11, 2007 3:36 am

ray.wurlod wrote:Parallel jobs do not permit "passive" stages to have both input and output link (reject link is a special case).

You need two jobs, possibly run under control of a job sequence. It's the only way. Ranga's solution does not achieve the dependency of the third file upon the output from the second.

Can I do it like this :

File(1)------> Transformer--------> File(2)

File(2))------> Transformer--------> File(3)

At least it reduces the number of jobs . Please suggest whether it is feasible.

DSguru2B · Post by **DSguru2B** » Wed Apr 11, 2007 7:32 am

It wont work. Your job will either fail, or process zero records from file2 if the file is present. Both the streams will fire off simultaneously.

pravin1581 · Post by **pravin1581** » Wed Apr 11, 2007 8:28 am

DSguru2B wrote:It wont work. Your job will either fail, or process zero records from file2 if the file is present. Both the streams will fire off simultaneously.

Thats true , the second file is reading 0 rows . Can something be done with the link ordering so that the second link is triggered after the completion of the first link. If this fails then it would increase the number of jobs tremendously.

ray.wurlod · Post by **ray.wurlod** » Wed Apr 11, 2007 3:50 pm

Yes.

You can put the second link into a second job.

Execute the two jobs consecutively from a job sequence.