funnel Stage issue

[xinhuang66]

I need to add one row head and one row tail to my output data file.

And I use funnel to combine 3 different inputs.

Input1 is the one row header,
Input2 is the body of data file
Input3 is the tail of the file.

I use sequence mode for funnel. partition use default.

I guess it is a partition problem, since the row of input1 and the row of input 3 are always in the middle of output sequencial file, not as I expected input1 is the first row, and input3 is the last row.

Anybody knows how to configure the partition or some other options to make it work .

thanks

[DSguru2B]

You dont need to build a whole seperate job to do this. You can simply provide a command line cat to achieve this

Code: Select all

cat file1 file2 file3 > CompleteFile

[ray.wurlod]

Or just

Code: Select all

cat file1 file2 file3 

as the Filter command in a Sequential File stage (the stage reads stdout from the filter command).

The next question is whether you are intending to read this file executing in parallel mode (that is, multiple readers per node), in which case the header and trailer lines will not be read by all readers.

If that's your intent, read the three files separately (as per your original design), but make sure that the partitioning for the header and trailer files is set to Entire. Do this on the relevant input links to the Funnel stage. That way, the header and trailer lines are visible on all partitions.

[xinhuang66]

Thanks !

And I will try it

[xinhuang66]

use cat file1 file2 file3 > outputfile is fine.

But based on my understanding, funnel stage should work if we set the property as sequence, it will read the data from the first input link firstly, then next, then next.

But actually, it didn't .

Is there any problems when funnel stage output the file to sequencial file, any the sequence goes chaos...

thanks

[gbusson]

There are actually two operators :

the funnel and the sort Merge (as you run the funnel in sequential mode).

Thaty is the reason of the "bug"

[xinhuang66]

Not very sure what you mean ?

Even if funnel = sort + funnel, it couldn't cause a bug..

Anyway,how to fix it up..

[dspxlearn]

If you have 3 different links, add dummy fields with values eg: 1,2,3 in each of the transformers header, detail,trailer respectively... then while linking this 3 links to funnel use this dummy field(common to all the 3 links) as sort field.
Then header will comes first, then detial racords and then the trailer.

file-->trans('Dumfield' value as '1')| |
file-->trans('Dumfield' value as '2')|-->Funnel (sort type:continous |target
file-->trans('Dumfield' value as '3')| sort field: 'Dumfield')|




Done1 :D :D

[xinhuang66]

That definitely work.

thanks

[vamsi.4a6]

This is for my understanding even though post is resolved.I think if you use sequence mode for funnel and execution mode as sequential in Funnel stage it will solve your problem.please correct me anybody if am worng?