funnel Stage issue
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 161
- Joined: Wed Aug 02, 2006 4:30 am
funnel Stage issue
I need to add one row head and one row tail to my output data file.
And I use funnel to combine 3 different inputs.
Input1 is the one row header,
Input2 is the body of data file
Input3 is the tail of the file.
I use sequence mode for funnel. partition use default.
I guess it is a partition problem, since the row of input1 and the row of input 3 are always in the middle of output sequencial file, not as I expected input1 is the first row, and input3 is the last row.
Anybody knows how to configure the partition or some other options to make it work .
thanks
And I use funnel to combine 3 different inputs.
Input1 is the one row header,
Input2 is the body of data file
Input3 is the tail of the file.
I use sequence mode for funnel. partition use default.
I guess it is a partition problem, since the row of input1 and the row of input 3 are always in the middle of output sequencial file, not as I expected input1 is the first row, and input3 is the last row.
Anybody knows how to configure the partition or some other options to make it work .
thanks
You dont need to build a whole seperate job to do this. You can simply provide a command line cat to achieve this
Code: Select all
cat file1 file2 file3 > CompleteFile
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Or just as the Filter command in a Sequential File stage (the stage reads stdout from the filter command).
The next question is whether you are intending to read this file executing in parallel mode (that is, multiple readers per node), in which case the header and trailer lines will not be read by all readers.
If that's your intent, read the three files separately (as per your original design), but make sure that the partitioning for the header and trailer files is set to Entire. Do this on the relevant input links to the Funnel stage. That way, the header and trailer lines are visible on all partitions.
Code: Select all
cat file1 file2 file3
The next question is whether you are intending to read this file executing in parallel mode (that is, multiple readers per node), in which case the header and trailer lines will not be read by all readers.
If that's your intent, read the three files separately (as per your original design), but make sure that the partitioning for the header and trailer files is set to Entire. Do this on the relevant input links to the Funnel stage. That way, the header and trailer lines are visible on all partitions.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 161
- Joined: Wed Aug 02, 2006 4:30 am
use cat file1 file2 file3 > outputfile is fine.
But based on my understanding, funnel stage should work if we set the property as sequence, it will read the data from the first input link firstly, then next, then next.
But actually, it didn't .
Is there any problems when funnel stage output the file to sequencial file, any the sequence goes chaos...
thanks
But based on my understanding, funnel stage should work if we set the property as sequence, it will read the data from the first input link firstly, then next, then next.
But actually, it didn't .
Is there any problems when funnel stage output the file to sequencial file, any the sequence goes chaos...
thanks
-
- Participant
- Posts: 161
- Joined: Wed Aug 02, 2006 4:30 am
If you have 3 different links, add dummy fields with values eg: 1,2,3 in each of the transformers header, detail,trailer respectively... then while linking this 3 links to funnel use this dummy field(common to all the 3 links) as sort field.
Then header will comes first, then detial racords and then the trailer.
file-->trans('Dumfield' value as '1')| |
file-->trans('Dumfield' value as '2')|-->Funnel (sort type:continous |target
file-->trans('Dumfield' value as '3')| sort field: 'Dumfield')|
Done1 :D :D
Then header will comes first, then detial racords and then the trailer.
file-->trans('Dumfield' value as '1')| |
file-->trans('Dumfield' value as '2')|-->Funnel (sort type:continous |target
file-->trans('Dumfield' value as '3')| sort field: 'Dumfield')|
Done1 :D :D
Thanks and Regards!!
dspxlearn
dspxlearn
Re: funnel Stage issue
This is for my understanding even though post is resolved.I think if you use sequence mode for funnel and execution mode as sequential in Funnel stage it will solve your problem.please correct me anybody if am worng?