Hi all,
One of the sources in my parallel job is a SQf that has a header. The job uses 4 nodes and before performing any operations on the SQF, I would like to remove the header and pass the remaining data to the stages.
I thought of using a UNIX script but the number of rows in SQF are very very high and thus performs poorly. Another option is to use a transformer. But I want to reduce the use a transformer in ||el jobs.
Can anyone suggest some good options for doing this.
Eleminating first row in SQF
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 13
- Joined: Wed Dec 15, 2004 2:19 am
Re: Eleminating first row in SQF
In Sequential file stage you have a option where you can set the first the line as column header. So DS will send the remaining data to next stage. You do not need to use unix script or transformer.
Regadrs,
Tushar
Regadrs,
Tushar
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Unless some very specific conditions apply, including that your sequential file is in fixed width format, it will not be processed on more than one node (that is, sequentially).
Therefore, whether you follow the Sequential File stage with a Filter stage, a Transformer stage or some other stage kind, if you force that stage to operate sequentially you can reliably ensure that there is but one first row.
Another possibility is to rely on the fact that the header row is unlikely to match the schema, so you could trap the header row in a reject link from the Sequential File stage.
Therefore, whether you follow the Sequential File stage with a Filter stage, a Transformer stage or some other stage kind, if you force that stage to operate sequentially you can reliably ensure that there is but one first row.
Another possibility is to rely on the fact that the header row is unlikely to match the schema, so you could trap the header row in a reject link from the Sequential File stage.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.