DSXchange

Hi all,

One of the sources in my parallel job is a SQf that has a header. The job uses 4 nodes and before performing any operations on the SQF, I would like to remove the header and pass the remaining data to the stages.

I thought of using a UNIX script but the number of rows in SQF are very very high and thus performs poorly. Another option is to use a transformer. But I want to reduce the use a transformer in ||el jobs.

Can anyone suggest some good options for doing this.

In Sequential file stage you have a option where you can set the first the line as column header. So DS will send the remaining data to next stage. You do not need to use unix script or transformer.

Regadrs,
Tushar

Unless some very specific conditions apply, including that your sequential file is in fixed width format, it will not be processed on more than one node (that is, sequentially).

Therefore, whether you follow the Sequential File stage with a Filter stage, a Transformer stage or some other stage kind, if you force that stage to operate sequentially you can reliably ensure that there is but one first row.

Another possibility is to rely on the fact that the header row is unlikely to match the schema, so you could trap the header row in a reject link from the Sequential File stage.

Thanks a lot for the help guys. I was working on 7.1 and this doesnot have an option to remove first row. DS7.5 does have such option.