Page 1 of 1

skip header records in a sequential file

Posted: Tue Apr 07, 2009 12:34 am
by Aggie99
hi guys,

I need your help on how to skip reading the first 10 rows of a sequential file.

I have a csv file, the first 10 records are metadata, the rest of the file is data records.

Metadata have different field format from non-metadata.

I try to read in the data, but not the metadata using sequential file stage. But I don't see an option that let me SKIP the first XX records so I can read from row 11 onward.

What can I do? thx for your help.
-Aggie

Posted: Tue Apr 07, 2009 12:53 am
by Kryt0n
Can you run Unix commands on a Windows DS server? (we can on our clients so would expect similar functionality on the server)

An alternative is to define the input as one column (type varchar) per row and drop the first 10 rows through a transformer or filter stage

Posted: Tue Apr 07, 2009 7:40 am
by chulett
Define the columns in the stage to match the data columns in the file, the records from 11 on. The stage will automatically discard the first 10 for not matching that.

Posted: Tue Apr 07, 2009 8:38 am
by Aggie99
that's what I am doing now, define the columns for records from row 11 onward. This resulted in a set of warnings in the log.
I guess I can downgrade the warnings. But I was hoping there're other options.

thx guys.

aggie

Posted: Tue Apr 07, 2009 9:32 am
by ray.wurlod
Put a reject link on the Sequential File stage. It will capture any row that does not match the metadata on the main output link.