Sequential file input with optional columns

edwds · Post by **edwds** » Tue Oct 03, 2006 8:26 am

In a server job you could make columns in sequential files optional in the metadata (columns) tab by converting them to nulls with the "INCOMPLETE COLUMN" property. Is there an equivalent in the PX Sequential stage. I have a delimted file that has some optional columns at the end of each record.

samsuf2002 · Post by **samsuf2002** » Tue Oct 03, 2006 8:56 am

u can use SetNull() in the transformer for optional columns and make their value null

edwds · Post by **edwds** » Tue Oct 03, 2006 9:02 am

I can't even get to the transformer since the Sequential file stage gives me errors on reading the source data. These optional columns at the end have no delimeters at this point. So for example record 1 could have 10 columns but record 2 has only 5. But after the fifth column there isn't any delimeters. So the sequential file stage doesn't know how to handle the record so it ignores it and doesn't even bring it in.

ray.wurlod · Post by **ray.wurlod** » Tue Oct 03, 2006 3:57 pm

I don't think you can do it with properties. You can read the entire record as a VarChar, parsing it subsequently using a Transformer stage.

shepli · Post by **shepli** » Wed Oct 04, 2006 9:32 am

I thinnk you need to set the "INCOMPLETE COLUMN" of all the optional columns to "replace".

ray.wurlod · Post by **ray.wurlod** » Wed Oct 04, 2006 2:42 pm

Please advise where to find the "Incomplete Column" property in a parallel job's Sequential File stage.

shepli · Post by **shepli** » Wed Oct 04, 2006 2:53 pm

Please advise where to find the "Incomplete Column" property in a parallel job's Sequential File stage.

My fault. I thought we are talking about server jobs. Sorry, Ray.