DSXchange

I was under the impression that the number of columns defined in a sequential file stage had to match the number of columns in the source file.

Recently a new column was erroneously added to one of our source files and our jobs didn't pick it up. Is this the default behavior? If so, how do I trap this mismatch and send it down a reject link.

Thanks!

Does your stage have runtime column propagation (RCP) enabled?

ray.wurlod wrote:Does your stage have runtime column propagation (RCP) enabled?

Hi Ray. RCP is disabled. I am a bit embarassed that I do not know this.

If your RCP is disabled then you need to map the new columns from source till target. What is your job design ?

samsuf2002 wrote:If your RCP is disabled then you need to map the new columns from source till target. What is your job design ?

Right. And my expectation with RCP being disabled is to reject rows with columns undefined in the sequential file stage. This is obviously incorrect.

Say I have a process consisting of a Sequential File --> Transformer --> Sequential an my column definition for the first sequentual file is last_name and first_name and my sequentual file looks like the following:

Smith,Joe,CUST1,Somewhere,CA

My process is reading the first two columns and ignoring the last there fields as I only have two defined in the sequential stage. This came as a surprise. I expected the row to reject due to a metadata mismatch and am wondering if there is a way to force that behavior within the stage itself. I'd rather not us awk in the filter option if at all possible.

Hmmm... on the Server side, there is an option to "Ignore row truncation warnings" which would give you that very behaviour - read the first x columns as defined in the stage and ignore the fact that there are more in the file. Is there an equivalent on the PX side?

kld05,
dint you get any warnings in the director when reading the file ???..

betterthanever wrote:kld05,
dint you get any warnings in the director when reading the file ???..

No I didn't. It's all green.

[quote="kld05"][quote="betterthanever"]kld05,
dint you get any warnings in the director when reading the file ???..[/quote]

No I didn't. It's all green.[/quote]

just tested writing 3 columns to a file and then tried to read the same file mentioning only 2 columns in the sequential stage.

gives me a warning in the director...
Sequential_File_7,0: Field "lastname" with 'delim=end' did not consume entire input, at offset: 14

betterthanever wrote:

kld05 wrote:

betterthanever wrote:kld05,
dint you get any warnings in the director when reading the file ???..

No I didn't. It's all green.

just tested writing 3 columns to a file and then tried to read the same file mentioning only 2 columns in the sequential stage.

gives me a warning in the director...
Sequential_File_7,0: Field "lastname" with 'delim=end' did not consume entire input, at offset: 14

Interesting. I wonder if it is a project level setting that is toggling this behavior off for me as chulett suggested.

kld05,
can you post the format tab properties set to that sequential file???

betterthanever wrote:kld05,
can you post the format tab properties set to that sequential file???

Sure. I have only three values set.

Final Delimiter = end
Delimiter = comma
Quote = none

when the option
Final Delimiter = end
then it would look for the "end" after the last column you mention in the columns tab.

if it does not see it, would report a warning in the director and the import would be unsuccessfull..that is what my experience reading sequential files..

betterthanever wrote:when the option
Final Delimiter = end
then it would look for the "end" after the last column you mention in the columns tab.

if it does not see it, would report a warning in the director and the import would be unsuccessfull..that is what my experience reading sequential files..

That is exactly what I thought and expected. For me it is not firing a warning and explicitly mapped columns are read in without issue.

Doesn't that depend on your "Reject Mode" setting? From the docs:

Reject Mode
Allows you to specify behavior if a read record does not match the expected schema. Choose from Continue to continue operation and discard any rejected rows, Fail to cease reading if any rows are rejected, or Save to send rejected rows down a reject link. Defaults to Continue.