Sequential File Columns
Moderators: chulett, rschirm, roy
Sequential File Columns
I was under the impression that the number of columns defined in a sequential file stage had to match the number of columns in the source file.
Recently a new column was erroneously added to one of our source files and our jobs didn't pick it up. Is this the default behavior? If so, how do I trap this mismatch and send it down a reject link.
Thanks!
Recently a new column was erroneously added to one of our source files and our jobs didn't pick it up. Is this the default behavior? If so, how do I trap this mismatch and send it down a reject link.
Thanks!
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Premium Member
- Posts: 397
- Joined: Wed Apr 12, 2006 2:28 pm
- Location: Tennesse
Right. And my expectation with RCP being disabled is to reject rows with columns undefined in the sequential file stage. This is obviously incorrect.samsuf2002 wrote:If your RCP is disabled then you need to map the new columns from source till target. What is your job design ?
Say I have a process consisting of a Sequential File --> Transformer --> Sequential an my column definition for the first sequentual file is last_name and first_name and my sequentual file looks like the following:
Smith,Joe,CUST1,Somewhere,CA
My process is reading the first two columns and ignoring the last there fields as I only have two defined in the sequential stage. This came as a surprise. I expected the row to reject due to a metadata mismatch and am wondering if there is a way to force that behavior within the stage itself. I'd rather not us awk in the filter option if at all possible.
Last edited by kld05 on Thu Jan 15, 2009 8:33 am, edited 1 time in total.
Hmmm... on the Server side, there is an option to "Ignore row truncation warnings" which would give you that very behaviour - read the first x columns as defined in the stage and ignore the fact that there are more in the file. Is there an equivalent on the PX side?
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 152
- Joined: Tue Jan 13, 2009 8:59 am
Re: Sequential File Columns
kld05,
dint you get any warnings in the director when reading the file ???..
dint you get any warnings in the director when reading the file ???..
Re: Sequential File Columns
No I didn't. It's all green.betterthanever wrote:kld05,
dint you get any warnings in the director when reading the file ???..
-
- Participant
- Posts: 152
- Joined: Tue Jan 13, 2009 8:59 am
Re: Sequential File Columns
[quote="kld05"][quote="betterthanever"]kld05,
dint you get any warnings in the director when reading the file ???..[/quote]
No I didn't. It's all green.[/quote]
just tested writing 3 columns to a file and then tried to read the same file mentioning only 2 columns in the sequential stage.
gives me a warning in the director...
Sequential_File_7,0: Field "lastname" with 'delim=end' did not consume entire input, at offset: 14
dint you get any warnings in the director when reading the file ???..[/quote]
No I didn't. It's all green.[/quote]
just tested writing 3 columns to a file and then tried to read the same file mentioning only 2 columns in the sequential stage.
gives me a warning in the director...
Sequential_File_7,0: Field "lastname" with 'delim=end' did not consume entire input, at offset: 14
Re: Sequential File Columns
Interesting. I wonder if it is a project level setting that is toggling this behavior off for me as chulett suggested.betterthanever wrote:just tested writing 3 columns to a file and then tried to read the same file mentioning only 2 columns in the sequential stage.kld05 wrote:No I didn't. It's all green.betterthanever wrote:kld05,
dint you get any warnings in the director when reading the file ???..
gives me a warning in the director...
Sequential_File_7,0: Field "lastname" with 'delim=end' did not consume entire input, at offset: 14
-
- Participant
- Posts: 152
- Joined: Tue Jan 13, 2009 8:59 am
Re: Sequential File Columns
kld05,
can you post the format tab properties set to that sequential file???
can you post the format tab properties set to that sequential file???
Re: Sequential File Columns
Sure. I have only three values set.betterthanever wrote:kld05,
can you post the format tab properties set to that sequential file???
Final Delimiter = end
Delimiter = comma
Quote = none
-
- Participant
- Posts: 152
- Joined: Tue Jan 13, 2009 8:59 am
Re: Sequential File Columns
when the option
Final Delimiter = end
then it would look for the "end" after the last column you mention in the columns tab.
if it does not see it, would report a warning in the director and the import would be unsuccessfull..that is what my experience reading sequential files..
Final Delimiter = end
then it would look for the "end" after the last column you mention in the columns tab.
if it does not see it, would report a warning in the director and the import would be unsuccessfull..that is what my experience reading sequential files..
Re: Sequential File Columns
That is exactly what I thought and expected. For me it is not firing a warning and explicitly mapped columns are read in without issue.betterthanever wrote:when the option
Final Delimiter = end
then it would look for the "end" after the last column you mention in the columns tab.
if it does not see it, would report a warning in the director and the import would be unsuccessfull..that is what my experience reading sequential files..
Doesn't that depend on your "Reject Mode" setting? From the docs:
Reject Mode
Allows you to specify behavior if a read record does not match the expected schema. Choose from Continue to continue operation and discard any rejected rows, Fail to cease reading if any rows are rejected, or Save to send rejected rows down a reject link. Defaults to Continue.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers