Sequential file without a first line
Moderators: chulett, rschirm, roy
Sequential file without a first line
I have a sequential file where the first line is to be ignored.
However, it is not a title line, and the number of fields in the first line is smaller then in the following data lines. This last fact makes the use of the use of the Sequential File stage, and the option "First line is column names" not usable: an error is generated on the first line, as fields are missing.
What would be the best way to read a sequential file of which the first line is to be ignored.
Thanks,
Wim
However, it is not a title line, and the number of fields in the first line is smaller then in the following data lines. This last fact makes the use of the use of the Sequential File stage, and the option "First line is column names" not usable: an error is generated on the first line, as fields are missing.
What would be the best way to read a sequential file of which the first line is to be ignored.
Thanks,
Wim
-
- Premium Member
- Posts: 385
- Joined: Wed Jun 16, 2004 12:43 pm
- Location: Virginia, USA
- Contact:
You can use these unix command as filters directly in the sequential stage. On the stage tab, check Stage uses filter commands, and on the Outputs / General tab enter the unix command in the Filter command: box.
Chuck Smith
www.anotheritco.com
www.anotheritco.com
You could still play the 'First line is column names' game. In addition to that, check off the 'Ignore row truncation warnings'... either that or slide the 'Columns' definition tab to the right and set the entries that control missing columns appropriately.
Or as noted in previous posts, use UNIX commands to remove the first record. I'd suggest doing this in the 'Filter' box on the Sequential File stage rather than messing with 'before' commands.
Edit: Chuck beat me to that one.
Or as noted in previous posts, use UNIX commands to remove the first record. I'd suggest doing this in the 'Filter' box on the Sequential File stage rather than messing with 'before' commands.
Edit: Chuck beat me to that one.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
I understand your answers, but they require that space is allocated for the intermediate file and that these files be handleded correctly.
A Named Pipe would also be a solution for the intermediate file, but this also needs to be set up and managed in a reliable way. That's not always easy to do.
All this 'overhead' this would be unnecessary if you remain within the Datastage Designer environment. I would prefer DataStage to take care of all the data piping.
Would you agree with that consideration?
Thanks,
Wim
A Named Pipe would also be a solution for the intermediate file, but this also needs to be set up and managed in a reliable way. That's not always easy to do.
All this 'overhead' this would be unnecessary if you remain within the Datastage Designer environment. I would prefer DataStage to take care of all the data piping.
Would you agree with that consideration?
Thanks,
Wim
Ok. Read the online help for the Filter option. There's no need to use a named pipe for this, unless you prefer to take that route.
Simply putting the appropriate command in the Filter box (without the redirection as it leverages standard out) gets you what you need.
Simply putting the appropriate command in the Filter box (without the redirection as it leverages standard out) gets you what you need.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers