Stripping header and trailer record from input files
Posted: Thu Jun 04, 2009 9:50 am
Hi,
There are many topics out there which almost answer my question, but none which actually do. Hopefully someone willbe good enough to give me a hand.
I have a job that process a number of files on each run. the job currently starts with a sequential file read by file pattern. each file consists of a header record, record number 1, and a trailer record, the last record. All other records are data records. the data records are comma delimited.
When i run the job the read rejects the header and trailers as they do not match the meta data. this currently writes a number of warnings to the logs which if i am processing enough files blows the warning limit and aborts the job.
i can reject the header and trailer of course, but this still writes the warnings to the logs. is there some way of stopping the warnings when writing to the logs?
Alternatively i can preprocess the file to remove the header and trailer. unfortunatley as the read is by file pattern the file filter option is not available and si i cannot do sed command to remove the first and last record.
i also tried to run the data through a filter stage, that lead to further data rejects in the intial read.
Could anyone save me from going bald by giving me a pointer or two please?
Thanks, as always.
John Fitz
There are many topics out there which almost answer my question, but none which actually do. Hopefully someone willbe good enough to give me a hand.
I have a job that process a number of files on each run. the job currently starts with a sequential file read by file pattern. each file consists of a header record, record number 1, and a trailer record, the last record. All other records are data records. the data records are comma delimited.
When i run the job the read rejects the header and trailers as they do not match the meta data. this currently writes a number of warnings to the logs which if i am processing enough files blows the warning limit and aborts the job.
i can reject the header and trailer of course, but this still writes the warnings to the logs. is there some way of stopping the warnings when writing to the logs?
Alternatively i can preprocess the file to remove the header and trailer. unfortunatley as the read is by file pattern the file filter option is not available and si i cannot do sed command to remove the first and last record.
i also tried to run the data through a filter stage, that lead to further data rejects in the intial read.
Could anyone save me from going bald by giving me a pointer or two please?
Thanks, as always.
John Fitz