I have a file with Header, Trailer as first and last records. I know about the sequential file sed "1d;\$d" aspect but I was wondering if there was a way to do this in the FTP stage.
Any help would be great.
Thanks,
Scott
Remove the First and Last Record of File in FTP
Moderators: chulett, rschirm, roy
What's the next function after the FTP? If you're landing the file on the server, it's easier to edit the file with a script or with the file stage you are using.
If you are streaming the data directly to the next stage, you might find it better to do the editing there, a filter or transformer perhaps.
If you are streaming the data directly to the next stage, you might find it better to do the editing there, a filter or transformer perhaps.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
Do the records have a unique identifier, which tells you what kind of record? If so, the transformer constraint can passively drop the header and trailer.
In the mainframe/COBOL world, record type is standard. If your file is not like that, you'll have to get creative with how to identify the records. Good luck.
In the mainframe/COBOL world, record type is standard. If your file is not like that, you'll have to get creative with how to identify the records. Good luck.
Franklin Evans
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
"Shared pain is lessened, shared joy increased. Thus do we refute entropy." -- Spider Robinson
Using mainframe data FAQ: viewtopic.php?t=143596 Using CFF FAQ: viewtopic.php?t=157872
By the way - there are some positives and negatives about using the FTP stage. Unless it has changed in version 11, it does "row-by-row" transmissions with an acknowledgement request after each row.
This makes the FTP stage able to detect a transmission problem, but much slower than the unix sftp command.
The negative about the sftp command that ships with most UNIX operating systems is that it does not typically report transmission problems back to the calling process, especially when you use the interactive version of sftp.
Though sftp is much faster than the FTP stage, it can lead to problems with your process not receiving the complete file and not realizing it due to a lack of an error code. One solution is to buy a cheap third-party sftp tool that reports transmission errors (most do - they understand that it is a problem). The other is to either check file sizes on the source system or have a trailer record to validate "Yup - we got it all".
Since your file has a trailer record that can be used to confirm complete receipt, you might want to try using the UNIX sftp command. That would be significantly faster and would allow you to use commands to check for header / trailer and then strip them off before processing the file via sequential file stage. It could also report an error code when it doesn't find a trailer (or a file!) so the job doesn't attempt to process an old or partial file.
This makes the FTP stage able to detect a transmission problem, but much slower than the unix sftp command.
The negative about the sftp command that ships with most UNIX operating systems is that it does not typically report transmission problems back to the calling process, especially when you use the interactive version of sftp.
Though sftp is much faster than the FTP stage, it can lead to problems with your process not receiving the complete file and not realizing it due to a lack of an error code. One solution is to buy a cheap third-party sftp tool that reports transmission errors (most do - they understand that it is a problem). The other is to either check file sizes on the source system or have a trailer record to validate "Yup - we got it all".
Since your file has a trailer record that can be used to confirm complete receipt, you might want to try using the UNIX sftp command. That would be significantly faster and would allow you to use commands to check for header / trailer and then strip them off before processing the file via sequential file stage. It could also report an error code when it doesn't find a trailer (or a file!) so the job doesn't attempt to process an old or partial file.