File merge

[pratapsriram]

Can we do any merge like this automatically? All the Audit.txt into a single Audit.txt? I need to remove the headers and trailers in each file too. This is now being done manually and want to implement an automated way either using any shell script or Perl or Nant as we use Nant.
The files are appearing in the share drive as:
\Log\20120915\Audit.txt
\Log\20120916\Audit.txt
\Log\20120917\Audit.txt
\Log\20120918\Audit.txt
\Log\20120919\Audit.txt

[ray.wurlod]

Why not just script the commands that are currently being performed manually?

[chandra.shekhar@tcs.com]

You can use the command as told by Ray and call it in the Before-Job Subroutine.

[ray.wurlod]

Or you could even make it a Filter command; the Sequential File stage reads stdout of the Filter command.

[chulett]

I'd wager that what is being done manually is the file is being opened in a editor, the header and footer are being deleted and the file is then saved. As to how to automate that, I'd look into something like sed which could remove the header & footer and then stream the result to standard out which (as noted) works nicely in the Filter option of the Sequential File stage.

The complication is the fact that multiple files are involved. So you may need to loop the processing job and work with each file one by one, or script the filter command to create on consolidated file that the job processes and then deletes. One question in my mind for that approach is do you need to know the name of the source file in your processing? That answer would help differentiate the viable paths for you.