I have a folder which has few Seqeuntial file names. The number of files names can vary every time. I need to merge data from all these files and store it in a new Seq file for further processing.
Can anybody help me on how to handle this? Thank you.
Kumar
Merging Data
Moderators: chulett, rschirm, roy
vskr72,
how about using a folder stage to selectively get the filenames you wish to process, then a simple SeqFile->SeqFile type job with the output file NOT being overwritten, but appended to. It lacks the intrinsic simplicity and elegance of Ken's solution [I'd recommend his suggestion, execute a single efficient UNIX command].
how about using a folder stage to selectively get the filenames you wish to process, then a simple SeqFile->SeqFile type job with the output file NOT being overwritten, but appended to. It lacks the intrinsic simplicity and elegance of Ken's solution [I'd recommend his suggestion, execute a single efficient UNIX command].
-
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom