Adding newlines to multiple files in stream
Posted: Wed Jul 27, 2016 1:17 pm
Hi,
I've got a directory full of gz compressed xml files that I want to stream through an xml transformer to parse them to delimited text. The xml transformer needs to know where each file ends in order to accept them as valid xml. But the xml within the gz files, which originate in an external Web service, don't end with a newline.
Currently, I'm reading the files in with a wildcard in an external source stage using zcat to stream the uncompressed xml. I've tried putting a sed command in that stage and in a subsequent external filter stage to add a newline at the end of the file, but have only been able to add the newline at the end of the whole set.
Does anyone know of a way to do this? (insert a newline at the end of each xml file) I'd like to avoid landing all the unzipped files on my server, and I'm afraid I can't affect the contents of the gz files before I get them.
Thanks for any help!
Kelly
I've got a directory full of gz compressed xml files that I want to stream through an xml transformer to parse them to delimited text. The xml transformer needs to know where each file ends in order to accept them as valid xml. But the xml within the gz files, which originate in an external Web service, don't end with a newline.
Currently, I'm reading the files in with a wildcard in an external source stage using zcat to stream the uncompressed xml. I've tried putting a sed command in that stage and in a subsequent external filter stage to add a newline at the end of the file, but have only been able to add the newline at the end of the whole set.
Does anyone know of a way to do this? (insert a newline at the end of each xml file) I'd like to avoid landing all the unzipped files on my server, and I'm afraid I can't affect the contents of the gz files before I get them.
Thanks for any help!
Kelly