DSXchange

Not really a DataStage question as I don't want a job solution, but a UNIX one.

I have 16M records in 51 identical files that need to be concatenated together for processing. I'd like to add a single letter as a trailing pipe delimited column to each record during the process, if possible. I know which files need what letter, so don't worry about that. Just wondering if there's some way to maintain the speed of a straight command line 'cat *.xxx > file' operation and add a column to the end of each record at the same time.

Thanks!

Look into sed, Craig. Something like You can perform this command on the files, either before concatenating them or after.
As for the speed, you will have to test it out. Sed is pretty fast. But dont know with 51 x 2M records.

Thanks - running some timing tests. I can pipe cats through sed:
Two birds, one stone. :D

This does work but adds some overhead. Cat'ing 2.6M records went from 23 seconds to 1 minute 23 seconds. I can live with that but still curious if there is something less impactive that could be done.

How about using the wildcard as stdin for sed (so they work one at a time) and append-redirecting the output into your file? You could also parallelize the sed operations with a bit more scripting.

Syncsort available?

kcbland wrote:Syncsort available?

Sadly, no.

ray.wurlod wrote:How about using the wildcard as stdin for sed (so they work one at a time) and append-redirecting the output into your file?

Err... meaning... this?
I'll give it a shot.

Edited to add: WTH?

The substitution syntax that worked fine in the other form won't parse in this one, in spite of me cribbing it directly from the man pages.

sed: Function s/$/|M cannot be parsed

Doesn't seem to matter what I put between the quotes, the dollar sign or pipe are not the issue here.

Try this:

When I transcribed the previous syntax I was using into the post I left off the trailing slash, which is why it could no longer be parsed. So now both are working and here are some timing tests for anyone interested:
This for 1.7M records. I'll go with the former, though either would be 'fine' in the long run. :D