Using for loop in sequential stage filter command

[sbass1]

I'm experimenting with the sequential stage filter command...

I've discovered I can do something like:

File name:
/dev/null

(It seems DS requires a file name for the filter command, although I don't see why it should. What if my filter command doesn't require a file?)

Filter command:
head -5 /tmp/deleteme

(I know I could just set row limits on the job execution - I'm just trying to learn more about how DS works)

I suspect DS is doing something equivalent to:

cat /dev/null | head -5 /tmp/deleteme

This opens up other possibilities. For example, I needed some quick test data, so I tried:

File name:
/dev/null

Filter command:

for i in 1 1 2 2 3 3 3 3 4 5 6 7 8 9 10; do echo $i; done

Since cat /dev/null | for i in 1 1 2 2 3 3 3 3 4 5 6 7 8 9 10; do echo $i; done

works fine in the Unix shell, why doesn't this return any rows from my filter command? Should I expect any multi-line filter command (i.e. contains a semicolon) to fail?

Finally, if my filter command returns a non-zero return code, is there any way to trap that in the sequential stage processing?

Thanks,
Scott

[chulett]

Yes, it requires a filename but like you've found you can use /dev/null when you "don't need one" or if you need to use the filename differently than it would otherwise do. Much like you've seen, I've found that it does seem to do a "cat <filename> | <filter>" under the covers, so you could have put /tmp/deleteme in the filename and head -5 in the filter to have this executed by the stage:

cat /tmp/deleteme | head -5

I've had weird issues with certain characters in the filter command, sometimes you'll find a truncated version of the command there after you save and reopen the job and other times it just doesn't work as if the filter is being truncated 'on the fly' when the job runs. I tend to try and keep things as simple as possible there. Never tried it, but for your multi-line command, I'd try putting it into a script and then running the script itself from the Filter.

I believe that if the Filter command fails that the stage will automatically abort the job by logging a fatal error. Should be simple enough to force a test and see.