DSXchange

Does the darn Filter command in the Sequential stage not support wildcards?

I need to process an unknown number of files in a directory using a wildcard pattern. Yes, all have the same metadata and there's no header row to worry about.

Rather than use a Folder stage with its potential size issues or a before-job script, I thought I could simply put the 'wildcarded' filename in the File Name field and put 'cat' in the Filter command. I tried putting '/dev/null' in the File Name and the full just 'cat #FILE_PATH#/XXX*.csv' command in the Filter command. Nope. I either get an Error 11 trying to View the data or it crashes my entire Designer session.

Has anyone done this? What is the trick?

In the meantime, I'm cat'ting the files before job to a fixed filename as a 'work around' but I'd really rather not do that if possible.

Did you try writing the cat in a shell script and calling the script from your job?

Hmmm... call the script in the Filter? Nope, didn't try that. Off we go.

Hi,
Seems weird, as if the stage ignores the command and forces the value in the file path and not accepting wild cards.
I'm on 7.5

Anyone got something different?

chulett wrote:Does the darn Filter command in the Sequential stage not support wildcards?

I am not aware of any darn command, therefore cannot advise how one might use it as a Filter command in a Sequential File stage. Is the darn command UNIX or DOS, or is it only used for SOX?

Argh! You said the 'S' word!

My sweet apple-cheek'd mother taught me to darn SOX.

Craig, (or anyone else)

did you find a solution to this? or only a work around? having the same problem as you had, very annoying!

I never did find a way to cat files to standard out such that the Filter could read them in as one continuous stream. I just fell back on cat to a fixed name either before job or more typically in a specific separate 'cat' job so that I don't need to repeat it if the processing job needs to be rerun.