Sequential stage and wildcards in the Filter
Moderators: chulett, rschirm, roy
Sequential stage and wildcards in the Filter
Does the darn Filter command in the Sequential stage not support wildcards?
I need to process an unknown number of files in a directory using a wildcard pattern. Yes, all have the same metadata and there's no header row to worry about.
Rather than use a Folder stage with its potential size issues or a before-job script, I thought I could simply put the 'wildcarded' filename in the File Name field and put 'cat' in the Filter command. I tried putting '/dev/null' in the File Name and the full just 'cat #FILE_PATH#/XXX*.csv' command in the Filter command. Nope. I either get an Error 11 trying to View the data or it crashes my entire Designer session.
Has anyone done this? What is the trick?
In the meantime, I'm cat'ting the files before job to a fixed filename as a 'work around' but I'd really rather not do that if possible.
I need to process an unknown number of files in a directory using a wildcard pattern. Yes, all have the same metadata and there's no header row to worry about.
Rather than use a Folder stage with its potential size issues or a before-job script, I thought I could simply put the 'wildcarded' filename in the File Name field and put 'cat' in the Filter command. I tried putting '/dev/null' in the File Name and the full just 'cat #FILE_PATH#/XXX*.csv' command in the Filter command. Nope. I either get an Error 11 trying to View the data or it crashes my entire Designer session.
Has anyone done this? What is the trick?
In the meantime, I'm cat'ting the files before job to a fixed filename as a 'work around' but I'd really rather not do that if possible.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 3337
- Joined: Mon Jan 17, 2005 4:49 am
- Location: United Kingdom
Hi,
Seems weird, as if the stage ignores the command and forces the value in the file path and not accepting wild cards.
I'm on 7.5
Anyone got something different?
Seems weird, as if the stage ignores the command and forces the value in the file path and not accepting wild cards.
I'm on 7.5
Anyone got something different?
Roy R.
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Re: Sequential stage and wildcards in the Filter
I am not aware of any darn command, therefore cannot advise how one might use it as a Filter command in a Sequential File stage. Is the darn command UNIX or DOS, or is it only used for SOX?chulett wrote:Does the darn Filter command in the Sequential stage not support wildcards?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
I never did find a way to cat files to standard out such that the Filter could read them in as one continuous stream. I just fell back on cat to a fixed name either before job or more typically in a specific separate 'cat' job so that I don't need to repeat it if the processing job needs to be rerun.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers