Hi,
Can anyone help me how to read the files of some patterm using sequential file stage in server stage. It seems like server version doesnt support wild cards unlike in parallel version.
The files will be at the location like AA_010.dat, AA_02.dat etc. I want to give like AA_*.dat so that the job picks up all the files.
Is there anyway to do it?
Thanks in advance.
Reading multiple files of same pattern
Moderators: chulett, rschirm, roy
No, the Server Sequential stage does not support wildcards, as least not as easily as the PX version does. Read up on the Filter option, there are times it can be leveraged for that. Use a command like:
there to send all files, one after another, to 'standard out'. The stage will then read standard out stream as if it was a single file.
You can also read up on the Folder stage, which can handle small files. It's typically used for XML files but could make do in a pinch. You could also play with named pipes for yet-another-way to accomplish this. Or concatenate all into one combo file for processing. Or...
Code: Select all
cat AA_*.dat
You can also read up on the Folder stage, which can handle small files. It's typically used for XML files but could make do in a pinch. You could also play with named pipes for yet-another-way to accomplish this. Or concatenate all into one combo file for processing. Or...
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
Re: Reading multiple files of same pattern
Hi,mouthou wrote: Can anyone help me how to read the files of some patterm using sequential file stage in server stage. It seems like server version doesnt support wild cards unlike in parallel version.
you can read files by a wildcard in server version using a Folder stage (Output tab -> Properties -> Wildcard).
It will read file names which you can process then and put into a hashed file or other sequential file which would read the files consecutively in a separate job (or a sequence).
Regards,
ggoli
Re: Reading multiple files of same pattern
I tried using Folder stage. It has got couple of limitations.
It takes exactly 2 columns and expecting a key column. And when it creates the final file, it adds one extra record as the header.
So, I gave COPY command as the before-job subroutine to concatenate the files and create a single file at the same location and reading that file in my job.
It takes exactly 2 columns and expecting a key column. And when it creates the final file, it adds one extra record as the header.
So, I gave COPY command as the before-job subroutine to concatenate the files and create a single file at the same location and reading that file in my job.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: