Reading multiple files of same pattern

[mouthou]

Hi,

Can anyone help me how to read the files of some patterm using sequential file stage in server stage. It seems like server version doesnt support wild cards unlike in parallel version.

The files will be at the location like AA_010.dat, AA_02.dat etc. I want to give like AA_*.dat so that the job picks up all the files.
Is there anyway to do it?

Thanks in advance.

[narasimha]

You'r best bet would be to concatenate all these files and then read them, as your file pattern is the same
or else build a Job Sequences using Loop Activiy Sequences to handle it.

[chulett]

No, the Server Sequential stage does not support wildcards, as least not as easily as the PX version does. Read up on the Filter option, there are times it can be leveraged for that. Use a command like:

Code: Select all

cat AA_*.dat

there to send all files, one after another, to 'standard out'. The stage will then read standard out stream as if it was a single file.

You can also read up on the Folder stage, which can handle small files. It's typically used for XML files but could make do in a pinch. You could also play with named pipes for yet-another-way to accomplish this. Or concatenate all into one combo file for processing. Or...

[g_goli]

Can anyone help me how to read the files of some patterm using sequential file stage in server stage. It seems like server version doesnt support wild cards unlike in parallel version.

Hi,

you can read files by a wildcard in server version using a Folder stage (Output tab -> Properties -> Wildcard).
It will read file names which you can process then and put into a hashed file or other sequential file which would read the files consecutively in a separate job (or a sequence).

Regards,
ggoli

[mouthou]

I tried using Folder stage. It has got couple of limitations.

It takes exactly 2 columns and expecting a key column. And when it creates the final file, it adds one extra record as the header.

So, I gave COPY command as the before-job subroutine to concatenate the files and create a single file at the same location and reading that file in my job.

[ray.wurlod]

Job sequence with StartLoop activity based on the regular expression?