filepattern - reading list of files

mekrreddy · Post by **mekrreddy** » Mon Jan 16, 2012 4:09 pm

Hi I have a file list which contains list of files to be read using sequential file with same metadata. Using file pattern, Can I just use the file name which contains the list of files to be read. and extract the data?

file_list.txt --> datafile1.dat
datafile2.dat

In other words, I have to use only the file_list.txt to read the data files. Please advise.

chulett · Post by **chulett** » Mon Jan 16, 2012 4:48 pm

No, file pattern means you'd need to supply a wildcard pattern that would match all of the files you'd want to process, not a specific list of filenames. Typical solution for a 'list of files' would be a looping Sequence job that runs a single job once per filename.

vmcburney · Post by **vmcburney** » Mon Jan 16, 2012 7:42 pm

I prefer the transformer looping to the wildcard approach as you can then log to a control table the names of the files you have processed and the success of that processing. If you use the wildcard approach and the job aborts half way through you have no idea which of the files have been processed.

ray.wurlod · Post by **ray.wurlod** » Mon Jan 16, 2012 8:14 pm

I usually prefer to do this looping in a controlling sequence.

vmcburney · Post by **vmcburney** » Mon Jan 16, 2012 8:42 pm

Sorry - when I said Transformer looping I meant Sequence looping! Wrong looping.

jwiles · Post by **jwiles** » Mon Jan 16, 2012 9:42 pm

The file pattern option allows you to also specify a file containing a list of files, or a shell command which will return a list of files.

Regards,

pandeesh · Post by **pandeesh** » Mon Jan 16, 2012 10:58 pm

In Sequential file stage?

chulett · Post by **chulett** » Tue Jan 17, 2012 12:06 am

Yup. Crap, how did I miss that? It's right there in black and white pixels in the documentation of all places:

File Pattern: Specifies a group of files to import. Specify file containing a list of files or a job parameter representing the file. The file could also contain be any valid shell expression, in Bourne shell syntax, that generates a list of file names.

To make matters worse, there's no "they musta just added that in the 8.x release" - I just found the same text in the 7.x docs as well. D'oh.

ray.wurlod · Post by **ray.wurlod** » Tue Jan 17, 2012 12:36 am

Worth logging in today. Learned something.

jwiles · Post by **jwiles** » Tue Jan 17, 2012 6:32 am

Careful, you'll make me feel like I accomplished something...

Kryt0n · Post by **Kryt0n** » Tue Jan 17, 2012 3:19 pm

I knew I had read about it somewhere but when I looked at the sequential file stage I was expecting a "Filename file" kind of option but could find nothing so gave up. Certainly didn't know about it available in 7.x... thought it was something new in 8.5!

Satwika · Post by **Satwika** » Wed Jan 18, 2012 7:14 am

Hi,

Can you please ellobarate on this. I tried to read from a file which contains the list of file names but it's not considering the columns in the listed files.

chulett · Post by **chulett** » Wed Jan 18, 2012 7:58 am

Please explain for us what "considering the columns" means. If by that you mean that it doesn't respect the "First record is column names" setting for any but the first file, that's a known limitation when reading multiple files using any of the available mechanisms. Unless they've managed to fix that rather silly issue and you mean something else entirely...

ray.wurlod · Post by **ray.wurlod** » Wed Jan 18, 2012 2:52 pm

Does the metadata in the link properties mention these columns that it's "not considering"?

Satwika · Post by **Satwika** » Fri Jan 20, 2012 12:13 am

[quote="chulett"]Yup. Crap, how did I miss that? It's right there in black and white pixels in the documentation of all places:

File Pattern: Specifies a group of files to import. Specify file containing a list of files or a job parameter representing the file. The file could also contain be any valid shell expression, in Bourne shell syntax, that generates a list of file names.

Hi chulett \ray

Good morning:

As per chulett, I have created the file which contains the list of file names and those files have the same metadata. And I created job which having :

Sequential file (File pattern and the file name which have file names) --> Tfr --> Sequential file.

Now I'm tring to load the data from the listed files. The job is not performaing as expected (Means it's not loading the data properly). It just taking the file names as values and outputing the same file names.
Now, i think you guys can understand the issue. Please clarrify me in this . Thank you....