filepattern - reading list of files
Moderators: chulett, rschirm, roy
filepattern - reading list of files
Hi I have a file list which contains list of files to be read using sequential file with same metadata. Using file pattern, Can I just use the file name which contains the list of files to be read. and extract the data?
file_list.txt --> datafile1.dat
datafile2.dat
In other words, I have to use only the file_list.txt to read the data files. Please advise.
file_list.txt --> datafile1.dat
datafile2.dat
In other words, I have to use only the file_list.txt to read the data files. Please advise.
No, file pattern means you'd need to supply a wildcard pattern that would match all of the files you'd want to process, not a specific list of filenames. Typical solution for a 'list of files' would be a looping Sequence job that runs a single job once per filename.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
I prefer the transformer looping to the wildcard approach as you can then log to a control table the names of the files you have processed and the success of that processing. If you use the wildcard approach and the job aborts half way through you have no idea which of the files have been processed.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
Sorry - when I said Transformer looping I meant Sequence looping! Wrong looping.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Yup. Crap, how did I miss that? It's right there in black and white pixels in the documentation of all places:
File Pattern: Specifies a group of files to import. Specify file containing a list of files or a job parameter representing the file. The file could also contain be any valid shell expression, in Bourne shell syntax, that generates a list of file names.
To make matters worse, there's no "they musta just added that in the 8.x release" - I just found the same text in the 7.x docs as well. D'oh.
File Pattern: Specifies a group of files to import. Specify file containing a list of files or a job parameter representing the file. The file could also contain be any valid shell expression, in Bourne shell syntax, that generates a list of file names.
To make matters worse, there's no "they musta just added that in the 8.x release" - I just found the same text in the 7.x docs as well. D'oh.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Please explain for us what "considering the columns" means. If by that you mean that it doesn't respect the "First record is column names" setting for any but the first file, that's a known limitation when reading multiple files using any of the available mechanisms. Unless they've managed to fix that rather silly issue and you mean something else entirely...
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
[quote="chulett"]Yup. Crap, how did I miss that? It's right there in black and white pixels in the documentation of all places:
File Pattern: Specifies a group of files to import. Specify file containing a list of files or a job parameter representing the file. The file could also contain be any valid shell expression, in Bourne shell syntax, that generates a list of file names.
Hi chulett \ray
Good morning:
As per chulett, I have created the file which contains the list of file names and those files have the same metadata. And I created job which having :
Sequential file (File pattern and the file name which have file names) --> Tfr --> Sequential file.
Now I'm tring to load the data from the listed files. The job is not performaing as expected (Means it's not loading the data properly). It just taking the file names as values and outputing the same file names.
Now, i think you guys can understand the issue. Please clarrify me in this . Thank you....
File Pattern: Specifies a group of files to import. Specify file containing a list of files or a job parameter representing the file. The file could also contain be any valid shell expression, in Bourne shell syntax, that generates a list of file names.
Hi chulett \ray
Good morning:
As per chulett, I have created the file which contains the list of file names and those files have the same metadata. And I created job which having :
Sequential file (File pattern and the file name which have file names) --> Tfr --> Sequential file.
Now I'm tring to load the data from the listed files. The job is not performaing as expected (Means it's not loading the data properly). It just taking the file names as values and outputing the same file names.
Now, i think you guys can understand the issue. Please clarrify me in this . Thank you....