Page 1 of 1
File Pattern problems !
Posted: Wed Sep 12, 2007 5:55 am
by hamzaqk
Hi,
just playing around with the functionality of 'file pattern'which you can set in sequential files to read multiple files in a folder.
i have four files in a folder a.txt , b.txt , c.txt and d.txt in folder E:\files
when i read them individually they work all ok
but when i change the property to file pattern and state E:\files\*.txt as the pattern and view the data . it says no files found at host XYZ E:\files\*.txt or
when i run the job it says :
Missing record delimiter "\n", saw EOF instead
i only have one value in all the files and that is 1 i.e.
a.txt has only one col column1 with value 1 in the file and same is the case for other files
Any ideas ?
Posted: Wed Sep 12, 2007 6:02 am
by Maveric
One of the files might be missing a delimiter. make four exact copies of a.txt and try.
Posted: Wed Sep 12, 2007 9:59 pm
by hamzaqk
this is exactly what i did before and how i created the test files. i have read them individually with the same options for all and they work fine. it just gives me a problem when i try to read them all at once. i have done all the basic checks, but it does'nt work. and gives me the same errors. it however does process only the first file.
Posted: Wed Sep 12, 2007 10:17 pm
by ArndW
Add a final \n to your file a.txt and see if the error wanders to file 2...
Posted: Wed Sep 12, 2007 10:33 pm
by hamzaqk
thanks. stil does'nt work. when i click on view data on seq file stage it gives me this:
Code: Select all
##I TOSH 000002 09:31:35(001) <main_program> orchgeneral: loaded
##I TOSH 000002 09:31:35(002) <main_program> orchsort: loaded
##I TOSH 000002 09:31:35(003) <main_program> orchstats: loaded
##W TCOS 000049 09:31:35(004) <main_program> Parameter specified but not used in flow: DSProjectMapName
##W TOIX 000000 09:31:35(005) <main_program> createFilesetFromPattern(): Couldn't find any files on host with pattern E:/test/*.txt.
>##E TOIX 000138 09:31:36(001) <Sequential_File_0> At least one filename or data source must be set in APT_FileImportOperator before use.
>##E TFSR 000019 09:31:36(005) <main_program> Could not check all operators because of previous error(s)
>##E TCOS 000029 09:31:36(006) <main_program> Creation of step finished with status = FAILED
Posted: Wed Sep 12, 2007 11:07 pm
by Minhajuddin
Try changing the backslashes in your path to forward slashes. It's a bug in the parallel edition.
E:\test/*.txt
Posted: Wed Sep 12, 2007 11:42 pm
by hamzaqk
ok i have tried all three of these and it still says the same !!
E:/test/*.txt
E:/test/*.txt
and E:\test/*.txt
what is an APT_FileImportOperator ?????
Posted: Thu Sep 13, 2007 1:37 am
by ray.wurlod
APT_FileImportOperator is the class from which Import objects (such as the operator generated by a Sequential File stage with an output link generates) are instantiated.
The most recent error message indicates that the wildcard handling is not finding any files matching the E:/test/*.txt pattern. What happen when you change back to Specific File and use four File properties to mention the four files explicitly?
We'll get back to the missing line terminator later; we need to solve one problem at a time.
Posted: Thu Sep 13, 2007 5:39 am
by hamzaqk
thanks ray i have done that and it works like a charm when i use specific files and add four files by using file property .
i don't know why when i change it to file pattern it's not working with the same files.
also i used the property first name column but its giving me the whole path of the file along with the file name and not just the file name in the target file..
Posted: Thu Sep 13, 2007 3:00 pm
by ray.wurlod
With File Pattern try using a job parameter for the directory part of the pathname, for example #jpFileDir#/*.txt
The behaviour of the File Name Column property (to return the complete pathname of the file) is as documented.
Posted: Thu Sep 13, 2007 10:07 pm
by hamzaqk
Thanks Ray! . it works fine when i specify the path as a parameter and not a hardcorded value. strange !
although when i try to view the data it still doesnt show it to me and still says that no files found on host and then about APT_FileImportOperator.
apart from this when i sepcify the file name property. it doesnt run the job properly. when i take it out it does. i think it sees that as a column in the file too and that why messes everything up when it comes to reading the data
Any ideas ?
Posted: Thu Sep 13, 2007 10:44 pm
by ray.wurlod
The File Name Column property must name a column that occurs on the stage's output link. This column should be a VarChar large enough to take a full pathname, for example VarChar(128). And, needless to say, the column must appear on the output link.