File Pattern problems !

[hamzaqk]

Hi,

just playing around with the functionality of 'file pattern'which you can set in sequential files to read multiple files in a folder.

i have four files in a folder a.txt , b.txt , c.txt and d.txt in folder E:\files

when i read them individually they work all ok
but when i change the property to file pattern and state E:\files\*.txt as the pattern and view the data . it says no files found at host XYZ E:\files\*.txt or

when i run the job it says :

Missing record delimiter "\n", saw EOF instead

i only have one value in all the files and that is 1 i.e.
a.txt has only one col column1 with value 1 in the file and same is the case for other files


Any ideas ?

[Maveric]

One of the files might be missing a delimiter. make four exact copies of a.txt and try.

[hamzaqk]

this is exactly what i did before and how i created the test files. i have read them individually with the same options for all and they work fine. it just gives me a problem when i try to read them all at once. i have done all the basic checks, but it does'nt work. and gives me the same errors. it however does process only the first file.

[ArndW]

Add a final \n to your file a.txt and see if the error wanders to file 2...

[hamzaqk]

thanks. stil does'nt work. when i click on view data on seq file stage it gives me this:

Code: Select all

 ##I TOSH 000002 09:31:35(001) <main_program> orchgeneral: loaded
 ##I TOSH 000002 09:31:35(002) <main_program> orchsort: loaded
 ##I TOSH 000002 09:31:35(003) <main_program> orchstats: loaded
 ##W TCOS 000049 09:31:35(004) <main_program> Parameter specified but not used in flow: DSProjectMapName
 ##W TOIX 000000 09:31:35(005) <main_program> createFilesetFromPattern(): Couldn't find any files on host  with pattern E:/test/*.txt.
>##E TOIX 000138 09:31:36(001) <Sequential_File_0> At least one filename or data source must be set in APT_FileImportOperator before use.
>##E TFSR 000019 09:31:36(005) <main_program> Could not check all operators because of previous error(s)
>##E TCOS 000029 09:31:36(006) <main_program> Creation of step finished with status = FAILED

[Minhajuddin]

Try changing the backslashes in your path to forward slashes. It's a bug in the parallel edition.

E:\test/*.txt

[hamzaqk]

ok i have tried all three of these and it still says the same !!

E:/test/*.txt

E:/test/*.txt

and E:\test/*.txt

what is an APT_FileImportOperator ?????

[ray.wurlod]

APT_FileImportOperator is the class from which Import objects (such as the operator generated by a Sequential File stage with an output link generates) are instantiated.

The most recent error message indicates that the wildcard handling is not finding any files matching the E:/test/*.txt pattern. What happen when you change back to Specific File and use four File properties to mention the four files explicitly?

We'll get back to the missing line terminator later; we need to solve one problem at a time.

[hamzaqk]

thanks ray i have done that and it works like a charm when i use specific files and add four files by using file property .

i don't know why when i change it to file pattern it's not working with the same files.

also i used the property first name column but its giving me the whole path of the file along with the file name and not just the file name in the target file..

[ray.wurlod]

With File Pattern try using a job parameter for the directory part of the pathname, for example #jpFileDir#/*.txt

The behaviour of the File Name Column property (to return the complete pathname of the file) is as documented.

[hamzaqk]

Thanks Ray! . it works fine when i specify the path as a parameter and not a hardcorded value. strange !

although when i try to view the data it still doesnt show it to me and still says that no files found on host and then about APT_FileImportOperator.

apart from this when i sepcify the file name property. it doesnt run the job properly. when i take it out it does. i think it sees that as a column in the file too and that why messes everything up when it comes to reading the data

Any ideas ?

[ray.wurlod]

The File Name Column property must name a column that occurs on the stage's output link. This column should be a VarChar large enough to take a full pathname, for example VarChar(128). And, needless to say, the column must appear on the output link.