File Name property in Sequential File Stage

[Django]

I have to read many .txt files from a directory and output to one single file. The output will be as follows ; First column is the name of the appropriate file and the remaining is the contents from each file.

In the sequential stage I have set

propertie -> Options -> File Name Column = 'FileNameCol1'
properties -> Source -> File = '/opt/datastage/*.txt'
properties -> Source -> Read Method = File Pattern

When I run the Job the FileNameCol1 column contains ''/opt/datastage/*.txt'' instead of the individual file names as follows

''/opt/datastage/one.txt'' xxx yyy
''/opt/datastage/one.txt'' zzz hhh
''/opt/datastage/two.txt'' aaa bbb

Any light ?

[naveen19]

Hi,

enable this option in seqeuntial file stage

First line is column is 'True'

Default is 'False'


Regards Naveen.K

[ray.wurlod]

Would you care to explain how this affects the File Name property?

[Django]

So whats the solution ?

If not through sequential stage properties is there any other way of reading the name of the files (of pattern *.txt) being used in the source stage.

[chulett]

There are other posts here of people with the same question that a search for "file pattern" would turn up. Best as I can tell, you need to set the APT_IMPORT_PATTERN_USES_FILESET environment variable to TRUE to get this to work as one would expect.

[Django]

Craig

Setting the APT_IMPORT_PATTERN_USES_FILESET to True resolved the issue.

Thx

[keshav0307]

set parameters
$APT_IMPORT_PATTERN_USES_FILESET = True
$APT_IMPORT_PATTERN_USES_FILESET_MOUNTED = True

[caterin_huang]

hi,
I'm facing the same problem here.. and I have changed the parameters into above. when I run the job, it has solved the problem. but suddenly, when I want to try to run the job again, it produces an error when I try to view data:

Code: Select all

##E IIS-DSEE-TFOP-00020 13:59:23(007) <rlgmfagun> Error occurred during initializeFromArgs().
##E IIS-DSEE-TOIX-00118 13:59:23(008) <rlgmfagun> In file set "/tmp/import_tmp_954582894fa4bc.fs": Parsing a dataset, expecting a fileset..
##E IIS-DSEE-TCOS-00029 13:59:23(009) <main_program> Creation of a step finished with status = FAILED.

what happen? and what should I do?

Thanks in advance

[chulett]

I've never seen the "MOUNTED" one mentioned before and have no idea what it means or does. What happens if you remove it from the job?

[caterin_huang]

if I removed the "MOUNTED", it also produces the same error..
thanks

[chulett]

OK... your filename ends in ".fs" which is for a fileset but it found a dataset where it pointed to instead. Any idea how that happened? Naming collision?

You really should start your own thread for something like this.

[caterin_huang]

no, my filename is ended with .txt and there is only .txt file at the source folder

fyi, my file location is /transport/*tblcustomer.txt
thanks,
Caterin

[chulett]

Not according to the error you posted. Perhaps your issue is now coming from elsewhere in the job, target rather than source side?

[caterin_huang]

nope, my job only has 3 stages.
1. sequential file as source file
which read file with pattern *customer.txt

2. transformer to map the columns
3. ODBC stage as the database target

if I set the $APT_IMPORT_PATTERN_USES_FILESET = False, it works and I can view data n run the job.. but the problem is, the file name column contains * and I still need the real file name for some reason.

thanks,
Caterin

[caterin_huang]

thanks God, everything has been solved..

if anyone find this post and facing the same problem, the solution is please check the directory and file name whether it is at the correct location. In my case, I have an additional / at my file path, and it makes the DS can't find the right file.

Why the error sounds like that? perhaps when DS import pattern use fileset,it will create a temporary fileset. and if DS can't find the correct filepath, it also can't parse the temporary fileset.

Thanks,
Caterin