Can DataStage search for a Pattern of Files in Sub-Direcotor

WBlaustein · Post by **WBlaustein** » Thu Jun 14, 2012 5:16 pm

Hello Everyone,
I have a scenario, In the Input folder we will have dynamic sub folders with XML Files. My questions is How can we read those xml files in the dymaic sub folders.

Ex: input/061212/ ---this has 15 files
input/0613/12/ -- this has 20 files

I want to read all these 35 files from Datastage.
and the sub-folder names will change dynamically may be for every 1hour or so..

Thanks

ray.wurlod · Post by **ray.wurlod** » Thu Jun 14, 2012 5:46 pm

Do you want to read the files' content or names?

Do you have any preferred mechanism for identifying these "dynamic" directory names?

chulett · Post by **chulett** » Thu Jun 14, 2012 5:55 pm

Since we're talking XML consumption here, I'm assuming the goal is to collect the various filenames so the file path can be passed to an XML Input stage where it will do the actual reading of the contents.

Yes, no, maybe so? If that's the case then off the top of my head I'd say perhaps a "find" command in an External Source stage?

WBlaustein · Post by **WBlaustein** » Fri Jun 15, 2012 9:27 am

I dont have any preferred mechanism. My job is to read those xml files and load its content to a database and after successfull completion of Loading, i have move those files to arcive directory.

WBlaustein · Post by **WBlaustein** » Fri Jun 15, 2012 11:50 am

I used the Find command to print all the XML files in the sub directories and redirect that output to a textfile and from that text file I am able to read those XML files too. BUT Any one can tell me the Move command to move those files that are present in the text file??

text files content
/IIS/data/Source/wra.xml
/IIS/data/Source2/from.xml
.
.

I want to move these files in the text file to go to another directory

ray.wurlod · Post by **ray.wurlod** » Fri Jun 15, 2012 5:51 pm

There's no single command, but you can write a shell script or DataStage sequence to loop through the file a line at a time reading the line and effecting the mv command.

WBlaustein · Post by **WBlaustein** » Tue Jun 19, 2012 12:22 pm

I wrote a small Script to do that (Moving files to Archive).
I am wondering is there any way to move the files while loading into the table??

ray.wurlod · Post by **ray.wurlod** » Tue Jun 19, 2012 4:31 pm

You could, of course, have DataStage write to the new location and simply delete the original on successful completion.

WBlaustein · Post by **WBlaustein** » Tue Jun 19, 2012 5:16 pm

How to do that! While i was writing to database how can i move the files from source to archive directory!!

Example: If my source file has 10 absolute paths:
iis/data/f1/1.xml
iis/data/f2/2.xml

while loading the 1.xml into the database how can we move the same file at the same time to another location!!

ray.wurlod · Post by **ray.wurlod** » Tue Jun 19, 2012 8:33 pm

Another link in the job design, perhaps from a Copy stage, to write into the new location.

WBlaustein · Post by **WBlaustein** » Wed Jun 20, 2012 9:46 am

But How can we pass that path?
I tried to use External Target Stage. The Path location is stored in a column.