read folder contents

sweetleaf · Post by **sweetleaf** » Thu Jan 30, 2003 8:42 am

Hi there,

Does anyone know of a DataStage function that will read the file names inside of a predefined directory, like a "dir" or "ls" command, then pass each file name into a variable in a given job.

Thanks much!

ray.wurlod · Post by **ray.wurlod** » Thu Jan 30, 2003 4:17 pm

There isn't one.
What exactly are you trying to accomplish? DataStage generally only works with one file at a time. Do you want to run a job N times, passing each time the name of one of N files?
This is done with a sequence or Job Control code.
I won't elaborate until I get a better feel for what you're trying to do.

Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518

ray.wurlod · Post by **ray.wurlod** » Fri Jan 31, 2003 5:15 pm

It has been brought to my attention that Ascential consultants have a function called GetNextFile in their toolkit. You might like to request a copy from them. It was written explicitly for HP-UX but ought to be easily adapted. Here's the description.

Scan a directory and return the name of a file. The file name will match the passed in pattern and only files that are not open by other processes will be returned. This makes the routine ideal for use with FTP as a file name will only be returned once the FTP file has arrived.
Returns - the file name or the character "/" to indicate that nothing was available.
Arg 1 - Directory to scan for new files
Arg 2 - Once a file is found it is moved to this directory for processing
Arg 3 - The file name to look file e.g. "SA*" will only find files starting SA.

Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518

endyl13 · Post by **endyl13** » Mon Feb 03, 2003 2:20 am

Hi there,

You can use Folder stage to get all the filename from your directory, and stored in a hash file (Job A).
And then, use another job (Job B) to read the content of the files and process it, use job parameters for all the filename variables. Then you need another job control (Job C) to do a looping based on the number of files in the directory, and pass the paramereter to the processing job (Job B).

I used this approach for my problem, and it works nicely...

HtH

Regards

ND

ray.wurlod · Post by **ray.wurlod** » Mon Feb 03, 2003 4:49 am

Yes, but the poster asked for a function! [:)]

sweetleaf · Post by **sweetleaf** » Wed Feb 05, 2003 8:07 am

Hi guys,

Thanks so much for all of your valuable input.
The route i've opted to take is to use the folder stage to read file names and their contents (as one large string) into the oracle backend.
Then write a seperate chunk of plsql to parse through the comma delimitted string and process each value accordingly into another oracle table.

Thanks gang! Ray I will look into the GetNextFile for the next piece that I'll be doing. cheers