Folder Stage

asvictor · Post by **asvictor** » Thu Oct 09, 2003 9:23 pm

Hi,

I tried using Folder stage in my job to use a set of Files in a folder and post in a seq File / Table. it is a CSV file. When I run the Job, I either get only the Column Names or Column Names and fields in the Resulting File. How do I use the Folder stage to Extract the required Fields

Victor Auxilium

raviyn · Post by **raviyn** » Fri Oct 10, 2003 9:22 pm

Does this set of files have same Metadata.Then you can go for a two step job like in the first job using folder stage u get the filenames , and then use this as an parameter to call the Second job where you would be Accessing the file and extracting releveant columns and putting into the table

ray.wurlod · Post by **ray.wurlod** » Sat Oct 11, 2003 1:55 am

Take a look at the properties of the Folder stage in Manager, or from the Repository view in Designer. There are two "columns". One is the "key" which, from the O/S point of view, is the file name. The other is the "record" which, from the O/S point of view, is the entire file. The second of these is of limited use, though can be parsed within a DataStage job. The Folder stage is really intended for the "see what file names are there" kind of operation.

Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518

mhester · Post by **mhester** » Sat Oct 11, 2003 6:12 am

Or moving many files of a certain type (or all) from one directory to another, which is how most customers use this stage.

Regards,

Michael Hester

thirupri · Post by **thirupri** » Mon Oct 13, 2003 3:27 am

Hi,

This is what I did as same as your requirement. First, I created a job using the Folder Stage with output column (1) Docname (filenames) to a sequential file then I transferred this to a hash file in the same Job. Then, I created another Job (Job3), which will accept the File name & File path as a Input parameter. The In a Script I read this Hash file and pass the file name to Job3.

Thiruma Valavan.

roy · Post by **roy** » Thu Oct 16, 2003 7:27 am

Hi,
I want to point out that the folder stage has a file size limit (probably configurable).
It will cause errors, if I'm not mistaken, in cases any file exeeds the limit.
Hence I would prefer to use any DSExecute() flavour via DSRoutine toget the list of file you need, especially if they might be really big ones.
( up to 100 MB is usually fine maybe even more but around 150-200+ MB your in trouble )

IHTH (I Hope This Helps)

ray.wurlod · Post by **ray.wurlod** » Fri Oct 17, 2003 7:23 pm

DataStage (and therefore DataStage BASIC) has the quirky ability to treat a directory (folder) as a "table". This is the underlying premise of the Folder stage. You can do the same thing in DataStage BASIC. If the O/S directory is seen by DS as a table, then each file name in that directory is seen by DS as a record key, and each file in that directory is seen by DS as a "record" - even though it's unlikely that every record has the same structure.
Use OpenPath to open the directory, Select to generate a list of file names, and ReadNext to process that list. For example:

Code: Select all

$INCLUDE UNIVERSE.INCLUDE FILENAMES.H
OpenPath "/usr3/datastage/hashedfiles" To filevariable
Then
   Clearselect 10
   Select filevariable To 10
   Loop
   While ReadNext FileName From 10
      FilePath = "/usr/datastage/hashedfiles/" : FileName
      Cmd = UV.BIN : UV.FSEP : "clearfile " : FilePath
      Call DSExecute("UV", Cmd, Output, ExitStatus)
      Call DSLogInfo(Output, "MyRoutine")
   Repeat
End
Else
   Msg = "Can not open /usr3/datastage/hashedfiles"
   Call DSLogWarn(Msg, "MyRoutine")
End