Folder Stage

edward_m · Post by **edward_m** » Mon Aug 21, 2006 8:31 am

ALL,

I want to read a file from directory and process that file and load into table.
The above directory contains only one file and everytime that file comes into that directory with different name..for this i need to read that file from the directory and load into table.
Could anybody explain me in detail how to read a file from folder stage and load that file data into table.

Thanks...

thumsup9 · Post by **thumsup9** » Mon Aug 21, 2006 9:30 am

Provide FolderPath in Stage Properties and in the Output Properties provide the Filename WildCard as say *.xml depending on your filename extension.

HTH

edward_m · Post by **edward_m** » Mon Aug 21, 2006 9:49 am

Thumsup9,

thumsup9 wrote:Provide FolderPath in Stage Properties and in the Output Properties provide the Filename WildCard as say *.xml depending on your filename extension.

HTH

Could you explain in detail..
How to define columns in outputs for folder stage.

chulett · Post by **chulett** » Mon Aug 21, 2006 9:59 am

Read the Help. There are only two columns allowed in the Folder stage: 1) The filename found in the folder and 2) the contents of the file.

If all you want is the filename, don't define the second column.

thumsup9 · Post by **thumsup9** » Mon Aug 21, 2006 10:04 am

How to define columns in outputs for folder stage.[/quote]

You will have to specify two columns or may be more based on your req

Say two:

1. Filename, Varchar 255 (this column must be defined as the key and receives the name of each file being read from the directory.)

2. Record, LongVarchar 999999999 (This column receives the actual contents of each file.)

This usually goes to XML inout stage. where you will mention under XML stage input page

XML Input Source as Record
Column Content as XML Document, if your source is XML

Does this help?

This column receives the actual contents of each file.

meena · Post by **meena** » Mon Aug 21, 2006 10:04 am

Hi,
I think the output coulmn is going to be your file name for example: File1(Column name) with datatype char or varchar of 250. And you can get columns data from the file1 column using "Field" function(ex:Field(DSlink.File1,"/",9,1)) in the transformer stage and load the data into table..Check it.

thumsup9 · Post by **thumsup9** » Mon Aug 21, 2006 10:08 am

Yes Chulet , it can allow only two columns...

chulett · Post by **chulett** » Mon Aug 21, 2006 10:15 am

I know, you know... the comment was for the OP. And it's very clearly laid out in the Help available, hence the suggestion they actually read it.

attili · Post by **attili** » Mon Aug 21, 2006 11:08 am

Hey We have the same issue over here, what we have done is we wrote a script which reads the incomming file and renames it to a standard format everytime you kickoff the sequencer.

#!/usr/bin/sh
DIR=<Directory Path>
cd $DIR
for I in `*.xml`
do
echo "######### $I Found ########" ><Log File Name>
cp $I <standard file name>.xml
mv $I <Archive folder path>
exit 0
done

Try implementing this....if it suit you

thanks and regards
satish

edward_m · Post by **edward_m** » Mon Aug 21, 2006 12:55 pm

You will have to specify two columns or may be more based on your req

Say two:

1. Filename, Varchar 255 (this column must be defined as the key and receives the name of each file being read from the directory.)

I defined this column Filename Varchar 255 and able to get filename from that directory.

2. Record, LongVarchar 999999999 (This column receives the actual contents of each file.)

I defined this column to accomadate file data..my question is how do we process this and load into table.

This usually goes to XML inout stage. where you will mention under XML stage input page
XML Input Source as Record
Column Content as XML Document, if your source is XML

My source is .csv file, do we have to have to use XML for this load.

Does this help?

Not completely.

FOlderStage----->Transformer-------->Table

Sorry to bother you..Could you explain in detail.

chulett · Post by **chulett** » Mon Aug 21, 2006 1:08 pm

The Folder stage is primarily for supporting the XML Input stage, hence the trick of feeding in the entire contents of the file in one field for subsequent parsing. It's not really appropriate for handling sequential files.

You are better off doing this via a Sequence job, one that leverages the new Start Loop, End Loop and User Variables stage. One of the pdf manuals goes into a decent amount of detail on setting something like that up so would be helpful if you found that and read through it.

Basically, the User Variables stage is used to gather up wildcarded filenames from your directory and build a list of them. This is then passed into a loop, created by the Start/End stages and your processing jod, which automatically pulls one filename off the list, loads it and then keeps going back for more until the list is exhausted. Your load job would just use the normal Sequential File stage and the filename would be a job parameter populated by the looping construct.

That's how I would setup something like this.