Folder Stage
Moderators: chulett, rschirm, roy
Folder Stage
ALL,
I want to read a file from directory and process that file and load into table.
The above directory contains only one file and everytime that file comes into that directory with different name..for this i need to read that file from the directory and load into table.
Could anybody explain me in detail how to read a file from folder stage and load that file data into table.
Thanks...
I want to read a file from directory and process that file and load into table.
The above directory contains only one file and everytime that file comes into that directory with different name..for this i need to read that file from the directory and load into table.
Could anybody explain me in detail how to read a file from folder stage and load that file data into table.
Thanks...
How to define columns in outputs for folder stage.[/quote]
You will have to specify two columns or may be more based on your req
Say two:
1. Filename, Varchar 255 (this column must be defined as the key and receives the name of each file being read from the directory.)
2. Record, LongVarchar 999999999 (This column receives the actual contents of each file.)
This usually goes to XML inout stage. where you will mention under XML stage input page
XML Input Source as Record
Column Content as XML Document, if your source is XML
Does this help?
This column receives the actual contents of each file.
You will have to specify two columns or may be more based on your req
Say two:
1. Filename, Varchar 255 (this column must be defined as the key and receives the name of each file being read from the directory.)
2. Record, LongVarchar 999999999 (This column receives the actual contents of each file.)
This usually goes to XML inout stage. where you will mention under XML stage input page
XML Input Source as Record
Column Content as XML Document, if your source is XML
Does this help?
This column receives the actual contents of each file.
Hi,
I think the output coulmn is going to be your file name for example: File1(Column name) with datatype char or varchar of 250. And you can get columns data from the file1 column using "Field" function(ex:Field(DSlink.File1,"/",9,1)) in the transformer stage and load the data into table..Check it.
I think the output coulmn is going to be your file name for example: File1(Column name) with datatype char or varchar of 250. And you can get columns data from the file1 column using "Field" function(ex:Field(DSlink.File1,"/",9,1)) in the transformer stage and load the data into table..Check it.
Hey We have the same issue over here, what we have done is we wrote a script which reads the incomming file and renames it to a standard format everytime you kickoff the sequencer.
#!/usr/bin/sh
DIR=<Directory Path>
cd $DIR
for I in `*.xml`
do
echo "######### $I Found ########" ><Log File Name>
cp $I <standard file name>.xml
mv $I <Archive folder path>
exit 0
done
Try implementing this....if it suit you
thanks and regards
satish
#!/usr/bin/sh
DIR=<Directory Path>
cd $DIR
for I in `*.xml`
do
echo "######### $I Found ########" ><Log File Name>
cp $I <standard file name>.xml
mv $I <Archive folder path>
exit 0
done
Try implementing this....if it suit you
thanks and regards
satish
...Skill and confidence are an unconquered army
You will have to specify two columns or may be more based on your req
Say two:
FOlderStage----->Transformer-------->Table
Sorry to bother you..Could you explain in detail.
Say two:
I defined this column Filename Varchar 255 and able to get filename from that directory.1. Filename, Varchar 255 (this column must be defined as the key and receives the name of each file being read from the directory.)
I defined this column to accomadate file data..my question is how do we process this and load into table.2. Record, LongVarchar 999999999 (This column receives the actual contents of each file.)
My source is .csv file, do we have to have to use XML for this load.This usually goes to XML inout stage. where you will mention under XML stage input page
XML Input Source as Record
Column Content as XML Document, if your source is XML
Not completely.Does this help?
FOlderStage----->Transformer-------->Table
Sorry to bother you..Could you explain in detail.
The Folder stage is primarily for supporting the XML Input stage, hence the trick of feeding in the entire contents of the file in one field for subsequent parsing. It's not really appropriate for handling sequential files.
You are better off doing this via a Sequence job, one that leverages the new Start Loop, End Loop and User Variables stage. One of the pdf manuals goes into a decent amount of detail on setting something like that up so would be helpful if you found that and read through it.
Basically, the User Variables stage is used to gather up wildcarded filenames from your directory and build a list of them. This is then passed into a loop, created by the Start/End stages and your processing jod, which automatically pulls one filename off the list, loads it and then keeps going back for more until the list is exhausted. Your load job would just use the normal Sequential File stage and the filename would be a job parameter populated by the looping construct.
That's how I would setup something like this.
You are better off doing this via a Sequence job, one that leverages the new Start Loop, End Loop and User Variables stage. One of the pdf manuals goes into a decent amount of detail on setting something like that up so would be helpful if you found that and read through it.
Basically, the User Variables stage is used to gather up wildcarded filenames from your directory and build a list of them. This is then passed into a loop, created by the Start/End stages and your processing jod, which automatically pulls one filename off the list, loads it and then keeps going back for more until the list is exhausted. Your load job would just use the normal Sequential File stage and the filename would be a job parameter populated by the looping construct.
That's how I would setup something like this.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers