Hi
I am trying to understand how to use the folder stage. I have developed a Job having in input a folder stage, after a trasformer one and finally a sequential stage. It works but I am not able to understand how I can split the rows returned from the folder stage.
I'll explain it better. In the folder that I use as input there are four delimited files. So when Jobs start the execution it read the contents of this folder, that it means it read the file's contents, and return the data as four blocks. How can I split this block in rows? Is there a way to read the data returned as single rows instead of a block or I have to split it after they returned from the folder stage. In this case how can I do that?
Thank you in advice.
Umberto
Ho to split the data blocks returned from a folder stage
Moderators: chulett, rschirm, roy
I've never tried to use the Folder stage with sequential files other than to get a list of file names, not contents. 
The Folder stage is meant to process XML files and so is typically followed by an XML Input stage which does the parsing into rows and columns.
The Folder stage is meant to process XML files and so is typically followed by an XML Input stage which does the parsing into rows and columns.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
I know that it is possible, and I have developed a Job to test it. It really works but I have the problem exposed in my post.chulett wrote:I've never tried to use the Folder stage with sequential files other than to get a list of file names, not contents.
The Folder stage is meant to process XML files and so is typically followed by an XML Input stage which does the parsing into rows and columns.
Ok... I know it is possible. My point is that (IMHO) it is the wrong approach - the Folder stage should be used with XML files.
Most people would either concatenate the four files together before-job and process them at once, or create an interative process where you loop four time and process each file one at a time. Why not take one of those roads?
Otherwise, you're going to have to go to alot of trouble to chunk up the output... first you'll need to cut it up into records and then each record will need to be parsed into fields. Seems to me what little you've saved using the Folder stage is more than used up doing all that post-processing.
Most people would either concatenate the four files together before-job and process them at once, or create an interative process where you loop four time and process each file one at a time. Why not take one of those roads?
Otherwise, you're going to have to go to alot of trouble to chunk up the output... first you'll need to cut it up into records and then each record will need to be parsed into fields. Seems to me what little you've saved using the Folder stage is more than used up doing all that post-processing.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
I understand what you mean. Thank you for the help.chulett wrote:Ok... I know it is possible. My point is that (IMHO) it is the wrong approach - the Folder stage should be used with XML files.
Most people would either concatenate the four files together before-job and process them at once, or create an interative process where you loop four time and process each file one at a time. Why not take one of those roads?
Otherwise, you're going to have to go to alot of trouble to chunk up the output... first you'll need to cut it up into records and then each record will need to be parsed into fields. Seems to me what little you've saved using the Folder stage is more than used up doing all that post-processing.
Have a nice day.
Umberto