Passing parameters to a nested loop

[kalpna]

Hi, Everyone

Can anyone suggest a solution to the following scenario.

I have more than 100 csv files as my sources. all of them are of same structure.
I need to extract the data from all these files into a table with the file name as an added field.
Number of files in the folder are not constant. and the file names may change every time .

flat file---->transformer----->table.
Can I put this job into a nested loop and pass the file name as parameter into it. and the loop should run till all the files in the folder are extracted(I need to use a job after sub routine to move the files after extraction).
Is this works? If works, How can we pass the file name parameters into the job?.

Thanks in Advance,
Kalpana.

[ray.wurlod]

Yes, but why would you? If you can match the file names with a regular expression you can read all files with a single Sequential File stage, using File Pattern as the method and the regular expression as the pattern.

[sreeni]

Ray,

I have two files with same structure, File names are test1.txt, test2.txt
test1.txt file structure is
ID Integer
Desc Varchar(100)

test2.txt file structure is
ID Integer
Desc Varchar(100)

Can I read these two files data in one sequetial file stage?

Could you please give us more explanation with the example.

Thanks
Sreeni

[OddJob]

If both files have the same structure...

In the sequential file stage add two 'File' properties, one for each file. Or, you can change the 'Read Method' to 'File Pattern' and use a value of 'test*.txt'

If you need to know which file the data came from add option 'File Name Column'

[sreeni]

Hi

Thanks for valueble information. It is working fine for parallel jobs.
Is it possible in server jobs?

Thanks
Sreeni

[ray.wurlod]

Yes, but done differently. In a server job use a Filter command cat file1 file2 in the Sequential File stage - the stage reads stdout of the cat command.

[GSIDSXGrp]

If both files have the same structure...

In the sequential file stage add two 'File' properties, one for each file. Or, you can change the 'Read Method' to 'File Pattern' and use a value of 'test*.txt'

If you need to know which file the data came from add option 'File Name Column'

When I use the this method in version 8.0.1.x the results is the file pattern in the column instead of the actual file name.