Loading two or more files in a single job run

DSRajesh · Post by **DSRajesh** » Wed Aug 08, 2012 3:04 am

Hi All,

I need to pick two 0r more than two files for which file name starts with similar word say file 1 name : HFDC_0101.csv,HDFC_08082012.csv.

Can we able to process these two files in a single job run and load into the target table?

ArndW · Post by **ArndW** » Wed Aug 08, 2012 3:30 am

Yes, take a look at the server "folder file" stage, that will let you use wildcards and other rules to select files to process from a directory.

ssreeni3 · Post by **ssreeni3** » Wed Aug 08, 2012 3:33 am

Hi,
1)ensure metadata is same?If Yes,use sequential file stage:File pattern option to import multiple files.specify the value is HFDC_*.csv
It will work.

-----Sreeni

ArndW · Post by **ArndW** » Wed Aug 08, 2012 3:47 am

ssreeni3 - This is posted in the server forum, and the server sequential file stage does not offer this option.

chulett · Post by **chulett** » Wed Aug 08, 2012 7:01 am

There are 'issues' with the Folder stage as it was built for XML processing and passes in the file contents in one lump. You could leverage the Filter option in the Sequential File stage to 'type' those files to standard out using a wildcard pattern.

DSRajesh · Post by **DSRajesh** » Wed Aug 08, 2012 8:17 am

Hi Chullet,

Do you mean sequential file stage will handle more than 1 file in a single job run?

sathya_anjan · Post by **sathya_anjan** » Wed Aug 08, 2012 9:27 am

The Answer to your above question is NO. Sequential file Stage cannot handle multiple input streams

However, you can combine the multiple streams of data (multiple sequential files) into a single stream and feed that into the Sequential File Stage, provided they have consistent Metadata. In this way you can handle multiple Sequential files

To acheive the above please follow the posts from ssreeni3 and chulett and use the Filter command option. This will create a single data stream from multiple inputs and hence can be parsed through the Sequential file stage

Thanks

ray.wurlod · Post by **ray.wurlod** » Wed Aug 08, 2012 3:12 pm

The Sequential File stage can readily handle multiple input files, provided that they have the same record structure.

The Sequential File stage can use a Filter command. In this case the filter command should be TYPE file1 file2 (or COPY file1 file2). (If you were on UNIX, the command would be cat file1 file2.) The stage then reads stdout from the command rather than trying to read the file named in the File property (which is still required, even though ignored).