Loading mutiple files using the same job

[tsktsk123]

Hi,

We have a requirement to do a loading from multiple source files i.e. the source system will send data based on extract date, if the extract date is less than the sysdate then we need to load the file as the reload process. Say if the sysdate is 05-Dec-2008 and we receive 3 different files from same source system ABC with three different extract date 02-Dec-2008,. 03-Dec-2008 and 04-Dec-2008 then we need to extract data from all three files and load it to target. We have a job curretly used to extract data from single file and load it target, I need your help to understand how the same can be extended if I use multiple files. The files coming from source will be dynamic i.e. sometimes it will send a single file for reload, some other time it may send 2 or 3. Currently we are using a UNIX script to see if the reporting date is less than sysdate then the file will be processed as reload file.

Appreciate any help on this.

[kumar_s]

Currently how are you identifying the files with previous date?
If its based on filename, you can still use the script to consolidate the file and pass it to Datastage. Are execute the same script in Datastage.
If the files are identified by Data, use datastage and discard the unecessary records.
You can use Filter option in Sequential file to read multiple files with wildcard.

[bkumar103]

You can use the file pattern to read from the multiple file provided the metadata is same for all the file.

[tsktsk123]

Hi,

Thanks for your reply, we are getting 3 different control files and 3 different data files for 3 different dates. We need to load extract date from the control file so we can't use file pattern as we have one control file for each data file.

You can use the file pattern to read from the multiple file provided the metadata is same for all the file.

[ray.wurlod]

How about a job sequence with a StartLoop activity specifying the "list of things" (file names), possibly initialized by an upstream command or routine into a user variable?

[kumar_s]

Atlest the does the control file has any patterns?
Read all the control files, check for the date.
I beleive the control file will have the date and the Data file name.
Based on the date, create a SourceProgram file for External Source stage. Which simply has cat <filename> and the sequence of it.

[Rubu]

Hi tsktsk123

I believe handling multiple files in same job will not be a good idea if you are maintaining process control metadata for processed file status. The solution will depend on the method of scheduling the jobs.

1. If you are scheduling through some scheduling tools(like control-M), Event based (file arrive) process triggering will solve the problem. Hopefully date is a part of the file names.

2. Otherwise going for loop activity as Ray specified will be good idea. You can write a routine to get file names as they arrive, return to a User Variable and use inside the loop.

[JoshGeorge]

Amend the Unix script to read the control files and call your existing job as multiple instances.

[tsktsk123]

Hello All,

thanks for all your help on this, i will try to implement the solution provided by you guys, will let you know if i need any help on that.

thanks