One parametric job to load 100 tables from 100 files

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
federico.l
Participant
Posts: 7
Joined: Wed Nov 10, 2004 10:42 am

One parametric job to load 100 tables from 100 files

Post by federico.l »

Hi,

I wish to create a Job that load a table from a file in a parametric way.

In other words, given the name of the file, the job should load the corresponding table in a straight way. (field names are identical between file and table and table has the same name of the file or anyway obtainable from the file).

Any idea?

Thanks.
jerome_rajan
Premium Member
Premium Member
Posts: 376
Joined: Sat Jan 07, 2012 12:25 pm
Location: Piscataway

Post by jerome_rajan »

1. Create a single RCP job with your source, target and any intermediate generic transformations.
2. Create a looped master sequencer with a delimited string of your file name/schema file name/ table name triplets in the start loop stage.
3. Use a user variable activity to extract each value into the 3 variables (i.e. File Name, Schema File Name, Table Name)
4. Pass these 3 variables as parameter values to your generic job in the sequencer.
5. The sequencer will run in a loop and load as many tables as you may have specified in the delimited string of triplets.

Hope this helps
Jerome
Data Integration Consultant at AWS
Connect With Me On LinkedIn

Life is really simple, but we insist on making it complicated.
federico.l
Participant
Posts: 7
Joined: Wed Nov 10, 2004 10:42 am

Post by federico.l »

Thanks Jerome,

but is it not necessary to define metadata of destination table within the job?
jerome_rajan
Premium Member
Premium Member
Posts: 376
Joined: Sat Jan 07, 2012 12:25 pm
Location: Piscataway

Post by jerome_rajan »

No. The metadata for your flat files will be derived from the schema files that you will have to create and place in a location on your server.
The metadata for the target tables will be directly available to Information Server provided you supply the right credentials. This is how RCP works.
Jerome
Data Integration Consultant at AWS
Connect With Me On LinkedIn

Life is really simple, but we insist on making it complicated.
WesL
Premium Member
Premium Member
Posts: 26
Joined: Wed Jun 28, 2006 1:05 pm
Location: Huntsville, AL

Post by WesL »

Here is a good tutorial on generic data processing.
federico.l
Participant
Posts: 7
Joined: Wed Nov 10, 2004 10:42 am

Post by federico.l »

Many thanks for the very useful infos.
Post Reply