A Generic/Base job to handle multiple file types

A forum for discussing DataStage<sup>®</sup> basics. If you're not sure where your question goes, start here.

Moderators: chulett, rschirm, roy

Post Reply
VCInDSX
Premium Member
Premium Member
Posts: 223
Joined: Fri Apr 13, 2007 10:02 am
Location: US

A Generic/Base job to handle multiple file types

Post by VCInDSX »

Hi,
Here is my use case.
We have several files from various providers on a daily basis. These files arrive at different times of a day.
Each of these files has its own pre-defined format, columns that have been documented.
Each of these files are parsed and loaded into separate Database tables dedicated for each file.
NOTE: Number of columns and field types do not change and are fixed.
File1 (20 Columns) ==> Parse ===> DB_Table1_File1
File2 (26 Columns) ==> Parse ===> DB_Table2_File2
File3 (22 Columns) ==> Parse ===> DB_Table3_File3
....
....
....
File25 (40 Columns) ==> Parse ===> DB_Table25_File25

I can create 25 different jobs that cater to the appropriate file and database table. If I have to add another provider in the future, I would make a copy from a template job and setup the metadata and schedule it for its appropriate time.
OR....
Can I create a common job that can take the input file name, its metadata, target database & table details - all via configuration files and not worry about maintaining N number of jobs?
My searches on Generic job returned some serious discussions on Shared Containers, RCP - Pros and Cons....
Is RCP the solution or is it going to be an overkill in the long run...?

I am sure several of you might have chartered through these waters before and would appreciate your invaluable suggestions, ideas, input.

If any additional information would help you help me, I will be more than willing to share here.

Thanks again for your time,
-V
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Use 25 separate jobs appropriate to the 25 different sets of metadata (record schemas). It's more maintainable in the long run.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Post Reply