A Generic/Base job to handle multiple file types
Posted: Fri Jun 15, 2007 9:55 am
Hi,
Here is my use case.
We have several files from various providers on a daily basis. These files arrive at different times of a day.
Each of these files has its own pre-defined format, columns that have been documented.
Each of these files are parsed and loaded into separate Database tables dedicated for each file.
NOTE: Number of columns and field types do not change and are fixed.
File1 (20 Columns) ==> Parse ===> DB_Table1_File1
File2 (26 Columns) ==> Parse ===> DB_Table2_File2
File3 (22 Columns) ==> Parse ===> DB_Table3_File3
....
....
....
File25 (40 Columns) ==> Parse ===> DB_Table25_File25
I can create 25 different jobs that cater to the appropriate file and database table. If I have to add another provider in the future, I would make a copy from a template job and setup the metadata and schedule it for its appropriate time.
OR....
Can I create a common job that can take the input file name, its metadata, target database & table details - all via configuration files and not worry about maintaining N number of jobs?
My searches on Generic job returned some serious discussions on Shared Containers, RCP - Pros and Cons....
Is RCP the solution or is it going to be an overkill in the long run...?
I am sure several of you might have chartered through these waters before and would appreciate your invaluable suggestions, ideas, input.
If any additional information would help you help me, I will be more than willing to share here.
Thanks again for your time,
Here is my use case.
We have several files from various providers on a daily basis. These files arrive at different times of a day.
Each of these files has its own pre-defined format, columns that have been documented.
Each of these files are parsed and loaded into separate Database tables dedicated for each file.
NOTE: Number of columns and field types do not change and are fixed.
File1 (20 Columns) ==> Parse ===> DB_Table1_File1
File2 (26 Columns) ==> Parse ===> DB_Table2_File2
File3 (22 Columns) ==> Parse ===> DB_Table3_File3
....
....
....
File25 (40 Columns) ==> Parse ===> DB_Table25_File25
I can create 25 different jobs that cater to the appropriate file and database table. If I have to add another provider in the future, I would make a copy from a template job and setup the metadata and schedule it for its appropriate time.
OR....
Can I create a common job that can take the input file name, its metadata, target database & table details - all via configuration files and not worry about maintaining N number of jobs?
My searches on Generic job returned some serious discussions on Shared Containers, RCP - Pros and Cons....
Is RCP the solution or is it going to be an overkill in the long run...?
I am sure several of you might have chartered through these waters before and would appreciate your invaluable suggestions, ideas, input.
If any additional information would help you help me, I will be more than willing to share here.
Thanks again for your time,