Framework for creating a generic job

dspxlearn · Post by **dspxlearn** » Mon Nov 03, 2014 12:06 pm

Hello all,

Prior to posting a question here, I searched the entire repository to find a solution to my question but no luck!! (may be my poor searching skills

)

Question:
We are trying to build a framework where a single job has to be build which should pick up the fields from the table dynamically. This job will pull the data based on SQL queries and load into target table with some intermediate joins and transformations also.
I believe, for simple data extraction and load, we can use RCP functionality.
But, in this case where transformations and joins will differ table to table, can we go still go with RCP?
Also, we need to use the Optim's Datamasking pack within the same job and the fields which are to be masked differs table to table.

Can you please suggest a solution for this? Appreciate you time for reading this.

Thanks!!

chulett · Post by **chulett** » Mon Nov 03, 2014 1:16 pm

Short answer? Not possible. Too many specifics for anything a single 'generic' job could handle. And RCP is out the window.

How many tables are you talking about?

dspxlearn · Post by **dspxlearn** » Mon Nov 03, 2014 7:15 pm

There could be more than 10 databases/applications and each would consist around 20 to 30 tables.

Can the .dsx file do any wonders here? I mean using Unix scripts to pass the table names/keys etc.. to the .dsx of a precreated datastage job and then import it back. This is just a thought and not sure if it works..

chulett · Post by **chulett** » Mon Nov 03, 2014 7:39 pm

Sure... I've helped build systems where 100s of jobs were generated from a template using source/target metadata to drive it. It's not something to take lightly and we had plenty of time to work all of the kinks out.

Unless you're comfortable with generating dsx code on the fly, you'd more than likely be better off just grinding out the jobs instead, an effort that would probably be complete before you perfected the job generation methodology.

Curious to hear what others think...

FranklinE · Post by **FranklinE** » Tue Nov 04, 2014 9:22 am

I have a severe bias against RCP. The conveniences are, for me, far outweighed by the difficulties in diagnosing problems. But that also prompts me to agree with Craig. The requirements as described are just not simple enough to fit into a generic job.

I am all for pattern recognition and coding efficiently from it. I use it myself, though not in ways that need RCP at any point.