how many tables can be loaded using single job

eostic · Post by **eostic** » Mon Jun 17, 2013 5:44 am

This is a great thread because it illustrates the flexibility in DataStage to create a solution in lots of ways, and also shows that there are important functional AND non-functional decisions that have to be made.

Obviously, there aren't any realistic limits here...you can write to a whole lot of tables, and as Arnd noted, 10 is not very many.

...but 10 jobs vs one Job? Clearly there are maintenance, support, debugging and management issues to consider, and the threads above highlight the fact that "one huge job" increases complexity. It might cost you far more in time and energy and future maintenance (you may not be the one who has to update or repair the job in the future) if you go with one Job. One large Job also takes away simple flexibility the first time someone says "I need you to re-run the load for ONLY table 7".

You didn't say how long it takes to run. There are great performance points up above regarding the source, re-reading it, etc. Depending on the source, performance could dictate the choice for you. If it were xml, for example, there are major benefits to reading an xml document only once and then parsing it many ways into different output links. And regardless of the source type, if the Job takes many hours to run, performance is a premium. ....but if the Job runs in 5 minutes or collectively as individual Jobs in 10 minutes, and you have a wide open batch window, opt for better long term maintenance and simpler debugging.

Ernie

sathyak098 · Post by **sathyak098** » Mon Jun 17, 2013 5:54 am

Hi,

How will it work?. Because target Datasets have diff metatada

SURA wrote:If i would be in your position, i will create a master job which will do all the transformations for all the target tables and my final targetwill be dataset files. Then have on multiple instance job with RCP to load the data.

SURA · Post by **SURA** » Mon Jun 17, 2013 4:49 pm

sathyak098 · Post by **sathyak098** » Wed Jun 19, 2013 4:05 am

Hi,
If possible, Could you be more specific?

SURA · Post by **SURA** » Wed Jun 19, 2013 5:15 pm

Datastage provides the flexibility to use the same job to load different metadata. Runtime column propagation allows you to do that. You can find this in the Developers guide provided by IBM.

Please understand how this will work ,ensure that how much this approach fits for your situation. If you are happy and comfort that's it! Give it a try.

Seniors & gurus also given their comments for you. Please do read the other answers carefully and choose your option.

ray.wurlod · Post by **ray.wurlod** » Wed Jun 19, 2013 5:16 pm

Runtime Column Propagation (RCP) is a technique for dealing with dynamic metadata in DataStage jobs. You can find more information in the Parallel Job Developer's Guide or in the IBM Information Center