DSXchange

Hi All,
I have designed a Parallel job which loads data into 15 oracle Targets tables and the data had been pulled out using 15 ODBC source stages (Source Main Frame DB)Some basic transformation was done using Transformer stage.Performance wise will there be any impact on the DS server and the resources it consume to run this job.Volume wise the data is quite high(in millions)
Is there any limit on the number of source stages and Target stages per job. Any ideal solution apart from splitting it into sub jobs

Thanks in Advance.

chpraveen.msc wrote:Hi All,

Is there any limit on the number of source stages and Target stages per job. Any ideal solution apart from splitting it into sub jobs

Thanks in Advance.

IMHO, there is NO limit to the number of target tables used in a single job.

saying that, you need to consider splitting the load into multiple jobs.
consider the following for doing so

a) modularization
b)handling dependent jobs better
c)exception/failure handling
b)taking advantage of parallel processes from OS
e)eliminating thread/process management at DS server level
f)eliminating resource allocation at DS server level for different jobs in the same job


regards,
Prabu

High volume, more stages will always be a challenge to performance.
Is it all one to one, or joined together and later diversified.
If its one source to one target table, its better to have different jobs for many beneficial factors. You can run the jobs parallely as well.

Bad design. Load in different jobs. You are loading 15 tables with data size ~ 30 Million. Restartability will be a problem. If a record fails in any one of the tables and the job aborts, its going to affect the other loads as well.