Page 1 of 1

DataStage Parallel Processing and DB I/O

Posted: Fri Mar 04, 2011 2:51 pm
by wfkurtz1
When a parallel job is configured to use more than one data partition (say 4) does DS create a seperate connection to the DB server (Oracle, DB2, etc) for each partition? Or is there only one connection between DS and the DB, and the data is not partitioned until it arrives on the DS server?

If the former then a DS parallel job has a larger impact on a DB server's connection pool than a server job does. Wouldn't you say?

_Wayne

Posted: Fri Mar 04, 2011 2:57 pm
by ray.wurlod
That will depend on which stage type you choose and whether or not it's capable of parallel processing. For example, the Oracle Enterprise stage is capable of parallel processing but (unless it's a partitioned table) Oracle will only deliver a sequential stream when reading. However, a DB2 Connector (for example) can SELECT in parallel. An ODBC Connector stage might establish multiple sessions into the database (which can have database licensing issues).

Posted: Fri Mar 04, 2011 8:19 pm
by wfkurtz1
Interesting. Looks like DS is optimized for DB2 ... not surprising. I guess if the Oracle result set is from a join then the input is sequential. Even with DB2 I guess only a single table (or view?) per SELECT is supported in parallel.

Thanks.

Posted: Sat Mar 05, 2011 1:02 am
by ray.wurlod
That you'd have to try - who knows what these database engines are capable of?

Posted: Tue Mar 08, 2011 12:35 pm
by wfkurtz1
Gregg - Thanks for the info. It should be very useful going forward.