DataStage Parallel Processing and DB I/O

wfkurtz1 · Post by **wfkurtz1** » Fri Mar 04, 2011 2:51 pm

When a parallel job is configured to use more than one data partition (say 4) does DS create a seperate connection to the DB server (Oracle, DB2, etc) for each partition? Or is there only one connection between DS and the DB, and the data is not partitioned until it arrives on the DS server?

If the former then a DS parallel job has a larger impact on a DB server's connection pool than a server job does. Wouldn't you say?

_Wayne

ray.wurlod · Post by **ray.wurlod** » Fri Mar 04, 2011 2:57 pm

That will depend on which stage type you choose and whether or not it's capable of parallel processing. For example, the Oracle Enterprise stage is capable of parallel processing but (unless it's a partitioned table) Oracle will only deliver a sequential stream when reading. However, a DB2 Connector (for example) can SELECT in parallel. An ODBC Connector stage might establish multiple sessions into the database (which can have database licensing issues).

wfkurtz1 · Post by **wfkurtz1** » Fri Mar 04, 2011 8:19 pm

Interesting. Looks like DS is optimized for DB2 ... not surprising. I guess if the Oracle result set is from a join then the input is sequential. Even with DB2 I guess only a single table (or view?) per SELECT is supported in parallel.

Thanks.

ray.wurlod · Post by **ray.wurlod** » Sat Mar 05, 2011 1:02 am

That you'd have to try - who knows what these database engines are capable of?

wfkurtz1 · Post by **wfkurtz1** » Tue Mar 08, 2011 12:35 pm

Gregg - Thanks for the info. It should be very useful going forward.