DB2 Extract Performance

shank · Post by **shank** » Fri Sep 12, 2014 1:50 pm

Hello All,

I have a simple EXtract job from DB2 wherein we need to dump the entire table into a flat file.

Currently I am using DB2_Connector Stage (DB2 Connector --> Copy Stage with Forced Option =True --> Sequential File ) and am able to write around 23K records per second in the target Sequential File.

Is it is the maximum performance I can get in Datastage ?

I have some tables which has 65 Million records in them and they are taking 1 hour each to get extracted.
Our goal is to complete a table extract in 15- 20 minutes.

Is there any better way to do this ? Please suggest

ray.wurlod · Post by **ray.wurlod** » Fri Sep 12, 2014 3:21 pm

Try using Table Read method in the DB2 Connector.

The Sequential File stage uses STREAMS I/O, so should not be a bottleneck. Rows/sec is a meaningless metric since we don't know how wide your rows are.

qt_ky · Post by **qt_ky** » Fri Sep 12, 2014 6:57 pm

Your job type says server but you mention the Copy stage. If it is actually a parallel job then you could run it on a different number of nodes to improve the extract performance. It still writes your text file sequentially.

chulett · Post by **chulett** » Fri Sep 12, 2014 10:23 pm

qt_ky wrote:Your job type says server but you mention the Copy stage.

Corrected.