ODBC Connector Stage for DB2 Performance issue

hi_manoj · Post by **hi_manoj** » Wed May 11, 2016 9:17 am

Hi,

I have a ODBC connector stage with with which i am reading data from a DB2 table. Table has multi million data.

When reading data from the table the number of rows is very low, it starts with 1000 and goes up to 7000.
Is there a way we can increase the performance of the ODBC connector stage. ?

I have tried changing array size and record count (from 2000 to 200000), but the result is same.

please help
Regards
Manoj

hi_manoj · Post by **hi_manoj** » Wed May 11, 2016 11:33 am

For one of the table, i tried with modulo partition (enable partition read-column name) , but seems like all the records are going into each partition. as if it doing entire partition.

for example -- if my source table has 5 records the output link shows 20 records
As I am using a 4 node cofig file.
Do I need to set any specific property.

Regards
Manoj

chulett · Post by **chulett** » Wed May 11, 2016 12:15 pm

Is your source table actually partitioned? Sounds like the answer is no.

Mike · Post by **Mike** » Thu May 12, 2016 7:33 am

The DB2 Connector can do partitioned reads even when the underlying source table is not partitioned.

I always prefer the DB2 Connector over the ODBC Connector, so I don't know about the ODBC Connector's parallel read capabilities. The fact that you're seeing all rows broadcasted to every processing node implies either a bug or a "feature" of that stage.

Mike

chulett · Post by **chulett** » Thu May 12, 2016 8:31 am

Just checked the docs and it doesn't seem that the ODBC Connector requires that either. So I would just double-check the partitioning method and column that you've chosen in the stage and then compare that to the partitioning that you are doing in the job itself.