Hello All,
We are trying to pull 250 Million Records from AS400 (DB2 I-Series), but ODBC Connector(to AS400) is pulling only 2000 Rows per Second. I played around transaction size and array size with no luck. I tried to figure out the bottleneck by directly using flatfile and dataset as target, and determined its the source ODBC Connector. I am using an 8 node config file and tried modulus partition as well with no luck. Did anyone do anything similar? Please let me know when you get a chance. Thanks for your help in advance.
Thanks
Koti
AS400 - ODBC Connector is very slow (2000 RPS)
Moderators: chulett, rschirm, roy
what is your row length?
Rows per second is ... a tricky beasty.
It's really bytes per second that is your speed. You simply care about rows because it is convenient to look at.
2000 rows per second at 10 bytes per row is horrible.
2000 rows per second at 4K bytes per row is not so bad.
Remember that your director monitor will calculate the rows per second based upon the job start time, job end time, and quantity of rows extracted. It is not an indication of the speed at the time of byte transmission.
Building the result set is factored into that rows per second calculation.
Any downstream ETL logic (none since you chopped it off with that dataset stage) is also factored into the rows per second when you look at those values after the job is done.
What is your DBA telling you about the database in question?
With your 8 node config, how many sessions out to odba do you see? 8 or more? Are you doing data repartitioning?
The modulus, was it on a KEY column?
Does the native DB2 Connector stage work against that AS/400 DB2 target?
Rows per second is ... a tricky beasty.
It's really bytes per second that is your speed. You simply care about rows because it is convenient to look at.
2000 rows per second at 10 bytes per row is horrible.
2000 rows per second at 4K bytes per row is not so bad.
Remember that your director monitor will calculate the rows per second based upon the job start time, job end time, and quantity of rows extracted. It is not an indication of the speed at the time of byte transmission.
Building the result set is factored into that rows per second calculation.
Any downstream ETL logic (none since you chopped it off with that dataset stage) is also factored into the rows per second when you look at those values after the job is done.
What is your DBA telling you about the database in question?
With your 8 node config, how many sessions out to odba do you see? 8 or more? Are you doing data repartitioning?
The modulus, was it on a KEY column?
Does the native DB2 Connector stage work against that AS/400 DB2 target?