Performance Boost
Posted: Fri Oct 03, 2003 2:24 pm
This is true for Tru64 implementation of DataStage 6.0.1 PX with the latest patch using data pulled from Oracle (you know, varchar, numbers, et cetera). Whether this can be true or not for other platforms is up to you.
We have discovered that Lookup stage are extremely slow in handling matches, while using Full Outer Join with a Transform filtering results (if you use the reject link) would produce 10-20x the performance (analysis still going on right now) over a simple Lookup stage. Naturally, if you do not need a reject link, you should use full inner join.
This was our major stumbling block in seeing how viable PX can be -- for some jobs, we're shifting 700+ million rows of data from a single table with 5+ million rows of 'lookup' data.
So if you guys have very large schemas, or is planning to do so -- keep in mind the dramatic difference in performance for Joins compared to Lookup.
This free advice have been brought to you by the letter "D", "S", and the number "42". :D
-T.J.
We have discovered that Lookup stage are extremely slow in handling matches, while using Full Outer Join with a Transform filtering results (if you use the reject link) would produce 10-20x the performance (analysis still going on right now) over a simple Lookup stage. Naturally, if you do not need a reject link, you should use full inner join.
This was our major stumbling block in seeing how viable PX can be -- for some jobs, we're shifting 700+ million rows of data from a single table with 5+ million rows of 'lookup' data.
So if you guys have very large schemas, or is planning to do so -- keep in mind the dramatic difference in performance for Joins compared to Lookup.
This free advice have been brought to you by the letter "D", "S", and the number "42". :D
-T.J.