The job is extremely slow after join
Posted: Thu Aug 02, 2007 8:56 pm
I have to Sequence File joined,then transformed,then upsert into Oracle.
Reading from Sequence File is about 25000 rows/sec,but after joining the link shows that it is about 50 rows/sec,it is intolerable.
Then I find that when upserting into Oracle many records are rejected.Maybe there is too much space in my data,so it is too long for the datatype(varchar) specified in the Oracle and this makes Oracle rejects records.
I set the Environment Variable APT_STRING_PADCHAR to 0x00,and use trim() in the Transformer.This works,the job runs fast.
But there is another problem,if I use trim() in the Transformer,because some trimmed fields of some records are null,so these records are dropped and this is not conformed to the business logic.
How can I solve this problem?thanks.
Reading from Sequence File is about 25000 rows/sec,but after joining the link shows that it is about 50 rows/sec,it is intolerable.
Then I find that when upserting into Oracle many records are rejected.Maybe there is too much space in my data,so it is too long for the datatype(varchar) specified in the Oracle and this makes Oracle rejects records.
I set the Environment Variable APT_STRING_PADCHAR to 0x00,and use trim() in the Transformer.This works,the job runs fast.
But there is another problem,if I use trim() in the Transformer,because some trimmed fields of some records are null,so these records are dropped and this is not conformed to the business logic.
How can I solve this problem?thanks.