Poor performance of Hash file?
Posted: Mon May 21, 2007 6:34 am
Hi all,
I am using 5 hashed files in single job and my hash file contains records around 50000 and my design looks like below.
seqfile---->TFM1----->TFM2----->TFM3----->Target
| | |
| | |
| | |
Lkp1 Lkp2 Lkp2
when i ran the job the first part i.e. from seqfile to TFM1 link was fetching at good speed around 3500 rows/sec and LKP1 file conatins around one million records.Now when it was running from TFM2 part the performance of reading and writing the records is too slow even though my Lkp2 file conatins only 50000 rows.The speed of reading and writing was only 50 rows/second(important thing here is that i am using the same LKP2
lookup at both TFM2 and TFM3)due to some business logic........
Can some one please guide me why the perfomance is too poor when it was reading the data and writing the data into the next link.
Thanks in advance.........
Regards,
skumar.
I am using 5 hashed files in single job and my hash file contains records around 50000 and my design looks like below.
seqfile---->TFM1----->TFM2----->TFM3----->Target
| | |
| | |
| | |
Lkp1 Lkp2 Lkp2
when i ran the job the first part i.e. from seqfile to TFM1 link was fetching at good speed around 3500 rows/sec and LKP1 file conatins around one million records.Now when it was running from TFM2 part the performance of reading and writing the records is too slow even though my Lkp2 file conatins only 50000 rows.The speed of reading and writing was only 50 rows/second(important thing here is that i am using the same LKP2
lookup at both TFM2 and TFM3)due to some business logic........
Can some one please guide me why the perfomance is too poor when it was reading the data and writing the data into the next link.
Thanks in advance.........
Regards,
skumar.