Poor performance of Hash file?

skumar · Post by **skumar** » Mon May 21, 2007 6:34 am

Hi all,

I am using 5 hashed files in single job and my hash file contains records around 50000 and my design looks like below.

seqfile---->TFM1----->TFM2----->TFM3----->Target
| | |
| | |
| | |
Lkp1 Lkp2 Lkp2

when i ran the job the first part i.e. from seqfile to TFM1 link was fetching at good speed around 3500 rows/sec and LKP1 file conatins around one million records.Now when it was running from TFM2 part the performance of reading and writing the records is too slow even though my Lkp2 file conatins only 50000 rows.The speed of reading and writing was only 50 rows/second(important thing here is that i am using the same LKP2
lookup at both TFM2 and TFM3)due to some business logic........

Can some one please guide me why the perfomance is too poor when it was reading the data and writing the data into the next link.
Thanks in advance.........

Regards,
skumar.

skumar · Post by **skumar** » Mon May 21, 2007 6:35 am

Please not that the Lkps are for TFM1,TFM2,TFM3 respectively not to sequentail file.

chulett · Post by **chulett** » Mon May 21, 2007 6:37 am

What is your Target? Have you confimed the lookups are the issue by replacing it with a Sequential File stage?

skumar · Post by **skumar** » Mon May 21, 2007 11:19 pm

craig,

Thanks for the reply.My target is also a sequential file.Thats why i wondered...........So,i am thinking the problem is with the lookup2 only.can you guide me on the same??

Regards,
skumar.

ray.wurlod · Post by **ray.wurlod** » Tue May 22, 2007 1:17 am

Are the hashed files read-cached? How large are the hashed files? How big have you set the read cache?

rafik2k · Post by **rafik2k** » Tue May 22, 2007 3:07 am

How complex is transformation rule in TFM2 and TFM3?
May be complex rule is eating your speed.

rafik2k · Post by **rafik2k** » Tue May 22, 2007 3:11 am

How complex is transformation rule in TFM2 and TFM3?
May be complex rule is eating your speed.

skumar · Post by **skumar** » Tue May 22, 2007 3:21 am

Rafik----I am not doing any transformations in the TFM2 & TFM3 . i am doing only the lookup based on one key which it is meant to be used for the same..

Ray,
Could you please tell me some thing more about the read cahche file size because i don't know that......How can we check whether the file is read cached or not???

Thanks,
skumar.

ray.wurlod · Post by **ray.wurlod** » Tue May 22, 2007 3:14 pm

There is a field in each Hsahed File stage in which caching can be enabled or disabled, if the stage's output is connected to a reference link.
Size of the hashed file cache for the project is set in the Administrator client on the Performance tab.