Link Collector and Link Partitioner Stages

bapajju · Post by **bapajju** » Wed Jun 01, 2005 12:34 pm

Hi,

Is there any limitation for the number of rows processed by a Link Collector or Link Partitioner stage. What I mean is is there any threshold limit in terms of number of records after which the efficiency of Link Collector and Link Partitioner stage goes down? Kindly let me know.

Thanks in advance

ray.wurlod · Post by **ray.wurlod** » Wed Jun 01, 2005 5:12 pm

None documented, and no reason why there should be. Rows are buffered (you can increase the buffer sizes), but the downstream process is draining the buffer ideally at least as fast as the upstream process is populating it. If your downstream processing can not go as fast as your upstream processing, then handshaking must occur when the buffer becomes full to tell the upstream process to stop sending - this is all managed internally and is not configurable. But it's not a function of the number of rows. Maybe you need to make your downstream processing go faster. Use statistics gathering on the active stages to identify hot spots.

bapajju · Post by **bapajju** » Tue Jun 07, 2005 6:01 pm

Thanks a lot Ray. The problem was really not in Link Collector. The problem was in the HASH lookup the Lookup was very slow.

Infact Link Collector works very well.

Thanks a ton for your suggestion

ray.wurlod wrote:None documented, and no reason why there should be. Rows are buffered (you can increase the buffer sizes), but the downstream process is draining the buffer ideally at least as fast as the upstream process is populating it. If your downstream processing can not go as fast as your upstream processing, then handshaking must occur when the buffer becomes full to tell the upstream process to stop sending - this is all managed internally and is not configurable. But it's not a function of the number of rows. Maybe you need to make your downstream processing go faster. Use statistics gathering on the active stages to identify hot spots.