Link Partitioner/Collector Performance issue

maffan76 · Post by **maffan76** » Tue Oct 11, 2005 2:17 am

Hi,
i am reading data from a flat file and passing it to Link Partitioner for performance reasons. and it feeds 4 transformers and i collect the rows back.

The problem i am facing is that in one run i was getting a speed of 2000 Rows/sec on each transformer and in another run i got a speed of 250 to 300 Rows/sec. so its kind of unpridictable behaviour which i am unable to understand.

I have assigend Interprocess memory to 1024K as well.

Any idea what i am doing wrong and how can i improve the performance.

We have a quad processor machine with 2 GB Ram and i am using 4 transformers to utilize the quad processors.

[/code][/list]

maffan76 · Post by **maffan76** » Tue Oct 11, 2005 2:29 am

Also when the file size feeding data to link partitioner is small it works on a very high speed. I am trying to load 60 Million records a files size of approximately 7 GB and i am experiencing very slow row rate per sec.

ArndW · Post by **ArndW** » Tue Oct 11, 2005 5:31 am

The interprocess memory is set to 1024Kb (so 1Gb), you have 4 of them so the system will allocate 4Gb of virtual memory to "speed things up".

Instead of being a good thing, this allocated memory needs to be swapped out to make room for user processes and associated memory. So it seems that by making these buffers too large you have made the machine swap (or perhaps even thrash). Use the default buffer sizes and most likely your performance will go up.

maffan76 · Post by **maffan76** » Tue Oct 11, 2005 7:35 am

How come it can be 1024 KB equals to 1GB. i am unable to grab it
can you please exlain this?