Write speed to seq file

[snassimr]

Hi , everybody .

How I can speed up the writing to seq file . I am sure that botlneck in writing to seq file . I am get about 3500 rows/sec.

Some programming tricks ?

[kandyshandy]

search for interprocess / inprocess / row buffering.

[snassimr]

I have already tried . I get 5-7 % improvement . May be anybody use some definitions inside SEQ FILE stage ?

[Mike]

Faster disk hardware... rows/sec is a meaningless metric... how many MB/sec are you writing?

Mike

[chulett]

I am sure that botlneck in writing to seq file . I am get about 3500 rows/sec.

What makes you so sure? No, there's no "trick" and I sincerely doubt the "bottleneck" is your sequential write speed unless you are writing to some crap disks or the underlying filesystem is very busy. Ask your SysAdmin about both points, they will either know or monitor that for you.

[kandyshandy]

I guess 3500rows/sec is good as well.

[ray.wurlod]

There are no programming tricks - the Sequential File stage already uses the STREAMS I/O module.

What else is happening? Are you getting contention on the disk, so that the heads keep being moved away from this file? Try a test where the file is the only thing being accessed on its spindle (an ideal situation, to be sure, but an interesting test) - however, this may not be possible on a storage array network (SAN).

[crouse]

Just to humor us, can you have your seq file stage write to the file /dev/null? And then are you still getting 3500 rps? If so, then your disk is the "problem". If greater than 3500 rps, then the problem is elsewhere.