Page 1 of 1
Write speed to seq file
Posted: Mon Jan 19, 2009 4:35 am
by snassimr
Hi , everybody .
How I can speed up the writing to seq file . I am sure that botlneck in writing to seq file . I am get about 3500 rows/sec.
Some programming tricks ?
Posted: Mon Jan 19, 2009 5:19 am
by kandyshandy
search for interprocess / inprocess / row buffering.
Posted: Mon Jan 19, 2009 5:39 am
by snassimr
I have already tried . I get 5-7 % improvement . May be anybody use some definitions inside SEQ FILE stage ?
Posted: Mon Jan 19, 2009 7:09 am
by Mike
Faster disk hardware... rows/sec is a meaningless metric... how many MB/sec are you writing?
Mike
Re: Write speed to seq file
Posted: Mon Jan 19, 2009 8:16 am
by chulett
snassimr wrote:I am sure that botlneck in writing to seq file . I am get about 3500 rows/sec.
What makes you so sure? No, there's no "trick" and I sincerely doubt the "bottleneck" is your sequential write speed unless you are writing to some crap disks or the underlying filesystem is very busy. Ask your SysAdmin about both points, they will either know or monitor that for you.
Posted: Mon Jan 19, 2009 9:51 am
by kandyshandy
I guess 3500rows/sec is good as well.
Posted: Mon Jan 19, 2009 3:36 pm
by ray.wurlod
There are no programming tricks - the Sequential File stage already uses the STREAMS I/O module.
What else is happening? Are you getting contention on the disk, so that the heads keep being moved away from this file? Try a test where the file is the only thing being accessed on its spindle (an ideal situation, to be sure, but an interesting test) - however, this may not be possible on a storage array network (SAN).
Posted: Mon Jan 19, 2009 4:02 pm
by crouse
Just to humor us, can you have your seq file stage write to the file /dev/null? And then are you still getting 3500 rps? If so, then your disk is the "problem". If greater than 3500 rps, then the problem is elsewhere.