sort stage

sonia jacob · Post by **sonia jacob** » Wed Sep 22, 2004 7:49 am

Hi,

I use SORT STAGE to sort the sequential file that is generated by my job. Just this stage itself takes 1 to 2 minutes. I did not get any improvement in performance even after giving a temporary directory (not sure whether it does improve performance in the first place).

Any suggestions.

thanks,
Sonia

raju_chvr · Post by **raju_chvr** » Wed Sep 22, 2004 8:18 am

Sonia,

What else are you doing in this job and how many number of rows are being processed by this stage for 1-2 minutes of time you mentioned ?

sonia jacob · Post by **sonia jacob** » Wed Sep 22, 2004 8:31 am

i have a simple transform and a sort in the container. The log shows that the processing happens pretty fast till the SORT. I process a couple of thousands of records.

raju_chvr · Post by **raju_chvr** » Wed Sep 22, 2004 8:56 am

On how many columns are you sorting the data in Sort specifications. This does matter.

Did you change any default settings in Sort stage like: "Maximum number of rows in Virtual Memory" ?

sonia jacob · Post by **sonia jacob** » Wed Sep 22, 2004 9:14 am

3 sort columns and everythig else have default values

chulett · Post by **chulett** » Wed Sep 22, 2004 9:33 am

sonia jacob wrote:I use SORT STAGE to sort the sequential file that is generated by my job. Just this stage itself takes 1 to 2 minutes. I did not get any improvement in performance even after giving a temporary directory (not sure whether it does improve performance in the first place).

The temporary directory isn't meant to improve performance, it just tells the stage where to put the temporary files it generates. Important note - if you leave that field blank, it will default to the current project. This may be less of an issue on Windows, but on UNIX I always make sure our jobs that use the Sort stage set this to "/tmp" or something else outside of the project. I've seen too many large sorts or runaway jobs fill up the disk with temp files and if you fill up your Project partition, that is what's known as a Very Bad Thing.

On the performance side, the Sort stage isn't the speediest, so one or two minutes ain't all that bad. The kind of things that would improve it are faster disks or faster processors or possibly more memory, other than that...

tonystark622 · Post by **tonystark622** » Wed Sep 22, 2004 9:40 am

One other thing, and it's been recommended in this forum before. The UNIX sort command is a _LOT_ faster than the sort stage, especially with lots of records. If you have more than a few thousand rows, it might be faster to write the data to a seq file and sort it with the UNIX sort command.

Good Luck,
Tony

chulett · Post by **chulett** » Wed Sep 22, 2004 9:53 am

Good advice, but Sonia is on an NT server.

tonystark622 · Post by **tonystark622** » Wed Sep 22, 2004 10:20 am

'S what I get for replying in a hurry. Yeah, what I meant was the Windows SORT command (open a command window and type 'help sort'). (I have no idea whether the Windows sort command would be faster than the Sort stage or not.... I was just trying to save face

)

tony

ariear · Post by **ariear** » Wed Sep 22, 2004 3:27 pm

DOS sort is a lot faster than DS Sort stage - not just that - DS sort failes ! when sorting huge Data (like 4GB).
...But DOS sort can't sort case-sensitive so if you're using DS aggregator stage after DOS sort and mark the sort order it'll fail on when encountering Unsorted charachters. Try using GNU sort or buy Sync sort

roy · Post by **roy** » Sun Sep 26, 2004 11:29 am

Hi,
If 1-2 minutes is out of the question, performance wise
I just wanted to point out that there is the SFU (Services For Unix) and probably others like it, which you can install on a windows platform that makes unix flavour OS command available in windows platforms hence enabling you to use unix os commands

this an alternative to the expensive co-sort and the likes in case they are beyond your budjet.

IHTH,

DSXchange

sort stage

sort stage

Re: sort stage

Re: sort stage