implicit Vs explicit sort

Edwink · Post by **Edwink** » Fri Jun 01, 2007 2:17 pm

Friends,
can any one suggest me?
i want to merge two files
whether i can use sort stage externally or leave the merge stage to take care of sorting before merging...
which will be better in performance ...?can u please explain the reson also?

Thank you

ray.wurlod · Post by **ray.wurlod** » Fri Jun 01, 2007 3:15 pm

A reson (literally "thingy") is one of the fundamental units of magic - it's a subdivision of the thaum. "U" is one of the participants on this forum: the second person personal pronoun is "you".

You provide no definition of performance, but ask us to comment on how something undefined can be changed. That is unfair.

Sort stage gives more options, particularly memory allocation, though that can be accomplished using the APT_TSORT_STRESS_BLOCKSIZE environment variable. All else being equal, the inserted tsort operator, the sort specified on the input link and the explicit Sort stage on that input link will all perform identical sort operation. Presumably, then, under those assumptions (all else being equal or, as the economists are fond of saying, ceteris paribus), your choice is irrelevant to "performance" (whatever that means).

Once you start tweaking properties, of course, all bets are off.

Edwink · Post by **Edwink** » Mon Jun 04, 2007 9:27 am

sorry

.

aakashahuja · Post by **aakashahuja** » Mon Jun 04, 2007 9:56 am

The decision to use external / implicit sort should be driven by many other factors apart from performance (the factors might end up driving performance). Some of them being

1. An external sort would definitely use the sort disk defined in your config file, but the implicit would not (this is what I have heard).
2. Would you need more flexibility in your sort, an explicit stage would provide more options than the implicit one.
3. The data you have.

Once you have the details, then only you can come to a conclusion as to what would you want and what would give you a better performance.

Cheers
Aakash

vmcburney · Post by **vmcburney** » Mon Jun 04, 2007 9:25 pm

The maintainable approach is to let the job do its own sorting and you just configure the merge. Parallel sorts are quite fast. You only need to make your design more complicated if you need more speed - so do a test run of the simple approach first to see if it meets your needs.

ray.wurlod · Post by **ray.wurlod** » Tue Jun 05, 2007 2:02 am

The Sort stage is very useful for preventing unnecessary sorting, using Sort Mode "Don't Sort (Previously Sorted)".

vmcburney · Post by **vmcburney** » Tue Jun 05, 2007 4:08 am

I like that. It satisfies the objective of keeping the design simple and it avoids playing around with the environment variables that try to prevent implicit sorts from being added to the job. Does it add any overhead to the job or does it get removed from the executing job? (Like a straight through copy stage).