Job abort during AGG OR SORT

Nagaraj · Post by **Nagaraj** » Thu Sep 15, 2011 1:29 pm

Job failed with the error saying "Required column missing"

pandeesh · Post by **pandeesh** » Thu Sep 15, 2011 2:44 pm

It might be the describe failed error.
Please check the presence of the missing column in source as well as target with same metadata

Nagaraj · Post by **Nagaraj** » Thu Sep 15, 2011 3:07 pm

Got it, it was the metadata error, with more quotes in other columns,
issue resolved, running the job again with Sort and Agg....post the results soon.

Nagaraj · Post by **Nagaraj** » Fri Sep 16, 2011 9:20 am

Okay the input link to the SORT stage has processed all the rows but the output link doesnt show any rows processed from a long time, does this mean it is not configured correctly????

Nagaraj · Post by **Nagaraj** » Fri Sep 16, 2011 9:48 am

Here is the settings i am using.

Rows coming in 1.2 million rows
Max Rows in VM = 49000
Max Open Files = 20
Temp File = explicit Full Path mentioned(i can also see files getting created therr)
tracing level = 0
Stable sort = no
esc char= \
column seperator = ,
Sort spec = Col1 ASC,COl2 ASC,Col3 ASC ...so on

Nagaraj · Post by **Nagaraj** » Fri Sep 16, 2011 12:38 pm

Still Running and no rows on the Sort Output link Yet, running from 3 hours.

Nagaraj · Post by **Nagaraj** » Fri Sep 16, 2011 1:26 pm

running from 4 hours.

ray.wurlod · Post by **ray.wurlod** » Fri Sep 16, 2011 3:48 pm

You will never get any rows output from a Sort stage until all the input rows have been consumed. This is common sense - it does not know where the final row will be in the sorted order until it has been processed.

chulett · Post by **chulett** » Fri Sep 16, 2011 4:06 pm

Nagaraj wrote:Okay the input link to the SORT stage has processed all the rows but the output link doesnt show any rows processed from a long time, does this mean it is not configured correctly????

Don't confuse my advice about how a properly configured Aggregator will run with what you see in the Sort stage, it must take in everything first. And the Server sort stage is pretty slow, in my experience I found an O/S level sort to be approximately 10x faster. YMMV.

Nagaraj · Post by **Nagaraj** » Fri Sep 16, 2011 6:19 pm

Okay will try to sort at the OS level and check the Processing in Job.