Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett , rschirm , roy
Marley777
Premium Member
Posts: 316 Joined: Tue Jan 27, 2004 3:26 pm
Post
by Marley777 » Mon Sep 19, 2011 7:40 am
Thanks for reading.
Getting warnings when using an agg stage.
Hash Table has grown to 32768
I did a forum search and found that sorting on the grouping keys will elimintae the warning. If my 'partitioning type' is set to 'auto', is sorting required...wouldn't partition type set to auto take care of any sorting needed?
fyi - Agg stage is using hash method.
battaliou
Participant
Posts: 155 Joined: Mon Feb 24, 2003 7:28 am
Location: London
Contact:
Post
by battaliou » Mon Sep 19, 2011 7:49 am
If you method is sort then datastage can release the grouped data as soon as it is processed. This is far more efficient.
3NF: Every non-key attribute must provide a fact about the key, the whole key, and nothing but the key. So help me Codd.
Marley777
Premium Member
Posts: 316 Joined: Tue Jan 27, 2004 3:26 pm
Post
by Marley777 » Mon Sep 19, 2011 7:53 am
So do I need to do my own sort or let datastage handle by using partition type = auto?
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Mon Sep 19, 2011 7:55 am
You need to ensure the data is sorted properly to support the aggregation being done.
-craig
"You can never have too many knives" -- Logan Nine Fingers
Marley777
Premium Member
Posts: 316 Joined: Tue Jan 27, 2004 3:26 pm
Post
by Marley777 » Mon Sep 19, 2011 8:20 am
Auto Partition should should take care of any sorting that needs to be done...correct?
always thought if you have the env variables below set to true DS will sort for you based on the grouping keys used in the agg stage by inserting tsort and partitioning operators?
APT_NO_SORT_INSERTION = TRUE
APT_NO_PART_INSERTION = TRUE
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Mon Sep 19, 2011 9:20 am
Setting "NO SORT INSERTION" to true means it is not allowed to insert tsort operators. Same goes for partitioning.
-craig
"You can never have too many knives" -- Logan Nine Fingers
Marley777
Premium Member
Posts: 316 Joined: Tue Jan 27, 2004 3:26 pm
Post
by Marley777 » Mon Sep 19, 2011 10:53 am
Yes..it should be set to false. Which means it should take care of the sorting correct?
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Mon Sep 19, 2011 11:11 am
I don't believe so, try it and let us know.
-craig
"You can never have too many knives" -- Logan Nine Fingers
Marley777
Premium Member
Posts: 316 Joined: Tue Jan 27, 2004 3:26 pm
Post
by Marley777 » Mon Sep 19, 2011 2:05 pm
Seems to be working for me. We use the aggregator stage to bring fields from multiple records with the same keys into one record. As of now the data looks good. In the agg stage I'm using the sort method versus hash. Warnings are gone.
Thanks for your help.