Hi,
I have fixed width file in Datastage mainframe job, When I used aggregator stage my job failes with a SORT error, despite of using SORT stage prior to aggreagtor stage along with a intermediate fixed width file the job fails with the same error message.
Can anyone tell me how to remove duplicates from a file in mainframe edition of datastage, as such we have very few processing stages in mainframe edition.
Remove duplicates in Datastage MVS edition
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 27
- Joined: Fri May 02, 2008 8:01 am
- Location: Mumbai
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Welcome aboard.
Add a sort ahead of the Aggregator stage, sorting by the grouping (duplicate identifier) keys. You can use a Sort stage or, if the data are coming from a relational table, specify the ordering in the extraction.
On the Output page General tab, select the Group By option rather than the Control Break option.
Add a sort ahead of the Aggregator stage, sorting by the grouping (duplicate identifier) keys. You can use a Sort stage or, if the data are coming from a relational table, specify the ordering in the extraction.
On the Output page General tab, select the Group By option rather than the Control Break option.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 27
- Joined: Fri May 02, 2008 8:01 am
- Location: Mumbai
[
Hi Ray,
Thanks for the support, I have tried using a sort stage before a aggregator stage, but it seems we cant have a sort stage before aggreator stage, since when I link sort stage followed by aggregator stage it gives me compilation error as input should be a file, relational table for aggregator stage. do I need to introduc another flat file after a sort stage?
quote="ray.wurlod"]Welcome aboard.
Add a sort ahead of the Aggregator stage, sorting by the grouping (duplicate identifier) keys. You can use a Sort stage or, if the data are coming from a relational table, specify ...[/quote]
Hi Ray,
Thanks for the support, I have tried using a sort stage before a aggregator stage, but it seems we cant have a sort stage before aggreator stage, since when I link sort stage followed by aggregator stage it gives me compilation error as input should be a file, relational table for aggregator stage. do I need to introduc another flat file after a sort stage?
quote="ray.wurlod"]Welcome aboard.
Add a sort ahead of the Aggregator stage, sorting by the grouping (duplicate identifier) keys. You can use a Sort stage or, if the data are coming from a relational table, specify ...[/quote]
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 27
- Joined: Fri May 02, 2008 8:01 am
- Location: Mumbai
Hi ,
I have already put a flat file after a sort stage then followed by a aggreagtor stage still gets the same sort error message. below is the job design for the same, Do I need to have 2 separate jobs one with a sort stage and other hob with an aggreagator stage or anything specific?
Flat file(i/p) --->Sort --->Flat file--->Aggreagator -->transfromer-->Lookup---> flat file
Thanks,
Sandeep S Pendem
I have already put a flat file after a sort stage then followed by a aggreagtor stage still gets the same sort error message. below is the job design for the same, Do I need to have 2 separate jobs one with a sort stage and other hob with an aggreagator stage or anything specific?
Flat file(i/p) --->Sort --->Flat file--->Aggreagator -->transfromer-->Lookup---> flat file
Thanks,
Sandeep S Pendem
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
As far as I can tell without seeing the detail of your design (sort keys etc) it should be able to remove duplicates satisfactorily. There ought to be no need for more than one job. Are you sorting and aggregating (grouping) the same keys (the ones that identify "duplicates"? How are you getting the other fields (if any) through the Aggregator stage?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact: