Multiple Rows into one single row, comma seperated.

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
iHijazi
Participant
Posts: 45
Joined: Sun Oct 24, 2010 12:05 am

Multiple Rows into one single row, comma seperated.

Post by iHijazi »

Hi,

This is my first post here, been reading a lot of posts on this website, pretty interesting.

Anyway, I am facing this issue:

I have the following data
<pre>
DM_OPERATION ID REASON_CODE
---------------------- -------------------
4 158
4 156
4 865
4 901
4 483
</pre>
I need to output them as
<pre>
DM_OPERATION ID REASON_CODE
---------------------- -------------------
4 158,156,865,901,483
</pre>
Read a couple of posts say to use Pivot, but I believe it's wrong. Not to mention that I am unable to configure it correctly.

The job is a bit complex, there are many stages before, and some after the above is done. I need the Operation_ID in order to merge it with other set of data. And I need the data to be comma separated because the customer asked me to! :)

I'd appreciate any advice. What stage to use, and quick example if possible.

Thanks.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Welcome.

It most certainly is a pivot, actually a "vertical pivot" of rows to columns. However, you are correct in that the actual Pivot stage only supports the other direction - columns to rows or a horizontal pivot - until the upcoming 8.5 release, at least from what I seem to remember I've read. So, in the meantime, you get to do it yourself!

If you've been searching the forums, you'll find most advice on vertical pivoting says to search the forum. Advice is out there, it's just the wheat is buried amongst a ton of chaff. :wink:

What comes to mind is transformer stage variable work along with an Aggregator. You need to do basic 'group change detection' so you know when your DM_OPERATION_ID value changes and you can do that all in stage variables or with a preceding Sort stage with a Key Change column added. If you go the latter route, make sure you tell it "Don't sort, already sorted".

In the transformer, accumulate REASON_CODE values in a single field and go ahead and output the single field. The Aggregator comes into play to capture the 'Last' record from each group of DM_OPERATION_ID values. So, output looking something like this:

Code: Select all

4   158
4   158,156
4   158,156,865
4   158,156,865,901
4   158,156,865,901,483 <- this row is passed by the Aggregator
5   184
5   184,163
-etc-
Use the key change column being 1 or 'true' to know when to reset the accumlating stage variable back to just the initial value. Every subsequent row where the key change column is 0 or 'false' you concatenate a comma and the current value onto the accumulating value.

Hope that helps and can be integrated into your current design. I'm sure there are other ways to skin this cat but that's the typical approach to this.
-craig

"You can never have too many knives" -- Logan Nine Fingers
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Perhaps you can use Remove Duplicates stage rather than Aggregator.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
ankesh
Participant
Posts: 8
Joined: Wed May 16, 2007 3:05 pm
Location: Mumbai

Use of stage variables

Post by ankesh »

HI,

Sort and Partition the data on first column, use stage variables to store second column. Concat the values of second column into the stage variable for repeatition of first column. (You might want to use a couple of stage variables to get the result).
You will get below result-
4 158
4 158,156
4 158,156,865
4 158,156,865,901
4 158,156,865,901,483
5 184
5 184,163

Then sort and remove duplicates to get this-

4 158,156,865,901,483
5 184,163
swapnilverma
Participant
Posts: 135
Joined: Tue Aug 14, 2007 4:27 am
Location: Mumbai

Post by swapnilverma »

do u still need help with logic ?
Thanks
Swapnil

"Whenever you find whole world against you just turn around and Lead the world"
iHijazi
Participant
Posts: 45
Joined: Sun Oct 24, 2010 12:05 am

Post by iHijazi »

Hey guys..

Thanks for your replies, you guys definitely gave me a better perspective.

I must say that ankesh solution worked like a charm, thanks buddy!! Simple and straightforward :)

And thanks again everybody.

Cheers (:
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

swapnilverma wrote:do u still need help with logic ?
Probably not. U did not ask the questions. (U is one of our DSXchange community.)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
marpadga18
Premium Member
Premium Member
Posts: 96
Joined: Fri Aug 20, 2010 8:51 am

Post by marpadga18 »

Hi Could you tell me how you did this any help really aprreciated
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

iHijazi wrote:I must say that ankesh solution worked like a charm, thanks buddy!! Simple and straightforward :)
marpadga18 - if that's not enough to help, please start your own topic and let us know what problem you are trying to solve and what issues you are having...
-craig

"You can never have too many knives" -- Logan Nine Fingers
marpadga18
Premium Member
Premium Member
Posts: 96
Joined: Fri Aug 20, 2010 8:51 am

Post by marpadga18 »

I tried with stage variable. I got it Thank you..
Post Reply