Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett , rschirm , roy
kittu.raja
Premium Member
Posts: 175 Joined: Tue Oct 14, 2008 1:48 pm
Post
by kittu.raja » Wed Jun 22, 2011 5:45 am
Hi,
I have 3 columns as input and I have to generate the 4th column using the 3 columns
For example my input 3 columns are
Code: Select all
Col1 Col2 Col3
1 a 1
2 b 1/2
3 c 1/2/3
4 d 1/2/3/4
I need to generate the 4th column as below
Code: Select all
Col1 Col2 Col3 Col4
1 a 1 a
2 b 1/2 a/b
3 c 1/2/3 a/b/c
4 d 1/2/3/4 a/b/c/d
can I achive this in a parallel job.
Thanks in advance
Rajesh Kumar
MT
Premium Member
Posts: 198 Joined: Fri Mar 09, 2007 3:51 am
Post
by MT » Wed Jun 22, 2011 5:59 am
Hi kittu.raja,
I have 3 columns as input and I have to generate the 4th column using the 3 columns
...
can I achive this in a parallel job.
Thanks in advance
As your posting subject already states "Logic needed"
you should first post the required logic instead of making us guessing it.
So the answer is short as your description:
Yes - use a transformer
regards
Michael
_chamak
Premium Member
Posts: 29 Joined: Tue Aug 24, 2010 10:29 am
Post
by _chamak » Wed Jun 22, 2011 8:59 am
ok Can you tell me how to implement it in transformer.
ray.wurlod
Participant
Posts: 54607 Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:
Post
by ray.wurlod » Wed Jun 22, 2011 4:34 pm
Who are you?
You're not the originator of this thread? Are you a colleague of kittu.raja? If not, how can we be sure that you're on an identical system?
In any case, until kittu.raja posts the logic required (in English) how could we suggest how to implement that logic in a Transformer - or any other - stage type?!!
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
vishal_rastogi
Participant
Posts: 47 Joined: Thu Dec 09, 2010 4:37 am
Post
by vishal_rastogi » Mon Jul 04, 2011 2:16 am
hi
with the help of 3 stage variables you can achieve the desired result
stgvar --> input column
stgvar1 --> stgvar2:'/':stgvar
stgvar2 --> stgvar1
the only thing is the result will come in the format
/a
/a/b
/a/b/c
/a/b/c/d
so you can remove that the first / easily
Vish
pandeesh
Premium Member
Posts: 1399 Joined: Sun Oct 24, 2010 5:15 am
Location: CHENNAI, TAMIL NADU
Post
by pandeesh » Mon Jul 04, 2011 7:55 am
vishal_rastogi wrote: hi
with the help of 3 stage variables you can achieve the desired result
stgvar --> input column
stgvar1 --> stgvar2:'/':stgvar
stgvar2 --> stgvar1
i have tested this with the input file contains the below data:
id
---
a
b
c
d
But the result is,
/c
/a
/a/d
/b
pandeeswaran
blewip
Participant
Posts: 81 Joined: Wed Nov 10, 2004 10:55 am
Location: London
Post
by blewip » Mon Jul 04, 2011 10:35 am
Run it sequentially
If the data get partitioned it won't do what you want.
Modern Life is Rubbish - Blur
ray.wurlod
Participant
Posts: 54607 Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:
Post
by ray.wurlod » Mon Jul 04, 2011 3:21 pm
blewip wrote: Run it sequentially
If the data get partitioned it won't do what you want.
False. If the data are partitioned properly it
will do what is required.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
vishal_rastogi
Participant
Posts: 47 Joined: Thu Dec 09, 2010 4:37 am
Post
by vishal_rastogi » Mon Jul 04, 2011 10:33 pm
HI pandeesh,
i really don't know why you are getting this result i am getting exactly the same i have mentioned in previous post . i have tested it again.
Vish
pandeesh
Premium Member
Posts: 1399 Joined: Sun Oct 24, 2010 5:15 am
Location: CHENNAI, TAMIL NADU
Post
by pandeesh » Tue Jul 05, 2011 1:16 am
Hi,
Previously in the source sequential file stage, the option 'Keep File partitions set to false'. i have changed that to true.
Now i am getting the required result.
Thanks
pandeeswaran
pandeesh
Premium Member
Posts: 1399 Joined: Sun Oct 24, 2010 5:15 am
Location: CHENNAI, TAMIL NADU
Post
by pandeesh » Tue Jul 05, 2011 1:41 am
Nice!
it's working great with the below derivation in transformer:
pandeeswaran
blewip
Participant
Posts: 81 Joined: Wed Nov 10, 2004 10:55 am
Location: London
Post
by blewip » Tue Jul 05, 2011 2:15 am
ray.wurlod wrote: blewip wrote: Run it sequentially
If the data get partitioned it won't do what you want.
False. If the data are partitioned properly it
will do what is required. ...
This is not false, this is what fixed the problem. The stated fix In "sequential file stage, the option 'Keep File partitions set to false'" keeps the file sequentially in one partition.
If the data was to get partitioned, it wouldn't work, exactly as I stated.
You have a sequential file input, what the brilliant way of partitioning this "properly" to get this to work?
Modern Life is Rubbish - Blur
ray.wurlod
Participant
Posts: 54607 Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:
Post
by ray.wurlod » Tue Jul 05, 2011 2:35 am
A key-based partitioning algorithm immediately downstream of the Sequential File stage.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
blewip
Participant
Posts: 81 Joined: Wed Nov 10, 2004 10:55 am
Location: London
Post
by blewip » Tue Jul 05, 2011 6:45 am
kittu.raja wrote:
For example my input 3 columns are
Code: Select all
Col1 Col2 Col3
1 a 1
2 b 1/2
3 c 1/2/3
4 d 1/2/3/4
And which Column would be used for the key?
Modern Life is Rubbish - Blur
soumya5891
Participant
Posts: 152 Joined: Mon Mar 07, 2011 6:16 am
Post
by soumya5891 » Tue Jul 05, 2011 7:44 am
Try with this
stagevar1 = col1
stagevar2=stagevar3:/:stagevar1(If Stagevar3 is null then Stagevar1)
stagevar3=stagevar2
Soumya