DSXchange

Hi,

I have 3 columns as input and I have to generate the 4th column using the 3 columns

For example my input 3 columns are
I need to generate the 4th column as below
can I achive this in a parallel job.

Thanks in advance

Hi kittu.raja,

I have 3 columns as input and I have to generate the 4th column using the 3 columns
...
can I achive this in a parallel job.

Thanks in advance

As your posting subject already states "Logic needed"
you should first post the required logic instead of making us guessing it.

So the answer is short as your description:
Yes - use a transformer

ok Can you tell me how to implement it in transformer.

Who are you?

You're not the originator of this thread? Are you a colleague of kittu.raja? If not, how can we be sure that you're on an identical system?


In any case, until kittu.raja posts the logic required (in English) how could we suggest how to implement that logic in a Transformer - or any other - stage type?!!

hi
with the help of 3 stage variables you can achieve the desired result
stgvar --> input column
stgvar1 --> stgvar2:'/':stgvar
stgvar2 --> stgvar1

the only thing is the result will come in the format
/a
/a/b
/a/b/c
/a/b/c/d

so you can remove that the first / easily

vishal_rastogi wrote:hi
with the help of 3 stage variables you can achieve the desired result
stgvar --> input column
stgvar1 --> stgvar2:'/':stgvar
stgvar2 --> stgvar1

i have tested this with the input file contains the below data:

id
---
a
b
c
d


But the result is,

/c
/a
/a/d
/b

Run it sequentially

If the data get partitioned it won't do what you want.

blewip wrote:Run it sequentially

If the data get partitioned it won't do what you want.

False. If the data are partitioned properly it will do what is required.

HI pandeesh,

i really don't know why you are getting this result i am getting exactly the same i have mentioned in previous post . i have tested it again.

Hi,

Previously in the source sequential file stage, the option 'Keep File partitions set to false'. i have changed that to true.
Now i am getting the required result.

Thanks

Nice!
it's working great with the below derivation in transformer:

ray.wurlod wrote:

blewip wrote:Run it sequentially

If the data get partitioned it won't do what you want.

False. If the data are partitioned properly it will do what is required. ...

This is not false, this is what fixed the problem. The stated fix In "sequential file stage, the option 'Keep File partitions set to false'" keeps the file sequentially in one partition.

If the data was to get partitioned, it wouldn't work, exactly as I stated.

You have a sequential file input, what the brilliant way of partitioning this "properly" to get this to work?

A key-based partitioning algorithm immediately downstream of the Sequential File stage.

kittu.raja wrote: For example my input 3 columns are

Code: Select all

Col1   Col2    Col3
  1      a     1
  2      b     1/2
  3      c     1/2/3
  4      d     1/2/3/4

And which Column would be used for the key?

Try with this
stagevar1 = col1
stagevar2=stagevar3:/:stagevar1(If Stagevar3 is null then Stagevar1)
stagevar3=stagevar2