Generating Surrogate Key In Basic Transformer Parallel Job

[DS_TM]

Hi,

When I use a BASIC transoformer in a parallel job, by default it runs in parallel on all the nodes as per the configuration file but it doesn't have partitionnum and numpartitions sytem varibales to generate a surrogate key. It only has @INROWNUM which is not sufficient when the job is running in parallel. Are there any altenatives to generate surrogate key in a BASIC transformer parallel job?

[Ravi.K]

Basic transformer runs under Server engine. So, there is no parallel concept here. So, you can use straightway @Inrownum system variable to generate the Surrogate Key.

You can also use "KeyMngGetNextValue" DS Transform Routinue for Surrogate Key Generation.

[kwwilliams]

Why use the basic transformer to create surrogate keys at all? Why would you not use the px transformer or the surrogate key generator?

[jwiles]

Hi,

When I use a BASIC transoformer in a parallel job, by default it runs in parallel on all the nodes as per the configuration file but it doesn't have partitionnum and numpartitions sytem varibales to generate a surrogate key. It only has @INROWNUM which is not sufficient when the job is running in parallel. Are there any altenatives to generate surrogate key in a BASIC transformer parallel job?

If you're using BASIC transformer in a parallel job solely to produce a surrogate key, you should seriously consider using either the parallel transformer or Surrogate Key Generator stages instead as has been suggested in this thread. This is especially true if you are dealing with moderate to high volumes of data as BASIC transformer will become a bottleneck to your processing speed in the parallel environment.

The parallel transformer has the partition number and partition count system variables you mention, but also can utilize Surrogate Key generator logic beginning with DS 8.0.

Regards,

[DS_TM]

Hello Everyone,
Just to be more precise about my requirement.

I have basic routines to run the in the transformer of a paralle job thus using BASIC transformer is must and also note that writing parallel routine is not in our scope at this moment.

While invoking routine for every row I need to generate a unique number in this BASIC transformer paralle job which runs on four node configuration. Also note that job can't be run on single node to generate the unique number though.

So, given all those conditions in my case, is there any way to generate a unique serial number in parallel basic transformer which runs on multiple node configuration?

[chulett]

There are many posts here on how to generate a unique surrogate key in a transformer using the various @PART related system variables. I supposed one could do the same by passing those values into a Server Shared Container that does the calculation there... but not sure you could do the same just using a BASIC Transformer.

[vmcburney]

You could drop the Surrogate Key Generator into your job. You need to get to know parallel routines via c++ and use those instead of BASIC routines for high volume jobs.