Hi,
When I use a BASIC transoformer in a parallel job, by default it runs in parallel on all the nodes as per the configuration file but it doesn't have partitionnum and numpartitions sytem varibales to generate a surrogate key. It only has @INROWNUM which is not sufficient when the job is running in parallel. Are there any altenatives to generate surrogate key in a BASIC transformer parallel job?
Generating Surrogate Key In Basic Transformer Parallel Job
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 437
- Joined: Fri Oct 21, 2005 10:00 pm
Why use the basic transformer to create surrogate keys at all? Why would you not use the px transformer or the surrogate key generator?
Keith Williams
keith@peacefieldinc.com
keith@peacefieldinc.com
Re: Generating Surrogate Key In Basic Transformer Parallel J
If you're using BASIC transformer in a parallel job solely to produce a surrogate key, you should seriously consider using either the parallel transformer or Surrogate Key Generator stages instead as has been suggested in this thread. This is especially true if you are dealing with moderate to high volumes of data as BASIC transformer will become a bottleneck to your processing speed in the parallel environment.DS_TM wrote:Hi,
When I use a BASIC transoformer in a parallel job, by default it runs in parallel on all the nodes as per the configuration file but it doesn't have partitionnum and numpartitions sytem varibales to generate a surrogate key. It only has @INROWNUM which is not sufficient when the job is running in parallel. Are there any altenatives to generate surrogate key in a BASIC transformer parallel job?
The parallel transformer has the partition number and partition count system variables you mention, but also can utilize Surrogate Key generator logic beginning with DS 8.0.
Regards,
- james wiles
All generalizations are false, including this one - Mark Twain.
All generalizations are false, including this one - Mark Twain.
Re: Generating Surrogate Key In Basic Transformer Parallel J
Hello Everyone,
Just to be more precise about my requirement.
I have basic routines to run the in the transformer of a paralle job thus using BASIC transformer is must and also note that writing parallel routine is not in our scope at this moment.
While invoking routine for every row I need to generate a unique number in this BASIC transformer paralle job which runs on four node configuration. Also note that job can't be run on single node to generate the unique number though.
So, given all those conditions in my case, is there any way to generate a unique serial number in parallel basic transformer which runs on multiple node configuration?
Just to be more precise about my requirement.
I have basic routines to run the in the transformer of a paralle job thus using BASIC transformer is must and also note that writing parallel routine is not in our scope at this moment.
While invoking routine for every row I need to generate a unique number in this BASIC transformer paralle job which runs on four node configuration. Also note that job can't be run on single node to generate the unique number though.
So, given all those conditions in my case, is there any way to generate a unique serial number in parallel basic transformer which runs on multiple node configuration?
There are many posts here on how to generate a unique surrogate key in a transformer using the various @PART related system variables. I supposed one could do the same by passing those values into a Server Shared Container that does the calculation there... but not sure you could do the same just using a BASIC Transformer.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
-
- Participant
- Posts: 3593
- Joined: Thu Jan 23, 2003 5:25 pm
- Location: Australia, Melbourne
- Contact:
You could drop the Surrogate Key Generator into your job. You need to get to know parallel routines via c++ and use those instead of BASIC routines for high volume jobs.
Certus Solutions
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn
Blog: Tooling Around in the InfoSphere
Twitter: @vmcburney
LinkedIn:Vincent McBurney LinkedIn