Combinability mode. What is it used for?
Moderators: chulett, rschirm, roy
Combinability mode. What is it used for?
Can anyone explain? The help says that it saves significant amount of data copying and preparation in passing data between operators. Which operators are we talking about? Stages? How can we save data copying and preparation time?
Each stage can be of single operator or of many. So if you have n stages in a job, with m nodes, you will end up with n*m process running simulteneously.
There are some operators which can be combined during runtime. By allowing the combinability to true you make them to combine and hence, the required memory and resource for each individual process is combined to single.
I wont say if 2 process got combined, the requried cpu and memory will be 1/2, but will be less that the previous.
But you cannot expect very good log out of this, during any fault analysis.
So for any debuggin purpose it is advisable to turn if off.
I recall, there is a discussion on this by roy. Search.
In some cases it acts the other ways too.
There are some operators which can be combined during runtime. By allowing the combinability to true you make them to combine and hence, the required memory and resource for each individual process is combined to single.
I wont say if 2 process got combined, the requried cpu and memory will be 1/2, but will be less that the previous.
But you cannot expect very good log out of this, during any fault analysis.
So for any debuggin purpose it is advisable to turn if off.
I recall, there is a discussion on this by roy. Search.
In some cases it acts the other ways too.
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
Hi,
By default the stage operators can be combined. If you want to override for a specfic stage you can set the combinability mode for the stage to Dont Combine.
For Fault analysis you have to use the environment variable APT_DISABLE_COMBINATION to True and run the job. The job log will show exactly which stage is causing the problem.
HTH
--Rich
By default the stage operators can be combined. If you want to override for a specfic stage you can set the combinability mode for the stage to Dont Combine.
For Fault analysis you have to use the environment variable APT_DISABLE_COMBINATION to True and run the job. The job log will show exactly which stage is causing the problem.
HTH
--Rich
Stages is GUI term used in front end, operator term used in the underlying design of datastage.
For example, sort operator, funnel operator, sortfunnel operator... Can be explicitly available as a stage, or will be supplied by datastage along with other stages like, funnel, join, sequential file Stage, when called implictly.
You can refer Orchestrate Operators Reference for more detials.
For example, sort operator, funnel operator, sortfunnel operator... Can be explicitly available as a stage, or will be supplied by datastage along with other stages like, funnel, join, sequential file Stage, when called implictly.
You can refer Orchestrate Operators Reference for more detials.
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'