Force Sequential mode
Moderators: chulett, rschirm, roy
Force Sequential mode
Hi my query is about force sequence mode in Job Parameters
The Job structure
Seq file----------> Transformer-------------------->Seq file
I ran the above job in 3 ways
1) Transformer in sequence -> Got Output A
2) Transformer in Parallel -> Got Output B
3) Force sequence mode enabled and Transformer Parallel -> Got Output B
I am clear with design 1) and 2)
But in design 3) since the force sequence mode is enabled why it didnt make the job to run sequentially. Ideally i should have got output A(though the tansformer in parallel) due to Force Sequential mode. Why it is returning output B instead of output A
Kindly help on this as this would clarify my various doubts
The Job structure
Seq file----------> Transformer-------------------->Seq file
I ran the above job in 3 ways
1) Transformer in sequence -> Got Output A
2) Transformer in Parallel -> Got Output B
3) Force sequence mode enabled and Transformer Parallel -> Got Output B
I am clear with design 1) and 2)
But in design 3) since the force sequence mode is enabled why it didnt make the job to run sequentially. Ideally i should have got output A(though the tansformer in parallel) due to Force Sequential mode. Why it is returning output B instead of output A
Kindly help on this as this would clarify my various doubts
-
- Premium Member
- Posts: 1735
- Joined: Thu Mar 01, 2007 5:44 am
- Location: Troy, MI
How are "Output A" and "Output B" different? Different order? Different number of rows? Different content?
Are you doing any sort of multi-row processing using stage variables in your transform stage or are you passing the data directly from input to output (with some transformations)?
The only thing that could be different is the ordering, if you sort A and sort B and then compare the files they will be identical regardless of how many partitions you use (assuming you are doing a 1:1 transfer in the transform stage.
Are you doing any sort of multi-row processing using stage variables in your transform stage or are you passing the data directly from input to output (with some transformations)?
The only thing that could be different is the ordering, if you sort A and sort B and then compare the files they will be identical regardless of how many partitions you use (assuming you are doing a 1:1 transfer in the transform stage.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
Input file:
empno
10
20
30
40
50
The Stage variables i used in Transformer
StageVar+ empno --- > StagegVar1
empno ----> StageVar (Intial value is 10)
Im taking output StageVar1
1) Transformer Sequentially output:
empno StageVar1
10,20
20,30
30,50
40,70
50,90
2) Transformer Parallel output:
10,20
50,70
40,50
30,40
20,30
The output will be obviously diff as the initial value set as 10 for second variable is node specific.
Kindly clear on the 3rd design
empno
10
20
30
40
50
The Stage variables i used in Transformer
StageVar+ empno --- > StagegVar1
empno ----> StageVar (Intial value is 10)
Im taking output StageVar1
1) Transformer Sequentially output:
empno StageVar1
10,20
20,30
30,50
40,70
50,90
2) Transformer Parallel output:
10,20
50,70
40,50
30,40
20,30
The output will be obviously diff as the initial value set as 10 for second variable is node specific.
Kindly clear on the 3rd design
Do you mean "Compile in Trace Mode" and then "Force Sequential Mode"?
I think that this isn't what you want, you need to go to the transform stage in the Designer, do Stage Properties -> Advanced -> and set the execution mode to "sequential".
I think that this isn't what you want, you need to go to the transform stage in the Designer, do Stage Properties -> Advanced -> and set the execution mode to "sequential".
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
ArndW,
You are right. I wanted to run the transformer sequentially to achive desired output.
I thought of searching for some thing in job level to make the job run sequentially.
But now i find that Force sequential mode in Job properties and Execution mode set to sequence in transformer are different.
Could you explain what is the difference between both. Is there any way to make the same work in Job level as well.
You are right. I wanted to run the transformer sequentially to achive desired output.
I thought of searching for some thing in job level to make the job run sequentially.
But now i find that Force sequential mode in Job properties and Execution mode set to sequence in transformer are different.
Could you explain what is the difference between both. Is there any way to make the same work in Job level as well.
I haven't used that debug/trace method, so haven't looked into detail at what it does "under the covers". Generally one should have a parameter to all parallel jobs for the APT_CONFIG_FILE and by using a 1-node configuration you can ensure that stages run sequentially. I don't know if there is a DSPARAMS global setting in addition to the APT_CONFIG_FILE which will force sequential mode, I kind of doubt it exists and I certainly haven't come across it.
Did you use the stage-level force sequential mode and did it work? If yes & yes you could close this thread off a resolved.
Did you use the stage-level force sequential mode and did it work? If yes & yes you could close this thread off a resolved.
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>
The Job Parameters don't contain that, the job properties do.
According to help, the sequential setting applies only to the conductor node. I couldn't find more information on the trace mode, but it probably applies to the c++ compiler and loads in more stack- and variable- tracing during compile and linking (allowing use of dbx and perhaps more details truss output).
According to help, the sequential setting applies only to the conductor node. I couldn't find more information on the trace mode, but it probably applies to the c++ compiler and loads in more stack- and variable- tracing during compile and linking (allowing use of dbx and perhaps more details truss output).
<a href=http://www.worldcommunitygrid.org/team/ ... TZ9H4CGVP1 target="WCGWin">
</a>
</a>