job running on single node
Moderators: chulett, rschirm, roy
job running on single node
Hi,
In confiuration file i have set two nodes.resource disk for first one i given d:/datastage/datasets,for second resorce disk for second node i given E:/datastage/datasets. my job contains
rowgenerator--->sort--->lookup(ref rowgeenrator)--->dataset
after executing that job on 2node but it showing singlenode.and i have set apt_dum_score is true in evironment variables.
but it showing like this
It runs 4 processes on 1 node.
what i have to do run in 2node. dataset also creating in first resource disk not created in second scratch disk
Thanks in Adavance
Vin...
In confiuration file i have set two nodes.resource disk for first one i given d:/datastage/datasets,for second resorce disk for second node i given E:/datastage/datasets. my job contains
rowgenerator--->sort--->lookup(ref rowgeenrator)--->dataset
after executing that job on 2node but it showing singlenode.and i have set apt_dum_score is true in evironment variables.
but it showing like this
It runs 4 processes on 1 node.
what i have to do run in 2node. dataset also creating in first resource disk not created in second scratch disk
Thanks in Adavance
Vin...
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
From your job log please post full details of:
(a) the configuration file entry
(b) the generated OSH
(c) the score
Please surround each with a set of Code tags.
(a) the configuration file entry
(b) the generated OSH
(c) the score
Please surround each with a set of Code tags.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Hi ray,
Please see this one
1.configfile
2.generated OSH
3.score
Thanks in Adavnce
vin..
Please see this one
1.configfile
Code: Select all
main_program: APT configuration file: C:/IBM/InformationServer/Server/Configurations/2node.apt
{
node "node1"
{
fastname "TOMEDW"
pools ""
resource disk "C:/IBM/InformationServer/Server/Datasets" {pools ""}
resource scratchdisk "C:/IBM/InformationServer/Server/Scratch" {pools ""}
}
}
{
node "node2"
{
fastname "TOMEDW"
pools ""
resource disk "E:/datastage/Datasets" {pools ""}
resource scratchdisk "C:/IBM/InformationServer/Server/Scratch" {pools ""}
}
Code: Select all
OSH script
# OSH / orchestrate script for Job Untitled3 compiled at 17:19:39 27 NOV 2009
#################################################################
#### STAGE: Row_Generator_0
## Operator
generator
## Operator options
-schema record
(
empid:nullable int32 {random={}};
ename:string[];
)
-records 10000000
## General options
[ident('Row_Generator_0'); jobmon_ident('Row_Generator_0'); par; nocombine]
## Outputs
0> [] 'Row_Generator_0:DSLink4.v'
;
#################################################################
#### STAGE: Row_Generator_1
## Operator
generator
## Operator options
-schema record
(
empid:nullable int32 {cycle={init=1, incr=1}};
)
-records 1000000
## General options
[ident('Row_Generator_1'); jobmon_ident('Row_Generator_1'); par; nocombine]
## Outputs
0> [] 'Row_Generator_1:DSLink5.v'
;
#################################################################
#### STAGE: Lookup_2
## Operator
lookup
## Operator options
-table
-key empid
-keep
-ifNotFound continue
## General options
[ident('Lookup_2'); jobmon_ident('Lookup_2'); nocombine]
## Inputs
0< [transfer( inputData excludes empid)] 'Sort_11:DSLink8.v'
1< [] 'Row_Generator_1:DSLink5.v'
## Outputs
0> [modify (
keep
empid;)] 'Lookup_2:DSLink6.v'
;
#################################################################
#### STAGE: Data_Set_3
## Operator
copy
## General options
[ident('Data_Set_3'); nocombine]
## Inputs
0< [] 'Lookup_2:DSLink6.v'
## Outputs
0>| [ds] 'D:\\test\\sas.ds'
;
#################################################################
#### STAGE: Sort_11
## Operator
tsort
## Operator options
-stable
-key 'empid'
-asc
## General options
[ident('Sort_11'); jobmon_ident('Sort_11'); nocombine]
## Inputs
0< [] 'Row_Generator_0:DSLink4.v'
## Outputs
0> [modify (
keep
empid,ename;
)] 'Sort_11:DSLink8.v'
;
# End of OSH code
Code: Select all
main_program: This step has 7 datasets:
ds0: {op0[1p] (parallel Row_Generator_0)
eOther(APT_HashPartitioner { key={ value=empid,
subArgs={ asc }
}
})->eCollectAny
op2[1p] (parallel Sort_11)}
ds1: {op1[1p] (parallel Row_Generator_1)
eEntire->eCollectAny
op3[1p] (parallel APT_LUTCreateOp in Lookup_2)}
ds2: {op2[1p] (parallel Sort_11)
[pp] eSame->eCollectAny
op4[1p] (parallel APT_LUTProcessOp in Lookup_2)}
ds3: {op3[1p] (parallel APT_LUTCreateOp in Lookup_2)
eEntire->eCollectAny
op4[1p] (parallel APT_LUTProcessOp in Lookup_2)}
ds4: {op3[1p] (parallel APT_LUTCreateOp in Lookup_2)
eAny->eCollectAny
op4[1p] (parallel APT_LUTProcessOp in Lookup_2)}
ds5: {op5[1p] (parallel delete data files in delete D:/test/sas.ds)
->eCollectAny
op6[1p] (sequential delete descriptor file in delete D:/test/sas.ds)}
ds6: {op4[1p] (parallel APT_LUTProcessOp in Lookup_2)
[pp] ->
D:/test/sas.ds}
It has 7 operators:
op0[1p] {(parallel Row_Generator_0)
on nodes (
node1[op0,p0]
)}
op1[1p] {(parallel Row_Generator_1)
on nodes (
node1[op1,p0]
)}
op2[1p] {(parallel Sort_11)
on nodes (
node1[op2,p0]
)}
op3[1p] {(parallel APT_LUTCreateOp in Lookup_2)
on nodes (
node1[op3,p0]
)}
op4[1p] {(parallel APT_LUTProcessOp in Lookup_2)
on nodes (
node1[op4,p0]
)}
op5[1p] {(parallel delete data files in delete D:/test/sas.ds)
on nodes (
node1[op5,p0]
)}
op6[1p] {(sequential delete descriptor file in delete D:/test/sas.ds)
on nodes (
node1[op6,p0]
)}
It runs 7 processes on 1 node.
vin..
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Premium Member
- Posts: 892
- Joined: Thu Oct 16, 2003 5:18 am
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Hi Ray,
I have changed Config file according to your instructions but its not running on two node.what trace mode at the time of compilation where i wil get that one.and if i changed dont Combinality mode but creating more datasets.what is the use of combinality mode in perfomance tunning..
Thanks
vin..
I have changed Config file according to your instructions but its not running on two node.what trace mode at the time of compilation where i wil get that one.and if i changed dont Combinality mode but creating more datasets.what is the use of combinality mode in perfomance tunning..
Thanks
vin..
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Combinability mode governs the extent to which operators will be combined into fewer processes. By default DataStage does this aggressively, which may be sub-optimal. But it is not relevant to the question running in this thread.
If you don't know where/how to compile in trace mode, then I daresay you have not done this, so would not worry about it.
Can you please advise what the setting of $APT_CONFIG_FILE is, but I specifically need the value extracted from the Director log of this job.
If you don't know where/how to compile in trace mode, then I daresay you have not done this, so would not worry about it.
Can you please advise what the setting of $APT_CONFIG_FILE is, but I specifically need the value extracted from the Director log of this job.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
in administrator i have set
APT_CONFIG_FILE as C:/IBM/InformationServer/Server/Configurations/default.apt
in job i have called as a job parameter and i have changed that one as a 2node.apt
[/code]
in this way i am getting in datastage director log regarding environmental variables
Thanks
Vin..
APT_CONFIG_FILE as C:/IBM/InformationServer/Server/Configurations/default.apt
in job i have called as a job parameter and i have changed that one as a 2node.apt
Code: Select all
Environment variable settings:
ALLUSERSPROFILE=C:\Documents and Settings\All Users
APT_COMPILEOPT=-W/TP -W/EHa -DAPT_USE_ANSI_IOSTREAMS -c
APT_COMPILER=cxx
APT_CONFIG_FILE=C:/IBM/InformationServer/Server/Configurations/2node.apt
APT_DUMP_SCORE=1
APT_ERROR_CONFIGURATION=severity, !vseverity, !jobid, moduleid, errorIndex, timestamp, !ipaddr, !nodeplayer, !nodename, opid, message
APT_LINKER=cxx
APT_LINKOPT=-s -W/dll -W/base:0x50000000
APT_MONITOR_MINTIME=10
APT_OPERATOR_REGISTRY_PATH=C:\IBM\InformationServer\Server\Projects\EBIA_v2\buildop
APT_ORCHHOME=C:/IBM/InformationServer/Server/PXEngine
APT_USE_CRLF=1
BELL=^G
CLASSPATH=C:\IBM\SQLLIB\java\db2java.zip;C:\IBM\SQLLIB\java\db2jcc.jar;C:\IBM\SQLLIB\java\sqlj.zip;C:\IBM\SQLLIB\java\db2jcc_license_cu.jar;C:\IBM\SQLLIB\bin;C:\IBM\SQLLIB\java\common.jar
in this way i am getting in datastage director log regarding environmental variables
Thanks
Vin..
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Try using backslashes ( \ ) in the pathname of the configuration file. I know Windows usually tolerates forward slashes, but please try.
Can you also please dump detail of the Director log entry that shows the configuration file contents?
Can you also please dump detail of the Director log entry that shows the configuration file contents?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Hi ray,
I have tried with back slash also it showing same.
i have attached detailed log of job.
Parallel job initiated
Parallel job default NLS map ASCL_MS1252, default locale OFF
Thanks
Vin..
I have tried with back slash also it showing same.
i have attached detailed log of job.
Code: Select all
Starting Job Partest.
$APT_CONFIG_FILE = C:\IBM\InformationServer\Server\Configurations\2node.apt
Code: Select all
Environment variable settings:
ALLUSERSPROFILE=C:\Documents and Settings\All Users
APT_COMPILEOPT=-W/TP -W/EHa -DAPT_USE_ANSI_IOSTREAMS -c
APT_COMPILER=cxx
APT_CONFIG_FILE=C:\IBM\InformationServer\Server\Configurations\2node.apt
APT_DUMP_SCORE=1
APT_ERROR_CONFIGURATION=severity, !vseverity, !jobid, moduleid, errorIndex, timestamp, !ipaddr, !nodeplayer, !nodename, opid, message
APT_LINKER=cxx
APT_LINKOPT=-s -W/dll -W/base:0x50000000
APT_MONITOR_MINTIME=10
APT_OPERATOR_REGISTRY_PATH=C:\IBM\InformationServer\Server\Projects\EBIA_v2\buildop
APT_ORCHHOME=C:/IBM/InformationServer/Server/PXEngine
APT_USE_CRLF=1
BELL=^G
CLASSPATH=C:\IBM\SQLLIB\java\db2java.zip;C:\IBM\SQLLIB\java\db2jcc.jar;C:\IBM\SQLLIB\java\sqlj.zip;C:\IBM\SQLLIB\java\db2jcc_license_cu.jar;C:\IBM\SQLLIB\bin;C:\IBM\SQLLIB\java\common.jar
ClusterLog=C:\WINDOWS\Cluster\cluster.log
COMPUTERNAME=TOMATO
CommonProgramFiles=C:\Program Files\Common Files
ComSpec=C:\WINDOWS\system32\cmd.exe
DB2INSTANCE=DB2INST
DISPLAY=:0.0
DS_ENABLE_RESERVED_CHAR_CONVERT=0
DS_OPERATOR_BUILDOP_DIR=buildop
DS_OPERATOR_WRAPPED_DIR=wrapped
DS_TDM_PIPE_OPEN_TIMEOUT=720
DS_TDM_TRACE_SUBROUTINE_CALLS=0
DS_USERNO=-1740
DSIPC_OPEN_TIMEOUT=30
FLAVOR=-1
FP_NO_HOST_CHECK=NO
INCLUDE=C:\IBM\SQLLIB\INCLUDE;C:\IBM\SQLLIB\LIB;C:\Program Files\Microsoft Visual Studio .NET 2003\SDK\v1.1\include\
LIB=C:\Program Files\Microsoft Visual Studio .NET 2003\SDK\v1.1\Lib\;C:\IBM\SQLLIB\LIB
MAN_CHM_INDEX=C:/PROGRA~1/MKSTOO~1/etc/chm/tkutil.idx;C:/PROGRA~1/MKSTOO~1/etc/chm/tkapi.idx;C:/PROGRA~1/MKSTOO~1/etc/chm/tcltk.idx;C:/PROGRA~1/MKSTOO~1/etc/chm/tkcurses.idx
MAN_HTM_PATHS=C:/PROGRA~1/MKSTOO~1/etc/htm/perl;C:/PROGRA~1/MKSTOO~1/etc/htm/perl/pod;C:/PROGRA~1/MKSTOO~1/etc/htm/perl/ext;C:/PROGRA~1/MKSTOO~1/etc/htm/perl/lib
MAN_TXT_INDEX=C:/PROGRA~1/MKSTOO~1/etc/tkutil.idx;C:/PROGRA~1/MKSTOO~1/etc/tkapi.idx;C:/PROGRA~1/MKSTOO~1/etc/tcltk.idx;C:/PROGRA~1/MKSTOO~1/etc/tkcurses.idx
NUMBER_OF_PROCESSORS=2
NUTCROOT=C:\PROGRA~1\MKSTOO~1
OS=Windows_NT
OSH_STDOUT_MSG=1
PATH=C:\IBM\InformationServer\Server\PXEngine\bin;C:\IBM\InformationServer\Server\PXEngine\osh_wrappers;C:\IBM\InformationServer\Server\PXEngine\user_osh_wrappers;C:\IBM\InformationServer\Server\PXEngine\user_osh_wrappers;C:\IBM\InformationServer\Server\DSComponents\bin;C:\IBM\InformationServer\Server\DSParallel;C:\IBM\InformationServer\Server\Projects\EBIA_v2\RT_BP385.O;C:\IBM\InformationServer\Server\DSEngine\bin;C:\IBM\InformationServer\Server\Projects\EBIA_v2\buildop;C:\IBM\InformationServer\Server\Projects\EBIA_v2\wrapped;C:\oracle\product\10.1.0\db_1\bin;C:\PROGRA~1\MKSTOO~1\mksnt;C:\PROGRA~1\MKSTOO~1\bin;C:\PROGRA~1\MKSTOO~1\bin\X11;C:\PROGRA~1\MKSTOO~1\mksnt;C:\IBM\InformationServer\ASBNode\lib\cpp;C:\IBM\InformationServer\ASBNode\apps\proxy\cpp\vc60\MT_dll\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Intel\DMIX;C:\IBM\SQLLIB\BIN;C:\IBM\SQLLIB\FUNCTION;C:\Program Files\Java\jdk1.5.0_22\bin;C:\Program Files\Java\jdk1.5.0_22\lib;C:\Program Files\Java\jre1.5.0_22\bin;C:\Program Files\Java\jre1.5.0_22\lib;C:\IBM\SQLLIB\SAMPLES\REPL;C:\IBM\SQLLIB\BIN;C:\Program Files\Common Files\DivX Shared\;C:\Program Files\MySQL\MySQL Server 5.0\bin
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.sh;.ksh;.csh;.sed;.awk;.pl
PERL5LIB=C:\oracle\product\10.1.0\db_1\perl\lib\5.6.1\MSWin32-x86;C:\oracle\product\10.1.0\db_1\perl\lib\5.6.1;C:\oracle\product\10.1.0\db_1\perl\5.6.1\lib\MSWin32-x86;C:\oracle\product\10.1.0\db_1\perl\site\5.6.1;C:\oracle\product\10.1.0\db_1\perl\site\5.6.1\lib;C:\oracle\product\10.1.0\db_1\sysman\admin\scripts
PIDTOK=7260
PROCESSOR_ARCHITECTURE=x86
PROCESSOR_IDENTIFIER=x86 Family 15 Model 4 Stepping 7, GenuineIntel
PROCESSOR_LEVEL=15
PROCESSOR_REVISION=0407
ProgramFiles=C:\Program Files
PWD=C:\IBM\InformationServer\Server\Projects\EBIA_v2
PX_DBCONNECTHOME=C:/IBM/InformationServer/Server/DSComponents
ROOTDIR=C:/PROGRA~1/MKSTOO~1
SHELL=C:/PROGRA~1/MKSTOO~1/mksnt/sh.exe
SystemDrive=C:
SystemRoot=C:\WINDOWS
TEMP=C:\WINDOWS\TEMP
TERM=console
TERMCAP=C:\PROGRA~1\MKSTOO~1\etc\termcap
TERMINFO=C:\PROGRA~1\MKSTOO~1\usr\lib\terminfo
TMP=C:\WINDOWS\TEMP
UNIVERSE_CONTROLLING_TERM=1
UNIVERSE_PARENT_PROCESS=7260
USER=TOMATO\Administrator
USERPROFILE=C:\Documents and Settings\Default User
VS71COMNTOOLS=C:\Program Files\Microsoft Visual Studio .NET 2003\Common7\Tools\
WHO=EBIA_v2
windir=C:\WINDOWS
Code: Select all
OSH script
# OSH / orchestrate script for Job Partest compiled at 11:47:47 30 NOV 2009
#################################################################
#### STAGE: Row_Generator_0
## Operator
generator
## Operator options
-schema record
(
empid:nullable int32 {random={}};
ename:string[];
)
-records 200
## General options
[ident('Row_Generator_0'); jobmon_ident('Row_Generator_0'); par]
## Outputs
0> [] 'Row_Generator_0:DSLink4.v'
;
#################################################################
#### STAGE: Row_Generator_1
## Operator
generator
## Operator options
-schema record
(
empid:nullable int32 {cycle={init=1, incr=1}};
)
-records 200
## General options
[ident('Row_Generator_1'); jobmon_ident('Row_Generator_1'); par]
## Outputs
0> [] 'Row_Generator_1:DSLink5.v'
;
#################################################################
#### STAGE: Lookup_2
## Operator
lookup
## Operator options
-table
-key empid
-keep
-ifNotFound continue
## General options
[ident('Lookup_2'); jobmon_ident('Lookup_2')]
## Inputs
0< [transfer( inputData excludes empid)] 'Sort_11:DSLink8.v'
1< [] 'Row_Generator_1:DSLink5.v'
## Outputs
0> [modify (
keep
empid;)] 'Lookup_2:DSLink6.v'
;
#################################################################
#### STAGE: Data_Set_3
## Operator
copy
## General options
[ident('Data_Set_3')]
## Inputs
0< [] 'Lookup_2:DSLink6.v'
## Outputs
0>| [ds] 'D:\\test\\sas.ds'
;
#################################################################
#### STAGE: Sort_11
## Operator
tsort
## Operator options
-stable
-key 'empid'
-asc
## General options
[ident('Sort_11'); jobmon_ident('Sort_11')]
## Inputs
0< [] 'Row_Generator_0:DSLink4.v'
## Outputs
0> [modify (
keep
empid,ename;
)] 'Sort_11:DSLink8.v'
;
# End of OSH code
Code: Select all
main_program: APT configuration file: C:/IBM/InformationServer/Server/Configurations/2node.apt
{
node "node1"
{
fastname "TOMATO"
pools ""
resource disk "C:/IBM/InformationServer/Server/Datasets" {pools ""}
resource scratchdisk "C:/IBM/InformationServer/Server/Scratch" {pools ""}
}
}
{
node "node2"
{
fastname "TOMATO"
pools ""
resource disk "E:/datastage/Datasets" {pools ""}
resource scratchdisk "C:/IBM/InformationServer/Server/Scratch" {pools ""}
}
}
Code: Select all
main_program: This step has 4 datasets:
ds0: {op0[1p] (parallel delete data files in delete D:/test/sas.ds)
->eCollectAny
op1[1p] (sequential delete descriptor file in delete D:/test/sas.ds)}
ds1: {op2[1p] (parallel APT_CombinedOperatorController(1):APT_LUTCreateOp in Lookup_2)
eEntire->eCollectAny
op3[1p] (parallel APT_CombinedOperatorController(0):APT_LUTProcessOp in Lookup_2)}
ds2: {op2[1p] (parallel APT_CombinedOperatorController(1):APT_LUTCreateOp in Lookup_2)
eAny->eCollectAny
op3[1p] (parallel APT_CombinedOperatorController(0):APT_LUTProcessOp in Lookup_2)}
ds3: {op3[1p] (parallel APT_CombinedOperatorController(0):APT_LUTProcessOp in Lookup_2)
[pp] ->
D:/test/sas.ds}
It has 4 operators:
op0[1p] {(parallel delete data files in delete D:/test/sas.ds)
on nodes (
node1[op0,p0]
)}
op1[1p] {(sequential delete descriptor file in delete D:/test/sas.ds)
on nodes (
node1[op1,p0]
)}
op2[1p] {(parallel APT_CombinedOperatorController(1):
(Row_Generator_1)
(APT_LUTCreateOp in Lookup_2)
) on nodes (
node1[op2,p0]
)}
op3[1p] {(parallel APT_CombinedOperatorController(0):
(Row_Generator_0)
(Sort_11)
(APT_LUTProcessOp in Lookup_2)
) on nodes (
node1[op3,p0]
)}
It runs 4 processes on 1 node.
Vin..