I have Datastage Job which does Null Validation, Duplicate Validation and finally loads into Dataset. Which takes around 50MINS to complete(27 million Rows). I would like to improve the performance can some one advise on?
ODBC Stage --> Transformer(Validation) Sort--> Transformer(Duplicate Removal)-->Datastage.
Apart from these, i have validation fail links writes into Datastage.
We have 2 nodes
Below are the environment Variables used in project level:
Environment variable settings:
APT_COMPILEOPT=/D APT_USE_ANSI_IOSTREAMS /D _WIN32 /D _MBCS /nologo /W3 /WX- /Gm- /EHa /MD /GS- /fp:precise /Zc:wchar_t- /Zc:forScope /Gd /TP /Zi /Oy- /c
APT_COMPILER=cl
APT_CONFIG_FILE=E:/IBM/InformationServer/Server/Configurations/default.apt
APT_DEFAULT_TRANSPORT_BLOCK_SIZE=13107200
APT_DISABLE_COMBINATION=1
APT_ERROR_CONFIGURATION=severity, !vseverity, !jobid, moduleid, errorIndex, timestamp, !ipaddr, !nodeplayer, !nodename, opid, message
APT_IO_MAP=1
APT_IO_NOMAP=1
APT_LINKER=link
APT_LINKOPT=/INCREMENTAL:NO /NOLOGO /DLL /DEBUG /SUBSYSTEM:CONSOLE /DYNAMICBASE:NO /MACHINE:X86
APT_MAX_DELIMITED_READ_SIZE=204800
APT_MAX_TRANSPORT_BLOCK_SIZE=104857600
APT_MONITOR_MINTIME=10
APT_NO_IOCOMM_OPTIMIZATION=1
APT_NO_ONE_NODE_COMBINING_OPTIMIZATION=1
APT_OPERATOR_REGISTRY_PATH=K:\DS_Projects\projects\international\INTL_DEV\buildop
APT_ORCHHOME=E:/IBM/InformationServer/Server/PXEngine
APT_PHYSICAL_DATASET_BLOCK_SIZE=9000000
APT_USE_CRLF=1
APT_USE_IPV4=1
DB2INSTANCE=DB2
DS_ENABLE_RESERVED_CHAR_CONVERT=0
DS_OPERATOR_BUILDOP_DIR=buildop
DS_OPERATOR_WRAPPED_DIR=wrapped
DS_OPTIMIZE_FILE_BROWSE=0
DS_PX_RESET=1
DS_TDM_PIPE_OPEN_TIMEOUT=720
DS_TDM_TRACE_SUBROUTINE_CALLS=0
DS_USERNO=-4024
ISUSER=ChalamN
NUTCROOT=C:\PROGRA~2\MKSTOO~1
TMP=C:\Windows\TEMP
UNIVERSE_CONTROLLING_TERM=1
UNIVERSE_PARENT_PROCESS=13308
USER=DSTAGE\dsadm
USERDOMAIN=WORKGROUP
USERNAME=DSTAGE$