I am running a Multi Instance Datastage in a loop by passing different parameters at run time sequencially using Unix Shell Script and DSJOB -run command. After running 4 to 5 iterations its gets out of loop and aborts. Does any one has any idea, is it related to some datastage resource issue.
Below is the output of Unix shell script and commands that I am using in Unix:
==================================================================
Script:
echo "Making entry in Audit table for below tables - CRZ"
echo "`grep "Loading data to table" $LocalAvscPath"controlfiles/bb5gsg_CRZ_Logs_"$SubGroup$Country.txt | cut -d " " -f8 | cut -d "_" -f6-7 --output-delimiter='|'`"
echo ""
echo "Starting Loop"
echo ""
for i in `grep "Loading data to table" $LocalAvscPath"controlfiles/bb5gsg_CRZ_Logs_"$SubGroup$Country.txt | cut -d " " -f8 | cut -d "_" -f6-7 --output-delimiter='|' |sort | uniq`
do
TABLANAME=`echo $i | cut -d'|' -f 1`
Version=`echo $i | cut -d'|' -f 2`
echo $TABLANAME
echo $Version
. /opt/IBM/InformationServer/Server/DSEngine/dsenv > /dev/null 2>&1
echo "/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=$Params -param ParamsHive_bb5gsg=$Params_Hive -param zone=$zone -param SubGroup=$SubGroup -param Country=$Country -param tablenm=$TABLANAME -param schemaVersion=$Version -param businessEffectiveDate=$businessEffectiveDate -param status=$status -param RecordCount=$RecordCount -param stream=$stream $Project $Job.$Group$SubGroup$CC$zone$TABLANAME"
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=$Params -param ParamsHive_bb5gsg=$Params_Hive -param zone=$zone -param SubGroup=$SubGroup -param Country=$Country -param tablenm=$TABLANAME -param schemaVersion=$Version -param businessEffectiveDate=$businessEffectiveDate -param status=$status -param RecordCount=$RecordCount -param stream=$stream $Project $Job.$Group$SubGroup$CC$zone$TABLANAME > /dev/null 2>&1
RETURNCODE=$?
if [ $RETURNCODE = 1 -o $RETURNCODE = 2 ]
then
echo "Job completed successfully"
echo ""
else
echo "Job aborting with Return Code :"$RETURNCODE
exit 3
fi;
sleep 3
done
echo "Entry in Audit table Finished - CRZ"
echo ""
=================================================================
Output of UNIX Shell Script:
Making entry in Audit table for below tables - TSZ
achbatch|2
adt495|1
fpm425|2
fpt010|2
fpt041|2
fpt043|2
fpx120|3
fpx362|2
fpx391|2
gpt340|2
ipt471|2
opm090|2
rpm090|4
Starting Loop
achbatch
2
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=PRD -param ParamsHive_bb5gsg=PRD -param zone=tsz -param SubGroup=S6 -param Country=do -param tablenm=achbatch -param schemaVersion=2 -param businessEffectiveDate=2019-09-04 -param status=S -param RecordCount=-1 -param stream=day BB5G_RDARR jbbb5gsgTSZAuditTableEntry.G3S6dotszachbatch
Job completed successfully
adt495
1
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=PRD -param ParamsHive_bb5gsg=PRD -param zone=tsz -param SubGroup=S6 -param Country=do -param tablenm=adt495 -param schemaVersion=1 -param businessEffectiveDate=2019-09-04 -param status=S -param RecordCount=-1 -param stream=day BB5G_RDARR jbbb5gsgTSZAuditTableEntry.G3S6dotszadt495
Job completed successfully
fpm425
2
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=PRD -param ParamsHive_bb5gsg=PRD -param zone=tsz -param SubGroup=S6 -param Country=do -param tablenm=fpm425 -param schemaVersion=2 -param businessEffectiveDate=2019-09-04 -param status=S -param RecordCount=-1 -param stream=day BB5G_RDARR jbbb5gsgTSZAuditTableEntry.G3S6dotszfpm425
Job completed successfully
fpt010
2
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=PRD -param ParamsHive_bb5gsg=PRD -param zone=tsz -param SubGroup=S6 -param Country=do -param tablenm=fpt010 -param schemaVersion=2 -param businessEffectiveDate=2019-09-04 -param status=S -param RecordCount=-1 -param stream=day BB5G_RDARR jbbb5gsgTSZAuditTableEntry.G3S6dotszfpt010
Job completed successfully
fpt041
2
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=PRD -param ParamsHive_bb5gsg=PRD -param zone=tsz -param SubGroup=S6 -param Country=do -param tablenm=fpt041 -param schemaVersion=2 -param businessEffectiveDate=2019-09-04 -param status=S -param RecordCount=-1 -param stream=day BB5G_RDARR jbbb5gsgTSZAuditTableEntry.G3S6dotszfpt041
Job completed successfully
fpt043
2
/opt/IBM/InformationServer/Server/DSEngine/bin/dsjob -run -jobstatus -warn 0 -param Params_bb5gsg=PRD -param ParamsHive_bb5gsg=PRD -param zone=tsz -param SubGroup=S6 -param Country=do -param tablenm=fpt043 -param schemaVersion=2 -param businessEffectiveDate=2019-09-04 -param status=S -param RecordCount=-1 -param stream=day BB5G_RDARR jbbb5gsgTSZAuditTableEntry.G3S6dotszfpt043
Job aborting with Return Code :255
Job aborting with Return Code :255
Moderators: chulett, rschirm, roy
Ask your datastage admin to look at the log purging setting in your project.
Under the Administrator tool, have him log in and tell you how the logs are purged.
There are two different settings. 1 is age based. Deleting log files or entries older than X amount of days.
The other is iteration based. Deleted log entries from jobs after N amount of runs. My guess is that this is what you are running into.
These log retention policies are set at a project level, not at the job level.
Under the Administrator tool, have him log in and tell you how the logs are purged.
There are two different settings. 1 is age based. Deleting log files or entries older than X amount of days.
The other is iteration based. Deleted log entries from jobs after N amount of runs. My guess is that this is what you are running into.
These log retention policies are set at a project level, not at the job level.
-
- Premium Member
- Posts: 425
- Joined: Sat Nov 19, 2005 9:26 am
- Location: New York City
- Contact:
How long it takes since the starting of the execution until it failed? It might be that the inactivity time out setting in DataStage Administrator is too low and the client connection that is running the job is closed... if this is the case then augmenting the value for the time out activity will do the trick
What happen if you take the command that is running the last job and run it manually, out of the loop, from the command line?
What happen if you take the command that is running the last job and run it manually, out of the loop, from the command line?
Julio Rodriguez
ETL Developer by choice
"Sure we have lots of reasons for being rude - But no excuses
ETL Developer by choice
"Sure we have lots of reasons for being rude - But no excuses