dsjob within ksh scripts
Posted: Tue Nov 29, 2011 1:05 pm
At our firmrun most of our DS jobs with one of a few ksh scripts.
There are a few standard audit-type steps that get executed before and after the main DS processing. These audit steps involve loading information to a few Oracle tables.
This process has been in place with very few problems for a few years. But lately, we've seen run times increase quite dramatically.
However, it's not the running of the datastage jobs themselves, but the wait time between them.
For example, as the last steps in one of our scripts, two jobs are called, one after the other, using dsjob.
The first is this:
DSCMD_BTCHAUDIT="$DSHOME/bin/dsjob -domain $Hostname -user $DsId -password $DsPwd -server $Hostname -run -wait -jobstatus $pBtchNm $pFileNm $pSysCde $pBtchNbr $pEffDt $pUserId $pBtchTyp $pBtchAuditSt $pStartTs $pEndTs $pEtlLdFileDir $pOdsDsn $pOdsId $pOdsPwd $ProjNm $BtchAuditJob"
Right after, is the second one:
DSCMD_INSETLREJ="$DSHOME/bin/dsjob -domain $Hostname -user $DsId -password $DsPwd -server $Hostname -run -wait -jobstatus $pBtchNm $pFileNm $pSysCde $pBtchNbr $pEffDt $pSrcFileDir $pLdFileDir $pTmpFileDir $pRejFileDir $pHashFileDir $pOdsDsn $pOdsId $pOdsPwd $ProjNm $InsEtlRejJob"
It's not unusual to see 15 - 20 minutes of wait time between the end of the first and the kickoff of the second.
One guess is that it may just be a system contstraint on our server, as I'm not able to replicate the issue in our Dev/UAT environment. But any suggestions as to something I could check would be greatly appreciated.
Thanks.
Tom
There are a few standard audit-type steps that get executed before and after the main DS processing. These audit steps involve loading information to a few Oracle tables.
This process has been in place with very few problems for a few years. But lately, we've seen run times increase quite dramatically.
However, it's not the running of the datastage jobs themselves, but the wait time between them.
For example, as the last steps in one of our scripts, two jobs are called, one after the other, using dsjob.
The first is this:
DSCMD_BTCHAUDIT="$DSHOME/bin/dsjob -domain $Hostname -user $DsId -password $DsPwd -server $Hostname -run -wait -jobstatus $pBtchNm $pFileNm $pSysCde $pBtchNbr $pEffDt $pUserId $pBtchTyp $pBtchAuditSt $pStartTs $pEndTs $pEtlLdFileDir $pOdsDsn $pOdsId $pOdsPwd $ProjNm $BtchAuditJob"
Right after, is the second one:
DSCMD_INSETLREJ="$DSHOME/bin/dsjob -domain $Hostname -user $DsId -password $DsPwd -server $Hostname -run -wait -jobstatus $pBtchNm $pFileNm $pSysCde $pBtchNbr $pEffDt $pSrcFileDir $pLdFileDir $pTmpFileDir $pRejFileDir $pHashFileDir $pOdsDsn $pOdsId $pOdsPwd $ProjNm $InsEtlRejJob"
It's not unusual to see 15 - 20 minutes of wait time between the end of the first and the kickoff of the second.
One guess is that it may just be a system contstraint on our server, as I'm not able to replicate the issue in our Dev/UAT environment. But any suggestions as to something I could check would be greatly appreciated.
Thanks.
Tom