Help needed for FTP stage...

sagar deshmukh · Post by **sagar deshmukh** » Fri Jan 02, 2009 7:59 am

I have a job......

sequential stage---->FTP stage

what i need to do is.....

1) job should run sucessfully...
2)it should not post any black file to remote server if the input has no value....

is it possible?

chulett · Post by **chulett** » Fri Jan 02, 2009 8:33 am

Guessing "black" = "blank".

Sequence job. Use a routine to check for an empty file and skip the FTP job if needed. Notification for empty file or job failure.

srinagesh · Post by **srinagesh** » Fri Jan 02, 2009 8:34 am

write a script to Do a wc-l to count the number of lines, FTP the file if file exists and number of lines is greater than zero.

sagar deshmukh · Post by **sagar deshmukh** » Fri Jan 02, 2009 8:46 am

i have a job control for this where i have sequenced it...
can you please share a code how to check the input is null or not in this case?

chulett · Post by **chulett** » Fri Jan 02, 2009 9:10 am

You can use DSGetLinkInfo() to check how many records went down the creation link or DSExecute() for O/S commands to check the actual file: "wc -l", "test -s", etc.

sagar deshmukh · Post by **sagar deshmukh** » Fri Jan 02, 2009 9:59 am

Code: Select all


      DEFFUN RunRestartableJob(JobName,HASHPATH,BatchNumber,SKIPPARAM) Calling "DSU.RunRestartableJob"
      DEFFUN EmailErrorMessage(HASHPATH,BatchNumber) Calling "DSU.EmailErrorMessage"
      DEFFUN JobControlReset(JobList) Calling "DSU.JobControlReset"
      DEFFUN RunBatch(JobList, Params) Calling "DSU.RunBatch"
      DEFFUN ConvDateYYYYMMDD.DashDeltoMMDDYYYY(DATE) Calling "DSU.ConvDateYYYYMMDD.DashDeltoMMDDYYYY"
      DEFFUN GetHashValueByKey(File,Key) Calling "DSU.GetHashValueByKey"

      BatchStartDts = DSGetJobInfo(DSJ.ME, DSJ.JOBSTARTTIMESTAMP)

      ** Run hashCheckControlDTS outside of restart framework - want to run on restart **
      JobName = "hashCheckControlDTS"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      BatchConfig<1> = BatchName         ; * current job name or job run id
      BatchConfig<2> = 1                 ; * Maximum jobs to have running in parallel (in this batch)
      BatchConfig<3> = "S"               ; * P = parallel batch, S = serial batch
      BatchConfig<4> = "JOB"             ; * Job List Type (JOB=running jobs, BATCH=running batches)
      BatchConfig<5> = "RUN"             ; * run type (RUN=run jobs, RESET=reset jobs, VALIDATE=validate)

      CONVERT @AM TO "," IN BatchConfig

      Errcode = RunBatch(JobName,BatchConfig)
      If Errcode Then
         GOTO ENDJOB
      End
      ** End of hashCheckControlDTS ***


     JavaBatchDTS = GetHashValueByKey(HASHPATH:"hshCCDTS","")
      If Not (ConvDateYYYYMMDD.DashDeltoMMDDYYYY(JavaBatchDTS) = ConvDateYYYYMMDD.DashDeltoMMDDYYYY(BatchStartDts)) Then
         *abort job
        Call DSLogFatal("Batch ran failed! Invoice Java Batch Job has not completed.", BatchName)
      End

      JobName = "hashProcessControlDTS"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts)
      If Errcode Then
         GOTO ENDJOB
      End

      JobName = "srcDlyCalcTxns_LN"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts)
      If Errcode Then
      GOTO ENDJOB
      result = DSGetLinkInfo(DSJ.ME,DSJ.ME,DSJ.ME,DSJ.LINKROWCOUNT)
      End
      If  Result <> 0 Then
      
      
      
      StgFileName = "LNREMIT"
      JobName = "xfmDlyCalcTxns_LN"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts)
      If Errcode Then
         GOTO ENDJOB
      End

      StgFileName = "LiveNationRemit_US"
      RemitFileName = "LiveNationRemit_US_":ConvDateYYYYMMDD.DashDeltoMMDDYYYY(BatchStartDts):".csv"
      JobName = "ftpLIveNationRemitFile_US"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts:",REMITFILENAME=":RemitFileName:",STGFILENAME=":StgFileName)
      If Errcode Then
         GOTO ENDJOB
      End

      StgFileName = "LiveNationRemit_CA"
      RemitFileName = "LiveNationRemit_CA_":ConvDateYYYYMMDD.DashDeltoMMDDYYYY(BatchStartDts):".csv"
      JobName = "ftpLiveNationRemitFile_CA"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts:",REMITFILENAME=":RemitFileName:",STGFILENAME=":StgFileName)
      If Errcode Then
         GOTO ENDJOB
      End

      JobName = "Batch::LNRemitMaintenance"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts)
      If Errcode Then
         GOTO ENDJOB
      End
      
      
      
      JobName = "ldiLNRemitProcessControl"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts)
      If Errcode Then
         GOTO ENDJOB
      End


ENDJOB:
* Setup DailyPCJobStatsLdiNew, run it, wait for it to finish, and test for success
      JobName = "DailyPCJobStatsLdiNew.":BATCHNAME

      Result = JobControlReset(JobName)
      hJob1 = DSAttachJob(JobName, DSJ.ERRFATAL)
      If NOT(hJob1) Then
         Call DSLogFatal("Job Attach Failed: DailyPCJobStatsLdiNew", "JobControl")
         Abort
      End

      Errcode1 = DSSetParam(hJob1, "HASHPATH", HASHPATH)
      Errcode1 = DSSetParam(hJob1, "STGPATH", PROJDIR:STGPATH)
      Errcode1 = DSSetParam(hJob1, "BATCHNUMBER", BATCHNUMBER)
      Errcode1 = DSSetParam(hJob1, "SRCDBNAME", SRCDBNAME)
      Errcode1 = DSSetParam(hJob1, "SRCUSERID", SRCUSERID)
      Errcode1 = DSSetParam(hJob1, "SRCPWD", SRCPWD)
      Errcode1 = DSRunJob(hJob1, DSJ.RUNNORMAL)
      Errcode1 = DSWaitForJob(hJob1)
      Status = DSGetJobInfo(hJob1, DSJ.JOBSTATUS)
      If Status = DSJS.RUNFAILED Or Status = DSJS.CRASHED Then
         * Fatal Error - No Return
         Call DSLogWarn("Job Failed: DailyPCJobStatsLdiNew", "JobControl")

      End

      If Errcode then
         *  Result = JobControlReset(BatchName)

         Message = EmailErrorMessage(HASHPATH,BATCHNUMBER)
         Reply = DSSendMail("From:":MSG_FROM:"\nTo:":MSG_TO:"\nServer:":SMTPSERVER:"\nSubject:":BatchName:" failed\nBody:":Message:"\n")
         Call DSLogFatal("Batch ran failed! (see warnings)", BatchName)
      End

*Clear Hash File.

      OPENPATH HASHPATH:"/":'hshJobInfo':BATCHNUMBER To hashPath Then
         CLEARFILE hashPath ON ERROR Call DSLogInfo("Cannot Clear the Log",BatchName)

      End




this is my batch job code.... 

i need to run xfmDlyCalcTxns_LN,LiveNationRemit_US,LiveNationRemit_CA along with below two jobs....only and only if the records flows through 1st job.... 
else i have to run only "ldiLNRemitProcessControl" after checking 1st job...... 

i have modified the code bt its not working.....any reason? 

or nay changes i need to do?

chulett · Post by **chulett** » Fri Jan 02, 2009 10:31 am

Which bit is the modified bit and what's not working about it?

chulett · Post by **chulett** » Fri Jan 02, 2009 10:38 am

nm - revisit the online help for the DSGetLinkInfo syntax, it needs Job Handle + Stage and Link names. DSJ.ME is the handle to the job invoking the call, btw - i.e. your job control.

sagar deshmukh · Post by **sagar deshmukh** » Fri Jan 02, 2009 10:41 am

DSGetLinkInfo syntax i checked....as per that i have modified bold part...

JobName = "srcDlyCalcTxns_LN"
BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts)
If Errcode Then
GOTO ENDJOB
result = DSGetLinkInfo(DSJ.ME,DSJ.ME,DSJ.ME,DSJ.LINKROWCOUNT)
End
If Result <> 0 Then

chulett · Post by **chulett** » Fri Jan 02, 2009 10:49 am

Check again. You need to establish a handle to the "srcDlyCalcTxns_LN" job and not use DSJ.ME at all. The 2nd & 3rd arguments are Stage Name and Link Name respectively from the job the handle is attached to. And then detach the handle when you are done with it.

sagar deshmukh · Post by **sagar deshmukh** » Fri Jan 02, 2009 11:10 am

Code: Select all

      DEFFUN RunRestartableJob(JobName,HASHPATH,BatchNumber,SKIPPARAM) Calling "DSU.RunRestartableJob"
      DEFFUN EmailErrorMessage(HASHPATH,BatchNumber) Calling "DSU.EmailErrorMessage"
      DEFFUN JobControlReset(JobList) Calling "DSU.JobControlReset"
      DEFFUN RunBatch(JobList, Params) Calling "DSU.RunBatch"
      DEFFUN ConvDateYYYYMMDD.DashDeltoMMDDYYYY(DATE) Calling "DSU.ConvDateYYYYMMDD.DashDeltoMMDDYYYY"
      DEFFUN GetHashValueByKey(File,Key) Calling "DSU.GetHashValueByKey"

      BatchStartDts = DSGetJobInfo(DSJ.ME, DSJ.JOBSTARTTIMESTAMP)

      ** Run hashCheckControlDTS outside of restart framework - want to run on restart **
      JobName = "hashCheckControlDTS"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      BatchConfig<1> = BatchName         ; * current job name or job run id
      BatchConfig<2> = 1                 ; * Maximum jobs to have running in parallel (in this batch)
      BatchConfig<3> = "S"               ; * P = parallel batch, S = serial batch
      BatchConfig<4> = "JOB"             ; * Job List Type (JOB=running jobs, BATCH=running batches)
      BatchConfig<5> = "RUN"             ; * run type (RUN=run jobs, RESET=reset jobs, VALIDATE=validate)

      CONVERT @AM TO "," IN BatchConfig

      Errcode = RunBatch(JobName,BatchConfig)
      If Errcode Then
         GOTO ENDJOB
      End
      ** End of hashCheckControlDTS ***


     JavaBatchDTS = GetHashValueByKey(HASHPATH:"hshCCDTS","")
      If Not (ConvDateYYYYMMDD.DashDeltoMMDDYYYY(JavaBatchDTS) = ConvDateYYYYMMDD.DashDeltoMMDDYYYY(BatchStartDts)) Then
         *abort job
        Call DSLogFatal("Batch ran failed! Invoice Java Batch Job has not completed.", BatchName)
      End

      JobName = "hashProcessControlDTS"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts)
      If Errcode Then
         GOTO ENDJOB
      End

  [b]    JobName = "srcDlyCalcTxns_LN"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts)
      If Errcode Then
      GOTO ENDJOB
      result = DSGetLinkInfo(DSJ.JOBNAME,DSJ.xfmDailyCalcTxns,DSJ.Extract_DailyCalcTxns,DSJ.LINKROWCOUNT)
      End[/b]
      
      
      
     
      StgFileName = "LNREMIT"
      JobName = "xfmDlyCalcTxns_LN"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts)
      If Errcode Then
         GOTO ENDJOB
      End
     
     
      
      
      JobName = "ldiLNRemitProcessControl"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts)
      If Errcode Then
         GOTO ENDJOB
      End


ENDJOB:
* Setup DailyPCJobStatsLdiNew, run it, wait for it to finish, and test for success
      JobName = "DailyPCJobStatsLdiNew.":BATCHNAME

      Result = JobControlReset(JobName)
      hJob1 = DSAttachJob(JobName, DSJ.ERRFATAL)
      If NOT(hJob1) Then
         Call DSLogFatal("Job Attach Failed: DailyPCJobStatsLdiNew", "JobControl")
         Abort
      End

      Errcode1 = DSSetParam(hJob1, "HASHPATH", HASHPATH)
      Errcode1 = DSSetParam(hJob1, "STGPATH", PROJDIR:STGPATH)
      Errcode1 = DSSetParam(hJob1, "BATCHNUMBER", BATCHNUMBER)
      Errcode1 = DSSetParam(hJob1, "SRCDBNAME", SRCDBNAME)
      Errcode1 = DSSetParam(hJob1, "SRCUSERID", SRCUSERID)
      Errcode1 = DSSetParam(hJob1, "SRCPWD", SRCPWD)
      Errcode1 = DSRunJob(hJob1, DSJ.RUNNORMAL)
      Errcode1 = DSWaitForJob(hJob1)
      Status = DSGetJobInfo(hJob1, DSJ.JOBSTATUS)
      If Status = DSJS.RUNFAILED Or Status = DSJS.CRASHED Then
         * Fatal Error - No Return
         Call DSLogWarn("Job Failed: DailyPCJobStatsLdiNew", "JobControl")

      End

      If Errcode then
         *  Result = JobControlReset(BatchName)

         Message = EmailErrorMessage(HASHPATH,BATCHNUMBER)
         Reply = DSSendMail("From:":MSG_FROM:"\nTo:":MSG_TO:"\nServer:":SMTPSERVER:"\nSubject:":BatchName:" failed\nBody:":Message:"\n")
         Call DSLogFatal("Batch ran failed! (see warnings)", BatchName)
      End

*Clear Hash File.

      OPENPATH HASHPATH:"/":'hshJobInfo':BATCHNUMBER To hashPath Then
         CLEARFILE hashPath ON ERROR Call DSLogInfo("Cannot Clear the Log",BatchName)

      End

Code: Select all

I have made the code little bit short..can you please tell why its not running now....any suggestion

[/quote]

chulett · Post by **chulett** » Fri Jan 02, 2009 11:20 am

First tell us what 'not running' means, what issues you are seeing where. I'm not about to guess what errors you may or may not be getting, especially with all those custom routines.

sagar deshmukh · Post by **sagar deshmukh** » Fri Jan 02, 2009 11:31 am

following Error is coming:----Batch::LNRemitMaster_exp..JobControl (fatal error from Batch::LNRemitMaster_exp): Batch ran failed! (see warnings)

Code: Select all

Is is not at all calling  following part of code...just failing....and dont care abt the routines.....those are restartability routines

JobName = "srcDlyCalcTxns_LN"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts)
      If Errcode Then
      GOTO ENDJOB
      result = DSGetLinkInfo(DSJ.JOBNAME,DSJ.xfmDailyCalcTxns,DSJ.Extract_DailyCalcTxns,DSJ.LINKROWCOUNT)
      End

chulett · Post by **chulett** » Fri Jan 02, 2009 11:45 am

No idea what your error has to do with the code you posted but your call to DSGetLinkInfo will never be executed:

Code: Select all

If Errcode Then 
   GOTO ENDJOB 
   result = DSGetLinkInfo(DSJ.JOBNAME,DSJ.xfmDailyCalcTxns,DSJ.Extract_DailyCalcTxns,DSJ.LINKROWCOUNT) 
End

Does that formatting help show you why?

sagar deshmukh · Post by **sagar deshmukh** » Fri Jan 02, 2009 11:58 am

Sorry for that stupidity...

Code: Select all

JobName = "srcDlyCalcTxns_LN"
      BatchName = DSGetJobInfo(DSJ.ME, DSJ.JOBNAME)
      result = DSGetLinkInfo(DSJ.JOBNAME,DSJ.ME,DSJ.ME,DSJ.LINKROWCOUNT)
      Errcode = RunRestartableJob(JobName,HASHPATH,BATCHNUMBER,"BATCHTIMESTAMP=":BatchStartDts)
      If Errcode Then
      GOTO ENDJOB
      
      End
      
      
      Can you tell how to use This "RESULT" value for skiping below jobs?