BATCH JOB & CSV File

karthi_gana · Post by **karthi_gana** » Sun Mar 18, 2012 11:11 pm

All,

We are using the server job to execute the parallel jobs which we designed.

The server job will contain only the annotation. We declared a parameter called "ParameterName" (may be any name) and assigned the path of .CSV file which internally have the job names as below.

Code: Select all

Job Name	Controlling Predecessors	Batch
TH_00_FTP_GET_SHARE		BATCH_MJC_PROCESS_XXX_FEEDS
JB_01_XXX_SHARE_LOAD	TH_00_FTP_GET_SHARE	BATCH_MJC_PROCESS_XXX_FEEDS
JB_02_XXX_SHARE_LOAD	JB_01_XXX_SHARE_LOAD	BATCH_MJC_PROCESS_XXX_FEEDS
JB_03_ETF_SHARE_LOAD	JB_02_XXX_SHARE_LOAD	BATCH_MJC_PROCESS_XXX_FEEDS
JB_01_XXX_PERF_LOAD_FA	JB_03_ETF_SHARE_LOAD	BATCH_MJC_PROCESS_XXX_FEEDS
JB_02_XXX_PERF_LOAD_FB	JB_01_XXX_PERF_LOAD_FA	BATCH_MJC_PROCESS_XXX_FEEDS
JB_03_XXX_PERF_LOAD_FC	JB_02_XXX_PERF_LOAD_FB	BATCH_MJC_PROCESS_XXX_FEEDS
JB_04_XXX_PERF_LOAD_F9	JB_03_XXX_PERF_LOAD_FC	BATCH_MJC_PROCESS_XXX_FEEDS
JB_05_XXX_PERF_LOAD	JB_04_XXX_PERF_LOAD_F9	BATCH_MJC_PROCESS_XXX_FEEDS
JB_06_ETF_PERF_LOAD	JB_05_XXX_PERF_LOAD	BATCH_MJC_PROCESS_XXX_FEEDS

All the above are parallel jobs.

How the server job is executing the .csv file?

ray.wurlod · Post by **ray.wurlod** » Sun Mar 18, 2012 11:31 pm

You tell us. It's your job!

Based on your description I would guess that it's using job control code. What does the annotation tell you?

karthi_gana · Post by **karthi_gana** » Mon Mar 19, 2012 1:51 am

yes. You are right!

Code: Select all

   DEFFUN KBAJobControl(A) Calling "DSU.KBAJobControl"
      DEFFUN KBAGetParameterArray(A) Calling 'DSU.KBAGetParameterArray'
      DEFFUN KBAGetParam(A,B) Calling "DSU.KBAGetParam"
      

      
      SystemType = (IF SYSTEM(91) = 0 THEN "UNIX" ELSE "NT")

      ParameterArray = KBAGetParameterArray(ParameterFile)

*
*  Retrieve the next batch processing id
*

      If BatchNumber = "" Then BatchNumber = @DATE:@TIME'R%5'
      If ProcessDate = "1000-01-01" Then ProcessDate = Oconv(@DATE,"D-YMD[4,2,2]")



*
*  Derive any runtime parameters not found in parameter file
*
        ExtraParameters = ""
           
*      ExtraParameters<-1> = "DerivedAtRuntimeParameter1=":Result1
*      ExtraParameters<-1> = "DerivedAtRuntimeParameter2=":Result2
*      ExtraParameters<-1> = "DerivedAtRuntimeParameter3=":Result3
      CONVERT @AM TO @VM IN ExtraParameters
*
*  Configure job control array
*
      JobControlArray = ""
      JobControlArray<1> = ParameterFile
      JobControlArray<2> = 15                ; * How many seconds delay between alarms (minutes * 60 seconds)
      JobControlArray<3> = 30                ; * How many seconds a job can run until the alarm rings(minutes * 60 seconds)
      JobControlArray<4> = 2                 ; * Polling interval in seconds
      JobControlArray<5> = 99                ; * Maximum jobs to have running in parallel (in this batch)
      JobControlArray<6> = "S"               ; * (S) Stop entire jobstream on first job failure (P) Proceed after job failure, truncating that particular sub-jobstream
      JobControlArray<7> = JobHierarchyFile  ; * Job hierarchy file
      JobControlArray<8> = ProcessDate       ; * Common date for all jobs
      JobControlArray<9> = BatchNumber       ; * Batch number
      JobControlArray<10> = SourceSystemList           ; * Source system list (optional)
      JobControlArray<11> = SubjectAreaList  ; * Subject area list (optional)
      JobControlArray<12> = StartingMilestone          ; * Starting milestone (optional)
      JobControlArray<13> = EndingMilestone  ; * Ending milestone (optional)
      JobControlArray<14> = DebugMode        ; * Y = extra output messages
      JobControlArray<15> = JobLinkStatisticChecksFile
      JobControlArray<16> = ExtraParameters  ; * Extra parameters not within the parameter file (optional subarray - @VM separated)
      JobControlArray<17> = ResurrectLogFile

      JobFailedCount = KBAJobControl(JobControlArray)
      If JobFailedCount > 0 Then 
          Call DSLogWarn("Jobs failed", "Msg")
          Abort
      End 
     Else 
         Call DSLogInfo("Jobs finished okay", "Msg")
         vRemoveCmd = "rm -f "
         aLogMsgs = "Deleting Resurrect Log File"
         vCmd = vRemoveCmd:ResurrectLogFile
         vSystemType = "UNIX"
         Call DSExecute(vSystemType, vRemoveCmd:ResurrectLogFile, pScreenOutput, pSystemReturnCode)
         Call DSLogInfo(vLogID:aLogMsgs:ResurrectLogFile:pScreenOutput:pSystemReturnCode:vCmd, vLogID)
      End

i don't understand this code at all. which is the key part to execute the jobs one by one ?

ray.wurlod · Post by **ray.wurlod** » Mon Mar 19, 2012 3:10 am

You have obtained a copy of Ken Bland Associates job control routines. Therefore, you should also have obtained a copy of the documentation that comes with it.
Track that down on your site, or you can find Ken's website by searching DSXchange.

The "key part" is the invocation of KBAJobControl() function.

karthi_gana · Post by **karthi_gana** » Tue Mar 20, 2012 1:53 am

Sure.

Now...If i don't have KBA and if i want to implement the same in parallel job itself instead of using server job with KBA routines...how should i achieve this?

chulett · Post by **chulett** » Tue Mar 20, 2012 5:29 am

You wouldn't.

ray.wurlod · Post by **ray.wurlod** » Tue Mar 20, 2012 2:54 pm

You couldn't (unless you created a couple of custom stage types and/or routines).

chulett · Post by **chulett** » Tue Mar 20, 2012 3:50 pm

There is a crap-ton of stuff under the covers that the 'KBA Utilities' encompasses... and I really don't think you could do anything equivalent as a straight parallel implementation. Even with a couple of custom stage types and/or routines.

ray.wurlod · Post by **ray.wurlod** » Tue Mar 20, 2012 5:10 pm

Also, quite manifestly, you DO have at least some of KBA utilities.

chulett · Post by **chulett** » Tue Mar 20, 2012 5:20 pm

Of course, but I assumed the question was can that all be pitched and replaced with strictly 'parallel' components... at least that's where my response was based on.

ray.wurlod · Post by **ray.wurlod** » Tue Mar 20, 2012 8:46 pm

... and that's just one of the reasons server jobs aren't going away any time soon.