DataStage Basic Routine behaves different in 8.7...

kaps · Post by **kaps** » Fri Jun 01, 2012 11:58 am

We have a routine (wrote in Basic) which works correct in 7.5 but not in 8.7. Are there any changes to the DataStage Basic in the newer version ? This rouutine reads a Master Mainframe EBCDIC file and breaks it into multiple files with base and riders.

We are in the process of upgrading from 7.5 Parallel to 8.7 and during the testing we found the routine creates files completely different from older version. Different meaning file size and even number of files it produces is different. Following is the routine.

Code: Select all

***DSLogInfo will display to Log in Director
      Call DSLogInfo("Processing Started ":TIMEDATE(),"JobControl")
      Call DSLogInfo("Process Parameters ":ProjectCategoryParmFile,"JobControl")
***Establish Parameter Values
      prm_project_path =FIELD(ProjectCategoryParmFile,';',1,1)
      prm_path_category=FIELD(ProjectCategoryParmFile,';',2,1)
      prm_source       =FIELD(ProjectCategoryParmFile,';',3,1)

****Exit on missing parameter value
      var_tmp="Parameter Missing: "
      IF prm_project_path="" OR ISNULL(prm_project_path)
      THEN Call DSLogFatal(var_tmp:"Project Path","JobControl")
      ELSE IF prm_path_category="" OR ISNULL(prm_path_category)
      THEN Call DSLogFatal(var_tmp:"Category Path","JobControl")
      ELSE IF prm_source="" OR ISNULL(prm_source)
      THEN Call DSLogFatal(var_tmp:"Source File Name","JobControl");

****Read Parameter File
      var_FileName=prm_project_path:prm_path_category:FIELD(prm_source,'.',1,1):"_MFP.prm"
      OPENSEQ var_FileName TO var_InFile ELSE Call DSLogFatal(var_FileName:" Parameter File OPEN ERROR","JobControl")
      var_EOF=0
      LOOP
         READSEQ var_InRec FROM var_InFile ELSE var_EOF=1
      UNTIL var_EOF
         IF var_InRec = "" OR var_InRec[1,1]="*" OR var_InRec[1,1]="#" ELSE
           var_param=FIELD(var_InRec,'=',1,1)
           var_value=FIELD(var_InRec,'=',2,1)
           BEGIN CASE
            CASE var_param = "prm_BaseLen"
               prm_BaseLen = var_value
            CASE var_param = "prm_BaseIdStart"
               prm_BaseIdStart = var_value
            CASE var_param = "prm_BaseIdLen"
               prm_BaseIdLen = var_value
            CASE var_param = "prm_KeyStart"
               prm_KeyStart = var_value
            CASE var_param = "prm_KeyLen"
               prm_KeyLen = var_value
            CASE var_param = "prm_SkipRecs"
               prm_SkipRecs = var_value
            CASE var_param = "prm_BaseIds"
               prm_BaseIds = var_value
            CASE var_param = "prm_TrailerIds"
               prm_TrailerIds = var_value
            CASE var_param = "prm_TrailerLens"
               prm_TrailerLens = var_value
            CASE var_param = "prm_TrailLenByteLen"
               prm_TrailLenByteLen = var_value
            CASE var_param = "prm_RecLen"
               prm_RecLen  = var_value
            CASE 1
               Call DSLogInfo("Unknown File Parameter ":var_param,"JobControl")
           END CASE
         END;
      REPEAT
      CLOSESEQ var_InFile

****Exit on missing or unassigned parameter value
      var_tmp="Parameter is Missing from Parameter File: "
      IF UNASSIGNED(prm_BaseLen) OR prm_BaseLen=""
      THEN Call DSLogFatal(var_tmp:"prm_BaseLen","JobControl")
      ELSE IF UNASSIGNED(prm_BaseIdStart) OR prm_BaseIdStart=""
      THEN Call DSLogFatal(var_tmp:"prm_BaseIdStart","JobControl")
      ELSE IF UNASSIGNED(prm_BaseIdLen) OR prm_BaseIdLen=""
      THEN Call DSLogFatal(var_tmp:"prm_BaseIdLen","JobControl")
      ELSE IF UNASSIGNED(prm_KeyStart) OR prm_KeyStart=""
      THEN Call DSLogFatal(var_tmp:"prm_KeyStart","JobControl")
      ELSE IF UNASSIGNED(prm_KeyLen) OR prm_KeyLen=""
      THEN Call DSLogFatal(var_tmp:"prm_KeyLen","JobControl")
      ELSE IF UNASSIGNED(prm_SkipRecs) OR prm_SkipRecs=""
      THEN Call DSLogFatal(var_tmp:"prm_SkipRecs","JobControl")
      ELSE IF UNASSIGNED(prm_BaseIds) OR prm_BaseIds=""
      THEN Call DSLogFatal(var_tmp:"prm_BaseIds","JobControl")
      ELSE IF UNASSIGNED(prm_TrailerIds) OR prm_TrailerIds=""
      THEN Call DSLogFatal(var_tmp:"prm_TrailerIds","JobControl")
      ELSE IF UNASSIGNED(prm_TrailerLens) OR prm_TrailerLens=""
      THEN Call DSLogFatal(var_tmp:"prm_TrailerLens","JobControl")
      ELSE IF UNASSIGNED(prm_TrailLenByteLen) OR prm_TrailLenByteLen=""
      THEN Call DSLogFatal(var_tmp:"prm_TrailLenByteLen","JobControl")
      ELSE IF UNASSIGNED(prm_RecLen)
      THEN prm_RecLen=0;

***If prm_RecLen is zero then use Record Descriptor Word(RDW) else process as fixed width file.
      var_RDW = (prm_RecLen=0)
***Add CR/LF (Windows) or LF (UNIX) length for fixed length format file.
      var_PhyRecLen=prm_RecLen + IF (System(91)) THEN 2 ELSE 1 

***Open Input File
      var_FileName=prm_project_path:prm_path_category:prm_source
      OPENSEQ var_FileName TO var_InFile ELSE Call DSLogFatal(var_FileName:" OPEN ERROR","JobControl")

***Skip Input File Records (prm_SkipRecs)
      var_EOF=0
      var_ReadCNT=0
      LOOP WHILE var_ReadCNT < prm_SkipRecs AND NOT(var_EOF)
****First two bytes for RDW are the RDW (length) in binary
         IF var_RDW THEN
            GET RDW,2 FROM var_InFile ELSE var_EOF=1
*****Binary conversion
            var_RecLen = (SEQ(RDW[1,1]) * 256) + SEQ(RDW[2,1])
*****Second two bytes could store extra data for the file
            GET var_ExtraData,2 FROM var_InFile ELSE var_EOF=1
*****Subtract 4 for the RDW on VB files
            var_RecLen = var_RecLen - 4
         END
         ELSE
            var_RecLen = var_PhyRecLen
         END
         GET INRECORD,var_RecLen FROM var_InFile ELSE var_EOF=1
         var_ReadCNT += 1
      REPEAT
      var_SkipCNT = var_ReadCNT

***Determine File Info Array Size
***Base file info is in array dimension 1 and trailer info is in 2
      DIM var_FileCNT(2)
      var_MaxCNT=0
      FOR I = 1 TO 2
         var_FileCNT(I) = IF I=1 THEN DCOUNT(prm_BaseIds,';') ELSE DCOUNT(prm_TrailerIds,';')
         IF var_FileCNT(I)=0 THEN var_FileCNT(I)=1
         IF var_FileCNT(I) > var_MaxCNT THEN var_MaxCNT = var_FileCNT(I)
      NEXT I

***Dimension File Handle & Count Arrays
      DIM var_FileHandle(2,var_MaxCNT), var_RecCNT(2,var_MaxCNT)
      MAT var_RecCNT=0

***Establish File Names & Handles Values
      var_FileName = prm_project_path:prm_path_category:prm_source
      FOR I = 1 TO 2
         var_FNPrefix = FIELD(var_FileName,'.',1,1):IF I=1 THEN "_Base" ELSE "_Trail"
         var_FNSuffix = FIELD(var_FileName,'.',2,1)
         var_FileIds  = IF I=1 THEN prm_BaseIds ELSE prm_TrailerIds
         FOR J=1 TO var_FileCNT(I)
            var_FNFull = var_FNPrefix:FIELD(FIELD(var_FileIds,';',J,1),',',1,1):'.':var_FNSuffix
            Call DSLogInfo("File Name(":I:",":J:") is ":var_FNFull,"JobControl")
            OPENSEQ var_FNFull TO var_FileHandle(I,J) THEN NAP 2
         NEXT J
      NEXT I
***Create Array to work with FIND statement
      var_BaseIds  = CONVERT(';,',@FM:@VM,prm_BaseIds)
      var_TrailIds = CONVERT(';,',@FM:@VM,prm_TrailerIds)

***Loop through Input File
      var_EOF=0
      LOOP
         IF var_RDW THEN
****First two bytes for RDW are the RDW (length) in binary
            GET RDW,2 FROM var_InFile ELSE var_EOF=1
*****Binary conversion
            var_RecLen = (SEQ(RDW[1,1]) * 256) + SEQ(RDW[2,1])
*****Second two bytes could store extra data for the file
            GET var_ExtraData,2 FROM var_InFile ELSE var_EOF=1
*****Subtract 4 for the RDW on VB files
            var_RecLen= var_RecLen - 4
         END
         ELSE
            var_RecLen= var_PhyRecLen
         END
         GET var_InRec,var_RecLen FROM var_InFile ELSE var_EOF=1
      UNTIL var_EOF
         IF Not(var_RDW) THEN var_RecLen=prm_RecLen
***Capture Base Information
         var_ReadCNT += 1
         var_BaseRec = var_InRec[1, prm_BaseLen]
         var_BaseKey = var_InRec[prm_KeyStart, prm_KeyLen]
         var_BaseNum = 1
         IF prm_BaseIdStart # 0 THEN
            var_BaseId = ASCII(var_InRec[prm_BaseIdStart,prm_BaseIdLen])
            FIND var_BaseId IN var_BaseIds SETTING var_BaseNum
               ELSE var_BaseNum=var_FileCNT(1)
         END
         WRITESEQ var_BaseRec TO var_FileHandle(1,var_BaseNum)
            ELSE Call DSLogFatal("Write Base ":var_BaseId:" Failed","JobControl")
         var_RecCNT(1,var_BaseNum) += 1
!For Debugging
!var_EOF = (var_RecCNT(1,var_BaseNum) > 9)
         var_CurPos = prm_BaseLen + 1
***Loop through Trailers
         LOOP WHILE var_CurPos < var_RecLen
            var_TrlLen = 0
            FOR I=0 TO (prm_TrailLenByteLen - 1)
               var_TrlLen += SEQ(var_InRec[var_CurPos + I, 1])
            NEXT I
            BEGIN CASE
               CASE var_TrlLen > 0 AND ((var_CurPos + var_TrlLen - 1) <= var_RecLen)
***In case of a valid Trailer
                  var_TrailRec = var_InRec[var_CurPos + prm_TrailLenByteLen, var_TrlLen - prm_TrailLenByteLen]
                  IF var_TrailRec # EBCDIC(STR(' ',var_TrlLen-prm_TrailLenByteLen)) THEN
                     var_TrailId = ASCII(var_TrailRec[1,1])
                     FIND var_TrailId IN var_TrailIds SETTING var_TrailNum
                     ELSE
                        var_TrailId = ASCII(var_TrailRec[1,2])
                        FIND var_TrailId IN var_TrailIds SETTING var_TrailNum
                           ELSE
                              var_TrailNum=var_FileCNT(2)
                           END
                     END
                     var_TrailRec = var_BaseKey:var_TrailRec
                     var_PadLen = FIELD(prm_TrailerLens,';',var_TrailNum,1) - LEN(var_TrailRec)
                     var_TrailRec := EBCDIC(STR(' ',var_PadLen))
                     WRITESEQ var_TrailRec TO var_FileHandle(2,var_TrailNum)
                        ELSE Call DSLogFatal("Write Trailer ":var_TrailId:" Failed","JobControl")
                     var_RecCNT(2,var_TrailNum) += 1
                  END
                  var_CurPos += var_TrlLen
               CASE 1
                  var_CurPos = var_RecLen
            END CASE
         REPEAT
      REPEAT

***Shutdown Process & Display Information
      Call DSLogInfo("Processing Complete ":TIMEDATE(),"JobControl")
      Call DSLogInfo("Read Count is ":var_ReadCNT,"JobControl")
      Call DSLogInfo("Skip Count is ":var_SkipCNT,"JobControl")
      FOR I=1 TO 2
         FOR J=1 TO var_FileCNT(I)
            CLOSESEQ var_FileHandle(I,J)
            var_Msg = IF I=1 THEN "Base" ELSE "Trailer":" "
            var_Msg := FIELD(FIELD(IF I=1 THEN var_BaseIds ELSE var_TrailIds,@FM,J,1),@VM,1,1)
            var_Msg := " Count is ":var_RecCNT(I,J)
            Call DSLogInfo(var_Msg,"JobControl")
         NEXT J
      NEXT I
      CLOSESEQ var_InFile

ErrorCode = 0      ;* set this to non-zero to stop the stage/job

Please let me know if you see something in this routine which will cause issues in new version.

Appreciate your time..

Thanks

chulett · Post by **chulett** » Fri Jun 01, 2012 1:56 pm

Looks much more better with 'code' tags rather than 'quote' tags, yes?

ray.wurlod · Post by **ray.wurlod** » Fri Jun 01, 2012 4:26 pm

Please explain how this behaved in the previous version and how you allege it behaves differently in the new version.

chulett · Post by **chulett** » Fri Jun 01, 2012 5:40 pm

And the more specifics you provide, the better. Saying that the "file size and even number of files it produces is different" doesn't really help us help you. Examples, both before and after, generally don't hurt as well.

kaps · Post by **kaps** » Mon Jun 04, 2012 5:00 pm

I have an input file which I have passed to the same routine in older version and newer version and both generate different sets of output files with different file sizes. I don't think it will be useful to paste the content of the files as it is still in EBCDIC format not ascii. There is another process which converts the files to ascii using cosort but we have not moved forward as we suspecting there is a problem in the parsing.

I am not sure what else I can provide. Please let me know if you need anything specific.

Thanks Ray and Craig.