Problem in PL/I Ebcdic File Read Using Parallel Seq Stage

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
ravivarma20
Participant
Posts: 22
Joined: Tue Sep 28, 2004 9:58 am

Problem in PL/I Ebcdic File Read Using Parallel Seq Stage

Post by ravivarma20 »

Hi,

I am trying to read a EBCDIC file using Parallel Sequential Stage (you can read Ebcdic files using this stage also). I specified Under general properties Character set as Ebcdic and Dataformat as 'Binary'.

1. The table definition for this file is in PL/I format.
2. My Ebcdic file has fixed length records, no field or record delimiters.
3. Each record is of 84 bytes size. It's throwing some errors when I try to read. It's saying " Formatted length of fields and record (106) greater than declared length (84) ". But my PL/I structure says that record length is 84 only.
4. When I specify record length as 106 it's reading the data but not in correct format and it's reading less no.of records actually my ebcdic file has 20 records.
4. The file doesn't have any packed fields. Just have comp,comp1 and pic x(1) fields.

PL/I Structure:

Code: Select all

/************* THIS IS PLSTRUC.UPCOUT ********************************/
DCL 1 UPCOUT,    /* UPC SELECT UNIFIED OUTPUT RECORD */
   5 FSP_UNITS          BIN FLOAT(21), /* LOYALTY SHOPPER UNITS      */
   5 UPC,
     6 SYSTEM           BIN FIXED(15),
     6 GENERATION       BIN FIXED(15),
     6 VENDOR           BIN FIXED(31),
     6 ITEM             BIN FIXED(31),
   5 FME_PRI_UNITS      BIN FIXED(15), /* PRIMARY PRICE MULTI_UNITS  */
   5 FME_SEC_UNITS      BIN FIXED(15), /* SCNDARY PRICE MULTI_UNITS  */
   5 WEEK               BIN FIXED(15),
   5 OLD_STORE          BIN FIXED(15), /* STORE NUMBER IF THE NUMBER */
                                       /* IS LESS THAN 32767. IF THE */
                                       /* THE NUMBER IS > 32767, THIS*/
                                       /* VALUE IS SET TO -1.        */
                                       /* 10/27/92 PRJTC             */
   5 UNITS              BIN FLOAT(21),
   5 CENTS              BIN FLOAT(21), /* TOTAL CENTS PAID           */
   5 STUBNO             BIN FIXED(31), /* STUB POINTER               */
   5 COUPON_ORIGIN      BIN FIXED(15), /* VALID FOR PURCHASE ONLY    */
   5 FME_PRI_FLAGS,                    /* ENHANCED FEATURE FLAGS     */
     6 PICTURE          BIT(1),        /* HAS A PICTURE              */
     6 LOCATION1        BIT(1),        /* LOCATION1                  */
     6 LOCATION2        BIT(1),        /* LOCATION2                  */
     6 PRICE_MULTI      BIT(1),        /* HAS A PRICE_MULTIPLE       */
     6 MULTI_SIZE       BIT(1),        /* MULTI_SIZE ITEMS           */
     6 LIMITS           BIT(1),        /* LIMITED TO A MAXIMUM       */
     6 MUST_BUY         BIT(1),        /* MUST_BUY                   */
     6 BOGO             BIT(1),        /* BUY ONE GET ONE            */
   5 FME_SEC_FLAGS,                    /* ENHANCED FEATURE FLAGS     */
     6 PICTURE          BIT(1),        /* HAS A PICTURE              */
     6 LOCATION1        BIT(1),        /* LOCATION1                  */
     6 LOCATION2        BIT(1),        /* LOCATION2                  */
     6 PRICE_MULTI      BIT(1),        /* HAS A PRICE_MULTIPLE       */
     6 MULTI_SIZE       BIT(1),        /* MULTI_SIZE ITEMS           */
     6 LIMITS           BIT(1),        /* LIMITED TO A MAXIMUM       */
     6 MUST_BUY         BIT(1),        /* MUST_BUY                   */
     6 BOGO             BIT(1),        /* BUY ONE GET ONE            */
   5 COUPON_FACE_VALUE  BIN FIXED(15), /* COUPON FACE VALUE, THIS    */
                                       /* FIELD IS DEFINED FOR       */
                                       /* PURCHASE ONLY.  IT IS NOT  */
                                       /* YET SUPPORTED.             */
   5 COUPON_VALUE       BIN FIXED(15), /* REDEMPTION VALUE           */
                                       /* REGISTER IF SHOPPERS TOTAL */
   5 PERCENT_OFF        BIN FIXED(15), /* TRADE REDUCTION PERCENTAGE */
   5 FEATURE_CODES      BIN FIXED(15),
   5 DISPLAY_CODES      BIN FIXED(15),
   5 DEAL,
     6 MDEAL            BIT(1),        /* MFG DEAL, PURCHASE ONLY    */
     6 SDEAL            BIT(1),        /* STORE DEAL                 */
     6 FDEAL            BIT(1),        /* FEATURE DEAL               */
     6 DDEAL            BIT(1),        /* DISPLAY DEAL               */
     6 RDEAL            BIT(1),        /* PRICE REDUCTION            */
     6 PFDEAL           BIT(1),        /* FEATURE DEAL IN PREV WEEK  */
     6 PDDEAL           BIT(1),        /* DISPLAY DEAL IN PREV WEEK  */
     6 GIFTLINK         BIT(1),        /* THIS IS A GIFTLINK RECORD  */
                                       /* THIS IS ALSO THE EDLP FLAG */
   5 FLAG,
     6 TEST_ITEM        BIT(1),
     6 PLU_XLATED       BIT(1),
     6 NOT_ON_FILE      BIT(1),
     6 FAKE_DATA        BIT(1),
     6 PROMO_AGG        BIT(1),        /* BASELINED IN PROMO AGG     */
                                       /* IN SPECIALPAK              */
     6 DELETED          BIT(1),        /* RECORD FLAGGED AS DELETED  */
     6 FEAT_CODE_MOD    BIT(1),        /* FEATURE CODE MOD FLAG      */
     6 LDEAL            BIT(1),        /* LOYALTY SHOPPER FLAG       */
   5 FSP_DOLLARS        BIN FLOAT(21), /* LOYALTY SHOPPER DOLLARS    */
   5 IDENT              BIN FIXED(15),
   5 FOOTPRINT,                        /* FOOTPRINT BIT MAP          */
     6 UPCMOD           BIT(1),        /* A POST UPCLOAD MOD HAS     */
                                       /*   BEEN APPLIED             */
     6 DICT_PRICE       BIT(1),        /* THE DICTIONARY PRICE WAS   */
                                       /*   USED FOR THIS ITEM       */
     6 DATAMOD          BIT(1),        /* THIS ITEM WAS MODIFIED     */
                                       /*   DURING DATA PRODUCTION   */
     6 IMPUTE           BIT(1),        /* THE VOLUME OF THIS ITEM    */
                                       /*   SHOULD BE IMPUTED (IT    */
                                       /*     WAS A FAILING SUBTRACT)*/
     6 PLU_XLATED       BIT(1),        /* THIS ITEM HAD ITS UPC      */
                                       /*   NUMBER TRANSLATED        */
     6 DICT_HICONE      BIT(1),        /* THIS ITEM WAS HICONE-      */
                                       /*   MODIFIED IN DICT PROC.   */
     6 FEAT_DISP        BIT(1),        /* THIS IS A FEATURE/DISPLAY  */
                                       /*   PSEUDO-MOVEMENT RECORD   */
     6 COMBINE          BIT(1),        /* THIS RECORD WAS MADE UP BY */
                                       /*   MORE THAN ONE INPUT RECS */
     6 SPLIT            BIT(1),        /* THIS RECORD WAS MADE UP BY */
                                       /*   SPLITTING AN INPUT RECORD*/
     6 SKIP_BASE        BIT(1),        /* DO NOT UPDATE THE BASELINE */
                                       /*   DATA SET FROM THIS RECORD*/
     6 FEAT_PRICE       BIT(1),        /* STORE PRICE WAS REPLACED   */
                                       /*   WITH PRICE FROM FEAT FILE*/
     6 PLU_ALLOCATED    BIT(1),        /* RECORD WAS CREATED FROM THE*/
                                       /* REALLOCATION OF SYSTEM 7   */
                                       /*   MOVEMENT                 */
     6 SUBTRACT         BIT(1),        /* RECORD WAS PROCESSED BY    */
                                       /*   SUBTRACT STORE LOGIC     */
     6 SUBTR_RESET      BIT(1),        /* POSSIBLE RESET CONDITION   */
                                       /*   FOR SUBTRACT STORES      */
     6 BASE_OUTLIER     BIT(1),        /* USED TO CAPTURE ITEMS THAT */
                                       /* WENT THROUGH BASEWK OUTLIER*/
                                       /* LOGIC                      */
     6 DELIVERY_ADD     BIT(1),        /* UNITS BUMPED UP BY NEGATIVE*/
                                       /* BUMPING ALGORITHM          */
   5 BASE_UNITS         BIN FLOAT(21),
   5 BASE_CENTS         BIN FLOAT(21),
   5 STORE              BIN FIXED(31), /* STORE NUMBER. THIS VALUE   */
                                       /* IS ALWAYS VALID, EVEN IF   */
                                       /* THE NUMBER IS > 32767      */
                                       /* 10/27/92 PRJTC             */
   5 FEATURE_CODES_NEW  BIN FIXED(15),
   5 SOURCE             BIN FIXED(15), /* NEW FIELD WLM */
   5 LOW_RPT_DOLLARS    BIN FLOAT(21), /* LOWEST REPORTED DOLLARS    */
   5 PRE_FSP_DOLLARS    BIN FLOAT(21); /* PRE FSP DOLARS             */
 /************* END OF PLSTRUC.UPCOUT ********************************/
BIN FIXED 15 - 2 Byte integer
BIN FIXED 31 - 4 Byte integer
BIN FLOAT 21 -4 Byte (float)
BIN BIT - 1 Bit

If you need more infomation I can email u the .dsx and Sample Ebcdic file.

Any help would be greatly appreciated..........

Thanks,
Ravi.
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

Ravi,

declare the record length as 106 by adding a char(18) dummy column to the end of your existing Pl/1 imported declaration. Run the job and output to a delimited sequential file. Then go through the columns one-by-one until you find the location where your imported declaration and the data begin to differ. Then if the difference isn't clear post the resultant problem onto this thread.
ravivarma20
Participant
Posts: 22
Joined: Tue Sep 28, 2004 9:58 am

reading bit fields

Post by ravivarma20 »

ArndW,

I manipulate the metadata to refelect 106 as u said. After importing metadata while reading it's asking for 122. Since I added 22 more bytes to 84 byte record. I think there is some padding is going on while DataStage reads 84 byte record. I am working with mainframe guys on this.

I have a probelm in reading bit fields from my EBCDIC file. See if you can help me . For example I have a char field in PL/I Struc i.e nothing but a 8 - bit field.
5 FME_PRI_FLAGS, /* ENHANCED FEATURE FLAGS */
6 PICTURE BIT(1), /* HAS A PICTURE */
6 LOCATION1 BIT(1), /* LOCATION1 */
6 LOCATION2 BIT(1), /* LOCATION2 */
6 PRICE_MULTI BIT(1), /* HAS A PRICE_MULTIPLE */
6 MULTI_SIZE BIT(1), /* MULTI_SIZE ITEMS */
6 LIMITS BIT(1), /* LIMITED TO A MAXIMUM */
6 MUST_BUY BIT(1), /* MUST_BUY */
6 BOGO BIT(1), /* BUY ONE GET ONE */
after importing PL/I structure the above fields become

Code: Select all

cloumnname           SQL type    Length    NULLABLE
FME_PRI_FLAGS                CHAR           1           NO
How should I map these bit fields among 8 different fields, without loosing it's original value?

Do I need to do any bit manipulation on the CHAR field after importing ?

Thnaks for ur time........ Ravi.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

You could always extract them using bitwise AND function BitAnd (see page B-4 of Parallel Job Developer's Guide for details)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

Another way to do this, in addition to Ray's suggestion, is to declare the bit-level fields as CHAR (of appropriate length) and then use the OCONV(In.BitString,"MB0C") to get a longer character string with each character being a 0 or a 1.
ravivarma20
Participant
Posts: 22
Joined: Tue Sep 28, 2004 9:58 am

Post by ravivarma20 »

Issue 1 :

As it is parallel job I am taking Ray's approach. Is this the way I should use BitAnd logical operator in my job to get the bit values.

Ex:

Code: Select all

BitAnd("My Char input field value",128) - field 8
      BitAnd("My Char input field value",64) - field 7
      BitAnd("My Char input field value",32) - field 6
      BitAnd("My Char input field value",16) - field 5
      BitAnd("My Char input field value",8)  - field 4
      BitAnd("My Char input field value",4) - field 3
      BitAnd("My Char input field value",2)  - field 2
      BitAnd("My Char input field value",1)   - field 1
After doing ebcdic conversion my char field should contain numeric vaule to use this BitAnd operator otherwise it won't work right?

Issue 2 :

Regarding the problem i discussed starting of the thread - my parallel Seq stage reading the 84 byte ebcdic record as a 106 byte record. I checked with the mainframe guys they said it's generating the ebcdic format correctly and even there is a C program i.e converting Ebcdic to Ascii working fine without any issues. So I am confused about the way Sequential stage reading. When I try reading the record as 106 byte almost all the fields it's reading incorrectly.

I tried to debug by reading the ASCII equivalent (Generated by C Program) of Ebcdic file and tried to export it as a EBCDIC . To my suprise it is wrting 106 bytes per record. If I try to view the data in Ebcdic file i.e generated by me, using Seq Stage it's viewing data properly.

Any ideas........

Thanks for ur time guys.
Ravi.
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

I think that DataStage is padding the Bit fields to a byte boundary and thus getting an incorrect length computation. If this data is EDCDIC then you will want to ensure that the data is NOT being converted to ASCII as this would change all the bit flags.

I would define this file as having just one column and reading it without any EBCDIC/ASCII conversion. In a transform I would then go about dissecting this string into it's components using the In.BigColumn[StartPos,EndPos] function and then specifically converting using the ASCII() function.

There is one bit group which is not 2 bytes long, but I am fairly certian that the PL/1 definition forces internal padding to maintain byte boundaries for subsequent groups/fields.
ravivarma20
Participant
Posts: 22
Joined: Tue Sep 28, 2004 9:58 am

Post by ravivarma20 »

That's a smart idea. But unfortunately, I can't try that option because it's a parallel job. Parallel job transformer doesn't have a ASCII to Ebcdic Conversion funtion. We have to take care this conversion at stage level itself.

Ravi.
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

Ravi,

you can change/override the conversion settings at a column level; so for the columns that are binary you just change the "import EBCDIC as ASCII". Just click on the column definition, right-mouse and "edit row" in the sequential file stage to get the screen where you can adjust the settings as desired.
ravivarma20
Participant
Posts: 22
Joined: Tue Sep 28, 2004 9:58 am

Post by ravivarma20 »

Whether you set at column level or Stage level. It's giving me the same error... saying that 84 byte to 106 byte. Because it's using the same Metadata definition every time.

As you said there is some padding is going on when DataStage try to read PL/1 structure. I am running out of ideas to work around this issue.

I am thinking this should be a bug in interperting PL/1 structure files in DataStage. We are planing to open a ticket with ascential support as a last option.

Ravi.
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

Ravi,

I don't think that you will have much luck opening a ticket, didn't you get a warning message regarding an unsupported datatype - that would account for the length issue and as DataStage internally only works at byte boundaries it will not be easily solved.

I think the earlier posts stated that it will not work, you need to change your metadata definition to account for what DS can interpret, specifically mapping the overlay of 8 bit structures into one binary byte.
mydsworld
Participant
Posts: 321
Joined: Thu Sep 07, 2006 3:55 am

Post by mydsworld »

Hi,

I have a file from Mainframe with the following data types

DEC FIXED(5,2)
BIN FIXED(15)

I am using Sequential File stage with EBCDIC character set. What DataStage data types I will use for those above two data types.

Thanks.
Post Reply