Issue with complex flat file output when integrated to MF

[ramakrishna459]

Hi all,

We are generating complex flat file output in which the layout is having the some columns with DECIMAL COMP-3(PIC S9(15) COMP-3.) and Binary COMP(PIC S9(9) COMP.).

The output setting used are Characterset as ASCII and Dataformat as Binary.

The issues we are facing are
1)In mainframe side when they read this output file by FTP using ascii mode they are not able to find the newline delimeter at the end of record and all the data is comming in single line.

2)For the Decimal columns in mainframe side we are getting invalid data

Could you please share your thougts regarding this.

[jwiles]

Are all of your data columns binary (comp-3, comp, etc.) or do you also have some that are character data?

If you are using FTP in ASCII-mode to transfer a file from a system that is ASCII-based (Unix/Windows) to a system that is EBCDIC-based (z/OS), FTP will attempt to convert, on a byte-by-byte basis, character data from ASCII to EBCDIC. This will corrupt non-character data because individual bytes in these fields often map directly to ASCII characters and will be translated to corresponding EBCDIC character values. For instance: the binary value x41 (maps to ASCII character 'A') will be translated to binary value xC1 (maps to EBCDIC character 'A'). Also, z/OS datasets do not use record delimiters as they are understood in Unix/Windows environments. An ASCII-mode FTP transfer from ASCII to EBCDIC systems will remove record delimiters. FTP does NOT know that a particular byte of the data is in a character vs binary data column or that it is the record delimiter...it simply follows what the translation tables dictate based upon the source and target systems and transfer mode.

If you have any non-character fields (i.e. binary/packed decimal/etc.), you should always use a BINARY mode FTP transfer otherwise your binary values will likely be corrupted. If you have a mix of character and binary data, you should probably have CFF write the character fields using the EBCDIC character set and use a BINARY mode FTP transfer. ONLY if all of your data is character (NO binary data) should you consider using an ASCII-mode FTP transfer.

Assuming that the file is being written to a z/OS dataset when FTP'd, make sure that the proper record format, record length and block length values are used for the destination on the mainframe.

Regards,

[ramakrishna459]

Are all of your data columns binary (comp-3, comp, etc.) or do you also have some that are character data?

In our layout we are having some columns as comp-3 decimal and some are character.At mainframe end they are FTP using ascii mode.So we are some how unable to see the data in the mainframe.

We are required to genarate fixed width file.
So we are using this format in cff output.

Record type:FIXED
Data format :Binary
Byte order:Native Endian
Charcter set:Ascii
No reord delimeter
and Padchar we are using as single space.

As we are required to genarate fixed width file for some integer and decimal columns we are getting the data less than the metadata length so we are appending zeros infront of the decimal and integer columns.Is this the right way are please share your suggestions.

[chulett]

In our layout we are having some columns as comp-3 decimal and some are character.At mainframe end they are FTP using ascii mode.So we are some how unable to see the data in the mainframe.

As James noted, if your file is being ftp'd to you using ASCII mode the packed fields are destroyed in the transfer. Make sure they switch to a binary transfer.

[ramakrishna459]

We are generating that files in binary mode and the mainframes is ftp in ascii mode.

[jwiles]

In our layout we are having some columns as comp-3 decimal and some are character

We are generating that files in binary mode and the mainframes is ftp in ascii mode

Generate the file so that character data is using the EBCDIC character set and there are no record delimiters (Complex Flat File can do this for you). This will create your file so that it the data is correctly formatted and needs no translation during FTP. Work with a knowledgeable mainframer to verify that the file is formatted correctly before the FTP is performed. Execute the FTP in BINARY mode, with the correct recfm, lrecl and blksize settings so that the file is transferred into a correct format on the mainframe (again, work with a knowledgeable mainframer--they will understand what these are and what they should be set to for your file).

Regards,

[FranklinE]

An edited (i.e. cleaner and more readable) version of this would be a nice addition to the Using Mainframe Data FAQ.

The hardware and OS configurations on mainframe systems are, in general, very different from PC and server based systems. The following are the key aspects, in no particular order.

1) DASD (direct access storage device) disk packs are organized around tracks and cylinders. The modern 3390 standard (high end capacity dependent on actual device) is 56,664 bytes per track and 15 tracks per cylinder. Data is blocked according to those boundaries. Optimized blocking fits the maximum number of records on one track and minimizes the number of unused bytes. The math is easy: If you have a logical record length (lrecl) of 100 bytes, the optimal blocksize is 56,600 (standard usage would be 28,300) and each track will have 64 unused bytes.

2) File management is in a catalog, which stores the lrecl, record format and blocksize. Processes that read the data retrieve those attributes from the catalog. In most IBM platforms -- z/OS is the current top-of-line standard -- datasets are initialized with a command line that sets the attributes in a DCB -- data control block.

3) Fixed-length rules all, but it is not reflected in the physical storage above the field level. Data is stored in contiguous bytes as written. No delimiters are needed because the next process to read the data is expected to have a copybook (table definition) that conforms exactly to how the data was written. The data in a file starts with the first byte and ends with the last byte in the file. A browser or editor will display the data according to the lrecl found in the catalog. Don't be fooled into thinking that this implies record delimiters.

Transferring data between platforms is dangerous unless you always conform to the platform requirements. Don't expect an EBCDIC file to be readily transformable to ASCII unless you follow transformation steps exactly. COMP-3 and COMP data have exact equivalents in DataStage.

Code: Select all

COMP-3
Properties
Decimal Type
Default = 0
Packed = yes
Allow all zeros = yes

COMP
Properties
Integer type
Byte order = big-endian (default, may need to be changed)
Data format = binary
Default = 0

That is a guideline that works most of the time. It will help you figure out what needs changing when an exception to the general is found.

[FranklinE]

We are generating that files in binary mode and the mainframes is ftp in ascii mode.

With the FTP step a separate process initiated from the mainframe, I suggest you make sure the CFF output file properties are identical to the mainframe storage formatting:

Code: Select all

Record type = implicit 
Delimiter = none 
Character set = EBCDIC 
Data format = binary 
Allow all zeroes = yes

Ideally, something I use in many jobs, it's better to use FTP Enterprise instead of a separate job or script. It handles all of the attributes, file and table definition, in one stage. Of course, all that depends on your system's security and user access configurations.