COMPARE with sequential/CFF

[kumar_s]

Hi,
i need to compare header value with no of records in the Detail file. i need to abort the flow if it doesnt matches. if i use sequential stage as the input and a compare stage for comparing the values, it works fine. if i use one as complex flat file(for header stage, since it is mainfram file), then there comes an error

Code: Select all

Compare_22,0: On partition [0], the first dataset had 0 records and the second dataset had 1 records.

Both file has records.
i thought the problem may be with partition, so i made to run in sequential mode, still it continues

Note:Header record is read by CFF which has only one record so i made Read First n rows = 1 under File option tab.

Can anybody help me out

[ashwin2912]

Can you do a "View Data" on the Header file?
You cant you use a sequential file for the same?

[kumar_s]

Can you do a "View Data" on the Header file?
You cant you use a sequential file for the same?

yes no probs with sequential file, but the reason i using CFF is header and Detial are in same data file.
In CFF if i give just the layout of header, it takes only the first line(header information) but in sequential it take the data file as well.

[kumar_s]

Is there any way to make sequential stage to read just the first line alone.
(rest of the line would be in different format)

[ashwin2912]

Is there any way to make sequential stage to read just the first line alone.
(rest of the line would be in different format)

Yes...In sequential file stage>Output>Properties>Options

Code: Select all

Read First Rows =1

[kumar_s]

Thanx Ahswin,

Code: Select all

Read First Rows =1

this worked, realy a smart move :D

[ashwin2912]

But hope it had an option to read the last row.

We had a situation to read the trailer....I had to split the header, detail and trailer from Unix and then read it in DataStage.

[kumar_s]

But hope it had an option to read the last row.

We had a situation to read the trailer....I had to split the header, detail and trailer from Unix and then read it in DataStage.

Least understanding ur requirment,
Though it is for debug why cant u use tail stage.
if it is sequential file, then u can extract, if it is ds or fs, u can make to run in sequential mode and u can extract. i tried it worked.

[thebird]

But hope it had an option to read the last row.

We had a situation to read the trailer....I had to split the header, detail and trailer from Unix and then read it in DataStage.

1. I believe you can use the Tail Stage for this purpose very efficiently.
2. Another apprach would be to use an External Source stage and then you can use the Unix tail command to retrieve the last 'n' lines from the source file.
3. There is still another option. The sequential file has a property - "Filter".
You can very well use the Grep command in here to extract only the Trailer records.

We regularly use the second and third approaches in our development works.

Regards,

The Bird.

[ashwin2912]

1. I believe you can use the Tail Stage for this purpose very efficiently.
2. Another apprach would be to use an External Source stage and then you can use the Unix tail command to retrieve the last 'n' lines from the source file.
3. There is still another option. The sequential file has a property - "Filter".
You can very well use the Grep command in here to extract only the Trailer records.

We regularly use the second and third approaches in our development works.

Regards,

The Bird.

Let me try these out (esp Tail stage)...Our environment is down today

BTW, My requirement is to read an EBCDIC file (so I guess I can't directly grep or tail).
I had used the dd command to read the first and last line.

[ashwin2912]

Ya...now I remember the problem I faced....

I have to read an EBCDIC file which can't be directly grep'd or tail'd from UNIX (unless its converted to ASCII...correct me if I am wrong)

The file has header, detail and trailer of foll format

Header:

Code: Select all

HDR-FILE-ID X(11).
HDR-FILE-DT X(10).
FILLER      X(271).

Detail:

Code: Select all

DATA1       S9(4) COMP.
DATA2       X(2).
.
.
.

Trailer:

Code: Select all

TRL-FILE-ID X(11).
TRL-FILE-DT X(10).
TRL-REC-CNT S9(11) COMP-3.
TRL-REC-AMT S9(17) COMP-3.
FILLER      X(256).

So I can read the header using "Read First Rows =1" or Head stage
and trailer using Tail stage.
But what about Detail records...cos the header has first column value "HDR-ABCDEFG" which when read with Detail metadata looks like "-17666" {this is not the not the actual number}.
I can still assume that DATA1 = "-17666" means header and filter this later and similarly filter trailer....Is this approach advisable?
Further, this approach will fall flat on its face where "HDR-ABCDEFG" is not fixed (i.e. it could be "HDR-HIJKLMN" the next time...I have got a few files like this too)

[kumar_s]

If i understand properly, u mean to say that while reading detail record "HDR-ABCDEFG" which is a header record value for HDR-FILE-ID are read in DATA1 (detail record column) instead.....

[ashwin2912]

If i understand properly, u mean to say that while reading detail record "HDR-ABCDEFG" which is a header record value for HDR-FILE-ID are read in DATA1 (detail record column) instead.....

Yes, and it looks like "-17666".

And the way I am doing it using dd UNIX command and creating 3 seperate files as follows:

Code: Select all

dd if=<i/p file name> bs=<rec length> skip=1 count=`expr <rec length> - 2` of=<o/p file name>_det.dat

dd if=<i/p file name> bs=<rec length> skip=0 count=1 of=<o/p file name>_hdr.dat

dd if=<i/p file name> bs=<rec length> skip=`expr <rec length> - 1` count=1 of=<o/p file name>_trl.dat

If you guys can think of a better method or doing it entirely from Datastage, your ideas are welcome.

[kumar_s]

Hi,
I could not under much of ur complexity, But
For header record, as per ur idea u can use

Code: Select all

Read First Rows =1

so that it reads only the first row.

For Detial you can mark

Code: Select all

First Line is Column name = True

so that it wont read the first record and starts with second record. Hope it wont read the trailer record sinc it is with different layout.
Since u r using tail stage u r getting the tailer record as well.
If u r using CFF stage it itself would convert the EBCDIC to ASCII....

[kumar_s]

Hi,
I could not under much of ur complexity, But
For header record, as per ur idea u can use

Code: Select all

Read First Rows =1

so that it reads only the first row.

For Detial you can mark

Code: Select all

First Line is Column name = True

so that it wont read the first record and starts with second record. Hope it wont read the trailer record sinc it is with different layout.
Since u r using tail stage u r getting the tailer record as well.
If u r using CFF stage it itself would convert the EBCDIC to ASCII....