Page 1 of 2
COMPARE with sequential/CFF
Posted: Fri Jul 15, 2005 5:49 am
by kumar_s
Hi,
i need to compare header value with no of records in the Detail file. i need to abort the flow if it doesnt matches. if i use sequential stage as the input and a compare stage for comparing the values, it works fine. if i use one as complex flat file(for header stage, since it is mainfram file), then there comes an error
Code: Select all
Compare_22,0: On partition [0], the first dataset had 0 records and the second dataset had 1 records.
Both file has records.
i thought the problem may be with partition, so i made to run in sequential mode, still it continues
Note:Header record is read by CFF which has only one record so i made Read First n rows = 1 under File option tab.
Can anybody help me out
Posted: Fri Jul 15, 2005 6:05 am
by ashwin2912
Can you do a "View Data" on the Header file?
You cant you use a sequential file for the same?
Posted: Fri Jul 15, 2005 6:11 am
by kumar_s
ashwin2912 wrote:Can you do a "View Data" on the Header file?
You cant you use a sequential file for the same?
yes no probs with sequential file, but the reason i using CFF is header and Detial are in same data file.
In CFF if i give just the layout of header, it takes only the first line(header information) but in sequential it take the data file as well.
Posted: Fri Jul 15, 2005 6:19 am
by kumar_s
Is there any way to make sequential stage to read just the first line alone.
(rest of the line would be in different format)
Posted: Fri Jul 15, 2005 6:23 am
by ashwin2912
kumar_s wrote:Is there any way to make sequential stage to read just the first line alone.
(rest of the line would be in different format)
Yes...In sequential file stage>Output>Properties>Options
Posted: Fri Jul 15, 2005 6:32 am
by kumar_s
Thanx Ahswin,
this worked, realy a smart move :D
Posted: Fri Jul 15, 2005 6:41 am
by ashwin2912
But hope it had an option to read the last row.
We had a situation to read the trailer....I had to split the header, detail and trailer from Unix and then read it in DataStage.
Posted: Fri Jul 15, 2005 7:15 am
by kumar_s
ashwin2912 wrote:But hope it had an option to read the last row.
We had a situation to read the trailer....I had to split the header, detail and trailer from Unix and then read it in DataStage.
Least understanding ur requirment,
Though it is for debug why cant u use tail stage.
if it is sequential file, then u can extract, if it is ds or fs, u can make to run in sequential mode and u can extract. i tried it worked.
Posted: Mon Jul 18, 2005 1:14 am
by thebird
ashwin2912 wrote:But hope it had an option to read the last row.
We had a situation to read the trailer....I had to split the header, detail and trailer from Unix and then read it in DataStage.
1. I believe you can use the Tail Stage for this purpose very efficiently.
2. Another apprach would be to use an External Source stage and then you can use the Unix tail command to retrieve the last 'n' lines from the source file.
3. There is still another option. The sequential file has a property - "Filter".
You can very well use the Grep command in here to extract only the Trailer records.
We regularly use the second and third approaches in our development works.
Regards,
The Bird.
Posted: Mon Jul 18, 2005 3:22 am
by ashwin2912
thebird wrote:
1. I believe you can use the Tail Stage for this purpose very efficiently.
2. Another apprach would be to use an External Source stage and then you can use the Unix tail command to retrieve the last 'n' lines from the source file.
3. There is still another option. The sequential file has a property - "Filter".
You can very well use the Grep command in here to extract only the Trailer records.
We regularly use the second and third approaches in our development works.
Regards,
The Bird.
Let me try these out (esp Tail stage)...Our environment is down today
BTW, My requirement is to read an EBCDIC file (so I guess I can't directly grep or tail).
I had used the dd command to read the first and last line.
Posted: Mon Jul 18, 2005 10:19 pm
by ashwin2912
Ya...now I remember the problem I faced....
I have to read an EBCDIC file which can't be directly grep'd or tail'd from UNIX (unless its converted to ASCII...correct me if I am wrong)
The file has header, detail and trailer of foll format
Header:
Code: Select all
HDR-FILE-ID X(11).
HDR-FILE-DT X(10).
FILLER X(271).
Detail:
Code: Select all
DATA1 S9(4) COMP.
DATA2 X(2).
.
.
.
Trailer:
Code: Select all
TRL-FILE-ID X(11).
TRL-FILE-DT X(10).
TRL-REC-CNT S9(11) COMP-3.
TRL-REC-AMT S9(17) COMP-3.
FILLER X(256).
So I can read the header using "Read First Rows =1" or Head stage
and trailer using Tail stage.
But what about Detail records...cos the header has first column value "HDR-ABCDEFG" which when read with Detail metadata looks like "-17666" {this is not the not the actual number}.
I can still assume that DATA1 = "-17666" means header and filter this later and similarly filter trailer....Is this approach advisable?
Further, this approach will fall flat on its face where "HDR-ABCDEFG" is not fixed (i.e. it could be "HDR-HIJKLMN" the next time...I have got a few files like this too)
Posted: Wed Jul 20, 2005 1:17 pm
by kumar_s
If i understand properly, u mean to say that while reading detail record "HDR-ABCDEFG" which is a header record value for HDR-FILE-ID are read in DATA1 (detail record column) instead.....
Posted: Wed Jul 20, 2005 9:52 pm
by ashwin2912
kumar_s wrote:If i understand properly, u mean to say that while reading detail record "HDR-ABCDEFG" which is a header record value for HDR-FILE-ID are read in DATA1 (detail record column) instead.....
Yes, and it looks like "-17666".
And the way I am doing it using dd UNIX command and creating 3 seperate files as follows:
Code: Select all
dd if=<i/p file name> bs=<rec length> skip=1 count=`expr <rec length> - 2` of=<o/p file name>_det.dat
dd if=<i/p file name> bs=<rec length> skip=0 count=1 of=<o/p file name>_hdr.dat
dd if=<i/p file name> bs=<rec length> skip=`expr <rec length> - 1` count=1 of=<o/p file name>_trl.dat
If you guys can think of a better method or doing it entirely from Datastage, your ideas are welcome.
Posted: Wed Jul 20, 2005 11:00 pm
by kumar_s
Hi,
I could not under much of ur complexity, But
For header record, as per ur idea u can use
so that it reads only the first row.
For Detial you can mark
so that it wont read the first record and starts with second record. Hope it wont read the trailer record sinc it is with different layout.
Since u r using tail stage u r getting the tailer record as well.
If u r using CFF stage it itself would convert the EBCDIC to ASCII....
Posted: Wed Jul 20, 2005 11:03 pm
by kumar_s
Hi,
I could not under much of ur complexity, But
For header record, as per ur idea u can use
so that it reads only the first row.
For Detial you can mark
so that it wont read the first record and starts with second record. Hope it wont read the trailer record sinc it is with different layout.
Since u r using tail stage u r getting the tailer record as well.
If u r using CFF stage it itself would convert the EBCDIC to ASCII....