header and footer

[arvind]

Hello Everyone,

I want to know how to remove the header and footer in datastage.
I have to read a file and load the detail data in the target file.

From source i'm getting the Header, detail and Footer records. I have to load only the detail records in the target.
Is there any way how to fillter Header and Footer in datastage, I know we can do it in unix but i want to know in datasateg.


Thanks in advance
Arvind

[chulett]

I would think in a PX job that would happen automatically. The sequential file stage will reject any records with a metadata mismatch and since you will be using the metadata for the detail rows, that should mean any header and/or trailer records won't make the cut and will automatically be 'filtered' out.

[francesco]

The sequential file stage rejects any records with a metadata mismatch but there is a warning message in the log like: "import unsuccessful at record.." corresponding to the header and footer records.
For the header I can use the option "First line is column names" but for the footer?
How can I shoot down this warning message?

Thanks to all
Frank

[kcbland]

Write a shell script to get a line count (wc -l) of the file, then issue a head -n(?-1) yourfile|tail -n(?-2) where ? is the result of the wc -l. Use this korn shell script as a filter on the file in the sequential stage. Tuck this shell script away somewhere and reuse whenever necessary.

[kumar_s]

Write a shell script to get a line count (wc -l) of the file, then issue a head -n(?-1) yourfile|tail -n(?-2) where ? is the result of the wc -l. Use this korn shell script as a filter on the file in the sequential stage. Tuck this shell script away somewhere and reuse whenever necessary.

But i really dont know why this alone works fine

Code: Select all

head -n (?-1) | tail -n (?-1)

regards
kumar

[rgandra]

If you know the format of the header and trailer record, you cna filter those records by adding contraint in the Transformer stage.

[kcbland]

If you know the format of the header and trailer record, you cna filter those records by adding contraint in the Transformer stage.

No, PX is very sensitive to metadata inconsistencies in the Sequential file stage.

[kcbland]

Okay, you can set a variable to be the result of a word count for lines on a data file. Then, subtract 1 from that value, that's the result to use with head. head gives you the top N lines in a file, thus original file line count minus 1 would stop at the trailer record. Pipe that to tail with the original line count minus 2 gives you the file missing the first line, because tail gives you the lines starting from the bottom of the file

Code: Select all

[C:/KBA/BOT/work/staging] ls -al
total 5
drwxrwxrwx   1 Kenneth+Bland   None                  0 Apr 23  2005 .
drwxrwxrwx   1 Kenneth+Bland   None                  0 Apr 23  2005 ..
-rwxrwxrwa   1 Kenneth+Bland   None               2406 Aug 17  2004 sample_rbc.t
xt
[C:/KBA/BOT/work/staging] wc -l sample_rbc.txt
     15    sample_rbc.txt
[C:/KBA/BOT/work/staging] head -n14 sample_rbc.txt|tail -n13|wc -l
     13
[C:/KBA/BOT/work/staging]

Now, you can use any method you like, a korn shell script like this is pretty easy to do.