Hello Everyone,
I want to know how to remove the header and footer in datastage.
I have to read a file and load the detail data in the target file.
From source i'm getting the Header, detail and Footer records. I have to load only the detail records in the target.
Is there any way how to fillter Header and Footer in datastage, I know we can do it in unix but i want to know in datasateg.
Thanks in advance
Arvind
header and footer
Moderators: chulett, rschirm, roy
I would think in a PX job that would happen automatically. The sequential file stage will reject any records with a metadata mismatch and since you will be using the metadata for the detail rows, that should mean any header and/or trailer records won't make the cut and will automatically be 'filtered' out.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers
The sequential file stage rejects any records with a metadata mismatch but there is a warning message in the log like: "import unsuccessful at record.." corresponding to the header and footer records.
For the header I can use the option "First line is column names" but for the footer?
How can I shoot down this warning message?
Thanks to all
Frank
For the header I can use the option "First line is column names" but for the footer?
How can I shoot down this warning message?
Thanks to all
Frank
Write a shell script to get a line count (wc -l) of the file, then issue a head -n(?-1) yourfile|tail -n(?-2) where ? is the result of the wc -l. Use this korn shell script as a filter on the file in the sequential stage. Tuck this shell script away somewhere and reuse whenever necessary.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
But i really dont know why this alone works finekcbland wrote:Write a shell script to get a line count (wc -l) of the file, then issue a head -n(?-1) yourfile|tail -n(?-2) where ? is the result of the wc -l. Use this korn shell script as a filter on the file in the sequential stage. Tuck this shell script away somewhere and reuse whenever necessary.
Code: Select all
head -n (?-1) | tail -n (?-1)
regards
kumar
Re: header and footer
If you know the format of the header and trailer record, you cna filter those records by adding contraint in the Transformer stage.
Re: header and footer
No, PX is very sensitive to metadata inconsistencies in the Sequential file stage.rgandra wrote:If you know the format of the header and trailer record, you cna filter those records by adding contraint in the Transformer stage.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Okay, you can set a variable to be the result of a word count for lines on a data file. Then, subtract 1 from that value, that's the result to use with head. head gives you the top N lines in a file, thus original file line count minus 1 would stop at the trailer record. Pipe that to tail with the original line count minus 2 gives you the file missing the first line, because tail gives you the lines starting from the bottom of the file
Now, you can use any method you like, a korn shell script like this is pretty easy to do.
Code: Select all
[C:/KBA/BOT/work/staging] ls -al
total 5
drwxrwxrwx 1 Kenneth+Bland None 0 Apr 23 2005 .
drwxrwxrwx 1 Kenneth+Bland None 0 Apr 23 2005 ..
-rwxrwxrwa 1 Kenneth+Bland None 2406 Aug 17 2004 sample_rbc.t
xt
[C:/KBA/BOT/work/staging] wc -l sample_rbc.txt
15 sample_rbc.txt
[C:/KBA/BOT/work/staging] head -n14 sample_rbc.txt|tail -n13|wc -l
13
[C:/KBA/BOT/work/staging]
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle