Remove header and footer(first and last line)

Ujjwal · Post by **Ujjwal** » Fri Nov 18, 2005 9:47 am

Hi all,
My source file having header in first line and tailer in last line
so i want to remove first and last record using DS(parallel Extender ) if any one know please suggest

Regards
Ujjwal

kcbland · Post by **kcbland** » Fri Nov 18, 2005 9:51 am

Write a ksh script that uses the results of a wc -l to issue the appropriate head and tail commands, streaming to stdout. Put that ksh into the filter on the Sequential stage.

kollurianu · Post by **kollurianu** » Fri Nov 18, 2005 4:04 pm

Hi Kenneth,

could you please give the steps briefly what is to be done in the shell script.to strip off the headers and and trailers , just curious

Thank you,
Purna.

chulett · Post by **chulett** » Fri Nov 18, 2005 4:08 pm

Do you really need to do that in PX? The metadata won't match for the header and trailer records (assuming you are using the metadata from the detail records) so the Sequential stage should automatically 'reject' those two... right?

ray.wurlod · Post by **ray.wurlod** » Fri Nov 18, 2005 7:30 pm

... then the reject link could feed a stream that accepts the header but rejects the footer, and these streams could then be processed appropriately!

kumar_s · Post by **kumar_s** » Fri Nov 18, 2005 8:57 pm

Hi Purna,
The scripts has been already provided in this forum. Pls do a search.

Hi Ujjwal,
The reject method work fine but with warnigs. If you are ready to accept warnigns, Ken's method is best.

-Kumar

chulett · Post by **chulett** » Fri Nov 18, 2005 9:24 pm

I don't recall the 'reject method' causing any warnings. It will rather happily process any records that fail the metadata check into oblivion without raising any kind of a fuss.

Perhaps it's only after you actually setup a reject link? In this case, it doesn't sound like that would be needed.

kumar_s · Post by **kumar_s** » Fri Nov 18, 2005 9:53 pm

But i wonder how iam getting the following warining if i approach reject mechanism.

Code: Select all

SEQ,0: Short input record
SEQ,0: Import warning at record 0.
SEQ,0: Import unsuccessful at record 0.

Is there any other smarter approach to get rid of Short input record - read.

-Kumar

kcbland · Post by **kcbland** » Fri Nov 18, 2005 11:25 pm

This is a point of style.

Relying on the stage to behave poorly (IMO), meaning to silently toss data, is poor style.

You should affirmatively handle all data. If you don't want a row, build the logic to purposely remove the remove. If yo do want specific rows, build the logic to purposely include only those rows.

Letting the job "lose" data because it's the header and footer means that potentially in the future Ascential could change the behavior of the stage and now because you don't have affirmative logic unexpected things start happening.

For example, if you use a constraint to say you only want rows where a code value is A thru Z except for X, a lazy programmer might say: inlink.column NE 'X'. This serves the purpose that Xs don't make it in, but also allows '!' and 'x' and 'XX' to go thru. The affirmative logic would be to name all values A to Z except X in the constraint.

It's style but the korn shell script is a reusable solution. It's a generic script that can always be used to strip off the header and trailer record. It's even more beneficial if the header and trailer has row counts and checksum information, because then the script can validate the file first and return a non-zero value stopping the job if the file is corrupted. See the expandabilities?

dspxlearn · Post by **dspxlearn** » Sat Nov 19, 2005 7:22 am

Hi all,

Do we have 'Stage using filter command' option in Parallel jobs in the sequential file stage like we have in Server jobs. If yes where can we enable that option.

chulett · Post by **chulett** » Sat Nov 19, 2005 9:47 am

dspxlearn - Please don't hijack other peoples threads. If you've got an new question, start a new post.

Ken, agree on your 'point of style'. Was primarily looking for a clarification / confirmation on how the stage actually works as I don't have access to a PX server. Just have a Ray Wurlod Guinea Pig Merit Badge, Texas Chapter.

ray.wurlod · Post by **ray.wurlod** » Sat Nov 19, 2005 1:09 pm

Spelling!!!

chulett · Post by **chulett** » Sat Nov 19, 2005 1:56 pm

Doh... at first I thought you meant 'Guinea' as in pig, but had checked that before posting. Finally dawned on me I was closer to Warlord than to Wurlod.

Sorry about that Chief... corrected.

ray.wurlod · Post by **ray.wurlod** » Sat Nov 19, 2005 5:37 pm

Merci beaucoup