skip header and tail while reading from seq file

[sunshine]

Skip header and tail while reading from a flat file.

source.txt

eg:
CountryName|Capital
USA|Washington
UK|London
Austria|Vienna
Generated on: 04/04/2006


I need to skip the header and tail while reading.

resulted output is


USA|Washington
UK|London
Austria|Vienna

i searched in DSX, someone told to use @INROWNUM

what are the possibilities of doing this.
i am novice in this..
where exactly i need to write the code.....


Thanks
sunshine

[zbethem]

n00b here too

@INROWNUM looks like the trick, but perhaps you could pair it with @OUTROWNUM

I'd build a job like this..

In transformer, add a constraint. The constraint would be something like:
@INROWNUM > 1 and @INROWNUM <> @OUTROWNUM

If the constraint evaluates to true, the output link should have the rows you want (i.e. not the first and not the last).

Hope it helps.

[ray.wurlod]

Use head tail and wc -l to strip off the header and trailer lines.

[sunshine]

[quote="ray.wurlod"]Use [b]head [/b][b]tail [/b]and [b]wc -l [/b]to strip off the header and trailer lines. ...[/quote]

where i can use this head and tail

can you explain me detail

Thanks
sunshine

[anujgarg]

Hi,

You can do this in before/after job job subroutine.

[sendmk]

head -$(expr $(wc -l | awk '{ print $1 }') \- 1) filename | tail +2

this would work

[vsi]

head -$(expr $(wc -l | awk '{ print $1 }') \- 1) filename | tail +2[/quote]

this would work,


ohhh, very confused...
If u have some time, Can you explain the command...

Thanks
vsi

[vsi]

when i tried the command,got an exception:

i copied the command to a file called "filter.txt"

head -$(expr $(wc -l | awk '{ print $1 }') \- 1) Countries.txt | tail +2

and in the filter command,i entered the path of the file

but throws exception:

Error in filter command "home/username/filter.txt" -
job1.. Sequentional_File_Dslink3: ds_seqopen() - Error in execv() -
Permission Denied
job1.. Sequentional_File_Dslink3: DSD.SEQOpen GCI
$DS.SEQOPEN error 11


any idea?


Thanks
vsi

[ArndW]

You need to enter the head/tail/awk command in the filter command.

you can view the UNIX man pages for head & tail & wc and the awk command has whole books written about it.

[vsi]

Hi

i used command "sed"

eg:sed -e '1d' -e '$d' filename.

any comments?

Thanks
vsi

[sendmk]

head -$(expr $(wc -l filename | awk '{ print $1 }') \- 1) filename | tail +2

try this [/quote]

[sendmk]

if i write the below script in a file and execute it in filter command say

"sh /home/kpraveen/test.sh"

how can i parameterize the "filename" in the script

"head -$(expr $(wc -l filename | awk '{ print $1 }') \- 1) filename | tail +2"

so that it takes the file which is given in the

filename

option of seqential file

any ideas guys

thx[/quote]

[ray.wurlod]

How are you invoking the script? If it's via ExecSH you can simply make the file name a job parameter and use the parameter reference in both locations. If through DSExecute in one of your own routines, you can simply build the command into a variable, having passed the file name as a routine argument for example.

[vinodhraj]

n00b here too

@INROWNUM looks like the trick, but perhaps you could pair it with @OUTROWNUM

I'd build a job like this..

In transformer, add a constraint. The constraint would be something like:
@INROWNUM > 1 and @INROWNUM <> @OUTROWNUM

If the constraint evaluates to true, the output link should have the rows you want (i.e. not the first and not the last).

Hope it helps.

Hi,

I tried with this logic, but only the header alone filtered out not the final record.

regards

vinod

[kumar_s]

Code: Select all

head -$(expr $(wc -l filename | awk '{ print $1 }') \- 1) filename | tail +2 > New_Filename.txt

Use the above in the before job subroutine with ExecSH.
Use te New_Filename.txt to read in the sequential stage.