How to build a External Filter Stage

[sidharth]

Hi All,
I am making an attempt to understand how to use a external filter stage. I know the below narrated one is a simple situation and can be handled with a filter stage. For simplicity, i have assumed the below situation for the study. Request your assistance in doing it...

Code: Select all

Input
-----
id 	val
-- 	---
1	A
2	B
3	C
4	A
5	B
6	A
7	A
8	C
9	C

I want to pass the records with val="A" alone

Output
------

id 	val
-- 	---
1	A
4	A
6	A
7	A

Code: Select all

> My DS Job   Seq File ----> External Filter Stage -----> Peek

> I have planned to use a shell script(grepcmd.sh) as a filter command in the External Filter Stage.

repcmd.sh => cat $1 | grep "A"

> In the  External Filter Stage, following are the optios set,
	Filter command = grepcmd.sh
	Arguments = val

# Should i need to pass all the columns as argumnets to the shell script(stdin), even though the filter critera is on a single column(val) ?
# Should i need to output(stdout) all the columns from the shell script ?

Code: Select all

With the above setup, i get the following error,
External_Filter_1,0: cat: val: The system cannot find the file specified. 
External_Filter_1,0: Wrapped Unix command 'grepcmd.sh val' terminated with error, exit code 1.External_Filter_1
External_Filter_1,0: subprocess failed with exit code 1,External_Filter_1
External_Filter_1,0: Operator's runLocally() failed.

I know i am missing things some where, pls help me in completing my understanding.

Bunch of thanks ,

-Sid

[ray.wurlod]

Our beloved moderator is extremely busy at the moment. How about you post in the parallel forum? Just copy your post above into the new one.

[sidharth]

Ray,
I have reposted this in the Parallel form,


Thanks,
Sid