filter command question ?

[kaps]

All

I am looking for help to use filter command...what I am trying to do is to execute a shell script from filter command which needs 3 parameter as input. first parameter is the name of the file need to be sorted which is what I am reading from DB2...

Job looks like this...

ODBC Stage-->ipc-->Seq file stage

I am using the following command in the filter of seq file stage.

ksh /xxxx/xxxx/xxx.sh stdin /xxxx/xxxx/xxxx /xxxx/xxxx/xxxx

I am getting this error.

Error in filter command .../usr/bin/ksh: : cannot open

Any help is appreciated....

Thanks

[ArndW]

The filter command doesn't use files, but stdin and stdout; it works row by row.

[ray.wurlod]

Try just putting the name of your script (that is, don't use ksh) into the Filter Command field, possibly with its three command line arguments.

Each row will be passed through that script before ending up in the target file.

That is, your script must be written to process its own stdin, on which rows will appear, to its own stdout, from which rows will be returned to the Sequential File stage for writing to the target file.

[kaps]

Thanks for your reply...Ray...

I still have problems...My shell script gets input from stdin and writing to stdout. If I run just the shell it runs fine but It fails when I call this from DataStage.

I have used /dev/null in the file name...

My Filter command look like this...

/xxx/xxx/xxx/xxx.sh stdin /xxx/xxx/xxx/xxx/xxx2 /xxx/xxx/xxx/xxx/xxx3

Here is the first warning I am getting...

[quote]test_filter..seq_xxx_T.L02_to_seq_file: ds_seqopen() - Error in filter command "/xxx/xxx/xxx/xxx.sh stdin /xxx/xxx/xxx/xxx/xxx2 /xxx/xxx/xxx/xxx/xxx3" -
[/quote]

Any Clue ?

Thanks

[ray.wurlod]

"Error in filter command /xxx/xxx/xxx/xxx.sh" seems fairly obvious as a diagnosis. Do you have execute permission to the shell script? Does the shell script exit with a status code of other than zero?

[kaps]

Ray -

Thanks....Yes. I do have execute permission to the shells. Now I have used a file name instead of /dev/null in file name column of sequntial file.
The shell returns the non zero value(return value is empty) which is written to the output file mentioned.

But It runs fine If I just ran the shell in AIX...

Please explain...

Thanks

[ArndW]

The job can have different results because your ENVironment settings can be different between running the job in a shell and running it from a DataStage job. Does your script require environment settings or paths to be set? If you check your Director logfile it will print out the job's environment settings at runtime so perhaps you could see if there are differences that might affect your script's output.

[kaps]

Thanks for the Inputs.

Please let me know what this error means...

DSD.SEQOpen GCI $DS.SEQOPEN error 11.

Thanks

[ArndW]

Kaps,

it means that your sequential file couldn't be opened for reading. Either the path/filename or the permissions are wrong.

[kaps]

Arnd

Thanks for the inputs. I checked the path, name and permissions and they look good. I don't know what the problem is...

Please tell me If I am doing anything meaningless...this is what I am doing...

ODBC Stage-->ipc-->Seq file stage

In the filter command of seq stage I am calling a shell script which inturn is calling a CoSORT script to sort the file. My shell script takes three parameters. First one comes from stdin which is actual file and I am passing other two parameters.

/xxx/xxx/xxx/xxx.sh stdin /xxx/xxx/xxx/xxx/xxx2 /xxx/xxx/xxx/xxx/xxx3

I know I can sort the file in many different ways. but we want to do it thro CoSORT because of the speed.

Is the problem is because this filter command acts row by row ?

[ArndW]

Hello Kaps,

since you have CoSORT you should use it - it really does improve sort performance.

On the other hand, I think your method of using a filter in this case isn't the optimal approach, at least initially. I would recommend starting off with writing the data to the sequential file and then using a distinct operation to sort it; either through the use of an after-job subroutine call or, if you have a sequencer calling the job, by adding an activity to the sequencer. You end up landing the data and extra time, but you can get your job designed so that it functions.

Once that is working (and you have the appropriate environment settings set up; since I remember that CoSORT needs specific library paths set, just as DS does and they are most likely not in your DS Env) then you can approach some tuning.

Since you are not calling CoSORT directly, the filter approach won't work for as you probably already surmise. One approach to not landing the interim unsorted file would be to create your own named pipe; starting off your script as part of the job initialization with a background process that sends this pipe to CoSORT as the input stream (i.e. the reader) and your DS Job would write to this sequential pipe. Thus the output from the DS Server job doesn't get written to disk but goes directly to the CoSORT input stream.

[ray.wurlod]

Thanks for the Inputs.

Please let me know what this error means...

DSD.SEQOpen GCI $DS.SEQOPEN error 11.

Thanks

This error means that the sequential file cannot be opened.
Check again. "Looks good" = insufficiently thorough. Do you have execute permission to every directory in the sequemntial file's pathname? Is the sequential file correctly spelled and cased.

This error is probably unrelated to any filter that you may have. $DS.SEQOPEN is the function that opens the text file itself.

Incidentally, what do you hope to achieve with an IPC stage between two passive stages?

[kaps]

Ray -

we are using IPC stage to improve the performance. Atleast thats what I was told...

Kaps

[kaps]

Ray -

In your post you said -

This error is probably unrelated to any filter that you may have. $DS.SEQOPEN is the function that opens the text file itself.

But If I remove the filter command and run the job, it writes to that file without any problem. Does this ring a bell ?

Arnd - Thanks for your input. I am thinking about it...

Thanks

[ray.wurlod]

There's not a lot more we can suggest by mind reading. Are you able to post the code of the shell script? Then we can see whether it's trying to do something that would result in 0 rows appearing on stdout.