Error in filter command...

kaps · Post by **kaps** » Fri Jun 16, 2006 4:45 pm

All

I am trying to execute the following command in filter command of seq file stage.

dsjob -report srcdev src_common_curr_ext.abwBase1 XML

But I am getting the following error..

[quote]
src_cog_test_filter_2..seq_TW1L0_curr_T.DSLink36: ds_seqopen() - Error in filter command "/opt/dsadm/Ascential/DataStage/DSEngine/bin/dsjob -report srcdev src_common_curr_ext.abwBase1 XML" -

Status code = 0
[/quote]

My job design is like this...

Seq File ------>Seq File

This is not my actual job design. but to narrow down the error I did it like this. I am mentioning this command in first seq file filter command.
I have /dev/null in my file name column. My seq file has only one column.

I even tried calling a shell script from filter command rather than executing the command...got same error...

any help is appreciated..

Thanks

chulett · Post by **chulett** » Fri Jun 16, 2006 5:16 pm

Have you read the help on the filter option? You can't do things like that there. You need to execute a command that writes to standard out and optionally also reads from standard in. The output of the command is streamed into the stage and read as if it were a 'virtual sequential file' so to speak, much like reading a pipe.

If you want to execute random stuffs or other kinds of 'commands', write a routine.

kduke · Post by **kduke** » Fri Jun 16, 2006 9:27 pm

Originally I downloaded DSJobReport from ADN as an example of how to use a filter command. It failed on a regular basis. Abort hard with goofy error messages like this. It was hard to debug. I figured out a lot of limitations of XML from trying to make this job stable. I changed this filter stage into a sequential file stage and did the dsjob command as a before job routine.

You have problems if try to pull a repeating group and it does not exist. The job fails hard. I wanted to extract parameter values and row counts. Parameter names control parameter values and link names control row counts. Both are repeating. (There can be more than one parameter or link in a job therefore it repeats.) I also wanted the start and end times for the job, these do not repeat. So if I tried to get job start and end times and there was no parameters then the job failed hard or "aborted". So I had to process job start and end times separately from parameter names which had to be processed separately from row counts. The new job became DSJobReportDb. This job is the basis for EtlStats. It is fairly simple if you look it. It is way more complicated than necessary because of the bugs in the XML plugin. Maybe the new plugin has fixed all this but what is there works in older versions of DS. It is vary stable now. Several of these jobs finish with warnings. Still kind of annoying but I can live with it.

DSJobReportDbDriver was written to get row counts for all jobs in a sequence. So you could call this one job at the end of your sequence. DSJobReportDbDriverNoWait was added so it would not wait to get the row counts. This is a kind of a slow process. It can take a hour to get row counts on a 1,000 jobs. Do you want to add a hour to your load times. The new version has macros for DSJobName and other things like that so you can just cut and paste this link into any sequence. Next weekend is a long weekend so maybe I will post the new version. It should handle UNIX and DOS versions easily.

The before command is:

#dsjobcmd# #projName# #jobName# XML >KimD/#jobName#.xml

You may need to change this to:

#dsjobcmd# #projName# #jobName# XML >KimD\#jobName#.xml

For DOS. You may also need to change the parameter default to:

..\..\Engine\bin\dsjob -report

or

../../Engine/bin/dsjob -report

You may need to hard code this value to the complete path like

${DSHOME}/bin/dsjob -report

I don't think this is very difficult.

Back to the filter issue. I think most of the time I would output the command to a sequential file and then process it. I don't know of too many situations where a filter cannot be handled this way.

kaps · Post by **kaps** » Mon Jun 19, 2006 8:16 am

Craig

I am familiar with the filter command option in sequential file.

[b]dsjob -report srcdev src_common_curr_ext.abwBase1 XML[/b] is writing it in standard output. so it should work...If I execute a simple echo or cat command in filter command instead of dsjob it's working fine. only with dsjob command it fails. I am thinking it mmight be because of the status code of dsjob command. I donno how to suppress the status code.

I am not able to read Kim's reply. It seems he has faced the similar kind of error.

Thanks anyway

chulett · Post by **chulett** » Mon Jun 19, 2006 8:28 am

How is your sequential metadata setup? Looks like you are generating XML and then asking it to be read and written as a sequential file. You'll need to be careful there and define a single large varchar column, I would think.

kumar_s · Post by **kumar_s** » Mon Jun 19, 2006 8:30 am

According to Kim's suggestion, divert the output of the command used in the filter into a sequential file. Like dsjob -report srcdev src_common_curr_ext.abwBase1 XML > OutFile.xml or OutFile.txt.

kaps · Post by **kaps** » Mon Jun 19, 2006 8:54 am

Craig

My job is designed the same way as you suggested. What's strange is that If I route the output to a file instead of stdout and then cat the file in shell script then the job works just fine. so I think it's not related to reading XML using Seq file.

Kumar

Thanks. I can do that but I have to break the job into two.

I was just trying to do the samething stated in [b]DSJobReport[/b] job in dev net to get the metadata information. but did not work !

Thanks