Need help with an External Filter problem
Posted: Mon Jul 25, 2005 2:54 pm
I've written a unix C program to be used as an external filter. The columns it returns are different than the columns fed to it (and many more rows are produced than are supplied to the filter).
When I run the DS job, the external filter is called three times (there are three input records) and, based on debug msgs I've written to stderr that show up in the director log, I'm getting the output I want. But the Data Set output stage that follows the filter is failing because it can't find one of the columns I supplied as input to the filter. I'm not outputting this column and it isn't in the output column list for the filter and the Runtime Column Propogation check box for the filter stage is not checked.
On a side note, the column being complained about is also one that I didn't want in my input to the filter, but DS insisted on supplying it. I finally gave up and accepted in on input, even though I don't use it.
Another side note - I'm processing all three input records before the error shows up. I can't tell if (a) the system waits until I've processed all the records before complaining about the first one, (b) only the last record has a problem or (c) if the system is complaining about something after the last record I put out.
As near as I can tell, I'm not writing anything extraneous at the end of the run. But even if I were, why would the stage complain about missing a field that I never told it should be there in the first place?
I'm pretty new at DS. Does anyone have any suggestions as to how to attack this problem?
(I previously posted this question in the server forum, but just realized that I should've posted it in the EE forum)
Thanks,
Bob Kendall
When I run the DS job, the external filter is called three times (there are three input records) and, based on debug msgs I've written to stderr that show up in the director log, I'm getting the output I want. But the Data Set output stage that follows the filter is failing because it can't find one of the columns I supplied as input to the filter. I'm not outputting this column and it isn't in the output column list for the filter and the Runtime Column Propogation check box for the filter stage is not checked.
On a side note, the column being complained about is also one that I didn't want in my input to the filter, but DS insisted on supplying it. I finally gave up and accepted in on input, even though I don't use it.
Another side note - I'm processing all three input records before the error shows up. I can't tell if (a) the system waits until I've processed all the records before complaining about the first one, (b) only the last record has a problem or (c) if the system is complaining about something after the last record I put out.
As near as I can tell, I'm not writing anything extraneous at the end of the run. But even if I were, why would the stage complain about missing a field that I never told it should be there in the first place?
I'm pretty new at DS. Does anyone have any suggestions as to how to attack this problem?
(I previously posted this question in the server forum, but just realized that I should've posted it in the EE forum)
Thanks,
Bob Kendall