Filter stage does not work

[splayer]

I have a simple job:
Dataset--> Filter --> Dataset

The Where clause I have in the Filter stage is:
StringToDecimal(InputLink.InputCol) > 5

It gives the following error:
Filter_12: Parse error: Expected comparison, 'between', or 'like' operator, got: "("Parse error: Expected boolean operator (and, or) got: "A"

I did a search on this site for this error but it doesn't seem like there is a solution.

[mansoor_nb]

you can't use functions in the Filter stage.Also while filtering provide just the column name do not specify link name. for example:InputCol> 5.

[splayer]

I can put in a transormer here and it'll do the job for me but I didn't want to do it because it seems like a filter is more appropriate for this purpose.

[ray.wurlod]

One task, one stage. (Transformer excepted but, if you examine the code it generates, there's a heap of Orchestrate operators that might appear there.)

The Transformer stage can, of course, do what the Filter stage does in addition to the transformation. So, if you lose the Filter stage and set up multiple outputs from the Transformer stage, your functional requirement is still met. If you're on 7.5.1A or later, you probably have a more efficient solution as well.

[bcarlson]

Some other options:

1) You could throw in a modify before the filter to handle the string to decimal conversion. Then in the filter you could use 'InputCol > 5'.

2) If there are only a few possible values, you could even leave it as a string. For example, if you just needed to find between 5 and 8 and the original type is a char(3), you could use the following:

Code: Select all

InputCol = '  5'
or
InputCol = '  6'
or
InputCol = '  7'
or
InputCol = '  8'

#2 is not the most glamorous way to do it, but will get the job done. Personally, if you are trying to avoid a Transformer, I think #1 might be the easiest.

Brad.

[koolnitz]

If you are on v7.5.1a or above, I'd personally go for single Transformer stage rather than using Modify plus Filter.

[vijayrc]

If you are on v7.5.1a or above, I'd personally go for single Transformer stage rather than using Modify plus Filter.

Yes I would second that. We had few Filters in our application because we felt Filter fits in the role, but the IBM experts suggested otherwise. [to use a Parallel Transformer instead of a Filter, which can improve the performance and at the same time can do almost all the tasks needed]

[koolnitz]

Helpful article: Is DS Parallel Transformer Evil?

Vincent quotes "Over my years of using parallel jobs I haven't noticed that much of an overhead and when you use the transformer for several tasks (transform, filter and copy) it can be faster."

[vijayrc]

Helpful article: Is DS Parallel Transformer Evil?

Vincent quotes "Over my years of using parallel jobs I haven't noticed that much of an overhead and when you use the transformer for several tasks (transform, filter and copy) it can be faster."

Exactly...Until the Parallel Transformer, yes the BASIC Transformer was a drag. but the Parallel Transformer does all-in-one, if that's what is needed.

[ray.wurlod]

IBM made several performance-related changes to the parallel Transformer stage in version 7.5.1A. If you have this or a later version prefer the Transformer stage to Filter or Switch. Modify and Copy, however, are still faster at doing the things they do.