Reject thresholds for reject links

[gsherry1]

Is there any functionality for setting abort thresholds on reject links. For example, if more than 50 records on reject link then abort job. Or even better, if > 1% of records end up on reject port, then abort job.

I understand that I can always join to the non reject link and do some calculation then cause the abort, but that's too much work and too processing intensive.

I am interested in such functionality in both EE and Server.

[RobertScarbrough]

If you use a transformer in EE you can click on the contraints and set a number. I'm pretty sure that this field does not allow a parameter value. To do the 1% thing you can use a transformer again. I know, I know, not the most efficient EE stage, but it's just an idea. You'll obviously need to determine the number of input rows you are starting with. In your stage variables keep track of the number of rows that have aborted. Have a special link coming out of the transformer that checks to see if the total that has aborted it equal to this tolerance. If it has then send a row down it to a peek stage. Change the contraints to abort after one row.

HTH.

-rls

[ray.wurlod]

The server job answer is easy; you can set this hard limit in the Constraints grid in a Transformer stage.
In parallel jobs you don't have this functionality, so need something downstream on the rejects link to count the number of rows and trigger abandonment of processing.

[ameyvaidya]

There is another way to set up aborts that I'd discovered by accident in 7.5.

I dont have access to DataStage at the moment but heres how:

When previewing data either from a Sequential file or database source, there is an option to view the generated OSH. In that OSH, you will find an abort operatot that takes an nrecs argument.

A custom stage can be wrapped around this operator:

1 input
0 outputs
and one stage level property that takes the number of records to abort after(can be parameterised)

When run in sequential mode and with RCP on(No need to map columns), this stage will read input rows until their number equals the value set in the stage property.

This has been tested DSEE 7.5.1.

Amey.

[thebird]

Hi,

We have done this in some of our jobs. We have a parameter RejectThreshold and if the number of rejects cross this limit, then the job aborts.

What we do is pass the reject link to a transformer running in sequential mode, and then pass two links to a Compare Stage.

One of these links is set to @INROWNUM and the other link checks if the Reject Threshold has been crossed or not. The derivation used is -

if @INROWNUM<=RejectThreshold then @INROWNUM else RejectThreshold

These two links are sent to the compare stage which also runs on a sequential mode. The compare stage has a property, Abort on Difference, which is set to True. Compare stage to a Peek Stage.

Since the Reject Threshold is parameterised and would be set to a low value, the sequential running of the transformer would not be much of a let down, I believe.

Regards,

The Bird.

[balki]

if @INROWNUM<=RejectThreshold then @INROWNUM else RejectThreshold

These two links are sent to the compare stage which also runs on a sequential mode. The compare stage has a property, Abort on Difference, which is set to True. Compare stage to a Peek Stage.

Regards,

The Bird.

Hi,

This is a good suggestion. But if I am not not wrong, we need to set the
'Abort on Difference ' property to False right?

Balki

[thebird]

This is a good suggestion. But if I am not not wrong, we need to set the
'Abort on Difference ' property to False right?

Balki

Hi Balki,

If the requirement is to abort the job if the threshold is crossed, then the 'Abort on Difference ' property has to be set to TRUE, only then shall the job abort, but if you dont want the job to abort, yes ofcourse, this property would have to be set to False.

Regards,

The Bird.