Is there any functionality for setting abort thresholds on reject links. For example, if more than 50 records on reject link then abort job. Or even better, if > 1% of records end up on reject port, then abort job.
I understand that I can always join to the non reject link and do some calculation then cause the abort, but that's too much work and too processing intensive.
I am interested in such functionality in both EE and Server.
Reject thresholds for reject links
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 24
- Joined: Fri Oct 01, 2004 1:43 pm
- Location: USA
Re: Reject thresholds for reject links
If you use a transformer in EE you can click on the contraints and set a number. I'm pretty sure that this field does not allow a parameter value. To do the 1% thing you can use a transformer again. I know, I know, not the most efficient EE stage, but it's just an idea. You'll obviously need to determine the number of input rows you are starting with. In your stage variables keep track of the number of rows that have aborted. Have a special link coming out of the transformer that checks to see if the total that has aborted it equal to this tolerance. If it has then send a row down it to a peek stage. Change the contraints to abort after one row.
HTH.
-rls
HTH.
-rls
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The server job answer is easy; you can set this hard limit in the Constraints grid in a Transformer stage.
In parallel jobs you don't have this functionality, so need something downstream on the rejects link to count the number of rows and trigger abandonment of processing.
In parallel jobs you don't have this functionality, so need something downstream on the rejects link to count the number of rows and trigger abandonment of processing.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Charter Member
- Posts: 166
- Joined: Wed Mar 16, 2005 6:52 am
- Location: Mumbai, India
An interesting little undocumented abort operator.
There is another way to set up aborts that I'd discovered by accident in 7.5.
I dont have access to DataStage at the moment but heres how:
When previewing data either from a Sequential file or database source, there is an option to view the generated OSH. In that OSH, you will find an abort operatot that takes an nrecs argument.
A custom stage can be wrapped around this operator:
1 input
0 outputs
and one stage level property that takes the number of records to abort after(can be parameterised)
When run in sequential mode and with RCP on(No need to map columns), this stage will read input rows until their number equals the value set in the stage property.
This has been tested DSEE 7.5.1.
Amey.
I dont have access to DataStage at the moment but heres how:
When previewing data either from a Sequential file or database source, there is an option to view the generated OSH. In that OSH, you will find an abort operatot that takes an nrecs argument.
A custom stage can be wrapped around this operator:
1 input
0 outputs
and one stage level property that takes the number of records to abort after(can be parameterised)
When run in sequential mode and with RCP on(No need to map columns), this stage will read input rows until their number equals the value set in the stage property.
This has been tested DSEE 7.5.1.
Amey.
Reject Threshold
Hi,
We have done this in some of our jobs. We have a parameter RejectThreshold and if the number of rejects cross this limit, then the job aborts.
What we do is pass the reject link to a transformer running in sequential mode, and then pass two links to a Compare Stage.
One of these links is set to @INROWNUM and the other link checks if the Reject Threshold has been crossed or not. The derivation used is -
Since the Reject Threshold is parameterised and would be set to a low value, the sequential running of the transformer would not be much of a let down, I believe.
Regards,
The Bird.
We have done this in some of our jobs. We have a parameter RejectThreshold and if the number of rejects cross this limit, then the job aborts.
What we do is pass the reject link to a transformer running in sequential mode, and then pass two links to a Compare Stage.
One of these links is set to @INROWNUM and the other link checks if the Reject Threshold has been crossed or not. The derivation used is -
These two links are sent to the compare stage which also runs on a sequential mode. The compare stage has a property, Abort on Difference, which is set to True. Compare stage to a Peek Stage.if @INROWNUM<=RejectThreshold then @INROWNUM else RejectThreshold
Since the Reject Threshold is parameterised and would be set to a low value, the sequential running of the transformer would not be much of a let down, I believe.
Regards,
The Bird.
Re: Reject Threshold
Hi,thebird wrote:These two links are sent to the compare stage which also runs on a sequential mode. The compare stage has a property, Abort on Difference, which is set to True. Compare stage to a Peek Stage.if @INROWNUM<=RejectThreshold then @INROWNUM else RejectThreshold
Regards,
The Bird.
This is a good suggestion. But if I am not not wrong, we need to set the
'Abort on Difference ' property to False right?
Balki
Re: Reject Threshold
Hi Balki,balki wrote: This is a good suggestion. But if I am not not wrong, we need to set the
'Abort on Difference ' property to False right?
Balki
If the requirement is to abort the job if the threshold is crossed, then the 'Abort on Difference ' property has to be set to TRUE, only then shall the job abort, but if you dont want the job to abort, yes ofcourse, this property would have to be set to False.
Regards,
The Bird.