writing a warning to the log in a parallel job

michaeld · Post by **michaeld** » Sat Feb 03, 2007 6:43 pm

Is there any way to write a warning to the log in a parallel job. Not server or sequence. I know how to do that. I need to know how to do it in a parallel job.

I tried the following, but they don't work in a parallel job transformer:

UtilityWarningToLog()

DSLogWarn()

Maybe you know of something that I can do in an generic stage using an orchestrate operator? I'm pretty much out if ideas...

Right now I use a compare stage with a dummy data set as input. That will cause it to send a warning to the log when the two input data sets don't have the same number of rows, but it's such a hack. I might as well cast a literal non numeric string to an integer... A proper solution would be nice.

DSguru2B · Post by **DSguru2B** » Sat Feb 03, 2007 7:07 pm

You can use a basic transformer if you want to raise a warning.
In a px transformer, you can abort a job. I am not sure if you can raise a warning.

kumar_s · Post by **kumar_s** » Sun Feb 04, 2007 12:56 am

What is the scenario in which you are trying to create this warning?

ray.wurlod · Post by **ray.wurlod** » Sun Feb 04, 2007 2:40 pm

All warnings in parallel jobs have to be logged by the conductor process. You have access to routines like UtilityWarningToLog() within before/after subroutines - you would need to create your own - or, as others have noted, using BASIC Transformer stage or server shared container.

The burning question, then, is how you wish to trigger this warning - that is, what artifact of the data or processing rules requires a warning to be logged - and whether it may need to be triggered on a row-by-row basis or only once per run. The answer to this question will influence the method that you choose to write the log entry.

michaeld · Post by **michaeld** » Mon Feb 12, 2007 7:55 am

Background:

I need to write this warning when the compare stage has two rows that do not match. Both input datasets will have one row each. The purpose is to see if the data matchs the balance row at the end of the file. The balance row would have a row count and a sum of some of the columns. Then I use an aggregate stage to get the same format from my data and use a compare stage to see if there is a difference. If there is then I want to send a warning to the log with will be interpreted by my sequence job and it will not continue. It will send an email saying that the input data did not balance.

Reply to your suggestion:

How can I use an after-job subroutine to check the value of the result column in the compare stage to see if I should write a warning to the log? I didn't know that after-job subroutines have access to the data in the result columns.

Rant:

I suppose I can write the difference to a file and then check it in the sequence job, but that's just more files to maintain. I wish parallel jobs were as flexible as server jobs... where you can write BASIC anywhere you want. I'd rather not use a BASIC transformer because it's just a legacy stage from server jobs... I wonder if DS8 will have BASIC transformers.. they should do something to allow more control over parallel jobs (writing a custom stage in C++ is not a good option - too much effort and maintenance )

ray.wurlod · Post by **ray.wurlod** » Mon Feb 12, 2007 2:50 pm

The Compare stage has a Warn on Record Count Mismatch property that will log a warning. While that's not your complete solution (nor is Abort on Mismatch), it's worth noting. A Compare stage only effects the comparison (philosophy: one stage, one task) and emits a coded value to indicate the result. You need to process the result downstream, for example using a Transformer, Switch or Filter stage.

BASIC Transformer is not the same as server Transformer - for one thing it doesn't do reference lookups. And, yes, BASIC Transformer does exist in DataStage version 8.0.

You could construct a link that is only taken if there is an imbalance and use a BASIC Transformer on that link - if it's not used it will not incur any performance overhead. One would hope that these imbalances are infrequent?

You're correct about writing the imbalances to a file and retrieving them using the after-job subroutine.

There are others (C programmers) who would argue that writing BASIC routines is too much maintenance compared to C++; it's a matter of your particular skill set.