Standardization job along with WISD IP/OP

divstands · Post by **divstands** » Mon Jul 06, 2009 10:25 am

I built a job for address standardization. The job looks like:
Seq File->Standardize(Country.set)->Filter->Standardize(MXADDR.set)->Seq File

Now in place of the input and output Sequential files, I attached WISD Input and WISD output Stages respectively.

The job with the above configuration is not giving an output. As per the error log, the job is aborted after 3 hrs.

5:43:38 PM
main_program: This step has 5 datasets:
ds0: {op0[1p] (sequential MDM_Input_Address)
eAny->eCollectAny
op1[1p] (parallel StandardizeCNTRY)}
ds1: {op1[1p] (parallel StandardizeCNTRY)
eAny->eCollectAny
op2[1p] (parallel APT_CombinedOperatorController:SelectMX)}
ds2: {op2[1p] (parallel APT_CombinedOperatorController:APT_TransformOperatorImplV18S3_ISD_MDMQS_Address_Standardization4_Transformer_59 in Transformer_59)
eAny->eCollectAny
op3[1p] (parallel StandardizeAddr)}
ds3: {op3[1p] (parallel StandardizeAddr)
eAny->eCollectAny
op4[1p] (parallel APT_TransformOperatorImplV18S4_ISD_MDMQS_Address_Standardization4_Transformer_73 in Transformer_73)}
ds4: {op4[1p] (parallel APT_TransformOperatorImplV18S4_ISD_MDMQS_Address_Standardization4_Transformer_73 in Transformer_73)
->eCollectAny
op5[1p] (sequential QS_Output_Address)}
It has 6 operators:
op0[1p] {(sequential MDM_Input_Address)
on nodes (
node1[op0,p0]
)}
op1[1p] {(parallel StandardizeCNTRY)
on nodes (
node1[op1,p0]
)}
op2[1p] {(parallel APT_CombinedOperatorController:
(SelectMX)
(APT_TransformOperatorImplV18S3_ISD_MDMQS_Address_Standardization4_Transformer_59 in Transformer_59)
) on nodes (
node1[op2,p0]
)}
op3[1p] {(parallel StandardizeAddr)
on nodes (
node1[op3,p0]
)}
op4[1p] {(parallel APT_TransformOperatorImplV18S4_ISD_MDMQS_Address_Standardization4_Transformer_73 in Transformer_73)
on nodes (
node1[op4,p0]
)}
op5[1p] {(sequential QS_Output_Address)
on nodes (
node1[op5,p0]
)}
It runs 6 processes on 1 node.

9:09:06 PM
Job control process (pid 1188020) has failed

I have tried this a couple of times. However I see that the default job which comes with IIS:
a). which does the act of Address Standardization for MDM, using Standardization Rules MDMUSA
b). and which also if feeded in the form of a container between the WISD input and WISD output stages
works fine.

Am I missing some configuration?

Note:
1. Having a doubt that the rulesets I am using(COUNTRY.set and MXADDR.set) should be there in the same sub-project where the job is there, I re-imported the rulesets at the appropriate location.

eostic · Post by **eostic** » Mon Jul 06, 2009 8:02 pm

Tell us more. Does it abort on its own, after you have deployed it via the wisd console and before you ever invoke it? ....after a single invocation? Does the job ever get up and running? How many instances have you set inside of the WISD console......?

Do you have other WISD jobs that still work?

It could be a data issue, or something more subtle in the deployment. If a data issue, I've seen cases where the APT_DEFAULT_PADCHAR or whatever it is called causes problems because it defaults to nulls..... so character columns returned might be an issue if not fully padded with spaces...they can blow up the SOAP engine......

Lots of possibilities.. Let us know as best you can "when" the error occurs.

Ernie

stuartjvnorton · Post by **stuartjvnorton** » Tue Jul 07, 2009 6:52 pm

Search the forums for "disable operator combination", update the job and re-run to get more useful error messages.

eostic · Post by **eostic** » Wed Jul 08, 2009 5:52 am

...missed your line about it working fine in the sample other job....

This sounds oddly like it could also be an end-of-wave problem. If it is still occuring, try this:

Right before your WISDOutput put in a Transformer with two output links. Send the first output link to a flat file or Peek ....the second to your WISDOutput as you have now.

Compile and Deploy it. Be sure it's instance is 'up and running' via the DS Director.

Invoke it....assuming you get no data in a reasonable time, go back to the WISD Console..... the _deployed_applications_workspace_ , select your application, click the "EDIT" button in the lower right corner, and notice that a new button appears towards the lower left...... for disable/enable ..... select this and "disable" the appliction. This will change the appearance of the icon next to it above in the tree.

Be sure that it is no longer running via the DS Director.

You have "cleanly" shut it down. If the issue is an end-of-wave problem, your Peek or seq file will now have your output row.

Ernie

divstands · Post by **divstands** » Wed Jul 08, 2009 11:02 am

eostic wrote:...missed your line about it working fine in the sample other job....

This sounds oddly like it could also be an end-of-wave problem. If it is still occuring, try this:

Right before your WISDOutput put in a Transformer with two output links. Send the first output link to a flat file or Peek ....the second to your WISDOutput as you have now.

Compile and Deploy it. Be sure it's instance is 'up and running' via the DS Director.

Invoke it....assuming you get no data in a reasonable time, go back to the WISD Console..... the _deployed_applications_workspace_ , select your application, click the "EDIT" button in the lower right corner, and notice that a new button appears towards the lower left...... for disable/enable ..... select this and "disable" the appliction. This will change the appearance of the icon next to it above in the tree.

Be sure that it is no longer running via the DS Director.

You have "cleanly" shut it down. If the issue is an end-of-wave problem, your Peek or seq file will now have your output row.

Ernie

IS it possible to throw an error to the connected application in real time as and when an address has unhandled patterns at the real time?
Will Peek stage do the above task?

eostic · Post by **eostic** » Wed Jul 08, 2009 3:41 pm

Aren't you always sending back the response? Wouldn't your unhandled pattern appear in either a direct stan or possibly investigate result? ...regardless? (assuming the process flow is finally working).

Ernie

divstands · Post by **divstands** » Wed Jul 08, 2009 5:04 pm

eostic wrote:Aren't you always sending back the response? Wouldn't your unhandled pattern appear in either a direct stan or possibly investigate result? ...regardless? (assuming the process flow is finally working).

Ernie

Yeah the process flow is finally working
but
the results of investigate and unhandled pattern reports of standardization are useful till only for batch load or in other words till a person like me is sitting here analyzing the unhandled stuff. But what during real time?

ray.wurlod · Post by **ray.wurlod** » Wed Jul 08, 2009 5:05 pm

During real time you pass back the standardized result, "for better or worse".

divstands · Post by **divstands** » Wed Jul 08, 2009 5:09 pm

ray.wurlod wrote:During real time you pass back the standardized result, "for better or worse". ...

yeah i understand that as the first thought solution(mean it), but currently the client wants to handle the error ifever there exist any unhandled part in the supplied address.

eostic · Post by **eostic** » Wed Jul 08, 2009 7:51 pm

You can get as creative as you want.... I suppose you could add such logic to the QS job....although "on the fly" fixing is not usually where I've seen standaridation and investigation being used. Exceptions are found, rules are tweaked, overrides created, and then one less exception results the next time around.......

...dynamically fixing it could be done downstream, but in a lot of cases QS is being called as a fine-grained basic level service to say "good" or "bad", and the receiving java or web service app decides whether to try and fix it and send it back for checking again, or to throw it into a review pile.

Ernie

divstands · Post by **divstands** » Thu Jul 09, 2009 12:01 am

eostic wrote:You can get as creative as you want.... I suppose you could add such logic to the QS job....although "on the fly" fixing is not usually where I've seen standaridation and investigation being used. Exceptions are found, rules are tweaked, overrides created, and then one less exception results the next time around.......

...dynamically fixing it could be done downstream, but in a lot of cases QS is being called as a fine-grained basic level service to say "good" or "bad", and the receiving java or web service app decides whether to try and fix it and send it back for checking again, or to throw it into a review pile.

Ernie

Yeah... so what i want to know is how do i tell the Java app that the address which is coming is not properly standradized(meaning: a NOT NULL value went into the "Unhandled Data" field)

My job looks like:

WISD INPUT->Container(standardization job)->WISD Output

eostic · Post by **eostic** » Thu Jul 09, 2009 6:51 am

Not sure what your goal is. The java program can certainly look at the unhandled data and flag the address as bad on its own. Hard to say how it might correct it --- indeed that could require clerical review.

Are you ultimately validating the address for actual location? Perhaps having QS perform a CASS or WAVES certification would be appropriate, thus passing back something more rich?

Ernie

eostic · Post by **eostic** » Thu Jul 09, 2009 10:11 pm

There is nothing automatic here. If you have not-null in your unhandled data, then craft an error message and send that back in your SOAP envelope in another column, with your own custom text message. That "message" column will be in the WSDL, and the java client can check that along with a flag or whatever else you might like to send back. There is no capability for a custom SOAP fault, if that's what you mean.

Ernie