Error in Match Reference Job

asyafrudin · Post by **asyafrudin** » Wed Jul 06, 2011 10:37 pm

Hi.

I'll get to the point. Is it possible that the condition of data inside the blocking or matching column(s) caused a match reference job to fail? Any characters that shouldn't be in the blocking or matching column(s)? I've been getting errors in my match reference job and I'm currently suspicious that the errors were caused by what's inside the column(s).

I hope the question is clear.

Thank you.

ray.wurlod · Post by **ray.wurlod** » Thu Jul 07, 2011 12:35 am

What errors are you getting? Do they relate, perhaps, to "block overflow"? That might mean that your blocking fields aren't discriminating well enough, so you are ending up with more records per block than can conveniently fit in the allocated amount of memory. Read more about block overflow in the QualityStage User Guide manual.

asyafrudin · Post by **asyafrudin** » Thu Jul 07, 2011 10:08 am

ray.wurlod wrote:What errors are you getting? Do they relate, perhaps, to "block overflow"? That might mean that your blocking fields aren't discriminating well enough, so you are ending up with more records per blo ...

I can't see all your replies. I don't have a premium account. Noted your suggestion on block overflow though. I'll look into it ASAP. It's just that in almost every match reference job that I made, I always end up having one or more block overflow warnings. But that was it; only warnings, not errors.

One more thing. This errors occurred every time I'm using columns from the output of Standardize stage. If I skip this stage and use raw data (non-Standardize) directly, my match reference job produced no errors. I suspected the cause of the error was the output from the Standardize stage. This is why I asked the above question. Is it possible that a certain type of data causes a match reference job to fail?

ray.wurlod · Post by **ray.wurlod** » Thu Jul 07, 2011 3:00 pm

It would be easier if you posted the exact error message(s). I have never seen errors caused by data (apart from NULL in the wrong place).

asyafrudin · Post by **asyafrudin** » Thu Jul 07, 2011 11:45 pm

ray.wurlod wrote:It would be easier if you posted the exact error message(s). I have never seen errors caused by data (apart from NULL in the wrong place). ...

Here it is:
Reference_Match_0,0: Fatal error from object or function QSM_MatJoinOp, code 1

I already posted this error message in another thread and nobody responded. I think the error message is too broad to trace.

FYI, the match reference job already produced match statistics. I can see them from the Director.

Thanks in advance.

ray.wurlod · Post by **ray.wurlod** » Fri Jul 08, 2011 6:55 pm

MatJoinOp is the operator that looks after the join on the combinations of blocking fields from the two sources. I presume you have specified these correctly, since the Match Designer won't really let you do otherwise. Is there any more detail to the error message? Might there be null values in the data?

asyafrudin · Post by **asyafrudin** » Fri Jul 08, 2011 7:01 pm

ray.wurlod wrote:MatJoinOp is the operator that looks after the join on the combinations of blocking fields from the two sources. I presume you have specified these correctly, since the Match Designer won't really le ...

Thanks for the help. Too bad I can't access premium content.

ray.wurlod · Post by **ray.wurlod** » Fri Jul 08, 2011 7:03 pm

Unblocked that one, since I'm asking questions rather than offering premium advice.

asyafrudin · Post by **asyafrudin** » Fri Jul 08, 2011 7:17 pm

ray.wurlod wrote:MatJoinOp is the operator that looks after the join on the combinations of blocking fields from the two sources. I presume you have specified these correctly, since the Match Designer won't really let you do otherwise. Is there any more detail to the error message? Might there be null values in the data?

I'm pretty sure there are no null values in the data. The data I'm using is the output of Standardize stage performed by a colleague. I guess I need to confirm this matter to him. Maybe he made changes to the rule set that produced NULL values. Unfortunately I don't have any more details to the error message. I'll look into it again and post more information ASAP.

Thank you for all the help.

asyafrudin · Post by **asyafrudin** » Mon Jul 18, 2011 9:56 pm

I'm back; with no workaround nor solution to my problem.

I think I need to go back and share more on my match specifications in this thread. Hopefully, someone could help me identify the source of my problem.

Here goes.

I have a Reference with columns r1 and r2. This Reference will be matched with a Data consisting of columns d1, d2. The match specification should be blocking by (r1,d1), matching by (r2,d2). This is the original match specification. Let's call it MSpec1. Nothing went wrong with MSpec1. Although the results were far from accurate, the job (let's call it Job1) was successful.

Next, I standardized columns r2 (in Reference) and d2 (in Data). The Reference now has columns r1, r2, r21, and r22 with r21 and r22 as the output from standardizing column r2. The Data (as expected) now has columns d1, d2, d21, and d22 with columns d21 and d22 is the output of standardizing column d2.

With the above changes, the match specification was also changed to blocking by (r1,d1), matching by (r21,d21)+(r22,d22). Let's call this MSpec2. The job (let's call it Job2) with MSpec2 failed to execute generating the error I mentioned in the first post on this thread.

Next, I tried using back MSpec1. The difference is that I'm using the columns already standardized. So I have r1, r2, r21, r22 in Reference and d1, d2, d21, d22 in Data. As mentioned above, blocking is by (r1,d1) and matching is by (r2,d2). Quite unexpected, this job (let's call it Job3) also failed and returns the same error as with MSpec2.

I thought for sure that running Job3 won't produce any errors because I'm using the same match specification as Job1. Turns out the mere existence of r21, r22, d21, and d22 was already enough to generate error. Now this is confusing for me. Why is it that including these output of standardize stage would generate the error, even though they aren't included in the match specification?

Now I'm facing a dilemma. Using the original columns only should be successful but the results are terrible. Using the standardized columns should definitely improve the match results but I can't get through the errors. Help?