job stop processing!

[wuruima]

dear all,
I meet a wierd situation that the parallel job stop processing in a lookup stage.

according to the performance statistics, I can see the 2 input links of the lookup stage have records pass in. However the output links(output and reject links) keep 0 rows all the time. I have no idea what's wrong.

I used another set of data to run the same job, it can complete successfully.

does anyone meet this situation? anyone knows what's wrong with the job?

Thanks in advance and wish you all the best.

Walter/

[ShaneMuir]

The obvious answer is that your key values do not match on the inputs. Have you proved that the input values are matching?

It could also be that your partitioning is incorrect so the values cannot find each other within the given partitions.

[wuruima]

I use another set of data to process the same job, it could complete.

after checking the director, I found another intersting thing.
The job is hold on below log.
hmftpattr_cut_out,0: Progress: 90 percent.
normally after that there is
hmftpattr_cut_out,0: Import complete; 6547564 records imported successfully, 0 rejected.
I am afraid the job is waiting the read file complete.

[wuruima]

When I force abort the job. This log appear immediately the then job abort.
hmftpattr_cut_out,0: Import complete; 6547564 records imported successfully, 0 rejected.

I have no idea why this will happen.

[ShaneMuir]

So if I follow you correctly - the job is not finishing - but rather hanging?

What is your stage after the lookup?

[wuruima]

it's transfomer after lookup.
I already note the problem. It's because the reference link of the lookup stage has too many records, the job is hang in the stage. it's better to use join instead of lookup.