Replacing lookup with Joiner

[deva]

Hi I am joining two tables , If I use lookup stage, due to volume of data it getting aborted, because of data limit is exceeding 4 gb.

that is why I am replacing Joiner over there. If I use joiner I am getting warning and job getting fail.

error
Join_67: When checking operator: On input data set 1: When binding input interface field "BSNS_SGMNT_ID" to field "BSNS_SGMNT_ID": Converting a nullable source to a non-nullable result;
a fatal runtime error could occur;
use a modify operator to specify the value to which the null should be converted.

for the BSNS_SGMNT_ID column left link is not null , right link is null (yes)
we are doing left outer join.

If I use lookup some times job getting success.

[vivekgadwal]

Hi I am joining two tables , If I use lookup stage, due to volume of data it getting aborted, because of data limit is exceeding 4 gb.

that is why I am replacing Joiner over there. If I use joiner I am getting warning and job getting fail.

error
Join_67: When checking operator: On input data set 1: When binding input interface field "BSNS_SGMNT_ID" to field "BSNS_SGMNT_ID": Converting a nullable source to a non-nullable result;
a fatal runtime error could occur;
use a modify operator to specify the value to which the null should be converted.

for the BSNS_SGMNT_ID column left link is not null , right link is null (yes)
we are doing left outer join.

If I use lookup some times job getting success.

Think about it - if you are doing a left outer join and a column in the left link is NOT NULLABLE, then why would you want that to be NULL in the output of the join stage? Have it as NOT NULL too.

[deva]

both source and lookup both are not null columns but no records are inserting into database. if I run the job using lookup records are inserting

[deva]

both source and lookup both are not null columns but no records are inserting into database. if I run the job using lookup records are inserting.

when I replacing lookup with join do i need to modify anything?

[Kryt0n]

Any output columns coming from the right side of your join need to be nullable.

On the input side, the columns on which you are joining should be non-nullable.

Also make sure your partitioning is the same for both input streams

[vivekgadwal]

Any output columns coming from the right side of your join need to be nullable.

On the input side, the columns on which you are joining should be non-nullable.

Also make sure your partitioning is the same for both input streams

Right...Keeping the partitioning scheme to Auto for a Join stage should do Key partition automatically. However, it is also a good practice if you explicitly mention the key partition.