Warning message about duplicate entry

[somu_june]

Hi,

When Iam running a job which has lookup stage that is looking to db2 table. Iam getting a warning . Please help me in getting out of this warning.

warning message:

Lookup,0: Ignoring duplicate entry at table record 5,216; no further warnings will be issued for this table


Thanks,
Somaraju

[kumar_s]

Hi,
This warning cant be cut from DS (Other that message handler ). DB2 source has duplicates on the key you specified. Try to avoid that, or specify composit key.

-Kumar

[somu_june]

Hi Kumar,

Thanks for your replys. They are really helping me to solve my problems .Can I use select distinct columm1,columm2 from table. so that I can over come this problem.


Thanks,
Somaraju

[kumar_s]

Yes you can, provided the lookup value is also duplicated. Else you shouldnt loose the valuble value. (By extracting first or last records).
You may also need to concentrate on the loading process of data into DB2. Root cause of the duplicated.

-Kumar

[somu_june]

Hi Kumar,

I selected Enter custom sql in DB2 stage and I provided sql as select Distinct xxx, uuu,yyy,zzzz fom ttttttttt where xxx= '230' . But Iam getting the same warning as I mentioned above. duplicate entry warning. You siad allow duplicate in look up stage .How to do that please guide me. Iam giving key as
xxx and zzzz in lookup



Thanks,
Somaraju

[kumar_s]

Hi,

You query extracts the distinct value among the combination of xxx, uuu,yyy,zzzz. Where as your key is only xxx and zzz. Where still there is the possibilites of duplicate of the key.
To ensue the non duplicates try with select count(*) from ttttttt group by xxx, zzz and ensure you are getting all 1s.
And more over i dont recall, that i told regarding the option of allow duplictes in Lookup.

[MTA]

Somu, You might as well consider using a sort stage between the db2 lookup stage and look you stage. The advantage of using sort stage is two fold.
1) you can remove duplicates
2) Having sorted data will help improve performence.