Invalid/Error Data handling

rohit_mca2003 · Post by **rohit_mca2003** » Wed Nov 24, 2010 5:16 am

Hi,

I have a record which is having nearly 15 fields/columns. We have to do some data values/format/nullability checks on some 10 fields. We decided to do these checks in Transformer stage. Till here no problem.

But we have to insert records in ERRORLOG table for each field validatin failure. Suppose 3 field validation failed then 3 records in ERRORLOG and if 8 fields validation failed then 8 records in ERRORLOG table.

Please suggest me solution/design of this job and performance should not be compromised.

Regards,
Rohit

ArndW · Post by **ArndW** » Wed Nov 24, 2010 7:34 am

One output link per error, then funnel the errors together to one stream for the error log. This is a common approach. We have one job at this site with many columns and each column can have multiple errors, so with one input link to a transform stage we have at least 50 output links and this job has the nickname of the "Broom Job" since the links sort of look like one.