Validate Datatypes

kogads · Post by **kogads** » Wed Oct 28, 2009 6:21 pm

Hello Everyone,
I have to validate input file datatypes and collect all the invalied records to the reject file.Presently iam using the stage variables, one for each field to validate. Valid = 0, Invalid = 1 and lastly sum them. Good row constraint: sum = 0, reject everything else.My question is how to make this job generic which can handle any metadata.Say if one input file have 3 date fields to validate and 2nd file have 10 date fields to validate is there a method to create only one job to validate all the files.Is it possible using schema file?If so how can we capture all the columns with a particular datatype say Date?

Thanks in advance

ArndW · Post by **ArndW** » Thu Oct 29, 2009 1:14 am

That is a difficult question, and I'm not sure it can be done. The schema could define default values for fields which don't validate and that value could be set to some common one outside the normal data range but that wouldn't work as a generic solution, unfortunately.

kogads · Post by **kogads** » Thu Oct 29, 2009 11:55 am

ArndW wrote:That is a difficult question, and I'm not sure it can be done. The schema could define default values for fields which don't validate and that value could be set to some common one outside the normal data range but that wouldn't work as a generic solution, unfortunately.

I really appreciate if you have any idea to implement this...

kogads · Post by **kogads** » Mon Nov 02, 2009 4:53 pm

kogads wrote:
ArndW wrote:That is a difficult question, and I'm not sure it can be done. The schema could define default values for fields which don't validate and that value could be set to some common one outside the normal data range but that wouldn't work as a generic solution, unfortunately.
I really appreciate if you have any idea to implement this...

Hi,
I just want to update this post so that it doesnt go back in the list.I am waiting for any reply from the experts.

Thanks.

ray.wurlod · Post by **ray.wurlod** » Mon Nov 02, 2009 5:57 pm

I agree with Arnd's assessment.

RAJARP · Post by **RAJARP** » Tue Nov 03, 2009 12:27 am

Hi,
I am not sure about the generic job and schema file.But i can share you a workaround which can be of help.
Load all your req column names and their data types/code for all datatype( eg:0 for char,1 for num, 2 for date and so on)

Use your source file as your driver and do a look up using column names as your key on the file you have just created.get the data type/ corresponding code from the look up file and do the validation in the next stage a transformer and reject all the records which din't meet the requirements.

Regards,
Raja R P