DSXchange

Hello Everyone,
I have to validate input file datatypes and collect all the invalied records to the reject file.Presently iam using the stage variables, one for each field to validate. Valid = 0, Invalid = 1 and lastly sum them. Good row constraint: sum = 0, reject everything else.My question is how to make this job generic which can handle any metadata.Say if one input file have 3 date fields to validate and 2nd file have 10 date fields to validate is there a method to create only one job to validate all the files.Is it possible using schema file?If so how can we capture all the columns with a particular datatype say Date?

Thanks in advance

That is a difficult question, and I'm not sure it can be done. The schema could define default values for fields which don't validate and that value could be set to some common one outside the normal data range but that wouldn't work as a generic solution, unfortunately.

ArndW wrote:That is a difficult question, and I'm not sure it can be done. The schema could define default values for fields which don't validate and that value could be set to some common one outside the normal data range but that wouldn't work as a generic solution, unfortunately.

I really appreciate if you have any idea to implement this...

kogads wrote:

ArndW wrote:That is a difficult question, and I'm not sure it can be done. The schema could define default values for fields which don't validate and that value could be set to some common one outside the normal data range but that wouldn't work as a generic solution, unfortunately.

I really appreciate if you have any idea to implement this...

Hi,
I just want to update this post so that it doesnt go back in the list.I am waiting for any reply from the experts.

Thanks.

I agree with Arnd's assessment.

Hi,
I am not sure about the generic job and schema file.But i can share you a workaround which can be of help.
Load all your req column names and their data types/code for all datatype( eg:0 for char,1 for num, 2 for date and so on)

Use your source file as your driver and do a look up using column names as your key on the file you have just created.get the data type/ corresponding code from the look up file and do the validation in the next stage a transformer and reject all the records which din't meet the requirements.


Regards,
Raja R P