DSXchange

Hello,

What is the best method of identifying records that generate warnings/errors? We have millions of records as input and we implement the StringToDate() function and some records generate the following warning in the log:
We know which stage is giving us the warning but would like to know the record that causes it. Thanks for the help.

You'll need to build in your own tests. They don't happen automatically.

OK. Thanks.

Have you/do you have access to any data profiling tools, such as Information Server Information Analyzer or InfoSphere Discovery (there are also a whole host of non ibm and open source options). These sorts of tools should be used during the onboarding/analysis of new data sources to identify the unique patterns (e.g. pattern YYYY-MM-DD occurs 70% of the time and DD-MMM-YYYY occurs 20% etc) as opposed to values found in the data so that transformation requirements can be identified upfront.

Else if you want to test in DataStage a precheck something like this post might allow you to trap the data and perform some additional transformations.