handling invalid data in dataset

[Poovalingam]

Hi gurus,
I need a suggestion or solution in a design for which Source is a data set and target is Oracle.
The meta data of dataset and the Oracle table is same.
When the dataset is being created with any invalid data to the corresponding data type, some ****** values are assigned to them.
Now, before reaching oracle, how I do can stop propagation of those invalid records into the Oracle?

thanks in advance,
Poova.

[ray.wurlod]

How the heck did you get invalid data into a Data Set in the first place?!!

These should be rejected in the job that populates the Data Set.

[gssr]

Hi gurus,
I need a suggestion or solution in a design for which Source is a data set and target is Oracle.
The meta data of dataset and the Oracle table is same.
When the dataset is being created with any invalid data to the corresponding data type, some ****** values are assigned to them.
Now, before reaching oracle, how I do can stop propagation of those invalid records into the Oracle?

thanks in advance,
Poova.

You can filter the records with invalid data (*****) using filter stage!

[sudhanmaad]

May be the Conversion that you have done is wrong , or format is wrong(like you have convertd into a Timestamp and in metadata definition you have given Date) . these are the scenarios where Data appears '**********' in Dataset. if this is not the case then Data is wrong.please filter properly.

[Poovalingam]

Thanks all.

To Prepare dataset, the source is a fixed width file and we captured all data as char data type. Then we used transformer for doing some transformation and to do required type conversion. But some data reaching as *** into the dataset.

Do we need to write filter condition for every column whichever i'm feeling that may contain bad value?

I want to know as well,
1. Doesn't the transformer reject the record when doing type conversion and transformation for invalid data?
2. When invalid data (***) enters into the data set, Will DS throw warning?

Regards,
Poova.

[priyadarshikunal]

seems like the data is being padded with '*' or during null handling *** is assigned. check the data flow.

[ETLJOB]

Probably you are missing out some valid data because of improper type conversion, null handling etc and you named it as "invalid" data. Throughly check your entire flow again...

[ETLJOB]

Also, let us know what your log says?

[sudhanmaad]

Just try mapping all the columns to dataset without any transformation done , and check whether now the data is still with '********'

[nagarjuna]

I think something is wrong in null handling of timestamp fields .Give the null field value . Is there any reason why you are reading with correct datatype in seq file .It will automatically reject invalid data .