DSXchange

The solution to this would be if you find a logic to reverse group the data. the data that you show is grouped by two fields.

This is known as Code to Name or Name to code Look-up. :!: :!: :shock: These translations are defined by the users. KAPS, you stated the question in a wrong way. This is not called cleanse/confirm. :x The most effecient and generic way to do this is using Hashed files in Server or Lookup stage in EE...

Tell me this samba,
when a record gets updated in your source table, is there any column there that stores a timestamp which tells you when this record was last updated?

Probably whats happening is that you are trying to load the Null values in a Sequential file. This will not be handled, hence DS is converting the nulls to a default.
I would recommend a peek stage to be added to all the jobs and observe the data there and post your observation.
Thanks.

Kumar,
Did u try handling it with the IF ISNULL(LINKNAME) logic?
I had this problem as well, you might want to check the logic for the condition too which is defaulted to Reject. ( this will be in a lookup stage).

This also often happens when the datatypes and the nullabilties of the columns are not in sync. As I mentioned in my other post, Datastage Enterprise Edition is very sensitive with handling nulls. And due to this you might see ripple effects in different stages. Better to follow the steps shown by K...

You should go by the book. :D

A lookup stage for looking up 30M rows is not a good idea. The processes will die very often.
As said by the DS Manuals and Burney, Join Stage should be your first option for this.

I have gone through this problem alot. This happens when the box configuration is not modified. Meaning. the Cache space for OSH should be increased from a default of probably 256mb to 1G which would be helpful enough. This shall be done by the unix admin.
Ray.. please comment or correct me.
Thanks

Another way to handle nulls in EE is that you have to take care of the nullability of the columns. EE is very sensitive in this case as compared to Server. So better take care of that, else it will result in numerous fatal errors.
Hope it help.

can you post the query please.

Does the extract even run?
Do select on the target and see if it loaded or not.

Here, try this. It works for me. I bet it will work for you

If trim(tpo_loan_orig_out.PRODUCT_DESCRIPTION) Matches "...alloo..." Then "Y" Else "N"

Actually i already did build a routine. For the very same reason that it is needed more than once in quite a number of different jobs.
Thanks everyone guys.

Thanks naveen. i basically did the exact same thing you mentioned but in the transformer itself. I guess it would be better to put it in a routine.
Thanks once again naveen.