Null handling for 200 / 300 columns at one go.

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
friend.kak@gmail.com
Participant
Posts: 28
Joined: Sat May 03, 2008 3:57 am
Location: chennai

Null handling for 200 / 300 columns at one go.

Post by friend.kak@gmail.com »

Hi,

I have a table with 400 columns and i may expect 200 columns with null data ( say 10% of cases ) , and my job will get aboret in dev / sit / production if no of warnings > 50 , if i want to handle this at one go instead of writing for each and every column, how to achieve this?

Any easier method to achieve the same.

Thanks,
friend.kak@gmail.com
- Dev
sanjay
Premium Member
Premium Member
Posts: 203
Joined: Fri Apr 23, 2004 2:22 am

Re: Null handling for 200 / 300 columns at one go.

Post by sanjay »

Hi

suppress the warnings from director .

Thanks
sanjay
friend.kak@gmail.com wrote:Hi,

I have a table with 400 columns and i may expect 200 columns with null data ( say 10% of cases ) , and my job will get aboret in dev / sit / production if no of warnings > 50 , if i want to handle this at one go instead of writing for each and every column, how to achieve this?

Any easier method to achieve the same.

Thanks,
friend.kak@gmail.com
nagarjuna
Premium Member
Premium Member
Posts: 533
Joined: Fri Jun 27, 2008 9:11 pm
Location: Chicago

Post by nagarjuna »

I also faced similar problem ...i think we have to edit the extended properties of each and every field ...This has been discussed in the forum sometime back ...
Nag
sanjay
Premium Member
Premium Member
Posts: 203
Joined: Fri Apr 23, 2004 2:22 am

Re: Null handling for 200 / 300 columns at one go.

Post by sanjay »

Hi

suppress the warnings from director .

Thanks
sanjay
friend.kak@gmail.com wrote:Hi,

I have a table with 400 columns and i may expect 200 columns with null data ( say 10% of cases ) , and my job will get aboret in dev / sit / production if no of warnings > 50 , if i want to handle this at one go instead of writing for each and every column, how to achieve this?

Any easier method to achieve the same.

Thanks,
friend.kak@gmail.com
nagarjuna
Premium Member
Premium Member
Posts: 533
Joined: Fri Jun 27, 2008 9:11 pm
Location: Chicago

Post by nagarjuna »

Oops..i thought that its a sequential file ....i didnt see the table ...

nagarjuna wrote:I also faced similar problem ...i think we have to edit the extended properties of each and every field ...This has been discussed in the forum sometime back ...
Nag
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Post by chulett »

Still, same advice applies. Do it for every field that needs it, 4 or 400. Or perhaps back up in the source database instead, with something like NVL for Oracle. In any case, handle it appropriately regardless of the number.
-craig

"You can never have too many knives" -- Logan Nine Fingers
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Research the Derivation Substitution utility in the Transformer stage.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Sainath.Srinivasan
Participant
Posts: 3337
Joined: Mon Jan 17, 2005 4:49 am
Location: United Kingdom

Post by Sainath.Srinivasan »

You can modify multiple columns in one go - if they have similar / identical derivations.
friend.kak@gmail.com
Participant
Posts: 28
Joined: Sat May 03, 2008 3:57 am
Location: chennai

Post by friend.kak@gmail.com »

Thanks for replies so far, but still I need a smart method of handling this at one go , may be handling at source side as suggested , or may be a routine which can help on this..... Yet to decide on this implementation. Thanks, friend.kak@gmail.com
- Dev
friend.kak@gmail.com
Participant
Posts: 28
Joined: Sat May 03, 2008 3:57 am
Location: chennai

Post by friend.kak@gmail.com »

Thanks for replies so far, but still I need a smart method of handling this at one go , may be handling at source side as suggested , or may be a routine which can help on this..... Yet to decide on this implementation. Thanks, friend.kak@gmail.com
- Dev
arvind_ds
Participant
Posts: 428
Joined: Thu Aug 16, 2007 11:38 pm
Location: Manali

Post by arvind_ds »

No Short Cut, Need to use Derivation Substitution as suggested by Ray.
Arvind
emma
Premium Member
Premium Member
Posts: 95
Joined: Fri Aug 08, 2003 10:30 am
Location: Montreal

Post by emma »

Did you try to handle the nulls into a schema file ?
Thanks,
Emma
Sreenivasulu
Premium Member
Premium Member
Posts: 892
Joined: Thu Oct 16, 2003 5:18 am

Post by Sreenivasulu »

Hi All,

I have tried to handle nulls for decimal and varchar field using schema files. It works wonderfully.
Just assign default="<value"> and then run an update for that "<value>"

Regards
Sreeni
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

Research "derivation substitution" here or in the manuals.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
friend.kak@gmail.com
Participant
Posts: 28
Joined: Sat May 03, 2008 3:57 am
Location: chennai

Post by friend.kak@gmail.com »

Hi Sreeni,

Thanks for info, Do you mean to say to create schema file like this and try.

col1: nullable timestamp {null_field=""} ;
col2: nullable timestamp {null_field=""} ;

...
col400: nullable timestamp {null_field=""}

Do you mean the same? if this is the same , then we are doing Null handling here itelf. i tried this. Thanks. friend.kak@gmail.com
- Dev
Post Reply