DSXchange

Posted: **Thu Apr 28, 2016 10:30 am**

Hi
I need to invoke Hive from Datastage, but I am having problem with passing in spaces in Hive parameters

At the operating i can pass in parameters with space in them, like this:
I have this simple Hive script. It just prints out the parameters passed in:

Code: Select all

 ==> cat param1.hql
!echo ============================================= ;
set PARAM1 ;
set PARAM2 ;
set PARAM3 ;
!echo ============================================= ;

And the Hive call and its output :

Code: Select all

 ==> hive -S -hiveconf PARAM1="A B C"  -hiveconf PARAM2="DEFG"   -f param1.hql
=============================================
PARAM1=A B C
PARAM2=DEFG
PARAM3 is undefined
=============================================

Note that PARAM1 has the correct spaces passed in.

Now I need to do the same thing in Datastage.

In our environment, We have to execute sudo then bsub, then hive, like this: (NOTE: NO space in parameters yet to proves that it works)

Code: Select all

sudo -u tedwload -i  "  bsub -I   /opt/ibm/biginsights/hive/bin/hive   "-hiveconf  PARAM1=ABC"     "-hiveconf  PARAM2=DEFG"  -f /data/dev/Scripts/T21.hql   " > /tmp/T21.log   2>/tmp/T21.err

This works, as it prints these 2 lines out:

Code: Select all

PARAM1=ABC	
PARAM2=DEFG

But now, I want to pass spaces into PARAM1.
I tried so many combinations, but it just loses all spaces. These are what I tried:

(I just include the PARAM1 part, the rest of the command are identical as above)

Code: Select all

"-hiveconf  PARAM1='ABC' "                          <--   it prints:  'A
"-hiveconf  PARAM1=\'A B C\'  "                     <--  it prints:  'A
"-hiveconf  PARAM1="A B C" "                       <--  it prints:  "A
"-hiveconf  PARAM1=\"A B C\"  "                    <--  it prints:  "A

Could you suggest a solution?

Thanks

Posted: **Thu Apr 28, 2016 5:35 pm**

Maybe (just a guess) escape the space characters themselves?

Code: Select all

A\ B\ C

Posted: **Thu Apr 28, 2016 7:09 pm**

Instead of space give %20 like

PARAM1='A%20B%20C'

Posted: **Thu Apr 28, 2016 9:35 pm**

Hi Ray
I tried that also. Didn't work either. This is what it passes into Hive:

Code: Select all

PARAM1=A\
PARAM2=DEFG

Thanks

Posted: **Thu Apr 28, 2016 9:54 pm**

Hi Jay

I just tried that.

" -hiveconf PARAM1="A%20B%20C" "

It passed in this:

PARAM1=A%20B%20C
PARAM2=DEFG

Same for this: " -hiveconf PARAM1='A%20B%20C' "

Your suggestion is similar to my backup solution:

Everybody agrees to some non-space token (such as *, or %20 as u suggested), then inside their Hive scripts they have to use regexp to convert them back to spaces for their queries. (This is assuming that those token are not part of the data itself.)

But this extra step is not ideal. I prefer to have a simple solution to pass in spaces as is.

Thanks

DSXchange

Pass space into Hive Config thru BSUB command

Pass space into Hive Config thru BSUB command

Re: Pass space into Hive Config thru BSUB command

Re: Pass space into Hive Config thru BSUB command