Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett , rschirm , roy
mydsworld
Participant
Posts: 321 Joined: Thu Sep 07, 2006 3:55 am
Post
by mydsworld » Thu Jun 04, 2015 12:16 am
Please let me know If DS 11.3 can connect to Hadoop file system (Cloudera 5) and to Hive/HBase tables. If 'Yes' how to configure.
Thanks in advance.
ray.wurlod
Participant
Posts: 54607 Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:
Post
by ray.wurlod » Thu Jun 04, 2015 1:05 am
Yes. Use the HDFS stage.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
mydsworld
Participant
Posts: 321 Joined: Thu Sep 07, 2006 3:55 am
Post
by mydsworld » Thu Jun 04, 2015 1:37 am
Hi Ray,
Thanks for your reply. I havn't used it ever, so wondering using HDFS stage I can access files on HDFS. Will I be able to access tables in Hive or HBase using 'HDFS stage' ?
Thanks.
mydsworld
Participant
Posts: 321 Joined: Thu Sep 07, 2006 3:55 am
Post
by mydsworld » Thu Jun 04, 2015 4:45 am
Ray or Others,
Is there any stage called 'HDFS stage' in DS 11.3. I thought it is 'Big Data' File stage to access files sitting on Hadoop.
Curious to know if one can set up the following.
1. ODBC for Hive tables and calling it in Datastage
2. JDBC for HBase tables and calling it in Datastage.
Thanks.
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Thu Jun 04, 2015 6:18 am
The 'HDFS stage' is the
Big Data File stage.
-craig
"You can never have too many knives" -- Logan Nine Fingers
mydsworld
Participant
Posts: 321 Joined: Thu Sep 07, 2006 3:55 am
Post
by mydsworld » Thu Jun 04, 2015 6:51 am
Thanks for the clarification. Still curious about the following.
ODBC/JDBC for Hive/HBase tables and calling it in Datastage
chulett
Charter Member
Posts: 43085 Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO
Post
by chulett » Thu Jun 04, 2015 7:05 am
If those access methods are supported they can be 'called' from DataStage. The JDBC side would take extra shenanigans because, well... Java. You could always ask your official support provider as well.
-craig
"You can never have too many knives" -- Logan Nine Fingers
JPalatianos
Premium Member
Posts: 306 Joined: Wed Jun 21, 2006 11:41 am
Post
by JPalatianos » Tue Dec 06, 2016 9:39 am
Hi,
I was just curious if you were ever able to configure your connectivity to Cloudera using the "Big Data File Stage"?
We have just been tasked with a similar exercise connecting to Cloudera from our 11.5 DataStage installation.
Thanks - - John
atulgoel
Participant
Posts: 84 Joined: Tue Feb 03, 2009 1:09 am
Location: Bangalore, India
Post
by atulgoel » Fri Dec 23, 2016 6:26 am
Hi .. just wanted to know if you are able to read the hive tables or hdfs files using Big data file stage...Even I have a similar requirement and doing research on the configuration part.
Atul