HTTP download stage

AugusRD · Post by **AugusRD** » Tue Nov 17, 2009 11:37 am

Problem:
I need to download a file(using datastage) from the internet using HTTP protocol. Please let me know which stage can be used for this purpose.

eg)
URL: http://news.google.com/news

In the above example, I need to download the data found under the URL to a file on my machine.

Thanks,

eostic · Post by **eostic** » Tue Nov 17, 2009 7:44 pm

JavaPack is an easy way to do it, assuming you are comfortable using HTTP within Java and then can spend some time building an interface for JavaPack ....I've also heard of people using PERL for this as well from within DataStage, although I haven't done it.......I could be wrong, but I seem to remember this being discussed relative to Java another time...maybe someone has built a class for this already.

Ernie

jhmckeever · Post by **jhmckeever** » Tue Nov 17, 2009 8:59 pm

Perhaps something employing the 'wget' command?

For text files you could employ the external source stage with a 'wget {URL}; echo {file}' combination to fetch from the web then echo the file.

chulett · Post by **chulett** » Tue Nov 17, 2009 10:32 pm

In other words, there's no built-in stage for this.

AugusRD · Post by **AugusRD** » Thu Nov 19, 2009 8:46 pm

Thanks Everyone. I am planning to go with wget for now. I did not feel very comfortable as it is not available by default on solaris 10.