How to load a file from a URL using DataStage

saraswati · Post by **saraswati** » Mon Jan 11, 2010 11:50 am

Hello friends,

As per my requirement I need to read a file(say file1 as also depicted below in the url) which would be posted on a URL.
The URL is somewha like:
http://abc.com/group/files/file1
I need to use DataStage Jobs to do this.

Kindly advise and let me know if I need to give further details.

Thanks in Advance!!

chulett · Post by **chulett** » Mon Jan 11, 2010 2:25 pm

What have you tried? There is a specific "URL/File pathname" option in the stage, have you tried using that?

saraswati · Post by **saraswati** » Mon Jan 11, 2010 11:40 pm

By stage what stage do you mean?

vinsashi · Post by **vinsashi** » Tue Jan 12, 2010 1:40 am

Use FTP stage. give URL , username and password then define metadata,specify file format(delimeter and all)

Thanks
vin..

chulett · Post by **chulett** » Tue Jan 12, 2010 6:41 am

For some reason I was thinking 'XML' when answering you, so what I meant was the XML Input stage... however, the FTP stage should be the correct answer here though personally I've never tried sourcing from a URL with it. Give it a shot, let us know.

saraswati · Post by **saraswati** » Wed Jan 13, 2010 6:20 am

The problem is the web url: http://abc.com/group/files/file1
is not a FTP server but an HTTP one and hence we donot have a username/password to access it. Its like an intranet url where files are posted and is hosted on an apache server and you can download files from there.
This is the reason I am unable to use an FTP stage to access the files.

Please help!! Thanks in Advance

eostic · Post by **eostic** » Wed Jan 13, 2010 7:25 am

I've seen it done with JavaPack and long ago with the old Perl Stage.

Today, write a java class that can get the document via HTTP and then you can parse it as appropriate and send the rows down an output link.

Ernie

chulett · Post by **chulett** » Wed Jan 13, 2010 7:53 am

Poor old ClickPack, we miss her.

eostic · Post by **eostic** » Wed Jan 13, 2010 9:26 am

Ah yes....it was a fun time in the early web-bubble days. The sessionizer was the most interesting.......thought it was going to make a comeback, but alas, not enough demand...into the bin with other cool stuff from the past like the Job Edit API.... ; )

Ernie

saraswati · Post by **saraswati** » Wed Jan 13, 2010 11:51 am

Well since I do not have the Java Pack is there really no other way to load a file from a http url, like the WebServices Pack or etc?

Did anyone else faced sort or this requirement and fixed it. Please advise! Thanks in Advance!!

eostic · Post by **eostic** » Wed Jan 13, 2010 12:02 pm

JavaPack is there....we don't call it "JavaPack" so much anymore because it is no longer a chargeable item. Look for the Java Transformer and Java Client Stages in your palette... Doc is in lots of places also, and with lots of entries here in this forum, and I have some samples at my blog noted below.

Ernie

lstsaur · Post by **lstsaur** » Wed Jan 13, 2010 12:21 pm

Yes, Java stages are included in your V8. You need to know how to use URLConnection, BufferReader in your Java stage to access the file on the URL.