Read a webpage data

devidotcom · Post by **devidotcom** » Sat May 24, 2008 5:27 am

Hi All,

I have a scenerio not sure if it is possible with DataStage.
I have a webpage URL address which contains records that I need to read and process. Is there a way we can read these records. I read about external source stage but not sure if it works with this.

The webpage would have that looks like

c1 c2 c3 c4
100 10 200 20
300 30 400 40
500 50 600 60

Once I read this webpage i need to convert columns into rows using a pivot stage and generate a flat file with the data given below

c1 c2
100 10
200 20
300 30
400 40
500 50
600 60

Any input appreciated. Thanks

Devi

chulett · Post by **chulett** » Sat May 24, 2008 2:47 pm

If the site hosts a web service that could deliver that data, then yes. Otherwise (AFAIK) you'd need to find some other way to retrieve that and either leverage it via an External Source stage or just land the data and process it from there like 'normal'.

eostic · Post by **eostic** » Sat May 24, 2008 6:06 pm

I haven't done it myself, but have talked with folks who have used JavaPack to do it, since java has fairly common libraries for http work...and at least one time heard that someone wrappered a perl script that did it, as perl (apparently?) has decent functions for http activity.

Ernie

devidotcom · Post by **devidotcom** » Sun May 25, 2008 11:19 pm

Thanks for the inputs.
I guess I will try with the perl option and update the post.

One more thing. This webpage could have more than what I am looking for like some descriptions, image... etc I would have to eliminate them and then look for the data i want.