DSXchange

Hi,

please any one clarify me, Is there any difference between a normal Ds job and web services used Ds job means performance wise. Because my currenta scenario like to get the reference data from one main database I have to use web services. The main database contains millions of records. Previously business was agreed to provide the views to refer the reference data. But suddenly they were saying instead of views use web services to refer the data by using web services.

Thanks in advance
Suresh

That would depend on several factors, not least being who wrote the web service how efficiently, and how efficiently the service has been deployed, how efficiently that service queries the database, and how much other traffic there is on the web service machine, the database server machine and on the network. In short, "how long is a piece of string?"

Also ....How many rows are they returning to you? Data in Web Services are shipped over the wire in XML, and those messages (called SOAP "envelopes" if SOAP over HTTP is used as a protocol) could get really big if there is a lot of data. A good industry practice is to keep the size of envelopes reasonably small. The bigger they are, the more impact on the network, and the more impact on both the provider (the host of the Web Service) and the client (in this case, DataStage) to process the XML.

If the packets are small, like single row lookups, great....but don't expect it to be anywhere near as performant as a local native database request, or even remotely close to an in memory lookup of already cached data. It's probably 10's, 100's or 1000's of times slower than the latter.

Then again, it may not matter if your overall Job only has to process a small quantity of data, and you have a sizable window.

Even a really fast Web Service (small packets, great network, efficient application server, low overhead service in well written java, etc.) might deliver a row back in 10 to 20 milliseconds......that's only 100 rows per second........... think in contrast about how fast some of your jobs with in-memory lookups can perform.

Web Services are great for re-usability, but one has to think carefully before using them in a pure "batch" mode.

Ernie