webservice tranformer stage

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
peep
Premium Member
Premium Member
Posts: 162
Joined: Mon Aug 20, 2012 6:52 pm

webservice tranformer stage

Post by peep »

we are using web service transformer stage to load data into mdm database.

Our datastage job which is using webservice transformer stage is processing very slow less than 10 row/sec.

our cpu ,memory , disk through out is absoutely at minimal usage.
let me know if there is any thing we can do to process messsages using WS transformer stage faster.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

You need to test the web service itself from some other kind of client to determine whether there's a limit on the rate of updates that the service itself can perform.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
eostic
Premium Member
Premium Member
Posts: 3838
Joined: Mon Oct 17, 2005 9:34 am

Post by eostic »

Web Services aren't a speedy protocol. Lots of things on both sides to deal with xml.

However...a couple of things you can look at....

a) as noted by Ray, see how the web service performs using another tool, to isolate any issues.

b) see if the target server can handle parallel streams of services being invoked....either from one Job (normal parallelism and use a multi-node config), from one Job and multiple paths formally drawn on the canvas, or perhaps simply multiple Jobs, either formally developed and named or using multi-Job instancing. All of these are options if the Server allows parallel input and doesn't care about the order of the services being called and the data rows that they are associated with...

c) How many columns? Is it a huge number? (many hundreds?).....and/or are you using xmlOutput or xmlStage to build a giant array for this service? There may be some things you can try if the volume of data per call is huge or if the number of columns is huge.

Ernie
Ernie Ostic

blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
Post Reply