RTI WS doesn't return data if above a certain amount of rows

[eostic]

...beyond that, my only solution for you is the programmatic one. Limit the rows.

Ernie

[MT]

....hmm..that's interesting. I didn't realize that you were calling this from another Job....

...using WSTransformer?

What happens when you invoke the service from an external tool, such as SOAPUI?

Ernie

Hi Ernie,

as wirtten before - the same error
wether we use a WS Client or WS Transformer.

We are going to test it with SOAPUI but it a looooong way to get this tool aproved and onto my machine - I can tell you...

regards
Michael

[eostic]

Well...looking thru our threads again, if the RTI job is dumping it, it's likely not something at issue inside of the calling Job. And if you right mouse click on the WS Transformer (for future reference) and select "grid" option for editing, you will see an entry in there that lets you adjust heap size (in case you ever have huge SOAP bodies that cause issues for the receiving Job).

I like having SOAPui available for ANY kind of testing and development of ISD jobs, so that I have an indepent tool that is separate from the java/.NET developers calling my service, or from the DS developers calling my service. So it's worth the paperwork in the end, even if it doesn't offer much for this scenario.

Ernie

[MT]

Well...looking thru our threads again, if the RTI job is dumping it, it's likely not something at issue inside of the calling Job. And if you right mouse click on the WS Transformer (for future reference) and select "grid" option for editing, you will see an entry in there that lets you adjust heap size (in case you ever have huge SOAP bodies that cause issues for the receiving Job).

I like having SOAPui available for ANY kind of testing and development of ISD jobs, so that I have an indepent tool that is separate from the java/.NET developers calling my service, or from the DS developers calling my service. So it's worth the paperwork in the end, even if it doesn't offer much for this scenario.

Ernie

Hi Ernie,

finally we got the SAOPUI installed and running on our development environment and it work fine - not matter how big the result set is.
I think that this proofs that the WebService is ok and the pronblem is
at the interface where the WS returns it result dfata to the calliing WSClient or WSTransformer stage.....

Different JVM options in the grid style view of the transformer did not show any effect.

regards
Michael

[eostic]

So much for it being an ISD thing. Oh well. Good debugging work.

Web Services pack is limited on what it can do. Having a tool like SOAPui is so critical when building an ISD app.....for future readers of this thread, you often have no control of the web service client --- whether it is DataStage or Java or C# or some other tool.........if it had blown up a C# process you'd be hunting in the wrong place.

Size is certainly an issue for WSPack. Did you set -Xms nnnM and -Xmx nnnM?

Ernie

[MT]

So much for it being an ISD thing. Oh well. Good debugging work.

Web Services pack is limited on what it can do. Having a tool like SOAPui is so critical when building an ISD app.....for future readers of this thread, you often have no control of the web service client --- whether it is DataStage or Java or C# or some other tool.........if it had blown up a C# process you'd be hunting in the wrong place.

Size is certainly an issue for WSPack. Did you set -Xms nnnM and -Xmx nnnM?

Ernie

Hi Ernie,

yes we tried both JVM setting - i.e. -Xms100M, -Xmx500M
the error changes to something like "can not create JVM" - but it dooes not work....

regards
Michael

[eostic]

3.6M is still really small.... If it were 100M, and pushing on the limits of XMLPack, I'd be concerned...but something else would seem to be going on here.

Do you get "cannot create the JVM" when just setting those values and running "any" service, or when you set those values and then try to run the service with the large response?

...because if you are getting the errors when you try to set the values, then it is probably just something syntactical, and then not yet a valid test for your large payload. I haven't had a chance to dig up an old job that uses them...I can't recall if the syntax requires the dashes, and/or whether I used to separate them with anything....

Also, which property did you put them in when you chose grid style?

Ernie

[MT]

3.6M is still really small.... If it were 100M, and pushing on the limits of XMLPack, I'd be concerned...but something else would seem to be going on here.

Do you get "cannot create the JVM" when just setting those values and running "any" service, or when you set those values and then try to run the service with the large response?

...because if you are getting the errors when you try to set the values, then it is probably just something syntactical, and then not yet a valid test for your large payload. I haven't had a chance to dig up an old job that uses them...I can't recall if the syntax requires the dashes, and/or whether I used to separate them with anything....

Also, which property did you put them in when you chose grid style?

Ernie

Hi Ernie,

well good question - actually the job aborts with small and big payloads when I set the Xmx limit above 207m (-Xmx207m). Below that it works for small amounts of data.
Anything above that returns (for small payloads)
"Abnormal termination of stage TestJobWS_SelZAD..SF_DummyOut.IDENT1 detected" (which is the dummy file stage right after the WSClient stage - the only two stages in the test job).


I set the JVM options property.

Do you have any idea what is limiting the Xmx setting (besides physical ram)?

Thanks in advance
regards
Michael

[eostic]

Trying to think of other things that could be an issue......

Let's talk about the Web Service itself and the method of calling it. It's a WISD based Service...

Can I assume it is configurd to return an array? (lots of rows=lots of data)
What is the topology of the Job? Is it WSClient at the start?
What are you doing with the large payload? ...just sending out rows? What is the "dummy" file stage downstream?

It might be worth trying the WSTransformer. Send in a dummy "row" of values (instead of fixed job parameter values like you are probably doing in the WSClient -- set your values in an upstream Derivation)...and then, especially for initial testing, just put a huge single column on the output lnk of the WSTransformer (like "mySOAPbody", longvarchar, length of 99999) and send it to a Seq Stage.

Ernie

[MT]

Trying to think of other things that could be an issue......

Let's talk about the Web Service itself and the method of calling it. It's a WISD based Service...

Can I assume it is configurd to return an array? (lots of rows=lots of data)
What is the topology of the Job? Is it WSClient at the start?
What are you doing with the large payload? ...just sending out rows? What is the "dummy" file stage downstream?

It might be worth trying the WSTransformer. Send in a dummy "row" of values (instead of fixed job parameter values like you are probably doing in the WSClient -- set your values in an upstream Derivation)...and then, especially for initial testing, just put a huge single column on the output lnk of the WSTransformer (like "mySOAPbody", longvarchar, length of 99999) and send it to a Seq Stage.

Ernie

Hi Ernie,

the WS is a passive WebService from DataStage like
DB2 -> TX -> RTIOut

Yes - the WS returns an array.

The topology of the calling test job (originally it was a more complex job - but to concentrate on the major problem we stripped it down) is
WSClient -> SqFile

I call the SqFile just 2dummy" because it is in the test job just for proofing if anything has been returned - it does not look nice but it need not to.

Regarding a WSTRansformer - as written above - we tried it with the same result (error).


kind regards
Michael

[eostic]

Bummer. I first learned about using the JVM options at a site when WSPack was brand new --- we were reading from a BEA based Web Service that was returning nearly 20Meg.......but that was many releases and many years ago.

...still seems odd that your volume is so small (big but not anywhere near extreme) and yet this still has an issue.

WSTransformer blew up the same way when you made the jvm option changes?

Ernie

[MT]

Bummer. I first learned about using the JVM options at a site when WSPack was brand new --- we were reading from a BEA based Web Service that was returning nearly 20Meg.......but that was many releases and many years ago.

...still seems odd that your volume is so small (big but not anywhere near extreme) and yet this still has an issue.

WSTransformer blew up the same way when you made the jvm option changes?

Ernie

Hi Ernie,

yes the WSTransformer blews the same way.
Running it without any JVM option is works up to a certain amount - and dumps above that amount.
Specifying -Xmx210M results in an error even with the small payload. In this case the transformer does not even call the WS but fails with
"Abnormal termination of stage TestJobWS_KFZPMR_Transformer..XFM detected"

There is a mysterious limit somewhere - I have no clue where....

regards
Michael

[eostic]

Can I assume that you are including both Xmx and Xms at the same time? One is for initial size and the other is for max....

Ernie

[MT]

Can I assume that you are including both Xmx and Xms at the same time? One is for initial size and the other is for max....

Ernie

Hi Ernie,

we actually tested both - with just -Xmx210M and with -Xms100M -Xmx210M
both failed with Xmx210M and both run with Xmx200M

crazy stuff

regards
Michael