RTI WS doesn't return data if above a certain amount of rows

[MT]

Hi,

I have a WebService selecting data from a DB2 table.
The WHERE Condition of the statement is a parameter for the WS.
My Testjob uses a WSClient stage and returns its output to a flat file.

Specifying a WHERE Condition that returns < 2000 rows works fine but
above a certain limit I do not get any results in my testjob. The log of the web service in the director does not show any error but shows the number of retrieved rows as if everything is ok.
The log of the test job does not show any error either but lists 0 rows moved from the WSClient to the file (which is empty).

Any ideas?

Many thanks in advance
Michael

[chulett]

What 7.x version?

[MT]

Hi

DS Version 7.5.3
RTI Console 7.5.2.8

regards
Michael

[chulett]

OK, so the latest and greatest 'pre-8' release. Tell us more about the service - for instance, is it 'always on'? There are considerations when writing to flat files from a service in that they don't fully flush to disk until the service is stopped. Is that anything like what you're seeing?

[MT]

Hi,

well this actually not a always on service because that wouldn't allow me to pass the WHERE condition directly into the SQL statement.
So there should not be any problem passing the information down to the file...

It works fine for when the WHERRE condition reduces the result set to about 2000 rows. Limiting the number to 2500 I get a heapdump and javacore and RTI returns a Java memory error (takes quite some time to write it).
Above a certain second limit I see no results but no error as well (but no rows are an error in fact) - the DS job is "green" - not even a warning in the DS log and nothing to find in the WAS logs (and it is fast again because it does not have to wait for the dumps to be written)...

Sometimes (at 2500 rows) I get following error in the DS Log:
TestJobWS_SelZAD..SF_DummyOut.IDENT1: java.lang.OutOfMemoryError
at java.lang.StringBuffer.expandCapacity(StringBuffer.java(Compiled Code))
at java.lang.StringBuffer.append(StringBuffer.java(Inlined Compiled Code))
at java.io.StringWriter.write(StringWriter.java(Compiled Code))
at org.apache.axis.encoding.SerializationContextImpl.endElement(SerializationContextImpl.java(Compiled Code))
at org.apache.axis.message.SAXOutputter.endElement(SAXOutputter.java(Compiled Code))
at org.apache.axis.message.SAX2EventRecorder.replay(SAX2EventRecorder.java(Compiled Code))
at org.apache.axis.message.MessageElement.output(MessageElement.java:743)
at org.apache.axis.message.MessageElement.getAsString(MessageElement.java:706)
at org.apache.axis.message.MessageElement.getAsDocument(MessageElement.java:685)
at org.apache.axis.message.MessageElement.getAsDOM(MessageElement.java:714)
at com.ascentialsoftware.wsclient.ResponseHandler.getAsDOM(ResponseHandler.java:233)
at com.ascentialsoftware.wsclient.ResponseHandler.buildBody(ResponseHandler.java:126)
at com.ascentialsoftware.wsclient.OperationCall.invoke(OperationCall.java:339)
at com.ascentialsoftware.wsclient.WebService.callServiceInternal(WebService.java:1659)
at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1566)
at com.ascentialsoftware.wsclient.WebService.callService(WebService.java:1519)
at com.ascentialsoftware.wspack.Transformer.process(Transformer.java:102)
at com.ascentialsoftware.jds.StageFactory.runStage(StageFactory.java:544)
at com.ascentialsoftware.jds.StageFactory.runSource(StageFactory.java:335)


Well It took me a while that the behaviour is so dependend on the amount of data ....

The WAS admins have already resized the Java memory with no effect as far as I can see and also the -Xmx option for the JVM as suggested from Ernie in other postings seem to matter here...

kind reagrds
Michael

[chulett]

If nothing else is forthcoming from the forum, I would suggest you get the errors in front of your official support provider and see if it's something they've seen before - if you haven't done so already, of course.

[MT]

Hi Chulett

thanks for your thoughts - yes it is something for support - I opened a case already but sometimes it is good to hear if someone else experiences the same thing or has any other idea....
There are not so many guys having RTI experience anyway

Thanks a lot.

regards
Michael

[chulett]

There are not so many guys having RTI experience anyway

Unfortunately true. Perhaps Ernie will add something once he comes back from his self-imposed exile.

ps. There is a dedicated "RTI" forum here. FYI.

[eostic]

How big is the answer set? There are defintely limits to the size, although I can attest to bringing back responses that are in the 10M range. On major key here though, is looking at the whole infrastructure......your Web Service is a lot more than just RTI.......there are a WHOLE lot of places where large SOAP responses can get squirrely (for example, are you building the payload yourself via xmlOutput Stage, are there limits on your choice of Application Server [which one are you using, btw?], are there limits at your SOAP client, proxies that it is going thru, etc. etc.)...

Keeping SOAP payloads small is a widely accepted practice, regardless of the provider.

A lot of providers put automated "row return" controls in place.....unfortunately RTI or ISD dont' have that capability (such services preserve state, and return a "handle" to you along with the total number of rows, and then you issue subsequent calls to get more "batches" in succession), but you could "invent" such a thing with temporary structures or some type of counter that returned a selective number of rows. It would be more difficult to code for dynamic sources, but doing it for a static source wouldn't be too bad.....

Ernie

[MT]

How big is the answer set? There are defintely limits to the size, although I can attest to bringing back responses that are in the 10M range. On major key here though, is looking at the whole infrastructure......your Web Service is a lot more than just RTI.......there are a WHOLE lot of places where large SOAP responses can get squirrely (for example, are you building the payload yourself via xmlOutput Stage, are there limits on your choice of Application Server [which one are you using, btw?], are there limits at your SOAP client, proxies that it is going thru, etc. etc.)...

Keeping SOAP payloads small is a widely accepted practice, regardless of the provider.

A lot of providers put automated "row return" controls in place.....unfortunately RTI or ISD dont' have that capability (such services preserve state, and return a "handle" to you along with the total number of rows, and then you issue subsequent calls to get more "batches" in succession), but you could "invent" such a thing with temporary structures or some type of counter that returned a selective number of rows. It would be more difficult to code for dynamic sources, but doing it for a static source wouldn't be too bad.....

Ernie

Hi Ernie,

thanks for answering...
Well the "critical" answer set is about 2100 rows - which is about 3,6 MB.
Good to hear that 10 MB worked in your environment.
We use WebSphere as the application server.

I would like to avoid any "experimental" solution with extra data packaging to "right size" it if possible. I think this should simply work.
But even support has not returned any answers yet....

kind regards
Michael

[eostic]

Hard to say...could be many different things. Some things to try:

a) Server Job and not EE. Server is more flexible in large datatype areas....you are in the Server forum, so I will assume that you already are using a Server job, but just wanted to check...

b) Package up the array before you send it. Just in case it's something in RTI's ability to package the rows. This is vastly less attractive, as the wsdl will no longer reflect the payload as anything but a giant string, but it's worth looking into, if only as a debugging strategy. Use XMLOutput Stage, aggregate the rows to a single xml document, send to a single column on the output link (use a longvarchar and 999999 or similar length and put a single slash in the Description)....send that single large column to RTIoutput.

c) Increase the heap size in the start up file for your WAS.

d) Try other client tools. Use SOAPUI or other such tools to eliminate or isolate RTI.

e) Try other data. There are more things than size that will crush RTI. Any binary zeros (hex 00) or other garbage in the extra rows could blow it up also.

Does the Job abort?
Does the RTIAgent die?

Hopefully support can help you fire up the logs for the RTIAgent...it's been too long since I've had a 7.x environment running to be able to advise you where to set those up.

Ernie

[MT]

Hard to say...could be many different things. Some things to try:

a) Server Job and not EE. Server is more flexible in large datatype areas....you are in the Server forum, so I will assume that you already are using a Server job, but just wanted to check...

b) Package up the array before you send it. Just in case it's something in RTI's ability to package the rows. This is vastly less attractive, as the wsdl will no longer reflect the payload as anything but a giant string, but it's worth looking into, if only as a debugging strategy. Use XMLOutput Stage, aggregate the rows to a single xml document, send to a single column on the output link (use a longvarchar and 999999 or similar length and put a single slash in the Description)....send that single large column to RTIoutput.

c) Increase the heap size in the start up file for your WAS.

d) Try other client tools. Use SOAPUI or other such tools to eliminate or isolate RTI.

e) Try other data. There are more things than size that will crush RTI. Any binary zeros (hex 00) or other garbage in the extra rows could blow it up also.

Does the Job abort?
Does the RTIAgent die?

Hopefully support can help you fire up the logs for the RTIAgent...it's been too long since I've had a 7.x environment running to be able to advise you where to set those up.

Ernie

Hi Ernie,

thanks for your answer
and to your questions:

a) well sure - it is a server job

b) you mean building a single structure (XML) for all rows or just one for all columns?
We actually concatenated all the results to a single column by concatenating the result columns to a single one in a transformer (not nice - just for testing purposes) and it really changed what we see - the error comes "later" now - above 10000 rows ....
Doing it with an XML Output stage we get the same error and dumps having one XML column per ROW - having one XML column per Resultset returns "row to big for inter stage row buffer"

c) done already twice - no real effect

d) we will try it - but the Webservice always runs fine and the correct number of rows are show in the log of the Webservice but the problem is the job calling the WS - wether we use a WS Client or WS Transformer...

e) did that test already - other data - same effect

Well support did not answer yet - so no advise however.

kind regards
Michael

[eostic]

Let's talk about "b". Using XMLOutput. You say that you get the same error......do you mean that a test job (not RTI) wth XMLOutput gets the same error (ie...when trying to build the xml payload and send to say, a flat file)?

Just looking for clarification in our discussion. You may have already done exactly the steps below, but for review....

....you should aim for an xml document that is a simple collection of all the rows.....so, it would have all the columns from upstream going to its input link, and then just one single column on the output link (with longvarchar and 99999 in the length). Put a single slash in the description for that single column. Go to a sequential file with "000" in the delimeter and the quote character at the Format Tab. On the Input link of the XMLOutput Stage, do something like this on the descriptions:

col1 /myRows/Row/col1/text()
col2 /myRows/Row/col2/text()
col3 /myRows/Row/col3/text()

Also on the output link of the xml stage, make sure that "formatted" is NOT checked, and that you specify "Aggregate". That will minimize the size of the payload.

You should get a nice clean xml document created at your Sequential Stage.... at 3 to 4meg it should work fine.

Then you put RTIOutput on that and re-publish the Job so that a brand new WSDL is created.

Ernie

[MT]

Let's talk about "b". Using XMLOutput. You say that you get the same error......do you mean that a test job (not RTI) wth XMLOutput gets the same error (ie...when trying to build the xml payload and send to say, a flat file)?

Just looking for clarification in our discussion. You may have already done exactly the steps below, but for review....

....you should aim for an xml document that is a simple collection of all the rows.....so, it would have all the columns from upstream going to its input link, and then just one single column on the output link (with longvarchar and 99999 in the length). Put a single slash in the description for that single column. Go to a sequential file with "000" in the delimeter and the quote character at the Format Tab. On the Input link of the XMLOutput Stage, do something like this on the descriptions:

col1 /myRows/Row/col1/text()
col2 /myRows/Row/col2/text()
col3 /myRows/Row/col3/text()

Also on the output link of the xml stage, make sure that "formatted" is NOT checked, and that you specify "Aggregate". That will minimize the size of the payload.

You should get a nice clean xml document created at your Sequential Stage.... at 3 to 4meg it should work fine.

Then you put RTIOutput on that and re-publish the Job so that a brand new WSDL is created.

Ernie

Hi Ernie,

well we tried all you described - it ran ok writing it into a file but running the WS we get again an error:

TestJobWS_KFZPMRSingleUnformattedXMLOut..SF_DummyOut.IDENT1: java.lang.OutOfMemoryError
...

And dumps are written again...

Looking in the /tmp directory of our AIX Server we see a big file had been generated there:
102400104 Apr 14 16:29 ade.VTPT7.VTPWS_XMLOutSingleUnformated.1271250031295.RTI_Output

These do not get deleted because our job aborted - the webservice looks fine but the calling job aborted...

Thanks in advance
Michael

[eostic]

....hmm..that's interesting. I didn't realize that you were calling this from another Job....

...using WSTransformer?

What happens when you invoke the service from an external tool, such as SOAPUI?

Ernie