Hierarchical Stage giving java 'index out of bounds' error

[robjones]

Hi all,

I'm new to using the Hierarchical Data Stage, but I'm persevering with it and slowly I'm getting the hang of it, but I'm seeing an error with one of my jobs that has me tearing my hair out.

Basically, I'm trying to create XML using the hierarchical stage, from around 20 datasets. DSs are all sorted, partitioned and keyed on the same 3 key columns, and are all joined using HJoin steps in the stage. There is one parent level, in this case 'Policy' and all the other input datasets create separate 1st level children of 'Policy'.

Policy is actually a section of a much larger XSD, for which I've created a view in the contract library manager.

Steps in the assembly are:

• - Input
  - Hjoin x 20 (all disk-based optimisation type)
  - XML composer (document root set as my 'policy' view, passing output as string to output step)
  - Output (to sequential file, with 'result_string' type set to LongVarChar)

The assembly is showing no errors or warnings at design time, and the job compiles no problem.

When running the job however, I'm getting the fatal error below. If anybody could give me any pointers as to the steps I should be taking to diagnose and remedy this, I'd be grateful, because right now I've kind of hit a brick wall.

I suspect the pertinent point from the larger message is the line:

Code: Select all

cause = 'java.lang.IndexOutOfBoundsException: Index: -1, Size: 0'

...but this suspicion hasn't got me any closer to a solution.


Complete error:

Code: Select all

hd_XML_MotorPolicy,0: Fatal Error: 2016-05-03 13:40:52,719 Fatal [com.ibm.e2.applications.runner.E2AssemblyRunner] [] caught fatal exception
com.ibm.e2.core.exceptions.OperatorException: OperatorTransitionFailedAtRuntime operator = 'UserRuntimeOperatorFrame:Op-RT-22[XML_Composer]',  cause = 'java.lang.IndexOutOfBoundsException: Index: -1, Size: 0'
	at com.ibm.e2.core.exceptions.OperatorException$FactoryImpl.operatorTransitionFailedAtRuntime(OperatorException$FactoryImpl.java:115)
	at com.ibm.e2.core.framework.runtime.OperatorController.runOperatorStep(OperatorController.java:72)
	at com.ibm.e2.core.framework.runtime.scheduler.OperatorTask.runFrame(OperatorTask.java:86)
	at com.ibm.e2.core.framework.runtime.scheduler.OperatorTask.execute(OperatorTask.java:40)
	at com.ibm.e2.core.framework.runtime.scheduler.AbstractTask.run(AbstractTask.java:29)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:853)
Caused by: 
java.lang.IndexOutOfBoundsException: Index: -1, Size: 0
	at java.util.LinkedList.checkElementIndex(LinkedList.java:564)
	at java.util.LinkedList.get(LinkedList.java:485)
	at com.ibm.e2.xml.writer.E2ToSAXNonChoiceChildHandler.itemBeginForDerivationBranch(E2ToSAXNonChoiceChildHandler.java:1368)
	at com.ibm.e2.xml.writer.E2ToSAXNonChoiceChildHandler.itemBegin(E2ToSAXNonChoiceChildHandler.java:1452)
	at com.ibm.e2.core.framework.runtime.generic.traversers.AbstractTraverser.handleEventForHandler(AbstractTraverser.java:668)
	at com.ibm.e2.core.framework.runtime.generic.traversers.AbstractTraverser.handleEvent(AbstractTraverser.java:534)
	at com.ibm.e2.core.framework.runtime.generic.traversers.ItemTraverser.handleCurrentState(ItemTraverser.java:80)
	at com.ibm.e2.core.framework.runtime.generic.traversers.AbstractTraverser.continueTraversal(AbstractTraverser.java:475)
	at com.ibm.e2.core.framework.runtime.generic.traversers.AbstractTraverser.startTraversal(AbstractTraverser.java:463)
	at com.ibm.e2.core.framework.runtime.generic.traversers.ItemTraverser.itemBegin(ItemTraverser.java:122)
	at com.ibm.e2.core.framework.runtime.generic.traversers.AbstractTraverser.handleEventForHandler(AbstractTraverser.java:668)
	at com.ibm.e2.core.framework.runtime.generic.traversers.AbstractTraverser.handleEvent(AbstractTraverser.java:534)
	at com.ibm.e2.core.framework.runtime.generic.traversers.ItemTraverser.handleCurrentState(ItemTraverser.java:80)
	at com.ibm.e2.core.framework.runtime.generic.traversers.AbstractTraverser.continueTraversal(AbstractTraverser.java:475)
	at com.ibm.e2.core.framework.runtime.generic.traversers.AbstractTraverser.startTraversal(AbstractTraverser.java:463)
	at com.ibm.e2.core.framework.runtime.generic.traversers.ItemTraverser.itemBegin(ItemTraverser.java:122)
	at com.ibm.e2.core.framework.runtime.generic.traversers.AbstractTraverser.handleEventForHandler(AbstractTraverser.java:668)
	at com.ibm.e2.core.framework.runtime.generic.traversers.AbstractTraverser.handleEvent(AbstractTraverser.java:534)
	at com.ibm.e2.core.framework.runtime.generic.traversers.VectorTraverser.handleCurrentState(VectorTraverser.java:133)
	at com.ibm.e2.core.framework.runtime.generic.traversers.AbstractTraverser.continueTraversal(AbstractTraverser.java:475)
	at com.ibm.e2.core.framework.runtime.generic.traversers.AbstractTraverser.startTraversal(AbstractTraverser.java:463)
	at com.ibm.e2.core.framework.frames.AbstractRuntimeFrame.runTraverser(AbstractRuntimeFrame.java:1454)
	at com.ibm.e2.core.framework.frames.AbstractRuntimeFrame.runTraverser(AbstractRuntimeFrame.java:1437)
	at com.ibm.e2.core.framework.frames.UserRuntimeOperatorAdapter.runTraverser(UserRuntimeOperatorAdapter.java:138)
	at com.ibm.e2.core.api.operator.AbstractRuntimeOperator.runTraverser(AbstractRuntimeOperator.java:199)
	at com.ibm.e2.providers.xml.AbstractXMLProviderRuntime.process(AbstractXMLProviderRuntime.java:282)
	at com.ibm.e2.core.framework.frames.UserRuntimeOperatorFrame.process(UserRuntimeOperatorFrame.java:93)
	at com.ibm.e2.core.framework.runtime.OperatorController.callOperatorProcess(OperatorController.java:341)
	at com.ibm.e2.core.framework.runtime.OperatorController.runOperator(OperatorController.java:273)
	at com.ibm.e2.core.framework.runtime.OperatorController.doReadyToExecute(OperatorController.java:177)
	at com.ibm.e2.core.framework.runtime.OperatorController.runDataStateTransistion(OperatorController.java:132)
	at com.ibm.e2.core.framework.runtime.OperatorController.runTransitions(OperatorController.java:88)
	at com.ibm.e2.core.framework.runtime.OperatorController.runOperatorStep(OperatorController.java:66)
	... 6 more

[ray.wurlod]

That definitely sounds like one for a support call, and ultimately for a patch (since this is "their" Java code, not yours).

[robjones]

Thanks for that, Ray. I was afraid that was the reply I was going to get.

We've now raised a ticket with IBM, so I'll report back here once we get an answer.

[robjones]

Still waiting on a response from IBM, but it seems that this error only occurs when the xsd/view to which we are mapping contains elements that are sub-types.

For example, we have an XML element type 'Policy' with a number of columns which are inherited by all policy types.
We then have a number of different sub-types of 'Policy', each with their own columns unique to their type, such as 'Motor Policy', 'Household Policy' etc.

When the view to which we're mapping contains any relationship of this kind, we're seeing the index out of bounds java error mentioned above.

I can't believe that this is something IBM wouldn't have catered for when designing this hierarchical stage, so I'm assuming it's just an infuriating bug. I'll report back when I get a response.

I'm just hoping that response isn't "upgrade to 11.5".

[robjones]

So IBM have pretty much admitted that this is a bug.

Their 'solution' is to not use views of the XSD, and to just build the whole thing at once.

In our case, this just isn't practical. We have around 60 datasets being used to form the XML, and having 60 H-Join steps in the hierarchical stage isn't realistic.

So until it gets fixed, as much as it pains me, I've fallen back to using the old XML Output stage to build the XML a piece at a time.