Improve performance with multiple transformers

[myukassign]

are you doing same kind of transformation processing on all 100 elements?

if so, you might want to use one transformer job to do this processing on all the elements first at one go and then may be you can use a switch stage to send the output to different DS based on value of complex elements.

No dear...Its not the same transformation in all the transformers...

[agpt]

Can you please post the real example may be with one complex variable only like what it is and what transformation your are doing and based on what how you are taking output. It might help us in finding the solution.

[eostic]

How "complex" is each complex element? Does each of the "100" have it's own inner repeating nodes? Or is each of the 100 just a single set of additional elements that occurs one time? (or perhaps some of them have multiples while others do not)....

The XMLInput Stage can do all of that in one single process. I will admit, having 100 links coming out of a single XMLInput might be pushing it, but certainly 20 or 25 isn't too much. The same XMLStage and the same job can handle a lot of these, with one single read of your XML document and loading it only one time into memory.

If any of the "100" are truly single occurring, they can all be on the same link, even if they are independent complex nodes.

Done right and you might have 4 or 5 jobs instead of 100.....

Ernie

[myukassign]

Yes 90% of the complex elements are nested and can repeat any number of time and its cardinality is 0..N.

I cant take multiple complex element in a transformer because The XML stage will let you assign only one column as key column.

So incase If I try to fetch more than one complex element in one go , It is highly posiible that I may loose data.

I swet a lot to understand this fact. And also incase I assign the parent key as a key for a group of parent and nested elements, I will be able to take only one child

[antonyraj.deva]

One or two examples of the complex elements can make things easier to think about an alternate ETL design.

Thanks,
Tony

[myukassign]

Alrt....I am getting very good speed now.....

I did the following steps.....(I had no option)

1. I break the the jobs with 50 TRNS to smaller bits...like 10 each

ds--> TRS-----> DS

2. I called the job like binary tree from sequencer..... It is designed in a way, at one time at the max two jobs will run....

3. All the sorts in the jobs I did it while XML extraction itself and stored it in persistent datasets.

4. I did an analysis with production support and changed the batch schedule time. The most idle period of server I could get.


Someone asked me a smple of complex element... its looks like this...

<IDOC-ADATA>

<Dealer>

<DealerNo> XXX<DealerNo>
<DealerName> XXX<DealerName>

<ContactDet>
<Lane1> </Lane1>
<Ph> </Ph>
<Ph> </Ph>
</ContactDet>

<BranchLoc> <-- 0...N cardinality.--->
<Locaction>
</Location>

<Locaction>
</Location>

<Locaction>
</Location>



</BranchLoc>


</Dealer>

<Dealer>

<DealerNo> YYY<DealerNo>
<DealerName> YYY<DealerName>

<ContactDet>
<Lane1> </Lane1>
<Ph> </Ph>
<Ph> </Ph>
</ContactDet>

<BranchLoc> <-- 0...N cardinality.--->
<Locaction>
</Location>

<Locaction>
</Location>


</BranchLoc>


</Dealer>



</IDOC-ADATA>

[eostic]

The Repetition element is "one per output link"....so you should definitely be able to improve on your current scenario dramatically. One link for each detailed repeating node scenario.

Ernie

[chulett]

Yah, you've made this way more complicated than it needs to be.

[myukassign]

Yah, you've made this way more complicated than it needs to be.

But what other option I have other than this.

I tried everything that suggested in this thread but I did not get any improvement.

But now its working as per my expection and everyone at my end is finally happy. They allowed me to close this defect...Ahhhh... I am the fourth person that defect has assigned to...

[ETLJOB]

Congratssss...