We took a perfectly fine working job that uses hierarchical stage to parse transactions. We broke it by adding a Wave Generator stage in an attempt to get the Hierarchical stage to parse a single file at a time rather than an entire batch of 5000 files.
Our favorite error is currently: Xfm_EOM_Group,0: Failure during execution of operator logic.
Xfm_EOM_Group is the name of a transform following a funnel stage. This transform adds some default values and calculates a row number. The output goes to another Hierarchical stage that composes an XML file that is landed on a drive.
Waves with Hierarchical Stages
Moderators: chulett, rschirm, roy
I often use end-of-wave to "release" xml when I need independent documents based on some key value. ...but end-of-wave has different impacts on all kinds of Stages.....it has to be tested carefully with each to understand what its flushing behavior will mean.
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
To achieve independent documents for XML from the Composer step, we use a Regroup step prior to the Composer step in the assembly. The Regroup does a great job of separating the XML into appropriate levels/files.
I would like to use the waves through the first 95% of the job and then do the last 5% as a continuous stream.
Process individual XML files for the first 95% of the job.
Separate the new documents based on line item count limits on the final 5% of the job.
Would I be able to add a second Wave Generator with a higher record count to effectively erase the end-of-waves from the initial Wave Generator?
I would like to use the waves through the first 95% of the job and then do the last 5% as a continuous stream.
Process individual XML files for the first 95% of the job.
Separate the new documents based on line item count limits on the final 5% of the job.
Would I be able to add a second Wave Generator with a higher record count to effectively erase the end-of-waves from the initial Wave Generator?
Sorry. Don't know. Never tried it...... not sure what it will do.....
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>