Page 2 of 2
Posted: Fri Jan 19, 2007 2:51 pm
by anu123
'Auto' the default one.
DSguru2B wrote:In the transformer, go to the stage properties, what partitioning are you providing?
Posted: Fri Jan 19, 2007 2:54 pm
by DSguru2B
And thats where you are going wrong. You need to partition it by keys. Do this, provide Hash as partitioning and choose the three keys. Now run your job again. See what happens.
Posted: Fri Jan 19, 2007 3:01 pm
by DSguru2B
Coming back to the "Why you are not using the Aggregator stage", inconsistant and inaccurate data is not the fault of the aggregator stage. Its the fault of how it was used. Take a small set of data, use the aggregator stage to build a job out of it. Test it out. If you have problems, we are here to help you out. Once you get that working, then feed in a couple of millions. If that works out too then feed in your complete data feed. Prove to "them" that aggregator works just fine. Instead of reinventing the wheel. You do realize that doing all those current row to previous row comparisons will also take some time. Wont be drastically slow but still.
Posted: Fri Jan 19, 2007 3:08 pm
by anu123
I got it. thanks Guru.
one last question. Is there any way that we can write last record with count to output instead of having a sort and Remove Dupl. stages after transformer.
DSguru2B wrote:And thats where you are going wrong. You need to partition it by keys. Do this, provide Hash as partitioning and choose the three keys. Now run your job again. See what happens.
Posted: Fri Jan 19, 2007 3:18 pm
by DSguru2B
Not that i can think of right now. Maybe someone else knows and will shed some light on it.
Posted: Fri Jan 19, 2007 3:25 pm
by anu123
Anyway thank you Guru and all.
DSguru2B wrote:Not that i can think of right now. Maybe someone else knows and will shed some light on it.
Posted: Fri Jan 19, 2007 4:05 pm
by ray.wurlod
Tail stage set to 1 row and executing on one node only (perhaps in sequential mode).