DSXchange

Hi All,

My job design is as follows:

dataset--->transformer---->aggreagator---->ODBC conector

I have hash partiitoning and perform sort on the aggregator. I have 4 key columns(Not Null) and one column whose sum I am finding out. Somehow I get different count coming from the aggregator stage everytime i run. The job runs on 4 nodes. Could this be because the data is wrong or due to partitioning issues. Actually one of the key columns had some null values and spaces, however we have handled them in the transformer before the aggregator stage and so no nulls are passing. However i fear this is because of any special characters in that key column which might not have been handled.

Any help is appreciated.

It is in the nature of programs to do the same thing with the same data. If that is not happening, then usually some factor is affecting the outcome. In this case, are you 100% certain that your source dataset has the same cnotents and that every single parameter to the job is the same between runs? What about writing to a sequential file instead of the ODBC stage and doing a UNIX level "diff" of two runs so that you can simplify the problem.

Are you hash partitioning on those '4 key columns(Not Null)'?

If you have AUTO partitioning selected, would it not sort and partition before aggregating. I used to sort and partition the data prior to the aggregate and then I experimented with a few different ways. With AUTO my totals were identical even with 13 million rows.