Records Joining After Aggregate Stage

[stiwari]

Hello,

I need help in the problem that i am facing. My problem is as follows.

I have a Fact table say XYZ with the three fields CardNum,Activation_Date and Fact_Ky. all the three fields are of Bigint. I an using the aggregate stage to get the minimum Activation_Date and grouping it on Card_Num. The out put of my aggregate stage is the CardNum and minimum Activation_Date and this out put i am joining using the Joiner with the Fact Tbl on the basis of Card_Num and Activation_Date to get the Fact_Ky. Now the problem is that the aggregate stage is passing say 5000 records to the input of Joiner. Now logically it should join all the 5000 records with the Fact Tbl but its not joining all the records. Can any body help me why its not joining all the records.

Thanks

[SriKara]

One possibility is that, your other link to the join stage is not hash partitioned or sorted. Because of which, the keys are ending up in different partitions and not getting matched!!

[stiwari]

Hi

I have tried the sorter stage also but the problem remains the same.

Just to confirm in the aggregate stage to find the minimum value, i am using the following property

Minimum value of the output column

is this right???

[SriKara]

If you are using 'Minimum value' on the Activation_date column, and the datatype is BigInt, make sure the date format is 'YYYYMMDD'.

Also make sure the datatype/format of the column 'Activation_date' is same on both the input links to the Join stage. The join may fail if both are in different formats.

[thebird]

Check your partitioning. The join stage needs data to be pre-sorted and hash partitioned on the Keys.

Bird.