Problem with Joiner stage
Moderators: chulett, rschirm, roy
Problem with Joiner stage
Hi All,
I have a scenario where is I am joining 2 datasets in a joiner.The dataset A.ds has 479 records and dataset B.ds has 383 records.I am doing inner join which should give 479 records in output.
I also have sort stage before join stage.I am sorting on the key which I also use to join.I tried these things:
1. I hash partitioned the inputs based on the same key on which I sort/join which geve me 518 records as output.
2. I removed partitioning and just sorted data before joining..which gave me 14 records as output.
3. I used lookup just for testing and it gave me 479 records which is proper.
I am not able to understand whats wrong here when i use join stage.If someone has some idea about this please let me know.
Thanks in advance
Shiva
I have a scenario where is I am joining 2 datasets in a joiner.The dataset A.ds has 479 records and dataset B.ds has 383 records.I am doing inner join which should give 479 records in output.
I also have sort stage before join stage.I am sorting on the key which I also use to join.I tried these things:
1. I hash partitioned the inputs based on the same key on which I sort/join which geve me 518 records as output.
2. I removed partitioning and just sorted data before joining..which gave me 14 records as output.
3. I used lookup just for testing and it gave me 479 records which is proper.
I am not able to understand whats wrong here when i use join stage.If someone has some idea about this please let me know.
Thanks in advance
Shiva
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
The dataset in the left link has been created from MQ which has xml messages.So when I create it the dataset can have duplicates for the col which I use to join.As some of them will be upddated records from the xml message.I also have the logic for new and updated record in the later job.
So I need all the matching records from the left link though they are dups
Thanks
Shiva
So I need all the matching records from the left link though they are dups
Thanks
Shiva