DSXchange

I have a Dataset which contains approximately 2.0 million records. When I connect the dataset to a Join stage only 1.3 million of the records are being read. I can put a Transformer or Copy stage between the Dataset and the Join and all records from the Dataset will be read. Any ideas on why connection directly to a Join would cause records to be omitted? I am running version 8.5.

Thanks

How have you confirmed the record count in the Data Set?

If you are doing an inner join...

Once the stage has read all records from one input link, there is no need to continue reading from any other input links.

Mike

mgray412 wrote: Any ideas on why connection directly to a Join would cause records to be omitted

Data in one link and not in the other.
Improper sorting
Improper partitioning

Did you check the DS record count? Try using the $ORCHADMIN ll and check it.

Also how was the DS created - using a job or $ORCHADMIN cp. Once I had used $ORCHADMIN cp and the newly created DS has a similar issue i.e. records read were less without any warning. There was some issue with char encoding while writing to new. I recreated the DS and it worked.

Thanks for your responses. Here are the answers to some of the questions that were asked:

1) Confirmed dataset record count using the Data Set Management tool in Designer
2) Using a Left Outer Join
3) Dataset was created using a DataStage job

Thanks

Which now leaves you with Keith's suggested last two possibilities... improper partitioning or improper sorting.

Mke

On second thought... a left join will read all of the data from the left input. Improper partitioning or improper sorting will just lead to bad results. Double check the link order to make sure that the left input is really the one that you think it is.

Mike

Mike,

The left link is to the source file and the right is to the dataset.

Thanks

OK.

So there you have it. Once all of the data has been read from the left link of a left join (source file), there is no reason to keep reading from the right link (dataset).

Mike

Oh Ok. That makes sense now!! Thank You for your help!!!