Problems reading from a Dataset

vijay1234 · Post by **vijay1234** » Wed Apr 13, 2011 2:58 am

Hi All,

We have 2 jobs. In Job1 a dataset is created. In job2, dataset is read and then used for join.

There are 1058 records loading into dataset in job1. When i run job 2, only 171 records are being read.

According to log, I think data is getting rejected. I see something like this below

repartition(0),1: Input 0 consumed 271 records
repartition(0),1: Output 0 producecd 271 records

I do not see any reason why data should be rejected.

Your help is appreciated.

Regards,
Vijay

antonyraj.deva · Post by **antonyraj.deva** » Wed Apr 13, 2011 3:24 am

Metadata mismatch probably. Use a sequential file or dataset stage to capture & analyse the reject rows.

kkalyanrao@gmail.com · Wed Apr 13, 2011 3:38 am

Check if both jobs use same configuration file

zulfi123786 · Post by **zulfi123786** » Wed Apr 13, 2011 4:34 am

Use dataset management utility to find out how many records are loaded into the datastet.
If the dataset has the number of records that are loaded by job 1 then the problem is with job2 else the problem is with job1.

Even if the configuration file is not the same Datastage would repartion the data appropriately while reading it in job2.

Hema.k · Post by **Hema.k** » Thu Apr 14, 2011 2:29 am

This might be because of having constraints in the transformer stage. If so ,it process only the records which satisfies the constraint given.

Regards,
Hema K

skrispriya · Post by **skrispriya** » Thu Apr 14, 2011 3:02 am

This is because 271 out of 1058 records are available in lookup table.
Join stage denotes number of records being processed after joining with lookup table.I hope you should have used inner join.

Regards
Priya