Sequential File dropping records without any warning

DataQuality_IS8.1 · Post by **DataQuality_IS8.1** » Tue Nov 10, 2009 6:40 am

Hi All,
Please find my job design

Oracle Enterprise Stage---> Transformer--->Sequential file Stage

Oracle stage is fetching 115 records from database & storing them in sequential file( delimited by "|"). The fetched data does not contain special character "|". The job performance statistics/job log show that 115 records have fetched from DB & loaded to Sequential file. But when I used "View Data" it is showing only 37 records. It seems that sequential stage is dropping these 78 records.

In subsequent job where this input file is used , it is only processing 37 records.

Also note that I am using "NULL" as null handler for sequential file stage.

I am not sure why these 78 records have dropped/rejected without any warning in log file.

Your help is appreciated.

Thanks,

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Tue Nov 10, 2009 6:46 am

Check the job log in your director for any drop messages.

chulett · Post by **chulett** » Tue Nov 10, 2009 6:47 am

Metadata issues will cause the stage to 'drop' records. How many records does the actual UNIX file have? Check outside of DataStage.

DataQuality_IS8.1 · Post by **DataQuality_IS8.1** » Tue Nov 10, 2009 7:17 am

Hi All,

I looked the log file & found that there is no message showing that record have dropped.

But when I viewed the file using "VI" editor it is showing 248 records.

Thanks,

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Tue Nov 10, 2009 7:35 am

How did 115 records became 248 ?

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Tue Nov 10, 2009 7:36 am

How did 115 records became 248 ?

DataQuality_IS8.1 · Post by **DataQuality_IS8.1** » Tue Nov 10, 2009 8:06 am

The 248 record count is based on no of lines in sequential file. But these are 115 records which get spaned across 248 lins in sequential file.

chulett · Post by **chulett** » Tue Nov 10, 2009 8:35 am

As noted, your metadata in the stage only matches those 37 records. Or perhaps your Period in the Viewer is not set to 1. Or your Row Limit is too low. Or all three.

ray.wurlod · Post by **ray.wurlod** » Tue Nov 10, 2009 1:17 pm

Add a reject link to the Sequential File stage and report back whether the row counts reconcile with this design. If they do, then you will need to perform some of your own detective work to figure out why the rejected rows do not match the record schema you have given in the Sequential File stage.