We have a requirement for bad records processing.
1.We are getting an input sequential file of fixed record length format having unix new line as record delimeter.There are around 4000 columns.Some of the char columns may contain the unix line in the data
2.Bad records are defined as any records that have more or less record length compared to the fixed record length
3.Need to filter out the bad records in a bad file and send the good records further for processing
4.We have designed one job having one single varchar column having undefined length.Please note the record length is not fixed here.Then in the transformer in the same job we are checking for length of this column . if this length is less or more than the fixed length then we are sending to bad file else we are sending to a good file
5.The above approach works well except below scenario.Whenever there is a Unix new line in some char columns , data stage thinks that it has got a new record and instead of one single record and it processes that as two records. So lets say if the actual record size is 1000 and also if there is one more new line character at 500 position then datastage will think it as a new record and will split it into 2 records of length 500 each ..because of this the transformer we will send both of these records to bad records file since the lenghth is less than 1000. But originally this is a good record having 1000 record length only.
6.Please note that our regular extract job where we have defined the file as fixed length record does not treat the above record as two records of 500 length even though there also the record delimeter is Unix new line.
Can you please advise how can we overcome this issue
Bad record processing
Moderators: chulett, rschirm, roy
