Hi,
We have a job which reads multiple sequential files from a specific folder using FilePattern. Additionally these files are read using a Schema File option. The reject Mode on this Sequential File stage is Set to 'Output' and for now we have attached a peek to the reject link output. We have also added the 'File Name Column' to the output tab as we want to capture the full file names from which each record came. To get these Full File names we have set $APT_IMPORT_PATTERN_USES_FILESET = True. We also have the 'Reject Reason Column' mapped to a column on the output for capturing the reason for any rejects by the Seq File Stage.
Now for the problem:
The above job works fine for us in most cases, we are having Issues when some records are rejected by the Seq file stage. for each reject record we need the Reject Reason and the File Name from which that record originated from, as this information has to be stored in a error table.
The rejected records that are spit out on the reject link and make their way to the Job Log via Peek but for these reject records we are not able to get the corresponding FileNames. That is with each rejected record we need another column, which will indicate that rejected record came from which specific Source File.
To overcome this issue we tried a different design.
We read the entire contents of the Seq File as a single large Varchar Column, and also added the FileName Property to the output Tab. After this Seq File Stage we used the Column Import stage and used the Schema file with this stage. We set the reject mode to Output on Column Import stage, so now for the reject records it shows the entire record plus the filename column but not the reject reason column.
So with either design we are not able to get both the information together which we want for our reject handling process.
Can someone please suggest any alternatives to get the required information for the reject handling part.
Thanks
FileName in Reject Link of Sequential File Stage
Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 14
- Joined: Thu Dec 13, 2007 2:57 am
Return to “IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)”
Jump to
- Moderators' Choice
- ↳ Editor's BLOG Corner
- ↳ Ask the Experts! - Dads and Grads
- ↳ DSXchange Testimonials
- ↳ Cognos (IBM BI)
- FAQs
- ↳ FAQs
- ↳ FAQ Discussion
- DataStage
- ↳ General
- ↳ IBM<sup>®</sup> Infosphere DataStage Server Edition
- ↳ IBM<sup>®</sup> DataStage Enterprise Edition (Formerly Parallel Extender/PX)
- ↳ Archive of DataStage Users@Oliver.com
- IBM<sup>®</sup>Infosphere Products<sup></sup>
- ↳ Business Glossary
- Suggestions
- ↳ Site/Forum
- ↳ Enhancement Wish List
- Consulting
- ↳ Talent
- ↳ Looking for Talent
- Support
- ↳ Parameter Manager
- ↳ Compile All Plus
- Usergroup Forums
- ↳ Usergroup Central Forum
- ↳ Heartland Usergroup Forum
- The Written Word
- ↳ Articles, White Papers and Tips and Tricks
- ↳ Product Documentation
- Third Party Applications
- ↳ Third Party Applications
- Product Derivatives
- ↳ Functions
- ↳ Routines
- ↳ Jobs
- ↳ Logs
- Tools
- ↳ Tools Forum
- Category
- ↳ Infosphere Master Data Management
- ↳ Data Quality Best Practices
- ↳ IBM QualityStage
- ↳ Information Analyzer (formerly ProfileStage)
- ↳ IBM<sup>®</sup> SOA Editions (Formerly RTI Services)
- ↳ IBM<sup>®</sup> DataStage TX
- ↳ BI
- ↳ Data Integration