Identification of a rejected row

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
DS_FocusGroup
Premium Member
Premium Member
Posts: 197
Joined: Sun Jul 15, 2007 11:45 pm
Location: Prague

Identification of a rejected row

Post by DS_FocusGroup »

Hi all

i am trying to read multiple files from a folder and then writing it to a single table. is there any way to identify, if a row is rejected, which file it belonged to? my guess would be if i am able to load the name of the file while reading the file i can capture the name with the rejected row. not sure how though. maybe the name of the file in one column and the rest of its content in another?
thanks
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

What stage type are you using? What defines "rejected"?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
DS_FocusGroup
Premium Member
Premium Member
Posts: 197
Joined: Sun Jul 15, 2007 11:45 pm
Location: Prague

Post by DS_FocusGroup »

Teradata as a target. rejected rows can be defined as duplicate rows in the source which are dropped while being loaded or based on some lookup any rejected records.

the main purpose here would be to associate the file name with the source record.

thanks
mahadev.v
Participant
Posts: 111
Joined: Tue May 06, 2008 5:29 am
Location: Bangalore

Post by mahadev.v »

There is a property called File Name Column in sequential file stage. Try that out. You cannot propagate the field to the target stage. So probably take a steam out using copy stage and join them back with the rejects link records and then write to the target file.
"given enough eyeballs, all bugs are shallow" - Eric S. Raymond
ag_ram
Premium Member
Premium Member
Posts: 524
Joined: Wed Feb 28, 2007 3:51 am

Post by ag_ram »

mahadev.v,

Did you convey this message to DS_FocusGroup that

Enabling option "File Name Column" in the Sequential file Stage will require an Environment variable APT_IMPORT_PATTERN_USES_FILESET to give the File individual file name in the specified column(or default fileNameColumn) when file pattern is selected for retrieving multiple files from a folder with matching the specified pattern.
DS_FocusGroup wrote:rejected rows can be defined as duplicate rows in the source which are dropped while being loaded or based on some lookup any rejected records.
After getting a field as Source file name from the Sequential file Stage, you can further use Remove Duplicate logic in transformer or Sort-filter as to seperate the duplicate records from the Source.
DS_FocusGroup
Premium Member
Premium Member
Posts: 197
Joined: Sun Jul 15, 2007 11:45 pm
Location: Prague

Post by DS_FocusGroup »

thanks
Post Reply