PX XML Input stage dropping records. No warnings or errors.
Posted: Sun Nov 23, 2014 1:53 pm
We recently completed a DataStage upgrade from 7.5.2 to 9.1.2.
DS 7.5.2
Server: None
Parallel: None
DS 9.1.2 NLS:
Server: UTF8 (Project Default)
Parallel: ASCL_ISO8859-1 (Project Default)
Parallel Job Description:
External source stage provides filename to XML input stage
XML input stage uses filename to parse XML file and writes to data files.
Observations (Same data file)
DataStage 7.5.2, XML stage outputs 10088 records
DataStage 9.1.3, XML stage outputs 4,798 records. No warnings or errors.
I found that data file contained the following html entity codes: 

When decoded, this is a newline.
When I removed html entities from the file, all 10088 records are processed.
DataStage job contains the following: External source (sends file name) connected to XML Input connected to transform connected to sequential file.
Questions:
Why does DataStage not write errors or warnings to the log?
Is there a way to tell DataStage 9.1.2 to ignore html entities and treat them as text?
Is this an NLS issue? NLS = None is not an option?
Any suggestions on how to resolve this?
DS 7.5.2
Server: None
Parallel: None
DS 9.1.2 NLS:
Server: UTF8 (Project Default)
Parallel: ASCL_ISO8859-1 (Project Default)
Parallel Job Description:
External source stage provides filename to XML input stage
XML input stage uses filename to parse XML file and writes to data files.
Observations (Same data file)
DataStage 7.5.2, XML stage outputs 10088 records
DataStage 9.1.3, XML stage outputs 4,798 records. No warnings or errors.
I found that data file contained the following html entity codes: 

When decoded, this is a newline.
When I removed html entities from the file, all 10088 records are processed.
DataStage job contains the following: External source (sends file name) connected to XML Input connected to transform connected to sequential file.
Questions:
Why does DataStage not write errors or warnings to the log?
Is there a way to tell DataStage 9.1.2 to ignore html entities and treat them as text?
Is this an NLS issue? NLS = None is not an option?
Any suggestions on how to resolve this?