Datastage text handling
Posted: Thu Jun 15, 2017 6:19 pm
Hello,
I have worked with several ETL tools, and have seen there are different ways to handle text fields. On importing a table definition the datastage has converted a text field into Longvarchar. The process then has to write into a hdfs location via hdfs connector, basically a .dsv file.
Here are some errors I am finding:
The array size must be set to 1 when reading LOBs. This setting when done however gives blank values.
When I set the Longvarchar value to the max 2GB value, it throws the error:
Schema reconciliation detected a mismatch for the field.
I think this may need some admin setting :
reference:
http://www-01.ibm.com/support/docview.w ... wg21614781
Has anyone any better suggestions, solutions?
I have worked with several ETL tools, and have seen there are different ways to handle text fields. On importing a table definition the datastage has converted a text field into Longvarchar. The process then has to write into a hdfs location via hdfs connector, basically a .dsv file.
Here are some errors I am finding:
The array size must be set to 1 when reading LOBs. This setting when done however gives blank values.
When I set the Longvarchar value to the max 2GB value, it throws the error:
Schema reconciliation detected a mismatch for the field.
I think this may need some admin setting :
reference:
http://www-01.ibm.com/support/docview.w ... wg21614781
Has anyone any better suggestions, solutions?