Hi All , i am running a parallel job using source as a sequential file with pipe | as a delimiter but the data coming is bad like there are some carriage returns in some columns and some columns has data which contain pipe therefore DS is taking it as a new column results in rejection of records. i just want to know can we get rid of these issues using DS jobs if yes then how?
any help will be appreciated
thanks
data issues
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 397
- Joined: Wed Apr 12, 2006 2:28 pm
- Location: Tennesse
data issues
hi sam here
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Are the text strings that contain delimiter and terminator characters quoted? If so, DataStage can manage them, if not it can not, so you will need "them" to create a file in a legal format, or pre-process it yourself.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
- Premium Member
- Posts: 397
- Joined: Wed Apr 12, 2006 2:28 pm
- Location: Tennesse
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
If you specify the quote character as a field level property, it will find the opening quote and scan characters into the field until the closing quote is found. Then it will resume its search for delimiter characters.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
could be a low value(non-printable characters like Enter)
I had some issues similar to this. This could be a LOW VALUE with ASCII value less than 32. you can write a condition to see if the input ASCII value is less than CHAR(32) but that is bad data. By the way are you getting this from the source table?
-
- Premium Member
- Posts: 397
- Joined: Wed Apr 12, 2006 2:28 pm
- Location: Tennesse
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
You need to know what every byte in the stream is. The advice from rparimi is sound, and you need to be able to handle what ever is in the file. If that means forcing "them" to supply validly quoted character strings, then that's what it will take.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.