double quotes inside data

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
dnat
Participant
Posts: 200
Joined: Thu Sep 06, 2007 2:06 am

double quotes inside data

Post by dnat »

Hi,

I have double quotes inside the data.

While reading, i am getting this error

with 'delim=end' did not consume entire input, at offset: 317

Ex

"1"|"9999"|"Exbis"|"MC"kh""|""

I searched the forum and i couldnt find any resolution to handle this

I have the Quote character as double quote now, i changed it to none, but then all the records are getting rejected.

Any help?
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

dnat - your source file is badly formed, the correct thing to do is to correct the file. If that is not possible then another option would work if you are certain that your column separator, the '|' symbol, does not occur in the data. In that case, don't use a quote character at all and read in the strings, then strip leading and trailing double quotes from each string. Another option is to "fix" the file. Declare it as just one (big) column, then replace all double-quotes with two double quotes where the quote is not preceded by or followed by a pipe.
DST
Participant
Posts: 7
Joined: Wed Nov 02, 2005 6:40 am

Re: double quotes inside data

Post by DST »

dnat

i have the same problem. in server jobs this works correct.
here is the answer from IBM support:

"Yes, Parallel job doesn't manage quote included in a field as a Server job for the reading of sequential file (CSV file).

I confirm Parallel job doesn't manage quote included in a field as a Server job for the reading of sequential file (CSV file).
To read this type of CSV file, they could:
- keep a "Server" job to read data, and include this job in a job sequence.
- write a specific 'buildop' to split the record in fields, and use this 'buildop' reading 1 record=1 field."

I am going to achieve a recognition of it as a bug
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

DST - this is not a bug.

Code: Select all

"I am not " a well-formed sentence. But ""I"" am."
Embedded quotes must be doubled.
DST
Participant
Posts: 7
Joined: Wed Nov 02, 2005 6:40 am

Post by DST »

ArndW

ok,
another example:

1,"company name ""super company""",company1

is well-formed or not ? and if i get it from not-covered resource. what should i do ??
this ok in server jobs, why failed in parallel ?
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

[

Code: Select all

1,"company name ""super company""","company1"
is a WFF.

The server job is tolerant of this incorrect data format.
DST
Participant
Posts: 7
Joined: Wed Nov 02, 2005 6:40 am

Post by DST »

ArndW wrote:[

Code: Select all

1,"company name ""super company""","company1"
is a WFF.

The server job is tolerant of this incorrect data format. ...
... but parallel job not ((( why ???
ArndW
Participant
Posts: 16318
Joined: Tue Nov 16, 2004 9:08 am
Location: Germany
Contact:

Post by ArndW »

The appropriate question is not why it doesn't work in PX jobs but why it works (when it shouldn't) in Server jobs.
Post Reply