new line charecter in btwn double quotes in sequential file

sahityab · Post by **sahityab** » Wed Sep 21, 2011 8:14 am

Hi ,
I am having issues with handling new line charecters inbetween the double quotes in a sequenctail file.THe sequencial file has the follwoing setting
RECORD DELIM =UNIX NEW LINE
FILED DELIM=~
QUOTES=double
NULL=*

Here is the example of a record i am trying to handle

2011~~~"'eye infection' written tresaderm131.37
"~"3449 E Pacific Coast Hwy

My understanding is that as long as the filed is varchar and if the value is inbetween the double quotes datastage would not try to resolve the charecters in between the quotes...But datastage is rejecting the records .Can some one please help find a soulution for this
thanks in advance
S

ray.wurlod · Post by **ray.wurlod** » Wed Sep 21, 2011 3:50 pm

Parallel Sequential File stage seems to have trouble with these. Prefer a server Sequential File stage, perhaps in a server Shared Container in your parallel job.

qt_ky · Post by **qt_ky** » Wed Sep 21, 2011 9:31 pm

It sounds like the newline takes precedence over the double quote. I had a similar situation in a previous life and used transformer stage variables to concatenate rows like this together. The trick with that option is to be able to detect the situation correctly.

dspxlearn · Post by **dspxlearn** » Thu Sep 22, 2011 3:35 am

May be, a shell script can do the job to remove the new line characters before the double quotes in the file. Then, this script can be called in the before job subroutine and have the cleaned file ready for the job as a input.

ray.wurlod · Post by **ray.wurlod** » Thu Sep 22, 2011 3:21 pm

Why do people think they can change the client's data with impunity?!!

What if these linefeeds are meaningful to the client?

sahityab · Post by **sahityab** » Thu Sep 22, 2011 7:54 pm

Thank you for all your responses.
I my case we do not want the newline charecters to be there,it was a mistake on the source side that we are getting these charecters...any suggestion on how to remove the new line charecters are greatly aprreciated ....
i really cant use a transformer to do this as the records are getting rejected when trying to load to the seq file stage itself....
I am just concerned about the perofomance of the job to use a server job to do this......please let me know if you have any ideas
THanks
Sahi

sahityab · Post by **sahityab** » Thu Sep 22, 2011 8:01 pm

Hi Ray,
Can you please explain what you mean by using a server Sequential File stage or using a server Shared Container Do you mean to recomment creating a server job for this...please explain
Thanks
Sahi

ray.wurlod · Post by **ray.wurlod** » Thu Sep 22, 2011 9:07 pm

Yes, I recommend using a server Sequential File stage.

Depending on the volume of data, use a server job (small to medium volume, say up to 10 million rows) or a server Shared Container containing just a Sequential File stage in a parallel job if the volume of data is large.

sahityab · Post by **sahityab** » Tue Sep 27, 2011 1:36 pm

Hi Ray,
thank you for the response...I have not used shared containers earlier can you please explain how to create a shared ocntainer with just seq file in it....i beileve there should be output connection to the link fromt he seq file also...also i have like 10 files each with their own format in which these new line charecters come in .So Can i use one sharec container for all of them or sh ould i be creating one for each....
Thanks

ray.wurlod · Post by **ray.wurlod** » Tue Sep 27, 2011 4:10 pm

Here's what I have in mind.

Code: Select all

SharedContainer
+-----------------------------------+
|                                   |
|   SeqFile                Output   |
|   +-----+                +----+   |
|   |     |  ----------->  |    | --------->
|   +-----+                +----+   |
|                                   |
+-----------------------------------+

sahityab · Post by **sahityab** » Thu Oct 13, 2011 7:01 am

Thank you all for your responses...I was able to resolve the issue by removing the newline charecters using macros in the excel....