non delimiter file

kashif007 · Post by **kashif007** » Wed Aug 02, 2006 5:10 pm

I am not able to run a job. The compilation has no error. Now the reason is that the job cannot read the file from the source unix server. The reason for the error is that there is no delimiter in the two flat files the is acting as input. The job has a before job subroutine for concatenating two flat files. Both the files exist on the server, but they dont have any delimitation between the columns which usually a flat file has. The job is getting aborted and like i said no delimiter found is the reason.

can any one suggest me how to tackle a file with no delimiter.
something to do with the properties of before job subroutine i guess or whatever, please help me. i am stuck in my project.

meena · Post by **meena** » Wed Aug 02, 2006 5:15 pm

Hi kashif007,
Can you please post the subroutine and your job design.Based on these two I can suggest you......

Thank you

chulett · Post by **chulett** » Wed Aug 02, 2006 6:24 pm

kashif007 wrote:can any one suggest me how to tackle a file with no delimiter. something to do with the properties of before job subroutine i guess or whatever, please help me. i am stuck in my project.

It should have nothing to do with your concatenation or before job subroutine. It's all about properly defining the file's characteristics in the Sequential file stage.

What you have is called a 'Fixed Width' file and you should be able to enable that in the stage. You'll also need to make sure your metadata is correct so it knows how many bytes each field contains. On the Server side that would be controlled by the Display Size, not so sure on the PX version...

chulett · Post by **chulett** » Wed Aug 02, 2006 6:30 pm

Upon further review, there could be an issue with the concatenation in addition to my previous post. Do you know if each record is delimited, meaning there are EOR characters at the end of each record? If you do a 'wc -l' on the file, does it show a 1 or something else?

I'm wondering if there will be an issue with concatenating two files without record delimiters together at the 'seam'.

If there is, you can always skip the concatenation, as the PX Sequential File stage is perfectly capable of reading both files one after the other using a wildcard pattern.

ray.wurlod · Post by **ray.wurlod** » Wed Aug 02, 2006 9:57 pm

Why not FTP both files from the UNIX machine and load them into DataStage with a filter (type file1 file2) in a Sequential File stage?
Specify the stage properties as no line terminator, and the record as fixed-width format. That way the column display widths will completely specify the row structure, and you can then process the data.
This recommendation assumes that both files have the same record structure.

kumar_s · Post by **kumar_s** » Wed Aug 02, 2006 10:07 pm

May I know, who can we make the Windows DS stages to read unix files. I never used windows as server.

kashif007 · Post by **kashif007** » Thu Aug 03, 2006 8:54 am

if i do wc -l file name, it shows me 10 for one file and 3 for the other file
can u tell me how does it effect the delimitation

chulett wrote:Upon further review, there could be an issue with the concatenation in addition to my previous post. Do you know if each record is delimited, meaning there are EOR characters at the end of each record? If you do a 'wc -l' on the file, does it show a 1 or something else?

I'm wondering if there will be an issue with concatenating two files without record delimiters together at the 'seam'. If there is, you can always skip the concatenation, as the PX Sequential File stage is perfectly capable of reading both files one after the other using a wildcard pattern.

chulett · Post by **chulett** » Thu Aug 03, 2006 10:18 am

Then I would say you should be fine with either approach.

ray.wurlod · Post by **ray.wurlod** » Thu Aug 03, 2006 10:43 am

The wc -l command reports the number of lines in the file. If it can count lines, then there ARE line terminator characters. Your original suggestion, that there are none, is disproven.

You can get a Windows server to read UNIX files using a gateway such as samba. Or you can FTP the files from the UNIX machine to the Windows machine, and read DOS files. (Depending on your FTP utility, you may still need to set the record delimiter property to "UNIX style".)

chulett · Post by **chulett** » Thu Aug 03, 2006 11:20 am

There are record terminators, yes. I believe the question is regarding the lack of field delimiters, but now who knows. We'd need some clarification here.

kashif007 · Post by **kashif007** » Thu Aug 03, 2006 1:51 pm

right guys
i made it work
now i was reading the records from two files on unix server using the comman
$head -40 filename1 filename2 > filename3
$head -40 filename4 filename5 > filename6
now the format of the data was not saved properly that why delimiter error occured
so i tried "tail" instead of head, so now it works beautifully.
so the answer is:
$tail -40 filename1 filename2 > filename3
$tail -40 filename4 filename5 > filename6
its just a small thing but took me a while to figure out.
thanks guys for your help.

meena wrote:Hi kashif007,
Can you please post the subroutine and your job design.Based on these two I can suggest you......

Thank you

kumar_s · Post by **kumar_s** » Thu Aug 03, 2006 7:57 pm

head -40 should give you first 40lines, tail -40 should give your last 40 lines, apart from this, may I know what difference made your job to run fine?

kashif007 · Post by **kashif007** » Thu Aug 03, 2006 10:17 pm

some times there is the format of the file which does not show propely
so its better to take tail than head

kumar_s wrote:head -40 should give you first 40lines, tail -40 should give your last 40 lines, apart from this, may I know what difference made your job to run fine?

kumar_s · Post by **kumar_s** » Fri Aug 04, 2006 1:22 am

Check whether your file have any header information. Is it totally a 40 line file.

DSXchange

non delimiter file

non delimiter file

Re: non delimiter file