dos2unix - problem while converting

[durgaps]

Facing an issue while converting dos files to unix files using the dos2unix command. The sequence finishes with the following log message.

dos2unix /ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG

Reply=0
Output from command ====>
dos2unix: converting file /ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG to UNIX format ...
dos2unix: problems renaming './u2dtmpkESNiS' to '/ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG'
output file remains in './u2dtmpkESNiS'
dos2unix: problems converting file /ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG

The temporary file ./u2dtmpkESNiS is getting created in the DS Project directory and the actual file in the input folder is getting deleted.

I ran the same command at the prompt in Unix. It ran perfectly fine.

The issue is happening only when its being run in DS. Could there be a permission problem of writing the temp file from Project dir back to the actual dir?

Thanks,

[DSguru2B]

It might be permissions. Try storing the result in some other file, instead of the replacing the original file.

Code: Select all

dos2unix -n a.txt b.txt

The above command converts a.txt and stores it in b.txt

[ray.wurlod]

Do you (the user ID under which the DataStage job executes) have write permission to the /ds_data/METL/JOBS/WBTKEY/OUTPUT directory?

[durgaps]

Hi DSGuru & Ray,

Thanks for the reply.

a.) I tried writing to a different file but the same problem exists.
b.) The DS user has complete permissions to write to the folder.

I was mulling over the thought of how to redirect the temp file being created in the <Project> folder to /tmp folder. Isnt this supposed to happen by default. i.e. the creation of temp file in /tmp ?

[DSguru2B]

I did some more research and this is what i found on bugzilla. Check it out
https://bugzilla.redhat.com/bugzilla/sh ... ?id=150277

[mpouet]

Hi,

How do you run your job : debug, director or command line ?

Matthieu

[durgaps]

I run my DataStage job using DS Director. Will running of the job in different modes have a say in the output?

[DSguru2B]

It will not matter.

[bcarlson]

Can I ask a dumb question? Why are you bothering with dos2unix? DataStage can import it. The only thing you need to do is make sure your import takes the final Ctrl-M character into consideration. IMHO, this would be more efficient - one less step to deal with before your data is parallel-ized (is that a word) in DataStage.

If you really need to do it outside of DataStage, you could do it with anything - C, Perl, KSH, sed, awk, etc. Here's a simple Perl script:

Code: Select all

#!/usr/bin/perl

# File: mydos2unix.pl
# Description: Remove Ctrl-M chars from DOS files to use in Unix.

$^I = '';
LINE: while(<>) {
        s/\015//;               # Trim Ctrl-M characters
        print "$_";

}

Call it as follows:

Code: Select all

mydos2unix.pl datafile.dat

- or -

cat datafile.dat | mydos2unix.pl > new_datafile.dat

The first call does an in-place edit - that is, the original file will be modified. The second call will create a new file that has been edited, and leave the original file intact.

Hope this helps...


Brad.

[ray.wurlod]

Seems like a lot of work to reproduce a special case of the UNIX tr command, Brad!

Code: Select all

tr -d \r < filename > outputfile

[DSguru2B]

Its not a dumb question Brad. It never hit us. The OP can also use

Code: Select all

sed 's/.$//' mydosfile.txt > unixfile.txt

[bcarlson]

Seems like a lot of work to reproduce a special case of the UNIX tr command, Brad!

Code: Select all

tr -d \r < filename > outputfile

Aww, come on. I thought that was what the 'etc.' was for:

If you really need to do it outside of DataStage, you could do it with anything - C, Perl, KSH, sed, awk, etc.

And do your code samples do in-place edits? Besides, the computer nerd in me likes to play with Perl.

Brad.

[Yuan_Edward]

If you are very sure it's not concerned with UNIX permission, check the file name quotation: DataStage reads your command and may interpret and pass a different command to UNIX.

For example, './u2dtmpkESNiS' and "./u2dtmpkESNiS" are different things to UNIX and the last one is what you are after.

Facing an issue while converting dos files to unix files using the dos2unix command. The sequence finishes with the following log message.

dos2unix /ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG

Reply=0
Output from command ====>
dos2unix: converting file /ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG to UNIX format ...
dos2unix: problems renaming './u2dtmpkESNiS' to '/ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG'
output file remains in './u2dtmpkESNiS'
dos2unix: problems converting file /ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG

The temporary file ./u2dtmpkESNiS is getting created in the DS Project directory and the actual file in the input folder is getting deleted.

I ran the same command at the prompt in Unix. It ran perfectly fine.

The issue is happening only when its being run in DS. Could there be a permission problem of writing the temp file from Project dir back to the actual dir?

Thanks,

[mpouet]

Hi durgaps ,

As you were thinking about a permission problem, I asked the way you were running DS, because depending on this, the profile loaded can change. When you run DS in command line with your profile (witch is not dsadm), jobs are running with the dsadm environment variables. So you may need to create some environment variables in dsadm profile and/or user defined environment variables with the Administrator.

Matthieu