Page 1 of 1
dos2unix - problem while converting
Posted: Fri Jan 05, 2007 5:25 am
by durgaps
Facing an issue while converting dos files to unix files using the dos2unix command. The sequence finishes with the following log message.
dos2unix /ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG
Reply=0
Output from command ====>
dos2unix: converting file /ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG to UNIX format ...
dos2unix: problems renaming './u2dtmpkESNiS' to '/ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG'
output file remains in './u2dtmpkESNiS'
dos2unix: problems converting file /ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG
The temporary file
./u2dtmpkESNiS is getting created in the DS Project directory and the actual file in the input folder is getting deleted.
I ran the same command at the prompt in Unix. It ran perfectly fine.
The issue is happening only when its being run in DS. Could there be a permission problem of writing the temp file from Project dir back to the actual dir?
Thanks,
Posted: Fri Jan 05, 2007 7:57 am
by DSguru2B
It might be permissions. Try storing the result in some other file, instead of the replacing the original file.
The above command converts a.txt and stores it in b.txt
Posted: Fri Jan 05, 2007 2:29 pm
by ray.wurlod
Do you (the user ID under which the DataStage job executes) have write permission to the /ds_data/METL/JOBS/WBTKEY/OUTPUT directory?
Posted: Fri Jan 05, 2007 6:43 pm
by durgaps
Hi DSGuru & Ray,
Thanks for the reply.
a.) I tried writing to a different file but the same problem exists.
b.) The DS user has complete permissions to write to the folder.
I was mulling over the thought of how to redirect the temp file being created in the <Project> folder to /tmp folder. Isnt this supposed to happen by default. i.e. the creation of temp file in /tmp ?
Posted: Fri Jan 05, 2007 7:44 pm
by DSguru2B
I did some more research and this is what i found on bugzilla. Check it out
https://bugzilla.redhat.com/bugzilla/sh ... ?id=150277
Posted: Sat Jan 06, 2007 1:35 pm
by mpouet
Hi,
How do you run your job : debug, director or command line ?
Matthieu
Posted: Mon Jan 08, 2007 12:31 am
by durgaps
I run my DataStage job using DS Director. Will running of the job in different modes have a say in the output?
Posted: Mon Jan 08, 2007 7:49 am
by DSguru2B
It will not matter.
Posted: Tue Jan 09, 2007 4:17 pm
by bcarlson
Can I ask a dumb question? Why are you bothering with dos2unix? DataStage can import it. The only thing you need to do is make sure your import takes the final Ctrl-M character into consideration. IMHO, this would be more efficient - one less step to deal with before your data is parallel-ized (is that a word) in DataStage.
If you really need to do it outside of DataStage, you could do it with anything - C, Perl, KSH, sed, awk, etc. Here's a simple Perl script:
Code: Select all
#!/usr/bin/perl
# File: mydos2unix.pl
# Description: Remove Ctrl-M chars from DOS files to use in Unix.
$^I = '';
LINE: while(<>) {
s/\015//; # Trim Ctrl-M characters
print "$_";
}
Call it as follows:
Code: Select all
mydos2unix.pl datafile.dat
- or -
cat datafile.dat | mydos2unix.pl > new_datafile.dat
The first call does an in-place edit - that is, the original file will be modified. The second call will create a new file that has been edited, and leave the original file intact.
Hope this helps...
Brad.
Posted: Tue Jan 09, 2007 7:23 pm
by ray.wurlod
Seems like a lot of work to reproduce a special case of the UNIX
tr command, Brad!
Posted: Tue Jan 09, 2007 7:31 pm
by DSguru2B
Its not a dumb question Brad. It never hit us. The OP can also use
Code: Select all
sed 's/.$//' mydosfile.txt > unixfile.txt
Posted: Tue Jan 09, 2007 9:33 pm
by bcarlson
ray.wurlod wrote:Seems like a lot of work to reproduce a special case of the UNIX
tr command, Brad!
Aww, come on. I thought that was what the 'etc.' was for:
If you really need to do it outside of DataStage, you could do it with anything - C, Perl, KSH, sed, awk, etc.
And do your code samples do in-place edits? Besides, the computer nerd in me likes to play with Perl.
Brad.
Re: dos2unix - problem while converting
Posted: Wed Jan 10, 2007 12:56 am
by Yuan_Edward
If you are very sure it's not concerned with UNIX permission, check the file name quotation: DataStage reads your command and may interpret and pass a different command to UNIX.
For example, './u2dtmpkESNiS' and "./u2dtmpkESNiS" are different things to UNIX and the last one is what you are after.
durgaps wrote:Facing an issue while converting dos files to unix files using the dos2unix command. The sequence finishes with the following log message.
dos2unix /ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG
Reply=0
Output from command ====>
dos2unix: converting file /ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG to UNIX format ...
dos2unix: problems renaming './u2dtmpkESNiS' to '/ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG'
output file remains in './u2dtmpkESNiS'
dos2unix: problems converting file /ds_data/METL/JOBS/WBTKEY/OUTPUT/HPCMTULDKEY_20061127101636.LOG
The temporary file
./u2dtmpkESNiS is getting created in the DS Project directory and the actual file in the input folder is getting deleted.
I ran the same command at the prompt in Unix. It ran perfectly fine.
The issue is happening only when its being run in DS. Could there be a permission problem of writing the temp file from Project dir back to the actual dir?
Thanks,
Re: dos2unix - problem while converting
Posted: Mon Jan 15, 2007 7:58 am
by mpouet
Hi durgaps ,
As you were thinking about a permission problem, I asked the way you were running DS, because depending on this, the profile loaded can change. When you run DS in command line with your profile (witch is not dsadm), jobs are running with the dsadm environment variables. So you may need to create some environment variables in dsadm profile and/or user defined environment variables with the Administrator.
Matthieu