Problem in running unix ftp script as Before-job subroutine

[userds]

I am running a following unix ftp script as Before job subroutine.

Please find the following ftp script:
##############

#!/bin/ksh
#ftp script filename: nt2unixcp

print "script begins"

host='gskprod.abc.com'
USER='gsk'
PASSWORD='gsk589'
remote_dir='gsk_df'

ftp -ni "${host}" << EOF
user $USER $PASSWORD
binary
cd $remote_dir
mget *.txt

bye
EOF

print "script completed successfully"

##############

Following is the Event Detail.

PCJB006..BeforeJob (ExecSH): Executed command: /dsProjects/data/xd_data/GSKFTP/nt2unixcp.sh
*** Output from command was: ***
script begins
script completed successfully

Here in the event detail it shows the ftp script has been run sucessfully. But it didn't ftp any of the text files in the directory. The script has all the permission to get executed from Datastage. After running the Job if i check with the history command in unix server it doesn't show up in the history list.

But the script is correctly ftping all the text files when it is executed directly from unix server.

Any help will be highly appreciated for this issue.

TIA.

[davidnemirovsky]

I'm assuming you have selected 'ExecSH' in the 'Before Job Subroutine' drop down box. What is exact command you are using in the InputValue text box?

[ketfos]

Hi,
You should check status of ftp using something like this

ftp_return_code=`grep -i "226 Transfer Complete." /ardent/uv/DataStage/Projects`
if [[ $ftp_return_code -ne NUM]]
then

echo "IQ FILE CREATION FAILED" >

If there are 14 files for ftp then replace NUM by 14 in above script.

Also echo complete ftp process into a log file and see the log file to see if it is
1. actually opening ftp port
2. connecting to user
3. getting right password.
4. connecting to target machine.
5. reading from source.
6. writing to target.
7. exiting ftp process.

[userds]

Please find the below log file content. It actually opens the port and successfully ftping the files. But the files are not getting copied into the local unix directory.

PCJB006..BeforeJob (ExecSH): Executed command: /dsProjects/data/yga_data/nt2unixcp.sh
*** Output from command was: ***
script begins
Connected to gskprod.abc.com.
220 DGRPWADEB1 Microsoft FTP Service (Version 5.0).
---> USER dgaftp
331 Password required for dgaftp.
---> PASS XXXX
230 User dgaftp logged in.
---> TYPE A
200 Type set to A.
---> PWD
257 "/" is current directory.
---> CWD dga_ds
250 CWD command successful.
---> PWD
257 "/dga_ds" is current directory.
---> PORT 172,20,192,142,174,98
---> NLST *.txt
---> PORT 172,20,192,142,174,99
200 PORT command successful.
---> RETR pnba022r200405062.txt
150 Opening ASCII mode data connection for pnba022r200405062.txt(128828 bytes).
226 Transfer complete.
local: pnba022r200405062.txt remote: pnba022r200405062.txt
128828 bytes received in 26 seconds (4.84 Kbytes/s)
---> PORT 172,20,192,142,174,106
200 PORT command successful.
---> RETR PC_0016_03142004.txt
150 Opening ASCII mode data connection for PC_0016_03142004.txt(25602 bytes).
226 Transfer complete.
local: PC_0016_03142004.txt remote: PC_0016_03142004.txt
25602 bytes received in 4.2 seconds (6.00 Kbytes/s)
---> PORT 172,20,192,142,174,107
200 PORT command successful.
---> RETR PC_0019_03142004.txt
150 Opening ASCII mode data connection for PC_0019_03142004.txt(2681182 bytes).
226 Transfer complete.
local: PC_0019_03142004.txt remote: PC_0019_03142004.txt
2681182 bytes received in 12 seconds (221.39 Kbytes/s)
---> PORT 172,20,192,142,174,110
200 PORT command successful.
---> RETR PC_0035_03142004.txt
150 Opening ASCII mode data connection for PC_0035_03142004.txt(199593 bytes).
226 Transfer complete.
local: PC_0035_03142004.txt remote: PC_0035_03142004.txt
199593 bytes received in 10 seconds (19.26 Kbytes/s)
---> PORT 172,20,192,142,174,113
200 PORT command successful.
---> RETR PC_0068_03142004.txt
150 Opening ASCII mode data connection for PC_0068_03142004.txt(2527570 bytes).
226 Transfer complete.
local: PC_0068_03142004.txt remote: PC_0068_03142004.txt
2527570 bytes received in 9.9 seconds (250.19 Kbytes/s)
---> PORT 172,20,192,142,174,116
200 PORT command successful.
---> RETR PC_0113_03142004.txt
150 Opening ASCII mode data connection for PC_0113_03142004.txt(127624 bytes).
226 Transfer complete.
local: PC_0113_03142004.txt remote: PC_0113_03142004.txt
127624 bytes received in 8 seconds (15.50 Kbytes/s)
---> QUIT
221
script completed successfully

Even after all these successfull execution still i don't know where its going wrong.

I am just calling the unix script and not passing any parameter and i think its not necessary to pass any parameter when i call this unix script.

Datastage server calls the Unix script but it is not logged in the history command at shell prompt. This really surprises me.

Please help me out to fix this problem.

Thanks a lot for earlier suggestions.

[chulett]

Please find the below log file content. It actually opens the port and successfully ftping the files. But the files are not getting copied into the local unix directory.

A couple of points:

1) Your message 'script completed successfully' should be changed to 'script completed'. There's no check for success of the actual ftp command, so all you really know is it got to the end of the script.

2) Your 'mistake' is assuming you know what your 'current working directory' is when running scripts from a DataStage job. If you expect the files to be ftp'd to a specific directory, then you need to 'cd' to that 'local unix directory' before initiating the ftp command. You'll find that by default the current working directory of a DataStage job is the project in which it lives - which is where you'll find your files.

[userds]

Thanks a lot Craig. I specified the local directory, now it is ftping the files in that directory. I don't know we have to specify the local directory explicitly. When ftp script is running, its execution control is under Unix,
But how datastage server changed the location of the ftped files. This really surprises me.

[chulett]

One of the first things you need to learn when writing scripts for DataStage is - make no assumptions about anything.

Specify full paths, beware of relative paths unless you know exactly where you are, don't assume environment variables are set, etc, etc. The environment a script runs under in DataStage bears little resemblance to when you run it from the command line, even when running as "the same user".

One quick way to prove it to yourself is to have a job run a script that simply executes the 'set' command. Look in the log for 'Output from command' and compare that to what happens when you do the same thing from the command line.

[ketfos]

Hi Craig,
I think you brought forward a very good point on assumptions which we make while writing scripts in datastage
Yes, we ought not to make any assumptions like the path used, etc in the scripts.

Thanks
Ketfos