"Telnet Before Command" sporadically not running c
Moderators: chulett, rschirm, roy
"Telnet Before Command" sporadically not running c
I currently have a server job that contains an ftp stage. Within this ftp stage I use the "Telnet Before Command" to execute a unix shell script before it performs the file ftp. Inside this unix shell script I create a log file that contains everything the script has completed. When this server job runs, it will sporadically NOT run the shell script. Not log is generated fromthe script, and there is nothing in the datastage job that shows any errors. I even enabled tracing for the Telnet and no errors are displayed there either. Here is the log entries:
sjPPIR0038..FTP_fcst: Entered tnet_exec() function.
sjPPIR0038..FTP_fcst: calling tnet_connect()
sjPPIR0038..FTP_fcst: Connected to telnet socket.
sjPPIR0038..FTP_fcst: Looking for Telnet prompt number 1
sjPPIR0038..FTP_fcst: Total iterations in login loop : 3
sjPPIR0038..FTP_fcst: Prompt [login], received
sjPPIR0038..FTP_fcst: Looking for Telnet prompt number 2
sjPPIR0038..FTP_fcst: Total iterations in password loop : 2
sjPPIR0038..FTP_fcst: Prompt [password], received
sjPPIR0038..FTP_fcst: Prompt not received, keep reading
sjPPIR0038..FTP_fcst: Prompt not received, keep reading
sjPPIR0038..FTP_fcst: Prompt not received, keep reading
sjPPIR0038..FTP_fcst: Prompt received
sjPPIR0038..FTP_fcst: Total iterations in 'prompt' loop : 4
sjPPIR0038..FTP_fcst: Sending command: /usr/dw/utilities/ppir0055.ksh "FCST_" /usr/dw/nonsap/db_files "DAT" /usr/dw/tds/db_files
sjPPIR0038..FTP_fcst: Finished tnet_exec() function.
Any idea's or has anyone had this issue before??
sjPPIR0038..FTP_fcst: Entered tnet_exec() function.
sjPPIR0038..FTP_fcst: calling tnet_connect()
sjPPIR0038..FTP_fcst: Connected to telnet socket.
sjPPIR0038..FTP_fcst: Looking for Telnet prompt number 1
sjPPIR0038..FTP_fcst: Total iterations in login loop : 3
sjPPIR0038..FTP_fcst: Prompt [login], received
sjPPIR0038..FTP_fcst: Looking for Telnet prompt number 2
sjPPIR0038..FTP_fcst: Total iterations in password loop : 2
sjPPIR0038..FTP_fcst: Prompt [password], received
sjPPIR0038..FTP_fcst: Prompt not received, keep reading
sjPPIR0038..FTP_fcst: Prompt not received, keep reading
sjPPIR0038..FTP_fcst: Prompt not received, keep reading
sjPPIR0038..FTP_fcst: Prompt received
sjPPIR0038..FTP_fcst: Total iterations in 'prompt' loop : 4
sjPPIR0038..FTP_fcst: Sending command: /usr/dw/utilities/ppir0055.ksh "FCST_" /usr/dw/nonsap/db_files "DAT" /usr/dw/tds/db_files
sjPPIR0038..FTP_fcst: Finished tnet_exec() function.
Any idea's or has anyone had this issue before??
lkthoma
The command always works from the command line and sporatically works from the Telnet Before Command. As for not using the ExecSH before job subroutine, this was developed by a IBM contractor and I inherited the support/maintenance, so I cannot answer why they did what they did. I had the same thought as to using that solution, but I want to understand why this failing? Everything in the logs looks fine and it does execute successfully 99 percent of the time.
lkthoma
Tell me about the log that the shell script creates. When the shell doesnt execute, or you think it does not execute, is there any entry in the unix shell's log? If there is nothing, what does the director's log say.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
Well, if DataStage is used properly then it wouldnt frustrate you. I have been using DS for doing different things, from awk scripts to calling ftp scripts, from file size checking on windoze box to using c routines. Its been good to me. You could create a routine and execute the shell via DSExecute(). You can manipulate the end result and test it out with return codes.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
Ok, so where is the 'book' on 'used properly'? If you provide the functionality to call a script through a Telnet Before Command and there are 'got ya' things you shouldnt do, then a responsible software company would outline those. I've seen nothing in any manual that says NOT to do what this contractor wrote.
lkthoma
A professionally done job would have not used this stage or these features. I would have written a generic library of reusable scripts and avoided the FTP stage altogether. I'd use ssh and scp to satisfy security concerns, as well as promote a faster file transfer (dedicated transfer with optional pre-file compression) versus remote reading. In addition, better error trapping and retriability from a script.
Sorry to say, but the contractor is gone and not having to do the support. If everyone would design and build for maintenance and support, you'd be having an easier time.
Sorry to say, but the contractor is gone and not having to do the support. If everyone would design and build for maintenance and support, you'd be having an easier time.
Kenneth Bland
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Rank: Sempai
Belt: First degree black
Fight name: Captain Hook
Signature knockout: right upper cut followed by left hook
Signature submission: Crucifix combined with leg triangle
Not everything will be outlined. In DataStage there are more ways than one to achieve a task. Are they all good ? No, Are they all supported? maybe not. Telnet execution might not be able to react to the return code, but before/after job subroutine does. So can a routine. Some functions have some capabilities, while others dont. I can catch my nose from the front, or let it come from behind, i am achieving the same through both ways. Which one will make me look stupid, you decide.
IMHO.
IMHO.
Creativity is allowing yourself to make mistakes. Art is knowing which ones to keep.
Re: "Telnet Before Command" sporadically not runni
from the oline help r.e. FTP Plug-in
"Telnet Before Command. The telnet command to execute on the remote machine before starting a job.
Telnet After Command. Specifies the telnet command to execute on the remote machine after completing a job."
Sounds like your before-command will run a script on the remote
machine via the "telnet before command."
No log output from your script implies the script was never invoked.
What are the parameters in your "ftp" stage?
In particular, I'm curious about:
Command Timeout
Telnet Prompt 1
Telnet Reply 1
Telnet Prompt 2
Telnet Reply 2
...3, 4, 5, etc...
Telnet Prompt 9
Telnet Reply 9
Telnet Prompt 10
Telnet Reply 10
Because it is intermittent, I suspect heavy loads on
your remote machine are causing DataStage to give up
before "/usr/dw/utilities/ppir0055.ksh" finishes running.
In that case, I don't know telnet well enough to know
if it will kill "ppir0055.ksh" before it can flush any
output to your log file.
However... what happens if you change your telnet-command from this:
to this:
Breaking it down, this squeezes 3 commands into a single line...
Note that the use of /tmp/foo.log is just smth I made up, you'll
probably want to write it to your home directory or smth.
Note that the difference between > and >> is significant; the first one will overwrite the log file, the others append to it. In this example (above), if you just used > the only surviving data in your log would be "Done.".
The "2>&1" part will send stderr into your log file as well; maybe something
else is borking ppir0055.ksh and the original author didn't trap all output.
The theory here would be you'll get at least a "Welcome..." msg in the log file after the dust settles, maybe more.
That will help guide where you are looking: at DataStage vs. remote-system.
Good luck, sounds like a real joy to debug.
John G.
"Telnet Before Command. The telnet command to execute on the remote machine before starting a job.
Telnet After Command. Specifies the telnet command to execute on the remote machine after completing a job."
Sounds like your before-command will run a script on the remote
machine via the "telnet before command."
No log output from your script implies the script was never invoked.
What are the parameters in your "ftp" stage?
In particular, I'm curious about:
Command Timeout
Telnet Prompt 1
Telnet Reply 1
Telnet Prompt 2
Telnet Reply 2
...3, 4, 5, etc...
Telnet Prompt 9
Telnet Reply 9
Telnet Prompt 10
Telnet Reply 10
Because it is intermittent, I suspect heavy loads on
your remote machine are causing DataStage to give up
before "/usr/dw/utilities/ppir0055.ksh" finishes running.
In that case, I don't know telnet well enough to know
if it will kill "ppir0055.ksh" before it can flush any
output to your log file.
However... what happens if you change your telnet-command from this:
Code: Select all
/usr/dw/utilities/ppir0055.ksh "FCST_" /usr/dw/nonsap/db_files "DAT" /usr/dw/tds/db_files >> /tmp/foo.log 2>&1
Code: Select all
echo "Hi from DataStage." > /tmp/foo.log; /usr/dw/utilities/ppir0055.ksh "FCST_" /usr/dw/nonsap/db_files "DAT" /usr/dw/tds/db_files >> /tmp/foo.log 2>&1; echo "Done." >> /tmp/foo.log
Code: Select all
echo "Hi from DataStage." > /tmp/foo.log;
/usr/dw/utilities/ppir0055.ksh ...etc... >> /tmp/foo.log 2>&1;
echo "Done." >> /tmp/foo.log
probably want to write it to your home directory or smth.
Note that the difference between > and >> is significant; the first one will overwrite the log file, the others append to it. In this example (above), if you just used > the only surviving data in your log would be "Done.".
The "2>&1" part will send stderr into your log file as well; maybe something
else is borking ppir0055.ksh and the original author didn't trap all output.
The theory here would be you'll get at least a "Welcome..." msg in the log file after the dust settles, maybe more.
That will help guide where you are looking: at DataStage vs. remote-system.
Good luck, sounds like a real joy to debug.
John G.