"Telnet Before Command" sporadically not running c

[lkthoma]

I currently have a server job that contains an ftp stage. Within this ftp stage I use the "Telnet Before Command" to execute a unix shell script before it performs the file ftp. Inside this unix shell script I create a log file that contains everything the script has completed. When this server job runs, it will sporadically NOT run the shell script. Not log is generated fromthe script, and there is nothing in the datastage job that shows any errors. I even enabled tracing for the Telnet and no errors are displayed there either. Here is the log entries:

sjPPIR0038..FTP_fcst: Entered tnet_exec() function.
sjPPIR0038..FTP_fcst: calling tnet_connect()
sjPPIR0038..FTP_fcst: Connected to telnet socket.
sjPPIR0038..FTP_fcst: Looking for Telnet prompt number 1
sjPPIR0038..FTP_fcst: Total iterations in login loop : 3
sjPPIR0038..FTP_fcst: Prompt [login], received
sjPPIR0038..FTP_fcst: Looking for Telnet prompt number 2
sjPPIR0038..FTP_fcst: Total iterations in password loop : 2
sjPPIR0038..FTP_fcst: Prompt [password], received
sjPPIR0038..FTP_fcst: Prompt not received, keep reading
sjPPIR0038..FTP_fcst: Prompt not received, keep reading
sjPPIR0038..FTP_fcst: Prompt not received, keep reading
sjPPIR0038..FTP_fcst: Prompt received
sjPPIR0038..FTP_fcst: Total iterations in 'prompt' loop : 4
sjPPIR0038..FTP_fcst: Sending command: /usr/dw/utilities/ppir0055.ksh "FCST_" /usr/dw/nonsap/db_files "DAT" /usr/dw/tds/db_files
sjPPIR0038..FTP_fcst: Finished tnet_exec() function.


Any idea's or has anyone had this issue before??

[DSguru2B]

Does the script run from command line? May i ask why are you executing the script in "Telnet Before Command" and why not in the before job subroutine "ExecSH"?

[lkthoma]

The command always works from the command line and sporatically works from the Telnet Before Command. As for not using the ExecSH before job subroutine, this was developed by a IBM contractor and I inherited the support/maintenance, so I cannot answer why they did what they did. I had the same thought as to using that solution, but I want to understand why this failing? Everything in the logs looks fine and it does execute successfully 99 percent of the time.

[DSguru2B]

Tell me about the log that the shell script creates. When the shell doesnt execute, or you think it does not execute, is there any entry in the unix shell's log? If there is nothing, what does the director's log say.

[lkthoma]

That's what I tried to explain previously. The datastage log above says it ran just fine, but there is NO unix shell script log at all.

[DSguru2B]

I have no clue buddy. There is no return code either. Change the design, get it to execute in the before job subroutine. Atleast you can get a return code there inside the log to look at next time the script fails.

[lkthoma]

That's what I'll probably end up doing. Things like this are what frustrates me about DataStage. Not to mention the fact, that I also tested my unix script by having it do an exit 99 to see if the datastage job would abort...it didn't.

[DSguru2B]

Well, if DataStage is used properly then it wouldnt frustrate you. I have been using DS for doing different things, from awk scripts to calling ftp scripts, from file size checking on windoze box to using c routines. Its been good to me. You could create a routine and execute the shell via DSExecute(). You can manipulate the end result and test it out with return codes.

[lkthoma]

Ok, so where is the 'book' on 'used properly'? If you provide the functionality to call a script through a Telnet Before Command and there are 'got ya' things you shouldnt do, then a responsible software company would outline those. I've seen nothing in any manual that says NOT to do what this contractor wrote.

[kcbland]

A professionally done job would have not used this stage or these features. I would have written a generic library of reusable scripts and avoided the FTP stage altogether. I'd use ssh and scp to satisfy security concerns, as well as promote a faster file transfer (dedicated transfer with optional pre-file compression) versus remote reading. In addition, better error trapping and retriability from a script.

Sorry to say, but the contractor is gone and not having to do the support. If everyone would design and build for maintenance and support, you'd be having an easier time.

[DSguru2B]

Not everything will be outlined. In DataStage there are more ways than one to achieve a task. Are they all good ? No, Are they all supported? maybe not. Telnet execution might not be able to react to the return code, but before/after job subroutine does. So can a routine. Some functions have some capabilities, while others dont. I can catch my nose from the front, or let it come from behind, i am achieving the same through both ways. Which one will make me look stupid, you decide.
IMHO.

[jgreve]

from the oline help r.e. FTP Plug-in
"Telnet Before Command. The telnet command to execute on the remote machine before starting a job.
Telnet After Command. Specifies the telnet command to execute on the remote machine after completing a job."

Sounds like your before-command will run a script on the remote
machine via the "telnet before command."
No log output from your script implies the script was never invoked.

What are the parameters in your "ftp" stage?
In particular, I'm curious about:
Command Timeout
Telnet Prompt 1
Telnet Reply 1
Telnet Prompt 2
Telnet Reply 2
...3, 4, 5, etc...
Telnet Prompt 9
Telnet Reply 9
Telnet Prompt 10
Telnet Reply 10

Because it is intermittent, I suspect heavy loads on
your remote machine are causing DataStage to give up
before "/usr/dw/utilities/ppir0055.ksh" finishes running.
In that case, I don't know telnet well enough to know
if it will kill "ppir0055.ksh" before it can flush any
output to your log file.

However... what happens if you change your telnet-command from this:

Code: Select all

/usr/dw/utilities/ppir0055.ksh "FCST_" /usr/dw/nonsap/db_files "DAT" /usr/dw/tds/db_files >> /tmp/foo.log 2>&1

to this:

Code: Select all

echo "Hi from DataStage." > /tmp/foo.log; /usr/dw/utilities/ppir0055.ksh "FCST_" /usr/dw/nonsap/db_files "DAT" /usr/dw/tds/db_files >> /tmp/foo.log 2>&1; echo "Done." >> /tmp/foo.log

Breaking it down, this squeezes 3 commands into a single line...

Code: Select all

echo "Hi from DataStage." > /tmp/foo.log;
/usr/dw/utilities/ppir0055.ksh ...etc... >> /tmp/foo.log 2>&1;
echo "Done." >> /tmp/foo.log

Note that the use of /tmp/foo.log is just smth I made up, you'll
probably want to write it to your home directory or smth.

Note that the difference between > and >> is significant; the first one will overwrite the log file, the others append to it. In this example (above), if you just used > the only surviving data in your log would be "Done.".

The "2>&1" part will send stderr into your log file as well; maybe something
else is borking ppir0055.ksh and the original author didn't trap all output.

The theory here would be you'll get at least a "Welcome..." msg in the log file after the dust settles, maybe more.
That will help guide where you are looking: at DataStage vs. remote-system.

Good luck, sounds like a real joy to debug.
John G.

[jvmerc]

This sure looks like DataStage not DataStage TX......