Datastage returns fake return code

[cheerfu1]

Hi,

I am doing a project which requires me to process a number of files. However, this processing requires me to delete the input file one by one once it is processed. e.g.

I have input files: input1, input2, input3

After I process input1 using DS Job -wait -jobstatus
I delete input1 before processing input2 and so on

However, eventhough I have used "-wait", it seems sometimes the program delete the input1 even before DS Job starts running properly, thus will return an error code. The funny thing is, sometimes the error code is 2, sometimes it's 1 (which is successful). Thus, the next processing which is loading the data will fail because it can't find the output from the ds job. Anybody has experienced that problem before? This problem is quite intermitten, I tried to simulate it by writing down the steps, but it just does not work the same way everytime. Thanks a lot.

[roy]

Hi,
You need to supply more info regarding your logic.
posting the batch file might help.
Also a bit more detailed info on the logic around/in the jobs you run.

[cheerfu1]

Thanks...........

Code: Select all

Dim DSPROJ As String = "TEST_PROJECT"

Dim DSPWD As String = solutionPath + "ds_setting\ds_password.txt"
Dim DSSERVER_NAME As String = File.OpenText(solutionPath + "ds_setting\ds_server_name.txt").ReadLine()
'DSJob 's name
Dim JOBDS As String = groupType + recType + version
'DSJob 's command
Dim dsExec As String = ds_path

'Main filtering job
Dim dsParams As String = " -file " + DSPWD + " " + DSSERVER_NAME + " -run -jobstatus -wait -param INPUTFILE=" + fullPathInputFileName + " -param OUTPUTFILE=" + fullPathOutputFileName + filterConditions + " " + DSPROJ + " " + JOBDS
Dim dscommand As String = dsExec + dsParams
Console.WriteLine(dscommand)

Dim processInfo As New ProcessStartInfo
processInfo.FileName = dsExec
processInfo.Arguments = dsParams
processInfo.WindowStyle = ProcessWindowStyle.Hidden

Dim pr1 As Process = Process.Start(processInfo)
pr1.WaitForExit()

If pr1.ExitCode <> 1 Then
    Dim processRestartInfo As New ProcessStartInfo
    processRestartInfo.FileName = dsExec
    processRestartInfo.Arguments = " -file " + DSPWD + " " + DSSERVER_NAME + " -run -mode RESET " + DSPROJ + " " + JOBDS
    processRestartInfo.WindowStyle = ProcessWindowStyle.Hidden

    Dim pr2 As Process = Process.Start(processRestartInfo)
    pr2.WaitForExit()
    fileInStream.Close()
    Throw New Exception("DS Job failed, exit code = " + pr1.ExitCode.ToString)
Else
    Thread.Sleep(60000)
    File.Delete(fullPathInputFileName)
    fileName = fileInStream.ReadLine()
End If

[chulett]

Please do us all a favor and edit your post - there's a button to allow you to do that. I'm assuming your original code is properly indented - wrap all of it in Code tags so that it displays properly here as well.

Just makes it simpler all around.

Also, are there job parameters or arguments involved here - like, for example 'workid'. What sets that?

[kduke]

After I process input1 using DS Job -wait -jobstatus

There is no -jobstatus in code posted. I also think that DataStage needs time to clean up between jobs. You need to sleep a few seconds between jobs.

[cheerfu1]

After I process input1 using DS Job -wait -jobstatus

There is no -jobstatus in code posted. I also think that DataStage needs time to clean up between jobs. You need to sleep a few seconds between jobs.

Hi Duke,

I have let it sleep for a minute however the problem persists especially when processing bigger files.

jobstatus was intentionally removed. However, I have updated the code which includes the jobstatus with the same problem. Thanks for your replies. Please do advice me.

[chulett]

jobstatus was intentionally removed. However, I have updated the code which includes the jobstatus with the same problem.

A couple of things to keep in mind...

The -wait and -jobstatus commands are mutually exclusive as both tell it to wait before moving to the next step. You should use one of the other, not sure which takes precedence when you use both as they handle return codes differently.

If you are expecting a return code of 1 to indicate success, then you must use the -jobstatus option. When using -wait a zero return status indicates success and a non-zero return code indicates failure.

[cheerfu1]

jobstatus was intentionally removed. However, I have updated the code which includes the jobstatus with the same problem.

A couple of things to keep in mind...

The -wait and -jobstatus commands are mutually exclusive as both tell it to wait before moving to the next step. You should use one of the other, not sure which takes precedence when you use both as they handle return codes differently.

If you are expecting a return code of 1 to indicate success, then you must use the -jobstatus option. When using -wait a zero return status indicates success and a non-zero return code indicates failure.

Thank you for your reply. I still can't solve the problem. However, I do a workaround by busy-waiting the output file, checking if the file is in use by another process (which is DS process). If it is, then I will keep looping. So far it works. But I do want a better solution as I am not sure whether this method will work for all occasions.

[cheerfu1]

It still does not work properly. Any suggestion?

[ray.wurlod]

Tell us exactly in what way it doesn't work properly. The 7.5 readme mentions that dsjob puts the job status code on stderr. Did you take that into account?

[cheerfu1]

it seems that the -wait parameter passed to the dsjob executable does not work properly arbitrarily. I am supposed to process the input file, then delete the input file, after the DS JOB completes. However, it deletes the input file first before the DSJOB starts properly, eventhough I have used -wait.

[ray.wurlod]

Don't know. As a workaround, wrap your job in a job sequence or in job control code, and invoke that from dsjob. The wait does work reliably there. You can manage the deletion from the same job sequence.

[cheerfu1]

Where can I get job sequencer in DS 4.0?

[chulett]

By upgrading.

[ray.wurlod]

Where can I get job sequencer in DS 4.0?

You can't. But you can create job control code, which I also suggested.