Report on jobs currently running in DS server

[vigneshra]

Hi everyone

Is there any way to get a report in file format which has a list of all the jobs presently running in a particular DataStage server along with their project name and other details?

[ray.wurlod]

Use Director client. It has a Print option, which has a Print to File option. Director also has a Filter - filter on job status so that you get just the running jobs. in Status view, select "print detail".

[vigneshra]

Ray,

We need to schedule through Control-M scheduler and get the report once in every 15 minutes to find which job is occupying lot of ETL and database resources at that instance. Can it be done using Director Client?

[kcbland]

A lot of functionality is available within the dsjob command line program. You can query a project for a list of jobs, then cycle thru the list of jobs and request the current status of each.

[ray.wurlod]

There was nothing about Control M in the original post. I answered the original question. I'd probably answer along the same lines of Ken if you'd introduced an external scheduler as part of the requirement. Or a ps -ef command.

[anton]

any updates on this?

i got exactly the same scenario: i have several thousand jobs running throughout the day, and i need to take snapshots every few minutes automatically to see what is running on the server at the time.

then i go back to my performance stats and correlate running jobs with the performance spikes.

then i go and research individual jobs.

this is what was suggested above:

- get the list of projects
- for each project get the list of jobs
- for each job, retrieve current status

it is certainly doable, but seems to be an awfully awkward way to approach this, not to mention the time it would take to run this (so much for the snapshot).

are there any other ways to obtain this information or approach this problem (which i am sure is pretty common)?

thank you.

[chulett]

Depending on exactly what you need, as noted ps -ef with an appropriate grep filter can get you the running jobs. For example, all running Server jobs can be found by grepping for phantom to see all running processes, or DSD.RUN to get a unique list.

Not sure on the PX side of the house...

[ray.wurlod]

are there any other ways to obtain this information or approach this problem (which i am sure is pretty common)?

Open a Director client in Status view (usually the default).
Set the refresh interval to your snapshot interval (maybe 900 seconds).
In the View menu choose "disable categories".
Click in the Status column heading to sort by status. Click again to sort in reverse order. This should show Running (if any) at the top of the report.

OK, that's the manual scenario sorted. Now, how to automate?

Do the same as above, but with server-side tracing enabled. One or two cycles should do it. Then stop tracing and inspect the trace file. Create a routine that makes the same calls, and writes the information to your output file.

[anton]

are there any other ways to obtain this information or approach this problem (which i am sure is pretty common)?

Open a Director client in Status view (usually the default).
S ...

thank you for the response.

i am not a premium member, so i cannot see your reply.

but it looks like you are sending me in the general direction of the director.

keep in mind:

- it needs to run on unix in regular intervals 24/7
- it needs to be automated
- it needs to produce parseable output

so director or any other gui is not an option.

is there something universe db can do for me?

thank you.

[kduke]

I don't have a UNIX machine to test this but it should be close. If you finish it then post it.

Code: Select all

ps -ef | grep DSD.RUN | grep -v grep |
while read LINE
do
   set - $LINE
   JOBNAME=$4
   dsjob -report $JOBNAME >>jobstats.txt
done

I think this is what they wanted you to try to figure out. You can lookup the syntax of dsjob and figure out how to parse the job name out. This will get you close.

[ray.wurlod]

For less than $1/week you can be a premium member. My earlier answer shows a manual approach using Director, then a suggestion for how to automate that process. Even that ought to be work this week's $1.

[anton]

For less than $1/week you can be a premium member. My earlier answer shows a manual approach using Director, then a suggestion for how to automate that process. Even that ought to be work this week's $1.

nice try :) this is not what i was asking though. i'll pass. thank you.

[chulett]

Actually, it is what you were asking for. Specifically, the automation part he mentions.

[anton]

Actually, it is what you were asking for. Specifically, the automation part he mentions. ...

sorry, guys, i am not convinced.

and i am not willing to buy into something that has no proven its value to me.

thanks for trying.

[kcbland]

Well let's see if I can't take a fair shot at this.

1. Jobs can't be correlated to a project from the unix command line. The same job name in two projects can't be distinguished using a "ps -ef|grep phantom |grep DSD.RUN" statement. But, you can get a list of jobs executing.

2. Using dsjob, you can get a list of jobs in a project, and then query each jobs status. A simple ksh with looping logic can give you what you need. Sure, it may be slow, but that is if you have just one script running. A script per project will give you some parallelism and avoid any proprietary or hacks into internals. Cross reference against #1 and you reduce the number of jobs being investigated greatly.

3. If all the jobs are being started via dsjob command line, consider replacing dsjob with a shell script called the same that wrappers dsjob but has additional audits statements to log to a database anything being started.

4. If jobs are being started via Batch jobs, consider a one-time hit to update all batch jobs with a subroutine call right before a job is launched using DSRunJob. That subroutine can do any logging you need, and only adds 1 line prior to each job run.

5. If using Sequencers, you have some ability to parse Sequencer jobs only and peek into their logs to see what's running. That may be helpful, but a little tedious as there's lots of things going on in those.

Why don't you come back and stay awhile, you might find a friendly conversation and who knows, maybe a tip or two?