DSJob -logdetails Hanging

hamzaqk · Post by **hamzaqk** » Tue Dec 08, 2009 12:03 am

Hi i am trying to get the job log of the job from unix command prompt as the director bit is way to slow even after changing the refresh interval for it. When i run it through command prompt it just goes to sleep and does not return anything. i have tried other options like -jobinfo and it works fine. Just anything related to the job log does not seem to work. I am using the following command with hardcoded values for parameters.

Code: Select all

 ./dsjob -domain domain_name:9080 -server server_name -user dsadm -password dsadm -logdetail project_name job_name

If i abort the command ctl+c its show the following message

DSICTerminate, pthread_mutex_destroy failed: Device or resource busy

Any help on this please?

Many thanks!

hamzaqk · Post by **hamzaqk** » Tue Dec 08, 2009 12:11 am

Well it took ages to return the result so i guess this work around is as slow as the dirctor itself !! . Any faster way to retrieve this log?

Thanks

ray.wurlod · Post by **ray.wurlod** » Tue Dec 08, 2009 3:12 am

Where are your logs, local to the project or in the unified metadata repository? The former is considerably faster.

hamzaqk · Post by **hamzaqk** » Tue Dec 08, 2009 4:48 am

Well they are in the metadata repository. I did some research on it and found out that we can swtich to the old method. It is mentioned on the IBM site. Is this a recommended though ? or is there a way of making the UMR accesible in a faster manner?

I think if we have to switch back to the old method there was no point of coming with the new way of storing the metadata.
http://www-01.ibm.com/support/docview ... -8&lang=en

Thanks

chulett · Post by **chulett** » Tue Dec 08, 2009 6:49 am

You might want to ping your official support provider and see if you have an XMETA issue - bad or missing index, perhaps? Shouldn't be *that* slow.

ray.wurlod · Post by **ray.wurlod** » Tue Dec 08, 2009 1:31 pm

I don't think they've got it right yet - and the log viewer in Web Console still leaves a lot to be desired.

hamzaqk · Post by **hamzaqk** » Wed Dec 09, 2009 2:21 am

@ Ray yes i guess so. i hope they do something about it as its pretty frustrating.

@ Craig, Sounds like a good idea. I just opened the xmeta database and it has loads of tables in it. Which one should i be looking at for Datastage Logs and job related information? We have DS_JOBS etc in the Universe but i doubt they are keeping the same names in the new one.

Many thanks

chulett · Post by **chulett** » Wed Dec 09, 2009 7:46 am

The XMETA database is well and truly obfuscated, so you'd need to involve your official support provider for help with that.

Ultramundane · Post by **Ultramundane** » Fri Feb 05, 2010 9:07 am

chulett wrote:The XMETA database is well and truly obfuscated, so you'd need to involve your official support provider for help with that. ...

I ran a whole bunch of tracing on the DB2 UDB instance while pulling out log information.

Here is what I found:

#1: DB2 UDB is extremely inefficient at cursor handling. Specifically the cursor close event.

#2: Reading out the log information:

DataStage is opening a cursor and running a select statement for every event which is part of the error log. DataStage should not be running so many select statements and it certainly should not open so many cursors.

IMHO, DataStage needs to run one select statement and open one cursor (if even necessary) with the results returned ordered as necessary.

nagarjuna · Post by **nagarjuna** » Fri Feb 05, 2010 11:37 am

The job for which you are trying to see the log is a generic job with multiple instance ??

Ultramundane · Post by **Ultramundane** » Fri Feb 05, 2010 1:35 pm

nagarjuna wrote:The job for which you are trying to see the log is a generic job with multiple instance ??

The indexing is not very good on the XMETA tables. There are not any composite indexes.

This helps (about 20 times faster, but still too slow):
CREATE INDEX XMETA.IDX_1466CB5F_RYAN
ON XMETA.LOGGING_XMETAGEN_LOGGINGEVENT1466CB5F(cis14_xmeta, cis12_xmeta,cis13_xmeta,cis7_xmeta,categoryName_xmeta,deleted_xmeta)
ALLOW REVERSE SCANS
;

GRANT CONTROL ON INDEX XMETA.IDX_1466CB5F_RYAN TO USER XMETA
;

REORGCHK UPDATE STATISTICS on SCHEMA XMETA
;

Ultramundane · Post by **Ultramundane** » Fri Feb 05, 2010 2:04 pm

Ultramundane wrote: The indexing is not very good on the XMETA tables. There are not any composite indexes.

The data model is not very good either. Looks like it was just thrown together with a whole bunch of columns and whole bunch of indexes and should they be used, they'll be used.

Columns have bad names (so what though). Columns have terrible choices of datatypes (This is a big deal). Numeric datatypes should not be of a VARGRAPHIC type as it makes querying the repository difficult and for very expensive queries.

So, the data model first needs to be fixed and then the developers and DBA's need to work together to fix the code and enhance the schema with a good choice of indexes.

From what I can tell, the best thing you can do is to turn this feature off as described by the article you had posted.