slow DSFindFirstLogEntry (2 to 3 minutes)

[jgreve]

So I want to write a log activity scraper.
*Seems* easy enough...

Step 1: put together some C code that eventually does a loop like this:

Code: Select all

...
DSOpenJob()
result = DSFindFirstLogEntry()
while( result == DSJE_NO_ERROR ) {
   write info from logevent
   result = DSFindNextLogEntry()
}
...
DSCloseJob()

Step 2: compile on Windows,
I'm compiling two versions; one for 7.5 and one for 8.1.
Compile workstation: Windows Server 2003, runing DS 7.5 installed.

I copied the dsutil.h file over from my 8.1 server.
My compiler is MS vistudio .net 2003 (cl version 13.10.3077).

Step 3: test it on my test environment.
It works pretty well (quickly) in my test environment. I'm running my log extractor on a ds 8.1 workstation (windows xp).


Step 4: Run my log-extract program on a windows xp workstation with the 8.1 client software installed, and ask it to connect to a DataStage 8.1 server running on AIX.
This is where I start having trouble (yeah, oddly enough, where theory meets practice - go figure ).
Anyway, everything runs well.
a) The list of available projects goes quickly.
b) The list of jobs in a given project goes pretty quickly.
However: c) DSFindFirstLogEntry is taking a long time. About 3 minutes per job, which is impractically long for me to scan several hundred job logs.

I don't know yet whether the AIX ds8.1 server is using new-style (xmeta) or old-style log files (trying to find out now).

I'd welcome any advice for making this run faster.

So far the only ideas I have had are:
1) Maybe put an arbitrary limit of 60 days on "find first log entry"

2) Run several extractors in parallel, perhaps one per project.
Doesn't really make it "faster", just asks the server to work harder.
And I'm concerned my "choke point" in scanning history will become an even larger choke point.

John G.

[jgreve]

So far the only ideas I have had are:
1) Maybe put an arbitrary limit of 60 days on "find first log entry"

Well, adding a 60 day limit doesn't seem to makes no discernable difference. I've noticed I will get a flurry of output for 5 or 10 jobs, then it will freeze again (sometimes for 3 minutes, currently about 25 minutes).

The current job it is waiting on happens to also be freezing the log-summary in the DS Director gui-tool (that wasn't the case with some earlier samples), so that is encouraging. Perhaps this machine just has really backed up logs that need to be purged.

This is a multiple instance job, courtesy of RTI-style instance-numbers: web services director in 8.1, it happens to be waiting on a particular instance.

John G.