slow DSFindFirstLogEntry (2 to 3 minutes)
Posted: Tue Jun 29, 2010 11:19 am
So I want to write a log activity scraper.
*Seems* easy enough...
Step 1: put together some C code that eventually does a loop like this:
Step 2: compile on Windows,
I'm compiling two versions; one for 7.5 and one for 8.1.
Compile workstation: Windows Server 2003, runing DS 7.5 installed.
I copied the dsutil.h file over from my 8.1 server.
My compiler is MS vistudio .net 2003 (cl version 13.10.3077).
Step 3: test it on my test environment.
It works pretty well (quickly) in my test environment. I'm running my log extractor on a ds 8.1 workstation (windows xp).
Step 4: Run my log-extract program on a windows xp workstation with the 8.1 client software installed, and ask it to connect to a DataStage 8.1 server running on AIX.
This is where I start having trouble (yeah, oddly enough, where theory meets practice - go figure ).
Anyway, everything runs well.
a) The list of available projects goes quickly.
b) The list of jobs in a given project goes pretty quickly.
However: c) DSFindFirstLogEntry is taking a long time. About 3 minutes per job, which is impractically long for me to scan several hundred job logs.
I don't know yet whether the AIX ds8.1 server is using new-style (xmeta) or old-style log files (trying to find out now).
I'd welcome any advice for making this run faster.
So far the only ideas I have had are:
1) Maybe put an arbitrary limit of 60 days on "find first log entry"
2) Run several extractors in parallel, perhaps one per project.
Doesn't really make it "faster", just asks the server to work harder.
And I'm concerned my "choke point" in scanning history will become an even larger choke point.
John G.
*Seems* easy enough...
Step 1: put together some C code that eventually does a loop like this:
Code: Select all
...
DSOpenJob()
result = DSFindFirstLogEntry()
while( result == DSJE_NO_ERROR ) {
write info from logevent
result = DSFindNextLogEntry()
}
...
DSCloseJob()
I'm compiling two versions; one for 7.5 and one for 8.1.
Compile workstation: Windows Server 2003, runing DS 7.5 installed.
I copied the dsutil.h file over from my 8.1 server.
My compiler is MS vistudio .net 2003 (cl version 13.10.3077).
Step 3: test it on my test environment.
It works pretty well (quickly) in my test environment. I'm running my log extractor on a ds 8.1 workstation (windows xp).
Step 4: Run my log-extract program on a windows xp workstation with the 8.1 client software installed, and ask it to connect to a DataStage 8.1 server running on AIX.
This is where I start having trouble (yeah, oddly enough, where theory meets practice - go figure ).
Anyway, everything runs well.
a) The list of available projects goes quickly.
b) The list of jobs in a given project goes pretty quickly.
However: c) DSFindFirstLogEntry is taking a long time. About 3 minutes per job, which is impractically long for me to scan several hundred job logs.
I don't know yet whether the AIX ds8.1 server is using new-style (xmeta) or old-style log files (trying to find out now).
I'd welcome any advice for making this run faster.
So far the only ideas I have had are:
1) Maybe put an arbitrary limit of 60 days on "find first log entry"
2) Run several extractors in parallel, perhaps one per project.
Doesn't really make it "faster", just asks the server to work harder.
And I'm concerned my "choke point" in scanning history will become an even larger choke point.
John G.