Hi, I was wondering if anyone wrote the parser needed for the Operational Metadata generated XML files to feed splunk.
Also, since IBM bought Platform LSF and it is a recommended Grid Resource Manager for DataStage (should you chose to go grid), has anyone written a splunk parser for the LSF logs? The splunk site has a mention of this, but it's basically another user asking if anyone has done it yet.
Thanks in advance.
Anyone using Splunk to mine info from DataStage?
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 103
- Joined: Tue Jan 06, 2015 4:30 am
How about just using DS with xmlInput Stage to parse them? They are very simple xmls...
Ernie
Ernie
Ernie Ostic
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
blogit!
<a href="https://dsrealtime.wordpress.com/2015/0 ... ere/">Open IGC is Here!</a>
it's not the datastage side that was of any issue. It is the coupling of both products that I needed assistance on.
And no, I have not had the time to get splunk feeding off of the LSF logs.
I do not want to make a datastage log parser since the act of extracting a log is something I want to avoid.
Creating a datastage job to parse the XML files when you have 20K jobs running per day is also something I do not want to do since that means a ton more datastage jobs, and as a result could create an invinite loop if not coded properly.
So basically nobody out there is using splunk to mine any datastage metadata it would seem.
I have splunk set up for the NMON data dashboards. But something more akin to DataStage jobs would have been nice.
DSODB can be a source of the data, but the dashboards it presents are not 100% what I am looking for.
Would be nice to get a GANNT chart view of a daily project run. Sequencers + jobs with visual dependencies.
I might have to try and code something with highcharts and a table dump from DSODB. That stuff is just not my strong suite, I've managed to get by on my good looks alone so far.
And no, I have not had the time to get splunk feeding off of the LSF logs.
I do not want to make a datastage log parser since the act of extracting a log is something I want to avoid.
Creating a datastage job to parse the XML files when you have 20K jobs running per day is also something I do not want to do since that means a ton more datastage jobs, and as a result could create an invinite loop if not coded properly.
So basically nobody out there is using splunk to mine any datastage metadata it would seem.
I have splunk set up for the NMON data dashboards. But something more akin to DataStage jobs would have been nice.
DSODB can be a source of the data, but the dashboards it presents are not 100% what I am looking for.
Would be nice to get a GANNT chart view of a daily project run. Sequencers + jobs with visual dependencies.
I might have to try and code something with highcharts and a table dump from DSODB. That stuff is just not my strong suite, I've managed to get by on my good looks alone so far.