Operational ETL Dashboards incl ETLStats

stotman · Post by **stotman** » Tue Jan 20, 2009 5:09 pm

I'd be very grateful if anyone who's built a dashboard to display operational information from DataStage / QualityStage could spare some time to drop me an email.

I'd after the following details:

1) What information /reports did you include
2) What tool / application did you use to display this information
3) Was there any information you would have liked to include but couldn't get hold of
4) do you have any documentations covering the dashboard / screen shots you could share.

Cheers

Steve

steven.totman@uk.ibm.com

kduke · Post by **kduke** » Tue Jan 20, 2009 11:50 pm

We started to but never got to the dashboard. We created the equivalent in a report at a couple of sites. I am not a Cognos or reporting person so I could never finish the task. The canned reports I called "Completeness" was more detailed and good enough for CFOs. There are several posts on here about these audit type reports and processes. Most of the jobs are included in EtlStats that I give away. The concepts are built around star schemas and fact tables but it would work for any data integration project as well. We are currently building an ODS. I built the reports for it as well. The concept works as well for full loads or incrementals. You need to be more creative to get numbers on incremental loads. Your table structures need to be more robust. Need more audit concepts in your targets like last change date. Most people have these columns nowadays.

Sloppy designs tend not to audited very well anyway. Better yet their numbers are always off a couple percent. 2 percent high one day and 2 percent low the next. So the developers say it is "close enough" or it evens out over time. Always blame the "granularity" or some overused DW term. Love to follow those developers. They always dislike me especially when I balance to the penny after I fix their designs.

I think this is the most overlooked part of most EDW solutions. It is kind of sore spot with me. Sorry if I got off topic. I assume this is what you were talking about when you asked about dashboards. Is the data clean? A lot of dashboards are just red, green or yellow based on how it matches the source. Is this what you were looking for Steve?

kduke · Post by **kduke** » Tue Jan 20, 2009 11:59 pm

If you are talking about operational data only then we have some killer reports which compare row counts to CPU utilization and disk utilization. As well as number of jobs running and paging information. Very complete information. You may need to go through IBM management and get our management's approval to see a lot of it. We had very serious performance issues. IBM was on site. We also had HP consultants looking at the information.

If this is what you want then let me know. I may need to ask permission to release some it. I was think about presenting some of it at IOD. You might be able to get my management to let me work on something more generic which we could share at IOD.