Generating documentation

[akrzy]

Hi,

Do you know if I can generate complete project documenation (including jobs, transforms, routins) from DataStage.
I know that we can do that using Reporting Assistant and that MS Access is the default repository to create that report.
I changed DSN on ODBC - SQL Server and I did update the database structure. And... what is the next step to generate documentaion.

Regards,
ANka

[roy]

Hi,
if you went out of the ms access you need to build BI reports depending on the tools at your desposal.
why not use the command line to generate the new html docs foreach of your jobs?

hmm no version noted ?

IHTH,

[akrzy]

I can't use HTML document because I have to print this documentation.

[chulett]

HTML documentation is perfectly printable, do this all the time.

[akrzy]

ok, but if HTML documents can have foa description whole project or just selected catgores?

[chulett]

From the command line I'm not sure, check the docs. I generally do this one at a time via the 'Job Report' routine Ascential posted at ADN. What version of DataStage do you have, btw?

Kim Duke should be along sometime today to let you know about his offerings. He has written some very nice utilities, one of which generates reports like you are looking for for the whole project, from what I recall... perhaps even just selected categories.

[kduke]

JobReport is a routine which you can download from ADN. It was written by an Ascential employee named Tony Curcio. It will generate html documentation on one job. I wanted to document all jobs and all routines. So I wrote several jobs to do this.

SeqGenAllHtml -> sequence
Param: CategoryToDoc -> use "All" for all jobs and all routines.

KgdGenHtmlJobDocs -> job to document a job or all jobs
Param: CategoryToDoc -> load a category name here or a sequence name to document all jobs in a sequence.

Creates JobName.html. Modifies html to add bmp images from DSaveAsBmpDriver.bat. Creates All_Index.html or SeqName_Index.html or CategoryName_Index.html with links to all of the JobName.html.

This thing is super fast. Tony did a great job. It is very nice documentation. So I wrote the samething for routines.

KgdGenHtmlRoutineDocs -> job to document a routine or all routines.
Param: CategoryToDoc -> load a category name here or "All"

This calls these routines.

1. KgdGenHtmlHeader
2. KgdGenHtmlDiv
3. KgdGenHtmlTableHeader
4. KgdGenHtmlTableRows
5. KgdGenHtmlFooter

It generates java script. Divisions are created around each section which can be closed or opened. This looks the same as JobReport.

You can take almost any dynamic array and create a html document out of it with calls to these routines. You get source code to all of these except 2 routines when you download it from my tips page or ADN. It is called KgdGenHtml.zip.

I created new versions of the these for the new EtlStats without Kgd in front of the job names. Job GenHtmlFromSql will take any SQL statement and turn it into a html document. So I do all my row count reports this way. They are automatically emailed to you as attachments.

There are several jobs which call GenHtmlFromSql. GenHtmlFromSqlDriver has a set of "canned" SQL statements that I run to get job history of run times of all sequences and a couple dozen more. I think the nicest one is EtlChkSeqJobsNotRun. This will loop through all jobs called in a sequence job and check to see if they had warnings or aborted. If they did have problems then the log files are sent as attachments to me in a email message. They are these html documents.

The latest thing I added is GetEtlQaSqlStatDriver. This job will run validity checks or all our dimensions or what my boss calls "sanity checks". It will run any SQL against a source or target table and create a measure to compare a source measure against a target measure. At ASCL World they talked about completeness of your dimesions. So mostly I compare row counts. At the end it emails a completeness report which show 75% of all source records or a given dimension ended up in my target. This runs at the end of all our jobs in about 7 minutes. It could also calculate sums or more complicated measures. This is what I thought AuditStage was. My solution is very simple and very easy to understand the whole process.

All of this is free. Just download EtlStats.zip from my tips page or ADN. There are 3 versions out there now on ADN. Make sure you get the latest one. Not sure I will continue to update ADN. I keep posting it in the wrong category and then I cannot correct it. Do a search. Craig is correct, I have posted all this several times.

[sachinkc]

I like to export in XML format and then do whatever I want with its tags. XSL parsing and rendering, Javadoc, use any othe app to create PDF\Word docs from XML source with XSL rendering. Saves a lot of cost for me wit hthis, and also use some java resources at site. - Sachin

Hi,

Do you know if I can generate complete project documenation (including jobs, transforms, routins) from DataStage.
I know that we can do that using Reporting Assistant and that MS Access is the default repository to create that report.
I changed DSN on ODBC - SQL Server and I did update the database structure. And... what is the next step to generate documentaion.

Regards,
ANka

[StefL]

I've downloaded the JobReport routine from ADN and it seems to work fine. I'm just somewhat curious to how it differs from the standard 'generate html report' functionality that already exists in Designer? There's not that much difference from what I can tell, apart from that a bitmap picture of the job design is not automatically generated by JobReport.

What I'd like to know is if there's any functionality that automatically generates reports for say a Server Job called from within a Job Sequence when you generate a report for the sequence, and then includes a hyperlink to the job's report where the job is referenced in the sequence report.
That would truly make the documentation useful and complete, and it's a functionality I'd very much like to have.

[kduke]

The jobs I wrote do use JobReport. Before the new version came out I modifed the output of JobReport and adding the html to display bmp files created by /saveasbmp option to the Designer. Included is a DSaveAsBmp.bat which will loop through and create bmp files for all jobs. These bmp files are huge. You can easily convert them to jpg or gif. It is also easy to edit all html files and change .bmp to .gif.

These jobs will document the whole project in seconds. It will also create All_Index.html which links to all the jobs sorted by name and also sorted by category and name. This index page can be created for categories and sequences. All source code is included to all the jobs and all but 3 of the routines. So change it to do it any way you want. I have received emails where people added a column here are there. I could easily show you how to do an index by hash file name.

[vmcburney]

This is excellent stuff! I ran it on our project. There was some minor adjustments as it was a Unix server and I had to run the dsjob commands manually on unix to get the project and job lists before running the create bitmap command back on the Windows client.

Well done Kim and Tony!

[kduke]

Thanks Vincent