Write SEQ files

lucianopolinari · Post by **lucianopolinari** » Wed Apr 23, 2003 3:41 am

My problem is:

I should intercept messages or information between stages (eg. number of rows processed, transformation applied, ect) and write a sequential files to insert in my audit db.
How could i do ?

Thank you !!!!

ray.wurlod · Post by **ray.wurlod** » Thu Apr 24, 2003 5:23 pm

Just one extra point to add to what Mat has contributed.
Why do you need to do "transformations applied"? These will already be described in a design document, and locked down in your job design. And will be the same for every row processed.
In what form were you planning to store this? If you want to capture the input and output value of every column in every row, you can accomplish this using stage tracing (you enable this from Director), which automatically generates a text file. But it will, as Mat says, affect your throughput.
Personally, I think your problem is that you have not thought through the ramifications of performing what is clearly an unnecessary task; all the information you require (which is collectively known as "process metadata") is automatically captured by DataStage and recorded in the job log and, if you are operating in MetaStage proxy mode, into your MetaStage hub as well.

Ray Wurlod
Education and Consulting Services
ABN 57 092 448 518

lucianopolinari · Post by **lucianopolinari** » Mon Apr 28, 2003 9:43 am

I don't know MetaStage very well, could you tell me more about it.
I need these information:
- job name
- job stage name
- start date
- end date
- during time
- rowcount
- row rejected
ect

How can take them from Metastage while job is running ?

Thank you !!!

lucianopolinari · Post by **lucianopolinari** » Mon Apr 28, 2003 9:45 am

Thank you MAT, but I have never seen DS function DSMake Report, could you tell me more about it ?

Thanks

MAT · Post by **MAT** » Mon Apr 28, 2003 9:55 am

Hi,

Since I do not use MetaStage, I can't help you with your question on this but for DSMakeJobReport(...), here's how I use it:

DSMakeJobReport(JobHandle, ReportLevel, LineSeparator)

JobHandle is what you get from DSAttachJob
ReportLevel is a number 0=basic and 1=detailed
Lineseparator is the string used to separate values in the report. One of the following: "CRLF", "LF", "CR".

Be aware that this may not be available in earlier version of DS. I use 6.x

Hope this helps

MAT

lucianopolinari · Post by **lucianopolinari** » Tue Apr 29, 2003 4:16 am

Thank you Mat, DSMakeJobReport works well !!!

Have a nice day

kjanes · Post by **kjanes** » Tue Apr 29, 2003 9:10 am

MetaStage has the ability to collect "process" metadata from DataStage jobs while they are executing. The process metadata is then fed into the MetaStage repository. Process metadata gives you details about the job execution like Start Time, End Time, Rows proessed per stage, data lineage, etc...

The process of collecting process metadata is facilitated by a daemon running on the DataStage server that logs the process metadata and then a listener accepts that metadata and loads it into the MetaStage repository.

This process is external to your Job Design so the ability to leverage process metadata (via MetaStage) for your job to use is not integrated into the products. I suppose where there is a will there is a way. But to the best of my knowledge, there is not an out-of-the-box method to leverage MetaStage-based process metadata from within DataStage.

The DataStage methods mentioned in prior responses are your best bet.

Kevin Janes