Routine to View Data flow between two stages

jweir · Post by **jweir** » Thu Apr 14, 2011 8:58 am

Hello all,

Is there a transformer routine out there where I can view the data flow between two stages?

jwiles · Post by **jwiles** » Thu Apr 14, 2011 9:35 am

Define more clearly what you mean by viewing the data flow.

The Peek stage can be used for displaying data in the job log or to a file.

You can write logic within a transformer that can output formatted data to a link which is connected to a Peek stage.

Regards,

jweir · Post by **jweir** » Thu Apr 14, 2011 12:05 pm

Sorry for the general statement.

Here is what I would like to do: Have the row counts of the out put of every stage inside a parallel job. However, I can't alter the parallel job itself to acheive this. I would like to add a routine or something at the sequence level.

Any thoughts/ideas?

jwiles · Post by **jwiles** » Thu Apr 14, 2011 1:10 pm

You can get that information through BASIC routines inside the sequencer. Also, Kim Duke's ETLStats package can gather that as well.

Regards,

ajay.vaidyanathan · Post by **ajay.vaidyanathan** » Thu Apr 14, 2011 1:43 pm

Hi,
In a basic routine you can achieve this logic. You can get the number of records processed per Link in your job in the routine. Call this routine in your job/sequence and then pass the same to your parallel job.

Regards,
Ajay

ray.wurlod · Post by **ray.wurlod** » Thu Apr 14, 2011 4:42 pm

Given that you already are in a Transformer stage, why not just use an additional output link? This could even feature a constraint driven by a job parameter that indicates whether or not you want to capture data and, if so, how many rows.