how to find out howmany records were processed and rejected

ray.wurlod · Post by **ray.wurlod** » Mon Jun 05, 2006 4:12 pm

sudhakar_viswa wrote:My reqirement is to do everything in one job.

Why?

Sometimes (often) a more modular approach is far easier to maintain.

Take some advice from an experienced developer - this (everything in one job) should never be a requirement. If "they" try to impose it, challenge them to provide a good reason. If it's your personal requirement, take a good hard look at why you want to do it this way.

kduke · Post by **kduke** » Mon Jun 05, 2006 5:17 pm

I do get row counts at the end of a sequence. Much cleaner and easier. Unless you need row counts to make a decision like send an email if more than 1000 rejects then wait till the end of the sequence.

sudhakar_viswa · Post by **sudhakar_viswa** » Mon Jun 05, 2006 11:17 pm

Hi DSGuru,

Thanks a lot.And one more thing,if we are using hashed file how can get the count.

Thanks,
sudhakar

sb_akarmarkar · Post by **sb_akarmarkar** » Tue Jun 06, 2006 1:00 am

Use routine GetLinkCount(JobName, StageName, LinkName) that will give you count ...
You can create a multiple instance job where you can pass parameter to that job and get count for each job...

Thanks,
Anupam

sudhakar_viswa · Post by **sudhakar_viswa** » Tue Jun 06, 2006 1:52 am

Hi,

can i get the just count in the same job

Thanks,
sudhakar

loveojha2 · Post by **loveojha2** » Tue Jun 06, 2006 1:56 am

The system variables
@INROWNUM
and
@OUTROWNUM within the transformer can be used for getting the counts on links.

sudhakar_viswa · Post by **sudhakar_viswa** » Tue Jun 06, 2006 2:05 am

Hi loveojha2,

It will give the row by row.I need the total count and i have to store the count in database table.

Thanks,
sudhakar

loveojha2 · Post by **loveojha2** » Tue Jun 06, 2006 2:08 am

You can pass it to an Aggregator stage and take the max of the values out from the aggregator and send it to the database table.

sb_akarmarkar · Post by **sb_akarmarkar** » Tue Jun 06, 2006 3:27 am

I think you need following information
1) How many rows selected from source.
2) How many rows insert/updated to target.
3) How many rows reject.

I suggest better to go with GetLinkCount()
@INROWNUM and @OUTROWNUM will not get proper information

Thanks,
Anupam
sb_akarmarkar@smart-bridge.co.in

loveojha2 · Post by **loveojha2** » Tue Jun 06, 2006 3:37 am

How about using stage variables, which should maintain the counts for the rows processed for every link and keep on writing to an aggregator from within the transformer and taking the max values from the aggregator to the database table.

sb_akarmarkar · Post by **sb_akarmarkar** » Tue Jun 06, 2006 4:15 am

Suppose
Source Link name is SrcLink
Target Link name is TgtLink
Transformer name between is TransformerX
Link one more transformer name TransformerY to TransformerX
Put constraints @OUTROWNUM = 1

Give following in TransformerY for link count
DSGetLinkInfo(DSJ.ME, "TransformerX", "SrcLink" ,DSJ.LINKROWCOUNT)
DSGetLinkInfo(DSJ.ME, "TransformerX", "TgtLink" ,DSJ.LINKROWCOUNT)

Thanks,
Anupam
sb_akarmarkar@smart-bridge.co.in
[/code]

loveojha2 · Post by **loveojha2** » Tue Jun 06, 2006 4:28 am

sb_akarmarkar wrote:Suppose
Source Link name is SrcLink
Target Link name is TgtLink
Transformer name between is TransformerX
Link one more transformer name TransformerY to TransformerX
Put constraints @OUTROWNUM = 1

Give following in TransformerY for link count
DSGetLinkInfo(DSJ.ME, "TransformerX", "SrcLink" ,DSJ.LINKROWCOUNT)
DSGetLinkInfo(DSJ.ME, "TransformerX", "TgtLink" ,DSJ.LINKROWCOUNT)

Thanks,
Anupam
sb_akarmarkar@smart-bridge.co.in
[/code]

But with such a solution, you should be pretty sure of whether the target links are fully processed or not?

sb_akarmarkar · Post by **sb_akarmarkar** » Tue Jun 06, 2006 5:03 am

loveojha2 wrote: But with such a solution, you should be pretty sure of whether the target links are fully processed or not?

It will wait for link .....

Thanks,
Anupam
sb_akarmarkar@smart-bridge.co.in

DSguru2B · Post by **DSguru2B** » Tue Jun 06, 2006 6:48 am

Ray pointed out a very useful fact. Why do all this in the same job. You might want to think about this.
As far as doing a count on hashed file goes, you can access the hashed file, if they were created in your project's account, by using a universe stage. You can run sql queries on it as well. Just do a simple
select count(keycolumn) from "hashedFileName"

kollurianu · Post by **kollurianu** » Tue Jun 06, 2006 2:31 pm

hi ,

how to do it dsguru , can u be more clear , in the reject file , yes it has
stage tab in the that tab we check the option stage uses filter command,

then right to the path of the file , filter command box is enabled ,

there we have to type wc -l and how do we write the result of wc -l in the reject file with an extra field to get count, i mean what would be the derivation in for the extra field which has get the value of wc -l

could you please explain it more clearly.

Thank you,,