Wave Generator Stage

bala_135 · Post by **bala_135** » Wed Jul 18, 2012 6:30 am

Hi All,

Kindly help me in understanding the concept of the wave generator stage.
Requirement:
========
Need to abort the job after inserting 'n' number of records into the reject table.

Design:
=====
1.Filtered the records based on the reject criteria and threshold and inserted into the reject table using the combination of constraints and stage variables in the transformer.
2.Flagged the success or failure criteria of the rejects as 1 or 0 in another stage variable and passed it on to a newly added column[ColA] in the output of the transformer.This is to indicate the row status[if it is rejected or not.]
3.Created another output link in the transformer and hard coded "1" in the derivation of a newly created column[ColB].
4.Used the step2 output link as primary and step3 output link as reference and compared the values using ColA and ColB and assigned the lookup failure as "Fail".

Result:
====
The job got aborted but whichever rows passed the lookup condition got inserted into the target table.

I placed the Wave generator stage between the lookup stage and target table.
I used the "Columname[ColA] /Columnvalue[passed the value as 0] " property then the records were not getting inserted.

Testing
=====
I tested 3 records with the following testcases
1.All records got inserted when there are no rejects and the job was success.
2.No records got inserted even on 1 reject and the job got aborted.

Just before deep diving into further testing, wish to know the concept of wave generator stage.

Apologies for detailed explanation and any inputs would be most appreciated.

Thanks in advance.

Regards,
Bala.

eostic · Post by **eostic** » Wed Jul 18, 2012 7:21 am

The wave generator was designed for jobs that are "always on" and it forces a signal to be passed downstream that stages "think" means "end of job". The job keeps running, but most normal stage processing will "finish" as though the job had completed.

It is mostly critical for aggregators when there is no sorting (aggregate on the entire set of rows flowing thru) and xml targeting (write the xml for the entire set of rows flowing thru).

The source with an always on job is generally something like MQ series, an ISDinput Stage, javaPack using JMS, a named pipe, or other custom resource that never stops feeding data.

Can it be used in other situations? I suppose --- but I don't know if it is particularly applicable to your issue.

If you want to look at another explanation of end-of-wave, see this blog entry...

http://dsrealtime.wordpress.com/2007/12 ... ormations/

Ernie

chulett · Post by **chulett** » Wed Jul 18, 2012 7:26 am

I'm going to pull this one thing from Ernie's comprehensive post so you can see it, a link:

http://dsrealtime.wordpress.com/2007/12 ... ormations/

bala_135 · Post by **bala_135** » Thu Jul 19, 2012 6:02 am

Thanks Ernie and Craig.Valuable information.

My requirement is similar[performing inserts,updates and rejects from the same transformer source] but not exactly real time in nature.

Also, I need to wait for the unit of work[rejects] to get completed for the job abort to occur and at the same time stopping my downstream operations[inserts and updates].

Looks good to me but let me perform some testing with more number of records and evaluate before selling the concept to the stake holders.

Thanks once again.

Regards,
Bala.

sls_hari · Post by **sls_hari** » Mon May 20, 2013 9:27 pm

Hi Bala,
Did the wave generator work for your requirement ?