Hi,
Does anyone have a clue how I can determine the number of records in a Data Link in a server job, and use that number for my processing in a transformer stage?
Regards,
Manav
Keeping track of record counts
Moderators: chulett, rschirm, roy
-
manavkumarm
- Participant
- Posts: 11
- Joined: Fri Sep 05, 2003 3:45 am
- Location: India
Manav
After a job is done then DataStage supplies routines which will tell you many rows went down each link. You can search this forum because how to call those routines has been posted.
Kim.
Kim Duke
DsWebMon - Monitor over the web
www.Duke-Consulting.com
After a job is done then DataStage supplies routines which will tell you many rows went down each link. You can search this forum because how to call those routines has been posted.
Kim.
Kim Duke
DsWebMon - Monitor over the web
www.Duke-Consulting.com
-
manavkumarm
- Participant
- Posts: 11
- Joined: Fri Sep 05, 2003 3:45 am
- Location: India
Thank you Kim and Wolfgang for your inputs.
The particular requirement in our project is to keep a count of the number of rows in a sequential input file that failed any of the validations performed in a transformer stage. Then, I need to reject the input file if the percentage of error records in the file exceeds a percentage supplied as a parameter to the job.
The above is why I need to know how many rows were in the input link at any point so that I can calculate the percentage of error records. Keeping track of it in a stage variable doesn't help much because I cannot pass the stage variable to the next transformer stage where my logic to reject the file resides. Likewise, accessing the row count at the very end of the job doesn't help either since I want to abort the job midway instead of wasting any more CPU cycles on the job stages that follow my error percentage check.
I can accomplish this in a round-about way by using an Aggregator stage to count the no. of rows or passing a stage variable containing the count (@inrownum as suggested by Wolfgang) into the rows of the link being passed to the next stage. But I would love it if someone could tell me an easier way.
Datastage unfortunately does not seem to support any such concept as a "Job Variable" (similar to stage variables) and the Parameters passed to a job cannot be modified by the job anywhere for such global processing. Maybe someone should suggest to them that they include them in the next version
The particular requirement in our project is to keep a count of the number of rows in a sequential input file that failed any of the validations performed in a transformer stage. Then, I need to reject the input file if the percentage of error records in the file exceeds a percentage supplied as a parameter to the job.
The above is why I need to know how many rows were in the input link at any point so that I can calculate the percentage of error records. Keeping track of it in a stage variable doesn't help much because I cannot pass the stage variable to the next transformer stage where my logic to reject the file resides. Likewise, accessing the row count at the very end of the job doesn't help either since I want to abort the job midway instead of wasting any more CPU cycles on the job stages that follow my error percentage check.
I can accomplish this in a round-about way by using an Aggregator stage to count the no. of rows or passing a stage variable containing the count (@inrownum as suggested by Wolfgang) into the rows of the link being passed to the next stage. But I would love it if someone could tell me an easier way.
Datastage unfortunately does not seem to support any such concept as a "Job Variable" (similar to stage variables) and the Parameters passed to a job cannot be modified by the job anywhere for such global processing. Maybe someone should suggest to them that they include them in the next version
"Job Variable"? You can effectively pass "variables" between transformers by taking advantage of COMMON storage. Check the online help, or look at some of the SDK routines. Don't recall the exact names off the top of my head, but the ones that do running totals or check to see if a value has changed use it.
-craig
-craig