File Operations in Parallel routine
Moderators: chulett, rschirm, roy
File Operations in Parallel routine
Hi,
I need to read a file, populate an array and perform a lookup using that array as part of a routine which will be called per record in a transformer. I haven't used routines before. I would like to know if it is advisable to do file operations in a routine. Also, will the entire routine be executed for every row, specifically if the file operation will be performed for every row or only once to populate array.
Also, will it be better if i do the file read and populate array in before routine and then use the array in the routine for each row. Can that array created in before routine be accessed in the per record routine and if yes, how?
I need to read a file, populate an array and perform a lookup using that array as part of a routine which will be called per record in a transformer. I haven't used routines before. I would like to know if it is advisable to do file operations in a routine. Also, will the entire routine be executed for every row, specifically if the file operation will be performed for every row or only once to populate array.
Also, will it be better if i do the file read and populate array in before routine and then use the array in the routine for each row. Can that array created in before routine be accessed in the per record routine and if yes, how?
Hi and welcome aboard ,
If your talking Parallel Job why not build a lookup file and perform a lookup on it (instead of array in a routine)?
IHTH,
If your talking Parallel Job why not build a lookup file and perform a lookup on it (instead of array in a routine)?
IHTH,
Roy R.
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
And having a reject link going to a modify stage to handle_null() and another lookup?
This will result in 2 links that you can eventually load in the logic suiting you.
The specific job design has several variations to implement this idea, simply choose the one that suites you.
IHTH,
This will result in 2 links that you can eventually load in the logic suiting you.
The specific job design has several variations to implement this idea, simply choose the one that suites you.
IHTH,
Last edited by roy on Mon Apr 03, 2006 4:55 am, edited 1 time in total.
Roy R.
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
Time is money but when you don't have money time is all you can afford.
Search before posting:)
Join the DataStagers team effort at:
http://www.worldcommunitygrid.org
It may not be directly possible using single lookup stage. But by knowing you exact requirement, like what sort of data is to be looked up, what are the ranges of the discretion of data... the lookup table can be loaded with some manipulation so that a single column can be maintained to lookup. Else as suggested, reject the records and do another lookup and use modify stage to handle_null().
Impossible doesn't mean 'it is not possible' actually means... 'NOBODY HAS DONE IT SO FAR'
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Its done! Implemented by splitting th e cross ref file into 2 : all the rows wid default value for second lookup column put in second file and then pull values from the 2 datasets. One will correspond to actual value and second to default. Chk in following transformer which is the required combination.
Thanks all for ur help
Thanks all for ur help