Issue generating multiple surrogate key state files
Posted: Thu Dec 20, 2012 1:29 am
Hi,
We are currently generating 150 surrogate key state files to be used in our jobs.
Our current design is
Job1: get the filename and surrogate key value from the database and put all the data into a sequential file.
Job2:(sequence) -> Will read each record from that file and pass the filename and key value as a parameter to job3
Job3: Will create state files for values passed to it from the sequence mentioned above.
This design worked fine when we have 10 files but now we are creating 150 state files and this is taking close to 2 hours to finish. Thereby impacting our performance.
Would appreciate it if you can help me with a better design approach.
Thanks,
Abhinav
We are currently generating 150 surrogate key state files to be used in our jobs.
Our current design is
Job1: get the filename and surrogate key value from the database and put all the data into a sequential file.
Job2:(sequence) -> Will read each record from that file and pass the filename and key value as a parameter to job3
Job3: Will create state files for values passed to it from the sequence mentioned above.
This design worked fine when we have 10 files but now we are creating 150 state files and this is taking close to 2 hours to finish. Thereby impacting our performance.
Would appreciate it if you can help me with a better design approach.
Thanks,
Abhinav