Hi,
We are currently generating 150 surrogate key state files to be used in our jobs.
Our current design is
Job1: get the filename and surrogate key value from the database and put all the data into a sequential file.
Job2:(sequence) -> Will read each record from that file and pass the filename and key value as a parameter to job3
Job3: Will create state files for values passed to it from the sequence mentioned above.
This design worked fine when we have 10 files but now we are creating 150 state files and this is taking close to 2 hours to finish. Thereby impacting our performance.
Would appreciate it if you can help me with a better design approach.
Thanks,
Abhinav
Issue generating multiple surrogate key state files
Moderators: chulett, rschirm, roy
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
How many times are you proposing to run this? If only once then two hours ought not to be a problem.
You have provided no information about how you are creating the state files. For example, are their initial values obtained from a database query? That will take some time - to establish the connection, to run the query, to get the results and to free the connection. You're doing one of these every 25 seconds or so - including, I'd imagine, the startup time of the parallel jobs. Does that really sound so unreasonable?
You have provided no information about how you are creating the state files. For example, are their initial values obtained from a database query? That will take some time - to establish the connection, to run the query, to get the results and to free the connection. You're doing one of these every 25 seconds or so - including, I'd imagine, the startup time of the parallel jobs. Does that really sound so unreasonable?
Last edited by ray.wurlod on Thu Dec 20, 2012 4:13 pm, edited 1 time in total.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.