Hi,
I am working on a job design which looks like below.
Job 1 - loads dataset A, Job 2- Reads dataset A and does a look up against input data 1, Job 3 - Reads dataset A and does a look up against input data 2.
I am planning to run Job1 first and then run Job 2 and Job3 parallely.
My question is Can Job 1 and Job2 read the DataSet created in Job 1 parallely at the same time without any problem?
If we can do parallel reads from DataSets, Is there a limit on how many jobs can read parallely from a single DataSet?
Thanks,
Nagin.
Multiple Jobs Reading a single dataset concurrently
Moderators: chulett, rschirm, roy
-
- Premium Member
- Posts: 238
- Joined: Fri Jul 25, 2008 8:55 am
Re: Multiple Jobs Reading a single dataset concurrently
why don't you give a try.
It should be able to read a dataset simulationusly by many process.
It should be able to read a dataset simulationusly by many process.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
No, but Job2 and Job3 can concurrently read the Data Set created by Job1. There's no limit until you run out of memory - each loads the Data Set into virtual memory when it's on a reference input link to a Lookup stage.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
What happens if it is a Mege or Join stage I am using for looking up? In this case Data Set won't be loaded into virtual memory right?ray.wurlod wrote:No, but Job2 and Job3 can concurrently read the Data Set created by Job1. There's no limit until you run out of memory - each loads the Data Set into virtual memory when it's on a reference input link to a Lookup stage.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
Wrong. Separate images are loaded for each link. So if you have three links referring to the same Data Set, you get three copies of it in memory.
Note that using shared memory for Entire partitioning in an SMP architecture applies to each reference link - you still get one copy (in shared memory this time) per link.
Note that using shared memory for Entire partitioning in an SMP architecture applies to each reference link - you still get one copy (in shared memory this time) per link.
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Sorry Ray, We have version 8 parallel edition. I don't know how to move the post to parallel edition. Does the same answer apply to parallel edition as well?ray.wurlod wrote:Wrong. Separate images are loaded for each link. So if you have three links referring to the same Data Set, you get three copies of it in memory.
Note that using shared memory for Entire partitioning in an SMP architecture applies to each reference link - you still get one copy (in shared memory this time) per link.
-
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
I "moved" it... since it was already in the right forum, all that was needed was an edit of the original post and (fyi) you can always edit your own posts.Nagin wrote: Sorry Ray, We have version 8 parallel edition. I don't know how to move the post to parallel edition.
-craig
"You can never have too many knives" -- Logan Nine Fingers
"You can never have too many knives" -- Logan Nine Fingers