Can the state file be read parallely in different jobs?

Post questions here relative to DataStage Enterprise/PX Edition for such areas as Parallel job design, Parallel datasets, BuildOps, Wrappers, etc.

Moderators: chulett, rschirm, roy

Post Reply
srinivas.nettalam
Participant
Posts: 134
Joined: Tue Jun 15, 2010 2:10 am
Location: Bangalore

Can the state file be read parallely in different jobs?

Post by srinivas.nettalam »

Hi All,
I recently shifted my company and here I have been told that they are running few jobs that copy data from source to a landing area.All these jobs run parallely and read same state file.Also they mentioned if the surrogate key generator generates a number eg 100 in the first job it doesn't generate 100 in other job.I always believed that surrogate key generator works similarly to the formula
(@PARTITIONNUM + (@INROWNUM-1) * @NUMPARTITIONS).I am not convinced that it can be read parallel and I assume there would be lot of discrepencies in generating the sequence numbers in this scenario.Am I right? Please let me know on this
N.Srinivas
India.
ray.wurlod
Participant
Posts: 54607
Joined: Wed Oct 23, 2002 10:52 pm
Location: Sydney, Australia
Contact:

Post by ray.wurlod »

A state file must generate unique numbers, no matter how many processes are reading it. So if 100 occurs in one job you should not expect 100 to appear in any other job reading the same state file.
Why would you believe otherwise?
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
chulett
Charter Member
Charter Member
Posts: 43085
Joined: Tue Nov 12, 2002 4:34 pm
Location: Denver, CO

Re: Can the state file be read parallely in different jobs?

Post by chulett »

srinivas.nettalam wrote:I am not convinced that it can be read parallel and I assume there would be lot of discrepencies in generating the sequence numbers in this scenario. Am I right?
Short answer? Nope.
-craig

"You can never have too many knives" -- Logan Nine Fingers
srinivas.nettalam
Participant
Posts: 134
Joined: Tue Jun 15, 2010 2:10 am
Location: Bangalore

Re: Can the state file be read parallely in different jobs?

Post by srinivas.nettalam »

chulett wrote:
srinivas.nettalam wrote:I am not convinced that it can be read parallel and I assume there would be lot of discrepencies in generating the sequence numbers in this scenario. Am I right?
Short answer? Nope.
Thanks
N.Srinivas
India.
Post Reply