any limitation on no of jobs
any limitation on no of jobs
This is a topic for an orphaned message.
DataStage imposes no limit on the number of jobs. (Actually this is not true - there is, by default, a limit of 2GB on the size of the DS_JOBS table, which maps job names to job numbers, so the limit is just short of 2GB worth of job names. Doubt that youll hit this one! If you do, we can resize the DS_JOBS table to use 64bit addressing - providing the O/S permits (WinNT does not) - allowing theoretically 19 million TB.)
However, the operating system may limit you for another reason. Every job you create creates ten or twelve new entries in the project directory. These are tables with names like DS_JOBnn, RT_LOGnn, and so on, where nn is the job number. If the operating system has a limit on the number of entry names in a directory then you may hit this limit. Its not a DataStage limit.
The other possible limit is comprehensibility and searching. You have to be able to manage what youve got. There is a filter in status view of Director (release 3.6 and later) which helps, but there is no such filter in the drop-down list of job names in Designer and none in the Jobs branch of Manager. There is some good news in release 4.0, where jobs can be placed in Categories, like all the other components in the Repository can at earlier releases. This should ease the management burden. However, the cost of searching such a large directory will still carry some kind of penalty in the operating system, as a directory search is, to all intents and purposes, a linear scan.
> ----------
> From: David Barham[SMTP:David.Barham@anglocoal.com.au]
> Sent: Thursday, 17 August 2000 17:17
> To: sudhir.mahendru@db.com; David Barham
> Cc: ray.wurlod@informix.com
> Subject: RE: any limitation on no of jobs
>
> This is going to be a case of you tell me. We only have 500. I would
> suspect the limitations are going to be more manageability rather than
> technical. We have some fairly sophisticated job control routines to
> manage our overnight runs. Using the standard job control that
> DataStage generates would be a nightmare with this many jobs.
>
> Ill let Ray field this question.
>
> David Barham
> Information Technology Consultant
> CoalMIS Project
> Anglo Coal Australia Pty Ltd
> Brisbane, Australia
>
> -----Original Message-----
> From: sudhir.mahendru@db.com [SMTP:sudhir.mahendru@db.com]
> Sent: Thursday, 17 August 2000 16:45
> To: David.Barham@anglocoal.com.au
> Cc: ray.wurlod@informix.com
> Subject: any limitation on no of jobs
>
> David
> is there any limitation on the number of jobs in one datastage
> project. i already have more than 1000 jobs in one project.
>
> thanks and regards
> Sudhir
> APHOIT/CA$H Datawarehouse Team
> (65) 423-7449
>
>
>
> **********************************************************************
> ***
> This e-mail and any files transmitted with it may be confidential
> and are intended solely for the use of the individual or entity to
> whom they are addressed. If you have received this e-mail in
> error, please notify the sender by return e-mail, and delete this
> e-mail from your in-box. Do not copy it to anybody else
>
> **********************************************************************
> ***
>
However, the operating system may limit you for another reason. Every job you create creates ten or twelve new entries in the project directory. These are tables with names like DS_JOBnn, RT_LOGnn, and so on, where nn is the job number. If the operating system has a limit on the number of entry names in a directory then you may hit this limit. Its not a DataStage limit.
The other possible limit is comprehensibility and searching. You have to be able to manage what youve got. There is a filter in status view of Director (release 3.6 and later) which helps, but there is no such filter in the drop-down list of job names in Designer and none in the Jobs branch of Manager. There is some good news in release 4.0, where jobs can be placed in Categories, like all the other components in the Repository can at earlier releases. This should ease the management burden. However, the cost of searching such a large directory will still carry some kind of penalty in the operating system, as a directory search is, to all intents and purposes, a linear scan.
> ----------
> From: David Barham[SMTP:David.Barham@anglocoal.com.au]
> Sent: Thursday, 17 August 2000 17:17
> To: sudhir.mahendru@db.com; David Barham
> Cc: ray.wurlod@informix.com
> Subject: RE: any limitation on no of jobs
>
> This is going to be a case of you tell me. We only have 500. I would
> suspect the limitations are going to be more manageability rather than
> technical. We have some fairly sophisticated job control routines to
> manage our overnight runs. Using the standard job control that
> DataStage generates would be a nightmare with this many jobs.
>
> Ill let Ray field this question.
>
> David Barham
> Information Technology Consultant
> CoalMIS Project
> Anglo Coal Australia Pty Ltd
> Brisbane, Australia
>
> -----Original Message-----
> From: sudhir.mahendru@db.com [SMTP:sudhir.mahendru@db.com]
> Sent: Thursday, 17 August 2000 16:45
> To: David.Barham@anglocoal.com.au
> Cc: ray.wurlod@informix.com
> Subject: any limitation on no of jobs
>
> David
> is there any limitation on the number of jobs in one datastage
> project. i already have more than 1000 jobs in one project.
>
> thanks and regards
> Sudhir
> APHOIT/CA$H Datawarehouse Team
> (65) 423-7449
>
>
>
> **********************************************************************
> ***
> This e-mail and any files transmitted with it may be confidential
> and are intended solely for the use of the individual or entity to
> whom they are addressed. If you have received this e-mail in
> error, please notify the sender by return e-mail, and delete this
> e-mail from your in-box. Do not copy it to anybody else
>
> **********************************************************************
> ***
>
Most unixes have an INode limitation on a disk partition. And when you create lots of files (10+ per job) plus the size of the programs, plus tables greater than 64k (use INODE redirection), you start really burning INodes fairly quickly. Saying that, I have only run into this problem once.
Cheers,
Ray D
----- Original Message -----
From: "Ray Wurlod"
To: ; "David Barham"
Cc:
Sent: Thursday, August 17, 2000 7:39 PM
Subject: RE: any limitation on no of jobs
> DataStage imposes no limit on the number of jobs. (Actually this is
> not true - there is, by default, a limit of 2GB on the size of the
> DS_JOBS table, which maps job names to job numbers, so the limit is
> just short of 2GB worth of job names. Doubt that youll hit this one!
> If you do, we
can
> resize the DS_JOBS table to use 64bit addressing - providing the O/S
permits
> (WinNT does not) - allowing theoretically 19 million TB.)
>
> However, the operating system may limit you for another reason. Every
> job you create creates ten or twelve new entries in the project
> directory. These are tables with names like DS_JOBnn, RT_LOGnn, and so
> on, where nn is the job number. If the operating system has a limit on
> the number of entry names in a directory then you may hit this limit.
> Its not a DataStage limit.
>
> The other possible limit is comprehensibility and searching. You have
> to
be
> able to manage what youve got. There is a filter in status view of
Director
> (release 3.6 and later) which helps, but there is no such filter in
> the drop-down list of job names in Designer and none in the Jobs
> branch of Manager. There is some good news in release 4.0, where jobs
> can be placed
in
> Categories, like all the other components in the Repository can at
> earlier releases. This should ease the management burden. However,
> the cost of searching such a large directory will still carry some
> kind of penalty in the operating system, as a directory search is, to
> all intents and
purposes,
> a linear scan.
>
> > ----------
> > From: David Barham[SMTP:David.Barham@anglocoal.com.au]
> > Sent: Thursday, 17 August 2000 17:17
> > To: sudhir.mahendru@db.com; David Barham
> > Cc: ray.wurlod@informix.com
> > Subject: RE: any limitation on no of jobs
> >
> > This is going to be a case of you tell me. We only have 500. I
> > would suspect the limitations are going to be more manageability
> > rather than technical. We have some fairly sophisticated job
> > control routines to manage our overnight runs. Using the standard
> > job control that
DataStage
> > generates would be a nightmare with this many jobs.
> >
> > Ill let Ray field this question.
> >
> > David Barham
> > Information Technology Consultant
> > CoalMIS Project
> > Anglo Coal Australia Pty Ltd
> > Brisbane, Australia
> >
> > -----Original Message-----
> > From: sudhir.mahendru@db.com [SMTP:sudhir.mahendru@db.com]
> > Sent: Thursday, 17 August 2000 16:45
> > To: David.Barham@anglocoal.com.au
> > Cc: ray.wurlod@informix.com
> > Subject: any limitation on no of jobs
> >
> > David
> > is there any limitation on the number of jobs in one datastage
> > project. i already have more than 1000 jobs in one project.
> >
> > thanks and regards
> > Sudhir
> > APHOIT/CA$H Datawarehouse Team
> > (65) 423-7449
> >
> >
> >
> >
*************************************************************************
> > This e-mail and any files transmitted with it may be confidential
> > and are intended solely for the use of the individual or entity to
> > whom they are addressed. If you have received this e-mail in error,
> > please notify the sender by return e-mail, and delete this e-mail
> > from your in-box. Do not copy it to anybody else
> >
> >
*************************************************************************
> >
>
Cheers,
Ray D
----- Original Message -----
From: "Ray Wurlod"
To: ; "David Barham"
Cc:
Sent: Thursday, August 17, 2000 7:39 PM
Subject: RE: any limitation on no of jobs
> DataStage imposes no limit on the number of jobs. (Actually this is
> not true - there is, by default, a limit of 2GB on the size of the
> DS_JOBS table, which maps job names to job numbers, so the limit is
> just short of 2GB worth of job names. Doubt that youll hit this one!
> If you do, we
can
> resize the DS_JOBS table to use 64bit addressing - providing the O/S
permits
> (WinNT does not) - allowing theoretically 19 million TB.)
>
> However, the operating system may limit you for another reason. Every
> job you create creates ten or twelve new entries in the project
> directory. These are tables with names like DS_JOBnn, RT_LOGnn, and so
> on, where nn is the job number. If the operating system has a limit on
> the number of entry names in a directory then you may hit this limit.
> Its not a DataStage limit.
>
> The other possible limit is comprehensibility and searching. You have
> to
be
> able to manage what youve got. There is a filter in status view of
Director
> (release 3.6 and later) which helps, but there is no such filter in
> the drop-down list of job names in Designer and none in the Jobs
> branch of Manager. There is some good news in release 4.0, where jobs
> can be placed
in
> Categories, like all the other components in the Repository can at
> earlier releases. This should ease the management burden. However,
> the cost of searching such a large directory will still carry some
> kind of penalty in the operating system, as a directory search is, to
> all intents and
purposes,
> a linear scan.
>
> > ----------
> > From: David Barham[SMTP:David.Barham@anglocoal.com.au]
> > Sent: Thursday, 17 August 2000 17:17
> > To: sudhir.mahendru@db.com; David Barham
> > Cc: ray.wurlod@informix.com
> > Subject: RE: any limitation on no of jobs
> >
> > This is going to be a case of you tell me. We only have 500. I
> > would suspect the limitations are going to be more manageability
> > rather than technical. We have some fairly sophisticated job
> > control routines to manage our overnight runs. Using the standard
> > job control that
DataStage
> > generates would be a nightmare with this many jobs.
> >
> > Ill let Ray field this question.
> >
> > David Barham
> > Information Technology Consultant
> > CoalMIS Project
> > Anglo Coal Australia Pty Ltd
> > Brisbane, Australia
> >
> > -----Original Message-----
> > From: sudhir.mahendru@db.com [SMTP:sudhir.mahendru@db.com]
> > Sent: Thursday, 17 August 2000 16:45
> > To: David.Barham@anglocoal.com.au
> > Cc: ray.wurlod@informix.com
> > Subject: any limitation on no of jobs
> >
> > David
> > is there any limitation on the number of jobs in one datastage
> > project. i already have more than 1000 jobs in one project.
> >
> > thanks and regards
> > Sudhir
> > APHOIT/CA$H Datawarehouse Team
> > (65) 423-7449
> >
> >
> >
> >
*************************************************************************
> > This e-mail and any files transmitted with it may be confidential
> > and are intended solely for the use of the individual or entity to
> > whom they are addressed. If you have received this e-mail in error,
> > please notify the sender by return e-mail, and delete this e-mail
> > from your in-box. Do not copy it to anybody else
> >
> >
*************************************************************************
> >
>