How many projects needed for one ETL Stream

[yiminghu]

Hi,

My company is just starting to use Datastage to build ETL. And we have a debate regarding to how many projects you need for constructin one ETL stream.

Do you put all extrating, transformation, and loading jobs in one project ( but separated them into different categories), or you spearate them into different projects. If you separate them into different projects, how do you schedule them accross the projects? I mean if the last job in loading project finishes, how do you start the first job in transformation project right away.

Thanks,

Yiming

[1stpoint]

Do you put all extrating, transformation, and loading jobs in one project ( but separated them into different categories), or you spearate them into different projects. If you separate them into different projects, how do you schedule them accross the projects? I mean if the last job in loading project finishes, how do you start the first job in transformation project right away.

In short, I recommend that you have a 'development' and a 'production' project and yes you put all the ETL jobs in that project. If different subject areas are involved you can create Categories/Folders within the project. This is a much more manageable environment.

I recommend using multiple projects when:
* Other departments with developers are using DataStage.

* The ETL processes are completely disparate, in that they
server entirely separate functional area.

As your development environment grows, you may decide to break it up into different projects.

[kduke]

Yiming

You can do it from a shell script with dsjob. Do a search. The dsjob command can run jobs in different projects and even on different servers. This is definitely not recommended. 1stpoint use of folders are a better way to handle this but it can be done if you have to do it that way.

Kim.

[Teej]

I would recommend at LEAST three separate projects for each set of client/project/field.

One for Development. One for QA. One for Production.

We took it a step further and have one for UAT - testing with full dataset.

We have a set of projects for a specific client's ODS process, and another set for the same client's DWH process.

Whatever you decide, STICK TO IT. Don't change naming conventions because it is a pain in the butt as an administrator to keep things consistent.

-T.J.