Page 1 of 1
How many projects needed for one ETL Stream
Posted: Fri Nov 21, 2003 9:33 am
by yiminghu
Hi,
My company is just starting to use Datastage to build ETL. And we have a debate regarding to how many projects you need for constructin one ETL stream.
Do you put all extrating, transformation, and loading jobs in one project ( but separated them into different categories), or you spearate them into different projects. If you separate them into different projects, how do you schedule them accross the projects? I mean if the last job in loading project finishes, how do you start the first job in transformation project right away.
Thanks,
Yiming
Recommendation
Posted: Fri Nov 21, 2003 9:44 am
by 1stpoint
Do you put all extrating, transformation, and loading jobs in one project ( but separated them into different categories), or you spearate them into different projects. If you separate them into different projects, how do you schedule them accross the projects? I mean if the last job in loading project finishes, how do you start the first job in transformation project right away.
In short, I recommend that you have a 'development' and a 'production' project and yes you put all the ETL jobs in that project. If different subject areas are involved you can create Categories/Folders within the project. This is a much more manageable environment.
I recommend using multiple projects when:
* Other departments with developers are using DataStage.
* The ETL processes are completely disparate, in that they
server entirely separate functional area.
As your development environment grows, you may decide to break it up into different projects.
Posted: Fri Nov 21, 2003 10:09 am
by kduke
Yiming
You can do it from a shell script with dsjob. Do a search. The dsjob command can run jobs in different projects and even on different servers. This is definitely not recommended. 1stpoint use of folders are a better way to handle this but it can be done if you have to do it that way.
Kim.
Re: How many projects needed for one ETL Stream
Posted: Fri Nov 21, 2003 10:11 am
by Teej
I would recommend at LEAST three separate projects for each set of client/project/field.
One for Development. One for QA. One for Production.
We took it a step further and have one for UAT - testing with full dataset.
We have a set of projects for a specific client's ODS process, and another set for the same client's DWH process.
Whatever you decide, STICK TO IT. Don't change naming conventions because it is a pain in the butt as an administrator to keep things consistent.
-T.J.