Do you use best practices in DataStage?
Moderators: chulett, rschirm, roy
Do you use best practices in DataStage?
Hi folks, not coming from a DataStage or ETL background, and having just begun to architect a DataStage system, I am wondering... does anyone use best practices? Can they be used here? Have any been defined? If you answer yes to this poll, perhaps you can lend some insight into what your answer means.
You can define naming conventions and plenty of other things. You should define when to use a merge stae over a join or lookup. You should define when it is okay to use ODBC over an API. You should define a typical job for each target table type like dimension over fact or even SCD type 1,2,3 ... When it is okay to land the data. When it is okay to use staging tables. What shared containers are used.
You can have a whole other set of documents for admin tasks. Like backup standards.
You can have a whole other set of documents for admin tasks. Like backup standards.
Mamu Kim
-
ray.wurlod
- Participant
- Posts: 54607
- Joined: Wed Oct 23, 2002 10:52 pm
- Location: Sydney, Australia
- Contact:
I recall a presentation at an IOD conference (2008?) in which the presenter made the point that there aren't really any best practices that have universal applicability, though you could argue for a set of "least worst" practices. There are three key concepts, as I see things:
- don't do anything unnecessary
be consistent and systematic in all things (from design to troubleshooting) - having standards that are not too complex is valuable here
obey the "law of least astonishment" (don't embed "surprises" for future developers - prominently document anything unusual)
IBM Software Services Group
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
Any contribution to this forum is my own opinion and does not necessarily reflect any position that IBM may hold.
-
jcthornton
- Premium Member
- Posts: 79
- Joined: Thu Mar 22, 2007 4:58 pm
- Location: USA
I'll be a little bit surprised if you get any significant numbers who say they don't use best practices.
That said, best practices will cover a wide range of tasks - from development, to change management, to deployment, to administration and more.
I would expect that best practices that most people here will cite are going to be about just one or two of these smaller areas that each person deals with individually on a day-in day-out basis. The best practices will also consist of personal experience and what people do to avoid experiencing the same pain again.
I do like Ray's example of 'Least Worst' practices.
I may be more cynical, but my variation is to imagine everybody else is going to be an idiot. Best Practices at that point are things that make it as easy as possible to do things right and as hard as possible to do things wrong.
My experience has been in applying it to ETL design, code control and deployment. I have not had the pleasure, yet, of getting to apply it to change management, architecture or any significant amount of maintenance [beyond what design and development provide as a foundation].
Best practices can cover such a wide range of sub-topics. Are there any in particular that you have concerns or questions about? People here can talk in detail about all aspects - there are 50k topics with more than 280k posts.
While many of us are occasional contributors, there is a nice core of usual suspects that are knowledgeable and quite willing to share - but a brain dump from them would not only take days to read [weeks?] but would require a significant background in a number of technology areas.
So, my short answer is as follows:
"There are a significant number of best practices that individuals and groups have developed for DataStage. Most of these are oriented to specific sub-topics and are from the 'on the ground' experience of individuals on the board - of which the best have a breadth of knowledge large enough that it is best to tease that knowledge out through specific questions and not a request for a brain dump."
That said, best practices will cover a wide range of tasks - from development, to change management, to deployment, to administration and more.
I would expect that best practices that most people here will cite are going to be about just one or two of these smaller areas that each person deals with individually on a day-in day-out basis. The best practices will also consist of personal experience and what people do to avoid experiencing the same pain again.
I do like Ray's example of 'Least Worst' practices.
I may be more cynical, but my variation is to imagine everybody else is going to be an idiot. Best Practices at that point are things that make it as easy as possible to do things right and as hard as possible to do things wrong.
My experience has been in applying it to ETL design, code control and deployment. I have not had the pleasure, yet, of getting to apply it to change management, architecture or any significant amount of maintenance [beyond what design and development provide as a foundation].
Best practices can cover such a wide range of sub-topics. Are there any in particular that you have concerns or questions about? People here can talk in detail about all aspects - there are 50k topics with more than 280k posts.
While many of us are occasional contributors, there is a nice core of usual suspects that are knowledgeable and quite willing to share - but a brain dump from them would not only take days to read [weeks?] but would require a significant background in a number of technology areas.
So, my short answer is as follows:
"There are a significant number of best practices that individuals and groups have developed for DataStage. Most of these are oriented to specific sub-topics and are from the 'on the ground' experience of individuals on the board - of which the best have a breadth of knowledge large enough that it is best to tease that knowledge out through specific questions and not a request for a brain dump."
Jack Thornton
----------------
Spectacular achievement is always preceded by spectacular preparation - Robert H. Schuller
----------------
Spectacular achievement is always preceded by spectacular preparation - Robert H. Schuller
Hi folks thanks for your thoughtful answers. From the information presented so far, I think there are several possibilities:
1) There really aren't best practices. It's a term that describes lessons learned, but everyone has an opinion, and there are so many, and so many various use cases, it's pointless to try to categorize them or reconcile them. Best practices are in the eye of the beholder.
2) There are some key design patterns that have emerged, in some areas more than others, but it exists mostly as tribal knowledge and has not been pulled together in any sort of coherent fashion. Hunt for the answers. Cajole the experts.
3) A lot of things have been pulled together, but they remain proprietary. Get a consultant.
4) A lot of stuff has been pulled together, and it exists in a FAQ linked at the top of this page.
Seems like I can at least cross off #4, and I don't believe #1.
Thanks,
Doug
1) There really aren't best practices. It's a term that describes lessons learned, but everyone has an opinion, and there are so many, and so many various use cases, it's pointless to try to categorize them or reconcile them. Best practices are in the eye of the beholder.
2) There are some key design patterns that have emerged, in some areas more than others, but it exists mostly as tribal knowledge and has not been pulled together in any sort of coherent fashion. Hunt for the answers. Cajole the experts.
3) A lot of things have been pulled together, but they remain proprietary. Get a consultant.
4) A lot of stuff has been pulled together, and it exists in a FAQ linked at the top of this page.
Seems like I can at least cross off #4, and I don't believe #1.
viewtopic.php?t=134328Are there any in particular that you have concerns or questions about?
Thanks,
Doug