performance tips

bhargav_dd · Post by **bhargav_dd** » Fri Jul 16, 2010 5:37 am

Hi All,

I need to give a document on Performance tips and Coding standards before the project development starts in Datastage.

Since I am new to Datastage can you help me in this. By performance tips I mean things like:
1:Whether its better to use an SQL Join Statement or a Lookup stage or a Join stage, which of them is more faster and efficient.
2: Is it better to use Orderby or the sort stage etc.

Sainath.Srinivasan · Post by **Sainath.Srinivasan** » Fri Jul 16, 2010 5:41 am

There is no one fixed efficient method. If it had been, the software developers would not have developed other stages or operators.

If you are new to DataStage, why are you tasked with putting together such a 'performance tips and coding standards document' !!??

Shaanpriya · Post by **Shaanpriya** » Fri Jul 16, 2010 5:45 am

Sainath.Srinivasan wrote: If you are new to DataStage, why are you tasked with putting together such a 'performance tips and coding standards document' !!??

Exactly my thoughts!

chulett · Post by **chulett** » Fri Jul 16, 2010 6:10 am

Shaanpriya wrote:
Sainath.Srinivasan wrote:If you are new to DataStage, why are you tasked with putting together such a 'performance tips and coding standards document' !!??
Exactly my thoughts!

Truly mind boggling. How can anyone new to any tool/product be expected to put together any kind of 'tips' or 'best practices' document? You need to push back on that one. Hard.

chulett · Post by **chulett** » Fri Jul 16, 2010 6:14 am

bhargav_dd: just exactly how "new" to DataStage are you? You say you are here and yet in your profile you rank yourself a "3" which means "Confident to go solo" - something which implies a certain level of training and time with the product. Which is it?

It would help us to actually know where you are, experience wise, to know how (or even if) to answer the kind of questions you are asking.

kduke · Post by **kduke** » Fri Jul 16, 2010 10:09 am

When to use a lookup over a join or a merge is usually based on volume of records and hardware used by the client. More CPU power will change when one of these will out perform the other. The trick is to build a job 3 different ways and try them with more and more records. So maybe a million rows makes a lookup too slow or slower than a join. You decide.

ray.wurlod · Post by **ray.wurlod** » Fri Jul 16, 2010 2:37 pm

chulett wrote:
Shaanpriya wrote:
Sainath.Srinivasan wrote:If you are new to DataStage, why are you tasked with putting together such a 'performance tips and coding standards document' !!??
Exactly my thoughts!
Truly mind boggling. How can anyone new to any tool/product be expected to put together any kind of 'tips' or 'best practices' document? You need to push back on that one. Hard.

I agree entirely.

dougcl · Post by **dougcl** » Fri Jul 16, 2010 5:39 pm

Hi folks, it is appropriate and commendable that a site establish coding practices before going into mass production. It is also likely that some sites are new to DataStage. It is therefore likely that someone completely new to DataStage is tasked with establishing the design guide for the development team. This is precisely the situation I am in. I think it is appropriate that this forum address this need (indeed it does). There are obviously established design patterns in this environment. They apparently have not been published. Under the circumstances, from my perspective at least, it is appropriate for the original poster to ask these questions.

Thanks,
Doug

chulett · Post by **chulett** » Fri Jul 16, 2010 9:21 pm

Still would like to see my question answered.

ray.wurlod · Post by **ray.wurlod** » Sat Jul 17, 2010 3:28 am

I am hired from time to time, particularly for new sites, to set up best practices (even a Centre of Excellence in some cases) and to mentor the new people. This (hiring an expert) is a more intelligent approach to my mind than expecting a novice to provide this guidance.

kwwilliams · Post by **kwwilliams** » Mon Jul 19, 2010 2:28 pm

The problem with a staff of people new to DataStage writing best practices is going to cause you some long range issues, that you may not be able to foresee. Asking people on a forum, may get you some good tips but they won't be comprehensive or specific to the needs of your organization.

If you can't hire a professional to help you write these best practices, then you will likely need to start asking some specific questions of the people on this board. Realize that this still will not gain you insight into best practices for your organization.

ArndW · Post by **ArndW** » Tue Jul 20, 2010 3:12 am

I have to admit that I have mixed emotions on this subject.

The professional in me shudders to think of the damage caused by documents such as "best practices" being created by neophytes to the product.

The consultant in me gloats in the knowledge that activities such as this guarantee my continued income in the future.

DSShishya · Post by **DSShishya** » Tue Jul 20, 2010 11:27 am

Well, For those who are astonished or surprised as to why Bharghav inspite of being "New" to DataStage is being assigned such a responsible task. You will probably get a much clearer picture as to the exact situation he is in if you try and understand the system most companies have incorporated themselves in getting skilled labor with "no strings attached" policy ("Consulting" in corporate terms). And in order to supply this demand the kind of "Ethical" practices all the consulting/staffing companies indulge themselves in. After all its business.

If you all understand those two matters then you will know exactly why Bhargav is in such a situation inspite of being "New". Just my opinion

@Topic
Bharghav,
There are many posts within this forum which gives details about performance tuning. Kindly do a good search.

You can also check this link.
http://vivantech.net/BestPracticesDataStage.aspx

Goodluck!

kwwilliams · Post by **kwwilliams** » Tue Jul 20, 2010 2:07 pm

In my experience the companies stand up DataStage and don't want to pay for training and mentoring their employees, trying to save a few dollars because after all they are just running a business.

What these companies don't realize is if they would expend a few extra dollars up front, they could avoid much greater expenses down the road in rework (rework is far more expensive than getting it right the first time). I don't fault Bhargav, it is his management trying to save a few dollars by making the employees go through "On The Job Training."