What is the highest level of maturity in a Performance Center of Excellence? HP describes it as a Performance Authority. In 2004, Mercury published a whitepaper called “Application Delivery: Center of Excellence Evolution”. They described the move from a Service Utility (a consulting services offering within a larger Enterprise) to a Performance Authority as one of “institutionalizing tools, techniques, and practices”. At that point, the way the CoE handles performance testing is finally accepted as “the way things get done” within IT. This requires a solid track record of showing value, so don't think you are going to get there very quickly. However, this is a GOOD thing. Below is the graphic we are used to seeing from HP:

At HP Universe 2008, I saw this slide over and over where a CoE was talked about in various presentations. Stage 4 represents less than 1% of the customer base that already have LoadRunner or Performance Center. You have most likely seen my past articles about this on this web site. Well, either companies aren't getting it or we're not talking enough about it in the industry because these are the same numbers that have not changed in over four years! So, like Michael Jackson once sang, "I'm starting with the man in the mirror." Forgive me for quoting Michael Jackson, but I could not find a death metal song with the same reference...

     I think the biggest question that needs answering is, what does it look like? Would we know it if we saw it? Here are some characteristics of a CoE acting as a Performance Authority I have put together to help people understand what it looks like.

• The CoE has proven time and again that their testing process works in mitigating all the performance risks associated with production roll out, and this needs to be marketed to the business. Year over year metrics consistently show this is the best process, the best way of approaching performance, and exposes those projects who go around the process and the problems they experience verses the ones who go through the process. To get such metrics requires traceability, real-time visibility, and a consistent “marketing” style evangelism into all areas of IT to gain acceptance. It is a hard sell at first, and every day will be a battle.
  
• It requires a “charge back” model where all LOB’s are contributing to the budget of the Performance teams to fund resources and the hours worked on projects. Some companies cannot move to this model because of the way budgeting has been set up. The performance team is seen as a "service", but this means it needs to be able to handle whatever is asked of them at any time without regards to priority and complexity of a project. Under those conditions, a charge back –mentality- should be in place, and worked into the escalation model to management. For example, when a new project is requested to be tested by the CoE, instead of allocating specific budget money from that project to fund a resource, a well defined process for approval by upper management needs to be in place that establishes the use of the resources of the CoE. If too many projects hit at one time, it’s management responsibility to allocate more budget for third party contractors to handle overflow, fund additional resources, or prioritize projects and push them back on the calendar. With enough demands made from the business side, this usually causes management to spend an inordinate amount of time with the approval process, and the charge back model becomes a lot more attractive at that point. Companies without a good charge back model/mentality rarely ever become a Performance Authority.
  
• It requires a culture of sharing knowledge and not hoarding it. This means a testing “community” (not just made of testers, but all people who are on board to make Quality and Performance their personal responsibility). The testing community should be sharing ideas and advice among different groups, collaborating on a regular basis for joint-success. No one is concerned about losing mindshare if individual members of the testing teams leave the company, because the knowledge base remains behind. 
  
• The CoE testing team members have become known as industry experts, known outside the organization and sought after for their knowledge, frequently speaking at user groups and conferences. The skill sets are so wide and deep in the group that outside consultants rarely bring anything new to group, and are only used to complete the overflow of projects. Internal training on the products and process rival that of external formal training offered by the vendors themselves. 
  
• The CoE has proven by a history of successes that their centralized model, processes, and techniques have affected the IT budget in a positive way, and upper level management signs off on this officially. 

How do you reach this pinnacle of application performance testing? What are the things that need to happen to begin acting as a Performance Authority? Here are my recommendations:

• Set up a “Standards and Methodologies” panel, made up of experts and key players in the decision making process for IT. This would include QA and Performance managers, business management, the CTO, etc…. This group is responsible for quality and performance engineering, standards enforcement, and instituting formalized process improvement. A balance between rich process and what is practical needs to be established day one. There needs to be an amount of flexibility to meet the needs of all projects and LOB’s when setting this up. The less mission-critical, the more flexibility that can be allowed. 
• The Standards and Methodologies committee must have the final say in allowing an application to be released into the production environment. If it does not meet the standards, it doesn’t go to production. If the product managers or the business are allowed to override this or go through any “back door” channels, there is no Authority in place. Ultimately the BUSINESS has to grant this right, not the CIO. The people who own the business, the board of directors, and the shareholders need to be partakers in this decision. Obviously this is easier said than done. The danger of empowering the Authority is that too much process can bog down the SDLC and create a bloated, endless maze of rules that people will try to get around instead of championing. 
  
• Create the documentation (using a technical writer) that describes the process for meeting the bar set by the Authority and how to comply with auditing of that process. 
  
• Audit the LOB’s. This requires penalties for non-compliance. Penalties might include denying access to the testing process until the LOB complies, denial of production implementation, and negative marks on annual reviews for LOB management. 
  
• Remove individual authority of project teams in the individual lines of business. They cannot continue to do it their own way, or cut corners. This will be a huge challenge in places where the LOB’s do not wish to give up that level of control. If an LOB has something to offer the overall process and make it better, this is fine. There should always be an open mind for making things better. An LOB may have a STRICTER or more controlled process than the Authority requires. This means that what is expected of the LOB's has enough flexibility to allow for this.
  
• There needs to be a high level of visibility to the C-Level and to the business by way of a dashboard or other reporting mechanism that allows them to keep track of the progress, look at metrics, wins, and losses. Each year this information should be used to determine how well the Authority is working. If it ceases to do what it has promised the business it could do by having the authority given, this authority should be taken away. 

As you can tell, these things are not very easy to do in today’s environment for most medium to large companies. This is not just a check list of items that can be completed in a couple of months. The key word is “maturity”. This is what I meant by saying that things taking a while to accomplish is a good thing. When I read a statement on a company’s web site indicating they are CMM Level 5 and they are less than six months old, I know they are full of it. To trust that company for their product or service is the equivalent of handing a two year old a machine gun and letting them go play in the yard. Maturity comes by making mistakes, learning from those mistakes, and making improvements over time that allow easy navigation over all the road blocks and sand traps that are out there. 

If it were up to me, I would make IT and the business sign a “Quality and Performance” contract, with penalties on both sides. If IT breaks the contract, CIO’s and CTO’s get fired, bonuses are yanked, managers of projects who have defeated the process are let go. If the business breaks the contract (by making the processes be done away with due to an exception), the cost to them immediately triples or quadruples because of the effort required to do it over when it fails. If I wanted a project done and IT says, “to do it right is 10 dollars, to subvert the standard its 40 dollars, paid upfront and guaranteed” – which way do you think I will go? 

Unfortunately, businesses do not have to remain in business, and have the right to make bad decisions. Sometimes business decisions are made that are bad for IT because at the time, other things were more important than process. A better idea than a contract with penalties is a mechanism for overriding the process. This already exist in most companies – i.e. when product manager "Bob" decides he doesn’t want to do things the right way, he just does it his way. Obviously if there is a CoE in place, this mechanism should be more formal and there should be a higher requirement set for using it than before the Performance Authority was established. But this mechanism does need to exist, and be allowed when it is needed. There also needs to be something in place to capture metrics on what the cost of that decision (to use the mechanism) was – bringing it back to real dollars. The Standards Committee could say, “Because you went around the process, a 5 million dollar project ended up costing 10 million and an increase in support for the application rose by 150% of the average application”. CIO’s can build exceptions into the annual budget, planning on a certain percentage of projects that will bypass the Authority's process. If a dashboard is available that shows who bypassed the Performance Authority and what the extra cost was, a pattern of behavior which cost the company money could be a career limiting move for people like Bob. 

As you start setting your sights on the all-elusive Performance Authority, be prepared for the long haul. It can be achieved, but it won’t necessarily be the most pleasant experience in your career. What it will do is ensure that across all tiers in the organization and across all areas of the software lifecycle that you have completely embedded and permeated sound practices around performance in all layers of the IT realm. Congratulations. I’ll be on the front row for your presentation at HP Universe as you tell us how you did it.

For more information about Scott, check out the rest of this web site at
www.loadtester.com