Performance Testing Lessons Learned

Posted by Ben Simo

Web and client/server load testing can easily become a complex task. Most people I've met got started in load testing with only minimal training in using the test tools. This is how I got started in load testing -- although I had an advantage in that I had been exposed to load testing of communications systems. I also had experience with automated single-user performance testing. I had led some small-scale manual load tests with multiple testers on a conference call hitting the same client-server application at once. (And we found some show-stopping bugs doing that manual testing.) I had watched others perform load tests. I had read numerous load test plans and reports. However, I had never directly participated in executing automated load tests... then I was asked to lead a load testing project.

Through the years, I have made many mistakes designing, scripting, and executing load tests. Load testing easily becomes complex. Tool sales people sometime tell us that nearly anyone can create tests with their tools. (Yet buying test tools is sometimes just like buying a new car: the salesman tells you that the car is reliable and has a great warranty; then the finance person warns of everything that could go wrong that isn't covered in the warranty and trys to sell you an extended warranty and maintenance contract.) Learning the mechanics of how to use a tool are often the easy part. Its what you do with the tool that matters.

Here is the short list of some of the important performance/load testing lessons I have learned. Some I learned from my own experience. Some I learned from the failures of others.

• Bad assumptions waste time and effort
  
• Ask questions
• Performance testing is often exploratory
• Expect surprises
• Prepare to adapt
  
• Get to know the people in your neighborhood: no single person or group is likely to have all the required information
• Subject-matter experts
• Developers
• System administrators
• Database administrators
• Network engineers
  
• Don’t script too much too soon: you may end up tossing out much of what you script
  
• Applications change
• Real usage is difficult to estimate
• Tool limitations may be discovered
  
• Different processes have different impacts: what users do can be as, or more, important as how many users are doing it
  
• Include high-use processes (80/20 rule)
• Include high-risk processes
• Include “different” processes
  
• Modularize scripts: simplify script maintenance -- but only when you intend to run the script again
  
• Data randomization is not always a good thing: randomization can make result comparison difficult
  
• Code error detection and handling
  
• Don’t assume that your tool will handle errors
• Catch and report errors when and where they happen
• Realize that errors may change simulated user activity
  
• Know your tools and test environment
  
• Tool’s supported protocols and licensing
• Load generator and network resource usage
• Load balancing and caching mechanisms
• Beware of test and production environment differences
  
• Try to translate results into stories that matter to the applicable stakeholders
• Tests are run to answer questions: don't overwhelm your audience with hundreds of numbers if they just want a "yes" or "no" answer

and finally…

• Most performance and load-related problems are due to software code or configuration; not hardware


• Don’t throw more hardware at a software problem