How is your performance?

Many studies have shown that the cost of finding defects in applications increases exponentially the further the defect remains undetected through the lifecycle. Some studies (Microsoft, National Institute of Science and Technology) have estimated that this may be as much as 100 times more expensive to fix a post-deployment issue compared to finding it at the point the error was inserted.

Performance issues are pernicious because, while the application meets the functional requirements, it does not deliver that functionality within the expected (or demanded) time frame. But what really matters with performance issues is that the solution is frequently not correctable with code changes. Too often the remedy involves architectural reformation, changes to the footprint (memory usage typically) or demands for higher and more specific topologies with greater horsepower. This kind of repair is costly to do and can take a long time to effect.

In the case of a performance issue in embedded code it may just not be possible to solve without re-architecting the silicon. How many times have we been forced to upgrade the hardware to achieve reasonable performance from the software?

So performance testing early in the lifecycle is critical if project costs are to be managed. Many tools exist that can simulate load on the application as well as throttling the resources (bandwidth, data transfer speeds, latency) so that realistic expectations of the resulting performance can be observed.

Simulated normal loads are one thing but stressing an application is vital too. Seeing how an application performs under extreme loads will help us understand what the resulting performance envelope looks like so that we can design in ways to balance the load over time. It also helps us to identify where we might instrument the code so that we can give live telemetry from the application about when it is reaching execution limits that are known to cause the application to fall out of the operational envelope.

Load testing, stress testing, performance testing are all essential parts of the test lifecycle that we can do today with very sophisticated and capable tools across a wide range of platforms. With test automation it should be possible to make the code, build and test cycle fully repeatable so that performance issues are detected early enough to do something about them.