Should developers performance test their code before checking it in?

Posted on 2013-04-22 by

When is a bug a good thing?

One of the glorious ironies of IT is that the CIO, the executive with “Information” in their job title, is frequently the least well equipped member of the management team when it comes to data. Information about project status comes from meetings with optimists (developers) and pessimists (testers). Really they should be called the Chief Opinion Officer instead.

The other glorious irony of IT is that we manually run the automated test suite! Now I have a vision of how development should be: fair warning this is an idealistic and romantic dream of a far-away development future.

Wouldn’t it be interesting if the first thing a developer did was to look at the work assigned to her, her “tickets”, right there from within her IDE. No popping out to the ticketing systems to get the daily task list.

And what if every piece of code she then touched, every change she made, was automatically “stamped” with the ticket number to give a complete audit trail for root cause analysis later.

When it came time to check code in the build tool would kick off immediately and, if it failed, would open a new, high-priority, ticket in the developer’s inbox in her IDE. Maybe even with a social media beep to get her attention.

Of course if the build is successful the automated test suite would run, AUTOMATICALLY, exercising those pieces of code that had changed or were impacted. That would mean a correlation between test scripts and code changes and an impact analysis tool that could understand upstream and downstream dependencies.

Should any of the battery of tests, functional, performance, non-functional, fail then a new ticket would be opened in the developer’s inbox.

Notice that what this means that the developer never gets to hand over code that does not build or does not pass the battery of automated tests. So what happens to the Quality Assurance team? Well, they start to assure the quality by catching everything else that gets missed.

A programmer once told me “I just spent all morning debugging the code and now I am spending the afternoon rebugging it.” The idea that we can rely on someone else to find our bugs is out dated and a very poor use of our best talent. After all “testers” didn’t put the bugs in, can’t take them out and are nobody’s clean-up crew.

We have to learn to say “no” in IT. If the requirement doesn’t make sense – send it back. If the code is incomplete and buggy – send it back. If the test plan isn’t comprehensive and is not automated – don’t test until it is. Far too often we pass our problems on downstream and when the same is done to us we grin and we bear it.

My first boss said to me when I was a newly minted programmer, “if you don’t have time to do it right when do you have time to do it over?” We should make every effort to test every aspect of the code we can before we are allowed to check it in. We have the technology available today to do that. But do we have the will?

Posted in Business and Technology | Tagged , , , , , , , , | Leave a comment

What is the benefit of doing performance testing earlier in the application lifecycle?

Posted on 2013-04-19 by

How is your performance?

Many studies have shown that the cost of finding defects in applications increases exponentially the further the defect remains undetected through the lifecycle. Some studies (Microsoft, National Institute of Science and Technology) have estimated that this may be as much as 100 times more expensive to fix a post-deployment issue compared to finding it at the point the error was inserted.

Performance issues are pernicious because, while the application meets the functional requirements, it does not deliver that functionality within the expected (or demanded) time frame. But what really matters with performance issues is that the solution is frequently not correctable with code changes. Too often the remedy involves architectural reformation, changes to the footprint (memory usage typically) or demands for higher and more specific topologies with greater horsepower. This kind of repair is costly to do and can take a long time to effect.

In the case of a performance issue in embedded code it may just not be possible to solve without re-architecting the silicon. How many times have we been forced to upgrade the hardware to achieve reasonable performance from the software?

So performance testing early in the lifecycle is critical if project costs are to be managed. Many tools exist that can simulate load on the application as well as throttling the resources (bandwidth, data transfer speeds, latency) so that realistic expectations of the resulting performance can be observed.

Simulated normal loads are one thing but stressing an application is vital too. Seeing how an application performs under extreme loads will help us understand what the resulting performance envelope looks like so that we can design in ways to balance the load over time. It also helps us to identify where we might instrument the code so that we can give live telemetry from the application about when it is reaching execution limits that are known to cause the application to fall out of the operational envelope.

Load testing, stress testing, performance testing are all essential parts of the test lifecycle that we can do today with very sophisticated and capable tools across a wide range of platforms. With test automation it should be possible to make the code, build and test cycle fully repeatable so that performance issues are detected early enough to do something about them.

Posted in Business and Technology | Leave a comment

What is the value of Data?

Posted on 2013-04-18 by

My favorite event of the year is the Defence Information conference held annually in the UK. It brings together the leading Ministry of Defence and Uniformed Services IT executives to talk about a range of topics. This year’s theme was about the Valuing Information as an Asset.

This year I was honored to give a brief keynote and chose as my topic the value of interpretation of data in a session I called Being Evangelical about Data. I was inspired to talk about the difference between collecting data, analyzing data and interpreting data.

Like all military organizations across the world: the MoD is looking to improve the efficiency in the systems they use that support the missions they have. One significant change that has been implemented is a joint logistics force that delivers the food, equipment, spare parts, ammunition and everything that is needed in theaters of operation. As the Army, Air Force, Navy and Marines all use the same makes of equipment (such as the Apache Helicopter) it makes sense to have all the parts managed by one, new, service called the Joint Supply Chain (JSC).

The first question asked when all the inventory came under the control of the JSC was, “What is the value of the inventory managed?” The answer came back that it was about £40bn (about $64bn). Quite something, as this was not able to be calculated before.

So here we have Data giving Information. But further analysis showed that, while the MoD could tell how much the inventory was worth, they could not tell, for the most part, where it was located. And now we have Information giving Insight. And now, the next step in improving the logistical capabilities of the MoD, is to make inventory tracking much better.

My presentation took all of this to the next level and left the audience with one of my favorite quotes from a Serena customer in Chicago. Our customer implemented the Orchestrated-IT approach across the entire development lifecycle from requirements through development and testing into change and release management and all the way to deployment. The director of the Change and Release Management team told me “I used to spend all of my time fighting fires … I now spend all of my time finding out who is starting them”.

During the breaks many of the project leaders came to the Serena exhibit to talk about how they too can use process improvement and automation to give them control, repeatability and insight. It is so interesting to talk to customers about their organizations and their approach to developing and deploying applications. It is fascinating to have these discussions with members of our armed forces whose mission is complicated by deploying applications into the most hostile environments.

We thanked them for their service. We will be working with many of them over the next months so that they too can get insight from their data.

Posted in Business and Technology, Personal experiences | Tagged , , , , , , , , , , , , | Leave a comment

ALF, OSLC and SLI – enough of the soup, time for some meat

Posted on 2013-03-25 by

Alien Life Form but well integrated with the human species … not so much with cats however.

In TechCrunch today, Alex Williams reports on the new Eclipse project being started by TaskTop to create a common integration layer between and amongst the ALM vendors.

This has been tried before.

The Eclipse Application Lifecycle Framework project (ALF) was started to achieve the very same thing more than five years ago. Unfortunately for ALF major players in this space stayed away in order to preserve they unique stacks and not expose their solutions to interoperability.

The ALF project, led by the leading ALM vendor Serena Software, created an exceptional platform that is SOA-based to consume web-services based API’s and orchestrate the interactions between multiple technologies from a disparate collection of vendors.

Point-to-point integrations are fragile and vendors are not incented to keep their integrations up to date.So we do need a solution to tool integration that protects the users of the tools from vendors who do not keep their integrations up to date and who fail to tell their partners when their API’s are changing.

When ALF failed to gain any traction IBM weighed in with the OSLC (Open Services for Lifecycle Collaboration) in an attempt to make IBM tools more accessible to non-IBM stacks. The OSLC vendor list is impressive as one would expect when the project has the blessing, and the backing, of IBM.

The question is, do we need another open integration framework. I love our industry: we are the ONLY engineering discipline that thinks “standard” is a plural.Point-to-point integrations are fragile and vendors are not incented to keep their integrations up to date.So we do need a solution to tool integration that protects the users of the tools from vendors who do not keep their integrations up to date and who fail to tell their partners when their API’s are changing.

When ALF failed to gain any traction IBM weighed in with the OSLC (Open Services for Lifecycle Collaboration) in an attempt to make IBM tools more accessible to non-IBM stacks. The OSLC vendor list is impressive as one would expect when the project has the blessing, and the backing, of IBM.

The question is, do we need another open integration framework. I love our industry: we are the ONLY engineering discipline that thinks “standard” is a plural.

Posted in Business and Technology | Tagged , , , , , , , , , , | Leave a comment

Will we ever ‘just say “no”?’

Posted on 2013-02-28 by

Time for some tough love in IT

One of the glorious ironies of IT is that the CIO, the executive with “Information” in their job title, is frequently the least well equipped member of the management team when it comes to data. Information about project status comes from meetings with optimists (developers) and pessimists (testers). Really they should be called the Chief Opinion Officer instead.

The other glorious irony of IT is that we manually run the automated test suite! Now I have a vision of how development should be: fair warning this is an idealistic and romantic dream of a far-away development future.

Wouldn’t it be interesting if the first thing a developer did was to look at the work assigned to her, her “tickets”, right there from within her IDE. No popping out to the ticketing systems to get the daily task list.

And what if every piece of code she then touched, every change she made, was automatically “stamped” with the ticket number to give a complete audit trail for root cause analysis later.

When it came time to check code in the build tool would kick off immediately and, if it failed, would open a new, high-priority, ticket in the developer’s inbox in her IDE. Maybe even with a social media beep to get her attention.

Of course if the build is successful the automated test suite would run, AUTOMATICALLY, exercising those pieces of code that had changed or were impacted. That would mean a correlation between test scripts and code changes and an impact analysis tool that could understand upstream and downstream dependencies.

Should any of the battery of tests, functional, performance, non-functional, fail then a new ticket would be opened in the developer’s inbox.

Notice that what this means that the developer never gets to hand over code that does not build or does not pass the battery of automated tests. So what happens to the Quality Assurance team? Well, they start to assure the quality by catching everything else that gets missed.

A programmer once told me “I just spent all morning debugging the code and now I am spending the afternoon rebugging it.” The idea that we can rely on someone else to find our bugs is out dated and a very poor use of our best talent. After all “testers” didn’t put the bugs in, can’t take them out and are nobody’s clean-up crew.

We have to learn to say “no” in IT. If the requirement doesn’t make sense – send it back. If the code is incomplete and buggy – send it back. If the test plan isn’t comprehensive and is not automated – don’t test until it is. Far too often we pass our problems on downstream and when the same is done to us we grin and we bear it.

My first boss said to me when I was a newly minted programmer, “if you don’t have time to do it right when do you have time to do it over?” We should make every effort to test every aspect of the code we can before we are allowed to check it in. We have the technology available today to do that. But do we have the will?

Posted in Business and Technology | Tagged , , , , , , , , , | Leave a comment

Testing times for system designers

Posted on 2013-02-27 by

It’s never too early to test for performance

Many studies have shown that the cost of finding defects in applications increases exponentially the further the defect remains undetected through the lifecycle. Some studies (Microsoft, National Institute of Science and Technology) have estimated that this may be as much as 100 times more expensive to fix a post-deployment issue compared to finding it at the point the error was inserted.

Performance issues are pernicious because, while the application meets the functional requirements, it does not deliver that functionality within the expected (or demanded) time frame. But what really matters with performance issues is that the solution is frequently not correctable with code changes. Too often the remedy involves architectural reformation, changes to the footprint (memory usage typically) or demands for higher and more specific topologies with greater horsepower. This kind of repair is costly to do and can take a long time to effect.

In the case of a performance issue in embedded code it may just not be possible to solve without re-architecting the silicon. How many times have we been forced to upgrade the hardware to achieve reasonable performance from the software?

So performance testing early in the lifecycle is critical if project costs are to be managed. Many tools exist that can simulate load on the application as well as throttling the resources (bandwidth, data transfer speeds, latency) so that realistic expectations of the resulting performance can be observed.

Simulated normal loads are one thing but stressing an application is vital too. Seeing how an application performs under extreme loads will help us understand what the resulting performance envelope looks like so that we can design in ways to balance the load over time. It also helps us to identify where we might instrument the code so that we can give live telemetry from the application about when it is reaching execution limits that are known to cause the application to fall out of the operational envelope.

Load testing, stress testing, performance testing are all essential parts of the test lifecycle that we can do today with very sophisticated and capable tools across a wide range of platforms. With test automation it should be possible to make the code, build and test cycle fully repeatable so that performance issues are detected early enough to do something about them.

Posted in Business and Technology | Tagged , , , , , , , , , , | Leave a comment

Thinking about focus

Posted on 2013-02-25 by

Take your own path through the system

I’ve been out and about lately and seen an increasing number of organizations that are heralding their evolution into modern businesses because they have updated their infrastructure for speed and they have implemented systems to keep them aligned with the business.

In our rush to “optimize” and “standardize” we may have forgotten about “personalize” and “consumerize”. It has almost been a decade since we were talking about “The Long Tail” on a daily basis and at the board level. We did shift our businesses so that we could target our offering more directly at our individual customers. And, today, we are focused on the most personal of platforms by deploying solutions directly to the devices that our customers carry in their pockets and pocketbooks.

But did we miss something? Are we still trying to make our systems be one-size-fits-all even though the products and services those systems deliver are highly customized to individual needs?

I’d like to explore with you the idea that we should be able to personalize the process we we engage in when we do business.

For example: every time I check in at the kiosk at United it asks me if I am carrying an infant (never) and if I want to purchase additional miles (never). When I check out my groceries at Whole Foods it asks me if I want cash back (never). When I call Cigna about my healthcare from my cell phone it asks me my date-of-birth, last four of my social security number, my member number even though I only ever call from my cell phone which it knows is my cell phone.

What I am talking about here is the idea that processes need to be adaptive. They need to learn about the users and their habits and adjust and a adapt accordingly. Why should my experience, as a weekly United flyer, be the same as my mum’s, who flies once a year? Why can’t Cigna detect my phone and connect me straightaway without the phone tree maze? Why doesn’t Whole Foods give me back a few seconds every time I shop and not bother with the question about cash back?

What would a world look like where businesses adapt their processes to meet the way the customer wants to work? Maybe we need to shift our focus again.

 

 

Posted in Business and Technology | Tagged , , , , | Leave a comment