If at first you don’t succeed…try another pair…or two…or the whole team
We have had a performance issue floating around at work where one operation was taking about 15-25 times longer than it did on previous versions. Obviously an issue that needed to be addressed. A couple of months ago, when we first identified the slowdown, we wrote a story card to find out how to speed it up.
Pair #1 picked up the card, and identified a way to speed it up some, and implemented their idea. Well, it was a little faster, but it broke a bunch of tests. They go back in and make a couple more changes. Now only a few tests fail. Since we had already spent more time than we should have trying to fix the tests, we decide to back out the changes, and have another pair take a look at it.
Pair #2 makes similar changes, but this time they don’t break the tests. It speeds up the operation a bit, but it is still far slower than it should be. At this point we write up a card to see if there are any other ways that we can speed it up.
Pair #3 finds that the we are not accessing the database in the most efficient way, and implement changes to reduce the inefficiency. The operation is faster, but still not fast enough.
Then comes this week when we have Don Roberts in as a consultant. We figured that since this is why we bring in highly paid consultants, we should let him take a look at it. Sam and Don picked up the card and ran a profile to see where the slowdown was. Initially, looking at the profile generated no new ideas. Digging a little deeper, we found that it was updating around 1/3rd of the records in a table, much more than we would have expected. Bringing in our resident expert Kim, we find out that most of the values it was saving were values that we calculate anyway. So we change it to only save the values that we need to be able to calculate the others, and about 99% of the updates disappear. Now the operation is much closer, and in some cases faster than it was in previous versions.
This one issue is really a testament to the value of pair programming and the agile process. If we all programmed in our own little cubes, we probably would have eventually found the issue, but it would have taken so much more time.