One of my personal project that started off as a thought experiment is slowly turning into reality.  One of the ideas was to see what would happen if I turned up all the best practices to 11  and see what happened. The other part of it was that it was an excuse to use some technologies that I wasn’t that familiar with and learn something new. There were a couple of surprises, some of them very pleasant.  Among them was how much starting load and performance testing early impacted the architecture of the project.

The Expected

There are some things that go in the “of course” category when you think about doing early performance testing like thinking about the deployment environment and the ops toolkit. One of my goals for this project was to get seamless deploys to both a testing and production environment. Creating a simple set of scripts and automating the install process was straightforward.  Incrementally maintaining the scripts was much less of a time sink than creating a set of install scripts at the end.

Another of the expected items was that minimizing the amount of javascript required, and designing the javascript for testability improves performance.  Basically, this came about with the realization that the de-facto MVVM pattern was a performance drag.  Transforming a data object to a javascript model to get transformed again into HTML elements is a slower process. When you can transform it once from a data object into a final HTML form, it’s a lot faster. That doesn’t mean that you have to necessarily have to have the HTML directly displayed on the screen immediately.

The Unexpected

There were several things that I found out so far that I didn’t expect. Foremost among them was how quickly I could discover race conditions.

In the wild, race conditions tend to be rather more rare.  It’s not very often that two or more users are updating the same records at the same time. Since the project was an experiment in collaboration, race conditions would be a lot more common. Without automating some performance tests, there were race conditions that as a single user, I wouldn’t have been able to find easily.  However, when I could run a scenario with a simulated load of a couple of dozen users race conditions expose themselves more quickly.

The next unexpected part of the process was that having performance based unit tests impacted the design early in the process. Thinking about what would happen when there were multiple users attempting to update had a big effect on the design in the following ways:

1. Eliminating or removing potential race conditions without resorting to mutexes or other ways of forcing serialization, moving serialized access to resources out of band so that responses can be returned more quickly.
2. Redesigning interactions to be more atomic. Basically, one of the early problems discovered was that multiple actions that updated the same general set of data could interfere with each other.  The solution to this was to make the messages more targeted and atomic. Reducing the area of the dataset that the message interacts with reduces the chance of collision between messages, even if the messages are simultaneous.

In the End

Ultimately, this project has been an interesting experiment and it will be fun to see where it goes.  Adding performance testing early in the process had very beneficial changes to the architecture that would have been prohibitively expensive if they were made later on.  Refactoring messages would have been difficult to do at the end, but in the beginning the changes were minor. Like doing test driven development, adding performance testing into the mix earlier produces useful results.

Ultimately, like any other kind of testing, performance testing is only as good as the tests. If the tests don’t really mirror the scenarios that happen in real life, they’re not useful. One of the things that will have to be done with this project is to update the performance tests to mimic how the application actually gets used “in the wild”. The operating costs of any kind of non-trivial testing will need to be a line item to maintain going forward.