tigretigre.com

So you've built out your unit tests with mocks and everything is looking peachy. But the next thing you know you have to extend a class here or there, tweak a little functionality. Maybe the cases that you so carefully mocked are suddenly no longer valid. Or maybe a few assumptions crept in that never really are valid in the first place. If your test suite is comprehensive enough you'll certainly catch it eventually, but possibly only after a monstrous cascade of test failures that don't make it clear where the real problem is. How can you guard against this sort of class-mock drift?

I have an idea that I'd like to throw out and see of it gains any traction. For now I'm calling it Two-Way Interface Mocking (TWIM). The idea is that an interface specification could be used both to generate tests for a class as well as to generate mocks for that class that ensure that they are only being called in ways that the tests confirm are valid. You can codify the givens and the expected results for the test, and turn around and make a mock that expects certain inputs and produces given results.

The key trick to this is making sure that all givens can be inverted into expectations. In some cases, it's quite simple: check a value vs. set or pass in a value. However it will often require some careful coding to define other inverses. Getting the result of a complicated SQL is much simpler than inserting or updating the tables that would make such a result possible.

So borrowing from the Behavior Driven Development model, I suggest interface definitions to consist of classes, with methods that represent their use as "givens" and other methods to represent their use as "expectations". There may be some simpler cases where we could automatically perform the necessary inversion, but for most interesting interactions it will probably have to be pretty customized.

The Three-Body Problem

Now the real fun begins. Let's say we want to use two mocks defined in this way. We need to guarantee that the two are used in a consistent fashion or we've undermined our whole effort. This means that our mock interfaces need to be defined in a rich enough manner that we can model their interactions behind the scenes. This is sounding like a lot of work. Why bother using mocks at all if it's going to take basically reimplementing the whole system as a mock first?

I think if we are separating our systems along appropriate lines, we should be able to define our interactions in terms that do not require deep implementations. The trickiest part of this actually is when multiple systems use another shared system, such as a database, where actual interactions are worth testing rather than modeling. Anything more than the most trivial database query should be tested against an actual database schema, with some semblance of real data.

So if the system under test needs access to data that your mock interfaces must create, you can spend a lot of cycles getting that data right. My strong suggestion is to build up this data set programmatically rather than by way of fixtures or the like, as it is less brittle when the need arises to change how the data is constructed. This is especially common if you have multiple systems being mocked that are modifying related data. Trying to maintain parallel fixtures is a world of madness you don't want to touch, but syncing up programmatically generated data is much more maintainable in the long run.

Next Steps

I've been very short on details so far. Since I'm working in a couple different platforms, I don't at the moment have the resources to do a deep dive and hammer too much out. My guess is I'll end up making my first attempts at some of this in Rails, as I feel I'll have some good testing frameworks to leverage. If you want to join the effort, let me know!