Jeremy posts in his blog about his replacement for StoryTeller about the pain of FIT based tools. We have been using Fitnesse on the current project and while there has certainly been pain, I think that the pain has been that all tests experience when they start to run the system end-to-end. Suddenly all those technology framework concerns we eliminated in TDD so that we could just focus on testing our domain model, come back into play. Once again we are dealing with setting up and tearing down databases, touching the file system etc.
Our decision to use Fitnesse on this project came from two forces. First was the general lack of satisfaction with our functional testing process. Up to now functional testing had been a combination of xUnit integration tests, custom built tools, and manual testing. Previous attempts with FIT has always drawn a blank. The result was that 'release hardening' took a lot longer than we would desire and smelt of waterfall style testing phases. Second, we wanted to have a better way to communicate the acceptance criteria for the story from the users to BAs, testers and developers such that we were doing story test driven development or customer test driven development and pushing our code from the agreed acceptance criteria.
We got a real lift in this direction from Gojko Adzic. Gojko spoke for us at London .NET User Group. I got answers to a lot of the fixture issues I had had with FIT. After that talk I read through Gojko's book on Fitnesse, which reinforced and clarified the message, and was re-enthused by this tool. It's far from perfect, roll on better tools from Jeremy, but it does give us the placeholder for the story acceptance criteria to drive development.
Like everything it is taking a while for us to refine our use of Fitnesse. One key change for us came from internal conversations and discussions with Gojko - the idea of a three-way meeting. Gojko came in to deliver this presentation at my current outfit (worth watching the video) which explains his thinking. To give my understanding of Gojko's point: the important part of using a tool like Fitnesse for story test driven development is the communication that occurs between customer and implementers. In a case where you may have customer -> BA -> developer -> tester in the mix, the potential for lossy information transfer and misunderstandings is large. Remember the classic swing cartoon? While agile provides some relief by making sure that requirements->development->testing all happen together, thus shortening the feedback loop that leads to failed requirements intepretation, simple 'transcription' errors between people can lead to issues. By getting everyone to review an executable form of the specification then the possiblity of one of us going astray is far less. To facilitate this we now begin work on any story within the iteration with a three-way meeting between BA->developer->tester to ensure that everyone agrees on what the acceptance criteria for the story are. Of course if there are more stakeholders, you may need to increase the size of this meeting. The Fitnesse test becomes a record of that conversation that everyone involved can check, but in many ways the fact that using a tool like Fitnesse forces us to have the conversation and reach agreement may be as important as the result - the excutable specification - is.
The result of the three-way has been that we are finding it easier to implement Fitnesse tests. We still have some problems of our own making (related to how we seed data for tests), but this seems to have removed a lot of the pain around these for us. The suggestion might be that a number of our issues with story test tools has been simply a reflection of problems communicating requirements.