My team uses FitNesse extensively to create automated acceptance tests for our systems. Recently we've added Selenium RC running from within FitNesse to our toolbox. There's a lot of buzz lately over the creation of Domain Specific Languages (DSL), driven in no small part to create a syntax for the application code that is readable by the business partners and testers. We accomplish the same goal (or at least try to) by using FitNesse fixtures to effectively create a DSL for our applications that our tester can use to specify and verify the expected behavior of the system. FitNesse is both the "language" parser and a reporting framework that can already be plugged into NAnt and CC.Net, so there's no real need to create custom interpreters. Take that you Ruby cool kids with your "missing_method" metaprogramming magic!
Most of my last week and a half has been spent building the "Mother of all Automated Tests." The system under test was basically a new messaging workflow between our company and a partner company for something that we call a "Bundle." All told, the entire Bundle pipeline includes:
- A web application on our side to start the Bundle workflow and approve or reject work from our partner
- An extension to our messaging broker component that runs as a polling windows service
- A web service, that we're building, installed on server at our partner's that will receive notifications of a new Bundle for them to process
- A web service on our side that will receive a pair of notifications from the partner, including the finished product. This web service enforces a wide range of business rules
The communication pipeline is fairly complex, there isn't a user interface for much of it, the web service messages are non-trivial, and the validation rules are extensive. Creating a FitNesse grammar for us and our tester to simulate the communication has been a lifesaver. Using our DSL we can quickly throw a lot of different scenarios at the system in a repeatable manner.
The DoFixture is the Lynchpin
FIT (and FitNesse) has always had the ability to express tests as declarative tables. It works great when you can stay within the lines of x number of inputs leads to y outputs, but it breaks down for more complex scenarios like, well, basically anything in an enterprise application. It's also a mess to maintain state between fixture tables because all you have are static members. All of that changed because now we have the DoFixture. I think I've said before in this blog that I didn't particularly care much for FIT style testing until we got the DoFixture from Rick Mugridge's FitLibrary extension.
The DoFixture allows us to create custom textual languages to express tests. In other words, jagged tables. I'll leave it for a later post, but the DoFixture allows us to nest other fixture tables to do declarative assertions. Think of the DoFixture as an interpreter that enables us to quickly create grammars without getting bogged down by writing our own parsing and reporting functionality. Here's an example straight out of our tests on the Bundle workflow.
public class CreateBundleScreenFixture : DoFixture
{
public void CreateBundleFromToAsAWithInvoiceTotalWithAttachment(
string senderId,
string receiverId,
string communicationType,
decimal invoiceTotal,
string fileName)
{
// call into your application's service layer with the inputs above
}
}
That's the Fixture code, it's nothing special, just a method. Let's move onto the representation in a FitNesse table. In FitNesse's particular version of Wiki markup, "!" is a directive to start a new table. New table cells are started just by delimiting the contents with a "|" character. Just to avoid some unpleasantness, always remember to close each line with a pipe bar and no trailing spaces. I'm going to make all of the FitNesse sample in Wiki text because that's what you'll be editing (and the blog engine will screw up the table formatting).
!|CreateBundleScreenFixture|
| Create Bundle From | Sender1 | To | Receiver1 |
as a | Fax | With Invoice Total | 1018.23 |
With Attachment | Invoice1.pdf |
The DoFixture parses this command by treating the table row as an array of cells. Starting with the first cell, DoFixture iterates through every other cell, concatenates the text together, and removes all of the white space. The DoFixture uses this concatenated string as the method name, which is located through reflection (lots and lots of reflection). Back to the other cells that got skipped over, DoFixture assumes that these cells are the arguments to the method in order. The DoFixture is able to determine the type of the argument from the method signature and coerce the types. In this case the DoFixture automatically converts the string "1018.23" into a decimal type before pushing it into the invoiceTotal argument of the CreateBundleFromToAsAWithInvoiceTotalWithAttachment() method. FitNesse itself in .Net handles all of the normal primitive types, including DateTime values. In an annoying oversight, the .Net version misses enumeration types. If anybody is interested, I can show some code to plug into FitNesse to modify the way FitNesse parses and handles any particular type.
So that's how you create an action in a DoFixture. FitNesse will of course fail the test if the action blows up with an exception, but you also need to make assertions about the outcome of the action. The simplest thing to do is create a method that returns a Boolean. The DoFixture will automatically treat any method with a Boolean return value as an assertion. Without any arguments, that might look like:
public bool TheBundleWasCreatedSuccessfully()
{
return _lastBundleSucceeded;
}
!|CreateBundleScreenFixture|
| Create Bundle From | Sender1 | To |
Receiver1 | as a | Fax | With Invoice Total |
1018.23 | With Attachment | Invoice1.pdf |
| The Bundle Was Created Successfully |
Fancier assertions can be made by using the Check keyword. Let's add some grammars to test the Status of the newly created Bundle object.
public void AliasTheLastBundleAs(string bundleName)
{
// store the Id of the last Bundle created in the FitNesse fixture
// as bundleName
}
public string TheBundleStatusOfIs(string bundleName)
{
// look up the Status field from the Bundle aliased
// as [bundleName] and return
}
!|CreateBundleScreenFixture|
| Create Bundle From | Sender1 | To | Receiver1 |
as a | Fax | With Invoice Total | 1018.23 |
With Attachment | Invoice1.pdf |
| The Bundle Was Created Successfully |
| Alias the Last Bundle As | TestBundle1 |
| Check | The Bundle Status of | TestBundle1 | Is | New |
By starting the test row with "Check," I've directed DoFixture to assert that the value of the TheBundleStatusOfIs("TestBundle1") method is the value in the last cell. When the test is executed, FitNesse will color the last cell green if it succeeds, or red with an NUnit-like "expected blah, actual blah" text description. One of the best advantages to FitNesse testing is the visceral red/green output. It's usually pretty easy to spot the test failures (diagnosing the cause, is alas, as difficult as always).
More complex assertions usually call for a nested Fixture table. I'll demonstrate a couple examples in later sections. If you can't already guess, I love the DoFixture.
Environment Settings
At some point FitNesse and Selenium have to know some environment settings. You can accomplish this through the normal configuration techniques, but I've started to put a lot of this type of information in the FitNesse tests themselves. FitNesse allows for a "SetUp" test page that will be included inside each test page in a test suite. I used a little DoFixture that looks something like this:
public class MySystemEnvironmentFixture : DoFixture
{
// Yes, I do realize that I'm using a public static field
// It's okay -- in this context anyway
public static string ApplicationRoot = "http://localhost/MyApp/";
public static string WebServer = "http://localhost";
public static string BrowserName = "*firefox";
private static DefaultSelenium _selenium;
public void TheWebServerIs(string webServer)
{
WebServer = webServer;
}
public void TheApplicationIsRootedAt(string root)
{
ApplicationRoot = root;
}
public void TheBrowserIs(string browserName)
{
BrowserName = browserName;
}
}
Inside the SetUp page I have a table like this:
!|MySystemEnvironmentFixture|
| The Web Server Is | http://MyTestServer/ |
| The Application Is Rooted At | http://MyTestServer/TheTestingRoot/ |
| The Browser Is | *iexplore |
In one place, I can quickly shift the FitNesse fixtures from pointing from a local installation on my workstation, to the development servers, or finally to the testing servers. If you'll notice the third line, I can also switch the browser engine that Selenium is using (IE, Firefox, Opera, etc.). That's awfully important in the age of AJAX/Web 2.0 tricks with multiple browsers.
Integrating Selenium into a DoFixture
Finally, let's pull Selenium into the mix to drive the web pages. While you can use Selenium with its own FIT-like driver, or though NUnit, I like running Selenium from within FitNesse for acceptance tests. I like this approach because we get a unified reporting and execution infrastructure for both UI and Service Layer tests. Plus, and this cannot be underestimated, I can use regular FitNesse fixtures to setup and verify application state before and after the Selenium execution in a single self-contained test.
We're using the .Net driver for Selenium RC. I wrote a brief introduction to using Selenium RC from .Net that explains the basic mechanisms.
The first thing to do is create a new Selenium session and make it accessible to all of the Fixture's in a test. I'm working on an assumption that we will normally write a separate FitNesse fixture for each non-trivial screen, and share the Selenium driver object in a static member somewhere in memory. I'm going to start by adding some infrastructure to the MySystemEnvironmentFixture just to provide access to a DefaultSelenium object.
// Create a new Selenium session
private static void resetSelenium()
{
// Use the ApplicationRoot and BrowserName fields
_selenium = new DefaultSelenium("localhost", 4444, BrowserName, ApplicationRoot);
_selenium.Start();
}
// Provide access to the currently executing Selenium session
// to other screen fixtures
public static DefaultSelenium SeleniumDriver
{
get
{
if (_selenium == null)
{
resetSelenium();
}
return _selenium;
}
}
// Start a Selenium session, probably call this
// in SetUp
public void StartApplication()
{
resetSelenium();
}
// Shut down the Selenium session and free the
// resources. Either do this in TearDown or at
// the end of every, single test.
public void CloseApplication()
{
if (_selenium != null)
{
_selenium.Stop();
}
}
Since every call to "Start Application" will spawn a new browser instance, you should definitely remember to call "Close Application" at the end of each test or in the TearDown. Unless you don't mind out of memory exceptions on your build server;)
!| MySystemEnvironmentFixture |
| Start Application |
| Close Application |
Now, let's create a little fixture to for a screen. In our Bundle workflow user interface we have a screen that displays all of the data about a single Bundle and has buttons to perform common actions on a Bundle like "Reject" or "Approve."
public class BundleDetailsFixture
{
// Elements on the screen
private const string STATUS_LABEL = "xpath=//*[@testid='status']";
private const string APPROVE_BUTTON = "xpath=//*[@testid='status']";
private const string REJECT_BUTTON = "xpath=//*[@testid='status']";
private const string URL = "BundleDetails.aspx";
DefaultSelenium _selenium;
public BundleDetailsFixture()
{
// Go grab the current Selenium session object
_selenium = MySystemEnvironmentFixture.SeleniumDriver;
}
public void OpenTheBrowserToBundle(string bundleName)
{
// Find the Id for the Bundle aliased as [bundleName]
long bundleId = 0;
string url = string.Format("{0}{1}?Id={2}", MySystemEnvironmentFixture.ApplicationRoot, URL, bundleId);
_selenium.Open(url);
}
public string TheBundleStatusIs()
{
return _selenium.GetText(STATUS_LABEL);
}
public bool TheApproveButtonIsEnabled()
{
return _selenium.IsEditable(APPROVE_BUTTON);
}
public bool TheRejectButtonIsEnabled()
{
return _selenium.IsEditable(REJECT_BUTTON);
}
// Click the Reject button, and then click Cancel
// on the confirmation popup box
public void ClickTheRejectButtonAndCancel()
{
_selenium.ChooseCancelOnNextConfirmation();
_selenium.Click(REJECT_BUTTON);
}
public void ClickTheApproveButton()
{
_selenium.Click(APPROVE_BUTTON);
}
}
The simplest case is to call the service layer first to create a Bundle. Next we'll look at the Bundle Details screen to confirm the status of the Bundle, then approve the Bundle and check the status again.
!|CreateBundleScreenFixture|
| Create Bundle From | Sender1 | To | Receiver1 |
as a | Fax | With Invoice Total | 1018.23 |
With Attachment | Invoice1.pdf |
| The Bundle Was Created Successfully |
| Alias the Last Bundle As | TestBundle1 |
Now, call into the screen. I'm assuming that the Selenium session was created in the FitNesse SetUp
By the way, you can quite happily mix prose with your testing tables
!| BundleDetailsFixture |
| Open the Browser to Bundle | TestBundle1 |
| Check | The Bundle Status Is | New |
| The Approve Button Is Enabled |
| Click the Approve Button |
| Check | The Bundle Status Is | Approved |
That's pretty simple stuff. Just about everything you need to check or manipulate on a web page is accessible from the DefaultSelenium object. If it's not, you can send over your own Javascript to be evaluated in the browser. I haven't bumped into anything that required that technique yet. Selenium itself will largely take care of page transitions and postbacks for you. If need be, you can direct Selenium to wait for a page transition with a timeout, or poll for a certain condition if you're doing heavy AJAX work.
One of the other advantages for wrapping Selenium within FitNesse is to abstract away the page structure from the test writer. The test writer, who may or may not have knowledge to the application's internals, can focus on the English language definition of the test and defer the xpath or id location of elements to the developers. Because ASP.Net likes to munge up element id's of nested controls, we've adopted a convention of using an arbitrary "testid" attribute on an element to be the Selenium key for testing. We can find an element with a testid anywhere on the page by using a locator like "xpath=//*[@testid='the test id']."
The FitNesse wrapper gets really handy when you want to perform multiple steps in a single sentence, like repeatedly filling in a form in the same steps but with different input. Here's a cool example I haven't needed yet, but might soon. With most testing frameworks you have to setup an expectation for what to do when the next popup box shows up before you define the step that creates spawns the popup button. That's potentially confusing, and it certainly makes your testing harder to understand. With the DoFixture we can write a better grammar for that case:
!| BundleDetailsFixture |
| Open the Browser to Bundle | TestBundle1 |
| Check | The Bundle Status Is | New |
| Click the Reject Button and Cancel |
| Check | The Bundle Status Is | New |
Wrapping Up
I wrote this in a hurry and probably missed quite a few details. If you have any questions, or better yet, suggestions, feel free to leave a comment and I'll get to it quickly for once.
It's just not that hard. The hardest part of the giant end-to-end test was really the deployment scripts. All I've shown here is the mechanics of simple cases. In the next installment, I'll talk about the pattern I use to load data in FitNesse tests, and using nested Fixture's within a DoFixture to do more complex data entry and assertions. I'll also try to get a post in on lessons learned and best practices, but I'm going to wait until the big end-to-end testing infrastructure is completely verified;)