Aaron Jensen

Rhino.Mocks Performance Issue Fixed

aaronjensen — Fri, 09 May 2008 05:56:15 GMT

Just wanted to quickly note that I tracked down the performance issue in Rhino.Mocks and patched it. I also updated the original post with the new numbers. Enjoy!

Mock Framework Benchmarks

aaronjensen — Thu, 08 May 2008 15:50:50 GMT

UPDATE: I tracked down the issue and committed a patch to Rhino.Mocks. Rhino.Mocks is now much more competetive performance wise, our CI build time nearly halved, and about 4 minutes out of 7 of our test time has disappeared. New numbers below.

I've complained before that Mocking is Slow but I never really dove further into it. Today I decided to actually compare Rhino.Mocks to other mock frameworks on a pure performance basis to see if it was a global problem. I timed 2000 unit tests across 100 classes with 20 tests each. The results were a bit surprising:

Framework	TD.NET Time	nunit-console Time
Rhino.Mocks old trunk	57.36s	28.82s
Rhino.Mocks new trunk	22.94s	7.59s
Moq trunk	18.30s	5.91s
TypeMock 4.2.3 Reflective Mocks	15.36s	9.35s
TypeMock 4.2.3 Natural Mocks	16.92s	9.56s

That's right, according to these tests, Rhino.Mocks is at least 3 times slower than the other frameworks when under heavy load in TD.NET and five times slower in the console according to these tests. It's also interesting to note that TypeMock is faster than Moq in TD.NET, but slower in the console runner.

While running the Rhino.Mocks tests it is very clear that there is a degrading performance issue. All the other frameworks executed tests with a near constant speed per test, but Rhino.Mocks slowed down noticeably about half way through.

Please feel free to try it yourself, grab the project here. You should be able to just run the 4 strategy .bat files (run-rhino, run-moq, run-tmock-reflective, run-tmock-natural). Let me know if you find anything interesting.

Introducing Machine.Specifications (or MSpec for short)

aaronjensen — Thu, 08 May 2008 07:12:19 GMT

As some of you who follow me on twitter know, I've been working on Yet Another Context/Specification Framework as an experiment. Yeah, I know we already have NSpec and NBehave, and they're great and all, but MSpec takes things on from a slightly different angle, and it's just an experiment (for now). Here's a sample Description:

[Description]
public class Transferring_between_from_account_and_to_account
{
  static Account fromAccount;
  static Account toAccount;

  Context before_each =()=>
  {
    fromAccount = new Account {Balance = 1m};
    toAccount = new Account {Balance = 1m};
  };
  
  When the_transfer_is_made =()=>
  {
    fromAccount.Transfer(1m, toAccount);
  };
   
  It should_debit_the_from_account_by_the_amount_transferred =()=>
  {
    fromAccount.Balance.ShouldEqual(0m);
  };

  It should_credit_the_to_account_by_the_amount_transferred =()=>
  {
    toAccount.Balance.ShouldEqual(2m);
  };
}

And a TestDriven.NET run:

------ Test started: Assembly: Machine.Specifications.Example.dll ------

Transferring between from account and to account
  When the transfer is made
    * It should debit the from account by the amount transferred
    * It should credit the to account by the amount transferred


2 passed, 0 failed, 0 skipped, took 0.79 seconds.

Err, What?

Different eh? The idea was heavily inspired by Scott Bellware's SpecUnit.Net framework he showed at the ALT.NET conference. It also took heavy cues from RSpec and my insanity. I realize that the the code doesn't look much like C# code and I'm OK with that. Many have and will ask why I don't just use Boo or RSpec w/ IronRuby eventually or even one of the existing Context/Spec/BDD frameworks. Those are good questions, but my main motivations are tooling and syntax. I enjoy the tooling I get in C# and I personally like the syntax in this library considering the limitations imposed by C#.

How's it work?

The simplest way to describe it is to compare it to a normal *Unit style testing framework:

Description = TestContext
Context before_each = SetUp
Context before_all = SetUpFixture
Context after_each = TearDown
Context after_all = TearDownFixture
When = Also SetUp, but happens after Context before_each
It = Test

Rather than methods and attributes, MSpec uses named delegates and anonymous functions. The only reason for this is readability. You'll also notice that the fields used in the context are static. This is necessary so that the anonymous functions in the field initializers can access them. Probably the first thing you noticed is the =()=> construct. I won't mention the names that this was given on twitter, but I think it's an acceptable thing to have to deal with in exchange for the cleanliness of the rest of the syntax.

Ok, you're crazy, but how do I try it?

First, this is a very rough cut. Everything is subject to change as we experiment with the language. That said, here's how you play with it:

Grab the drop here.
Extract it somewhere. Put it somewhere semi-permanent because the TestDriven.NET runner will need a static location for the MSpec TDNet Runner.
If you want TestDriven.NET support, run InstallTDNetRunner.bat
Check out the example in Machine.Specifications.Example. Note that you can run with TD.NET.
Create a project of your own. Just add Machine.Specifications.dll and get started.
Send me feedback! Leave comments, email me, tweet me, whatever.

Also, this is part of Machine, so feel free to take a look at the code and/or submit patches. There's also a Gallio adapter in there, but I didn't include it in the release as it's not quite polished enough yet. If you're interested in it, talk to me. Special thanks to Scott Bellware, Jeff Brown and Jamie Cansdale for their help and support.

Mocking Events

aaronjensen — Sun, 13 Apr 2008 08:29:51 GMT

Currently in Rhino.Mocks, making mocks fire events and ensuring that an event on your SUT was fired are both awkward and verbose at best. Here is an example of both things at once:

[Test]
public void ViewFiresBeginDrag_Always_FiresChangedEvent()
{
  IEventRaiser raiser;
  bool eventFired = false;
  using (Record)
  {
    The<ICardView>().BeginDrag += null;
    raiser = LastCall.IgnoreArguments().GetEventRaiser();

    Target.Changed += (x,y) => eventFired = true;
  }

  using (Playback)
  {
    raiser.Raise(The<ICardView>(), EventArgs.Empty);
    Assert.IsTrue(eventFired);
  }
}

Nice eh? First, is there a better way to do either of these things that I'm missing? Please tell me if so. Next, if not, what can we do to clean this up?

Well, Ayende and I discussed this in the past and Ayende spiked it and asked for feedback. The feedback was mixed and for one reason or another it never made it into Rhino.Mocks that I know of.

Well, today while I was working on Machine.Testing and another side project, I decided to give something else a shot. What I ended up with is this:

[Test]
public void ViewFiresBeginDrag_Always_FiresChangedEvent()
{
  using (Record)
  {
    PrimeEventFiringOn<ICardView>(x => x.BeginDrag += null);
    Target.Changed += NewEventFireExpectation<EventHandler>();
  }

  using (Playback)
  {
    FireEventOn<ICardView>(EventArgs.Empty);
  }
}

Better, but I still don't think it's perfect. Also, it probably requires some explanation, so let's pick it apart piece by piece:

    PrimeEventFiringOn<ICardView>(x => x.BeginDrag += null);

=> +=??? Ugly huh? I really wish we could just refer to an event somehow without having to do +=/-=. At least I'm not using vb though... Alas, we cannot access them easily so we're stuck hacking away like this. So this particular method will basically get the mock or stub ICardView.BeginDrag ready to be fired. This needs to be done during the record phase it seems. You can only prime one event at a time per mock, so if you need to do more than one you can revert to the normal Rhino.Mocks syntax.

    Target.Changed += NewEventFireExpectation<EventHandler>();

This was a fun method. This method actually creates a new DynamicMethod in the signature required by the event and creates a new delegate. The method tracks whether or not it was fired, and at the end of the Playback phase in my fixture it will assert that all of the events were actually fired.

    FireEventOn<ICardView>(EventArgs.Empty);

This actually fires the event we set up in the Prime call. It can only be called after you've primed an event.

So, the whole thing is kind of "magic", but it's less code if you can accept the magic. I think we can make things even better though, but it'd require changes to Rhino.Mocks and it's time for bed so maybe Ayende can swing by the Eleutian office while in Seattle and we can work on it. Here's what I'm thinking:

[Test]
public void ViewFiresBeginDrag_Always_FiresChangedEvent()
{
  using (Record)
  {
    Target.Changed += Mocks.CreateEventHandler<EventHandler>();
  }

  using (Playback)
  {
    EventRaiser.Raise(() => The().BeginDrag += null, The<ICardView>(), EventArgs.Empty);
  }
}

There's probably more you can do with the CreateEventHandler syntax like add more specific expectations, assert its not fired, assert that it's fired X times, etc. The EventRaiser syntax is ugly, but it doesn't involve strings or the fire from the right hand side like the syntax I mentioned at the beginning of the post.

You can get the source here, but I warn you it's first draft and there are hacky bits.

Oh, and here's some example tests using the TestsFor fixture:

[TestFixture]
public class CardPresenterTests : TestsFor<CardPresenter>
{
  private Card _card;

  public override void SetupContainer()
  {
    Override<ICardView>(With.Stub);
  }

  public override void BeforeEachTest()
  {
    _card = new Card(0);
  }

  [Test]
  public void OnBeginDrag_Always_SetsIsInFluxToTrue()
  {
    using (Record)
    {
      PrimeEventFiringOn<ICardView>(x => x.BeginDrag += null);
    }

    using (Playback)
    {
      FireEventOn(EventArgs.Empty);

      Assert.That(The<ICardView>().IsInFlux, Is.True);
    }
  }

  [Test]
  public void OnBeginDrag_Always_FiresChangedEvent()
  {
    using (Record)
    {
      PrimeEventFiringOn<ICardView>(x => x.BeginDrag += null);
      Target.Changed += NewEventFireExpectation<EventHandler>();
    }

    using (Playback)
    {
      FireEventOn<ICardView>(EventArgs.Empty);
    }
  }
}

Book Review: Hacking Vim

aaronjensen — Thu, 10 Apr 2008 05:01:41 GMT

Some time ago, Packt Publishing sent me a copy of Hacking Vim: A Cookbook to get the Most out of the Latest Vim Editor by Kim Shulz to review. I read through it pretty quickly and I must say I thought it was a good book for the most part. The book is definitely focused at a very niche audience as there aren't a lot of us Vimmers around. Furthermore, I can't say I'd recommend the book for the casual Vimmer or those that use only ViEmu in Visual Studio. If, however, you are a moderate to hardcore Vim user or are interested in becoming one I would definitely recommend grabbing a copy.

There is a lot of information in this book. It wastes no time or space describing the many ways to make your Vim experience even more efficient than it already is. Because of its very cut and dry style it reads more like a reference book than anything else, but I think that's perfectly fine for this style of book. It is essentially a slightly more readable, more detailed and better written version of a lot of the Vim help. It's also nice because in many of the chapters, especially the first few ("Better Navigation" and "Production Boosters") you can pick up a lot of interesting tips. I even learned a few handy commands I had no idea about.

There's plenty of good information on writing Vim plugins and customizing Vim that I wish I would have had six years back when I wrote Vim plugins. Ah, looking at that code brings back fond memories, mostly involving learning Regular Expressions through trial and error and I actually used /// comments in C# code, but I digress.

I also liked the stuff on firing Ruby from Vim--it reminded me I need to get a Ruby environment set up in Vim. Oh, and just for fun it seems, Kim even included an appendix showcasing some of the ridiculous things that have been written for Vim like Nibbles and Rubik's Cube. Heck, I just found a Twitter client for Vim (yea, I twitter now in case you care).

There are a few things wrong with it though. Some things are too close to the Vim built-in help and you're probably better off just looking there. Also, the index isn't very useful and it'd be nice to have a searchable version of the book as well, so maybe the eBook version is a good idea. I wish all book companies would bundle eBooks with paper books, but, I digress again. Like I said, there is a lot of information in this book. It's not for everyone, but I think most Vim users will find plenty of useful stuff in its pages.

Separate Stub and Verify != Duplicate code necessarily

aaronjensen — Thu, 03 Apr 2008 16:43:00 GMT

Daniel Cazzulino, author of Moq posted a good comment on my last post where I suggested looking into a Mockito like syntax for .NET Mock Frameworks.

On the surface, Mockito's approach seems good. But if you do the "true" comparison, you'll see that stub(...) is exactly the same as mock.Expect(...) in Moq.

Then, when you do verify(...), you have to basically repeat the exact same expression you put in stub(...). This might work if you only have a couple calls to verify, but for anything else, it will be a lot of repeated code, I'm afraid.

I thought this too. See my comment here from a month ago. Szczepan made a good point and I've thought about it more since then.

When combined with my position on loose vs strict mocks (almost always use loose), I'd say that *most* of the time you are either stubbing or verifying. Meaning, if you're verifying you don't need to stub unless of course that method returns something that is critical to the flow of your test, in which case you don't really need to verify, because the flow would have verified. That's a mouthful, but does that make sense?

I haven't used mockito, and I know there are times I use Expect.Call with return values that matter (which would essentially require you to duplicate stub & verify), but maybe that's a smell? Maybe if you think you need that you can do state based testing or change your API?

Here's an example Test using Rhino.Mocks:

[Test]
public void SomeMethod_Always_CallsSendMail()
{
  IMailSender sender = mocks.DynamicMock();
  UnderTest underTest = new UnderTest(sender);

  using (mocks.Record())
  {
    Expect.Call(sender.SendMail()).Returns(true);
  }

  underTest.SomeMethod();

  mocks.Verify(sender);
}

And some code this is testing (obviously not test driven, but you get the idea):

public void SomeMethod()
{
  if (!_sender.SendMail())
  {
    throw new Exception("OH NOS");
  }
}

Notice that here we would need to stub and verify separately with Mockito like syntax. This would look something like this:

[Test]
public void SomeMethod_Always_CallsSendMail()
{
  IMailSender sender = mocks.DynamicMock();
  UnderTest underTest = new UnderTest(sender);

  Stub.That(() => sender.SendMail()).Returns(true);

  underTest.SomeMethod();

  Verify.That(() => sender.SendMail()).WasCalled();
}

This may violate DRY, but what if you designed your API differently? Maybe SendMail should throw an exception on failure instead of returning a boolean? This would make the return value unnecessary and remove the need for the Stub call. Clearly you can't always do this, especially with unwrapped legacy or API code, but it's something to think about.

Also, I think you shouldn't be verifying more than one method generally to go along with the one assert/test rule, so a single repeat would not be that horrendous. Heck, you could even do:

[Test]
public void SomeMethod_Always_CallsSendMail()
{
  IMailSender sender = mocks.DynamicMock();
  UnderTest underTest = new UnderTest(sender);

  Stub.That(var sendMail = () => sender.SendMail()).Returns(true);

  underTest.SomeMethod();

  Verify.That(sendMail).WasCalled();
}

I think the syntax would lead to better, more concise tests. But maybe it would just be too annoying? I wouldn't know until I tried it for a while I guess.

The Past, Present and Future (?) of Mocking

aaronjensen — Thu, 03 Apr 2008 04:12:26 GMT

(Note: I'm going to speak about .NET mock projects here for the most part, but most of them have Java quasi-equivalents.)

The original mocking frameworks like NMock required you to setup expectations by passing strings for method names. This was fragile and made refactoring more difficult.

A few mock frameworks now allow you to define expectations and mock results in a strongly typed manner. Rhino Mocks and TypeMock use a record/replay method to setup expectations. The record replay method is mostly necessary because the same calls are made on the same objects under two different scenarios. This leads to a few issues.

The first issue is confusion and barrier to entry. Many people have complained that the Record/Replay method is not straight forward and the whole paradigm is confusing. There are also complains about the naming, are you really recording and then replaying? It's just kind of a strange thing. Of course most of us learn to live with it, understand it, and accept it for what it is. Recently though, a few mock frameworks have popped up that do away with this model.

In the .NET world we have Moq. Moq gets rid of the need for record/replay because recordings have a very different syntax. They use lambdas instead of actual calls to the mock object. This allows the framework to know when you are recording an expectation and when you are fulfilling an expectation. It adds a bit of noise in the form of "() =>" but all in all it's not bad. Of course this requires C# 3.0, but it's good to keep looking ahead.

In the Java world we have Mockito. Mockito also does away with the record/replay model but it does it in a different way. At first I wasn't a fan, but thinking about it more, I like it. Mockito has two main apis, stub and verify. Stub is equivalent to SetupResult.For, and verify is equivalent to Expect.Call with a verify. The interesting bit is that the stubbing happens before the the class under test is invoked and the verifying (which includes describing the method to be verified) happens after the class under test is invoked. This is best shown with an example stolen from the Mockito site:

  
  //stubbing using built-in anyInt() argument matcher
  stub(mockedList.get(anyInt())).toReturn("element");
  
  //stubbing using hamcrest (let's say isValid() returns your own hamcrest matcher):
  stub(mockedList.contains(argThat(isValid()))).toReturn("element");
  
  //following prints "element"
  System.out.println(mockedList.get(999));
  
  //you can also verify using argument matcher
  verify(mockedList).get(anyInt());

Obviously it would take a bit of imagination to arrive at a .NET equivalent, but you get the idea. I like this because the normal test structure is Setup Stuff->Do Stuff to Test->Verify Stuff did what it should have. The normal record/replay model requires you to set up verifications before you actually Do Stuff (though you call VerifyAll afterwards). This is a bit less natural. I feel syntax like this (yeah, I like the new NUnit syntax) would be more intention revealing:

Assert.That(() => someMock.Foo(), Was.Called);

Or:

Verify.That(() => someMock.Foo()).WasCalled();

Then you would stub like this:

Stub.That(() => someMock.Bar()).Returns(3);

(Note: No idea if this is feasible or makes sense or not, my lambda experience is limited to light reading, but you get the idea. I'm sure the syntax could also be prettier.)

Rhino.Mocks is my current mock framework of choice. I'm used to it, I've lightly contributed to it, and I've been working with it for a while now. Despite that, I do think that there is definitely more to explore in the mocking arena especially with C# 3.0.

There are lots of other fun things to talk about too... like TypeMock's magic, but that's another day still...

Moq's MockBehavior

aaronjensen — Wed, 26 Mar 2008 16:29:14 GMT

Moq has been getting some press lately because it's the newest mock framework on the block. I think it's certainly interesting and I'll have more to say on it later, but I wanted to briefly complain about one aspect real quick.

Moq touts that it has a more simplified API when compared to something like Rhino.Mocks. The API has a single entry point which certainly aids discoverability, but I question one of the design decisions. I remember seeing someone say that Rhino had too many types of mocks and that that was confusing. Well, I don't think it has this many different Mock Behaviors:

  public enum MockBehavior
  {
    Strict, 
    Normal,
    Relaxed,
    Loose,
    Default = Normal,
  }

Why have this many? Does anyone know what they do just by looking at them? At least they're documented, but the docs are quite a mouthful:

  public enum MockBehavior
  {
    /// 
    /// Causes the mock to always throw 
    /// an exception for invocations that don't have a 
    /// corresponding expectation.
    /// 
    Strict, 
    /// 
    /// Matches the behavior of classes and interfaces 
    /// in equivalent manual mocks: abstract methods 
    /// need to have an expectation (override), as well 
    /// as all interface members. Other members (virtual 
    /// and non-virtual) can be called freely and will end up 
    /// invoking the implementation on the target type if available.
    /// 
    Normal,
    /// 
    /// Will only throw exceptions for abstract methods and 
    /// interface members which need to return a value and 
    /// don't have a corresponding expectation.
    /// 
    Relaxed,
    /// 
    /// Will never throw exceptions, returning default  
    /// values when necessary (null for reference types 
    /// or zero for value types).
    /// 
    Loose,
    /// 
    /// Default mock behavior, which equals .
    /// 
    Default = Normal,
  }

I'm of the opinion that you should only have one type of Mock, and that's what Rhino calls Dynamic and Moq calls Loose. I described why here. If I wanted to simplify mocking, I'd start here.

The Eleutian Guys join the CodeBetter Guys

aaronjensen — Wed, 19 Mar 2008 16:42:00 GMT

As Dave so promptly pointed out immediately after our grueling hazing that I still have not yet recovered from, Jacob and I have been invited to join CodeBetter. I'm very excited to do so, thanks to everyone at CodeBetter!

I'm going to cross-post to both our Eleutian Blog and my CodeBetter Blog for now, so don't worry about adjusting your subscriptions unless you start seeing double... and don't want to read my posts twice.

Thanks again to CodeBetter for having us!