Joshua Flanagan

Arrange Act Assert with StructureMap RhinoAutoMocker

Joshua Flanagan — Fri, 26 Sep 2008 03:03:18 GMT

Fresh on the heels of finally figuring out how to make the AAA syntax in Rhino.Mocks work, I've added support to StructureMap's RhinoAutoMocker. If you pass MockMode.AAA to the constructor of your RhinoAutoMocker, all of the mocks that it creates will be in replay mode. This is required to successfully use the AssertWasCalled extension method. By default, if you do not pass a MockMode to the constructor, mocks will be created in record mode so that you can continue to set expectations the old fashioned way.

[Test]
public void TheAutoMockerOptionallyPushesInMocksInReplayModeToAllowForAAAsyntax()
{
    var autoMocker = new RhinoAutoMocker<ConcreteClass>(MockMode.AAA);

    autoMocker.ClassUnderTest.CallService();

    autoMocker.Get<IMockedService>().AssertWasCalled(s => s.Go());
}

The code is in the trunk, and will be included in the eventual StructureMap 2.5 release.

Updated Oct 8, 2008: The MockMode enumeration to enable AAA syntax has been changed to MockMode.AAA to be more self-explanatory.

The Rhino.Mocks' AssertWasCalled method does work

Joshua Flanagan — Thu, 25 Sep 2008 15:08:28 GMT

This behavior is probably clearly specified somewhere, but somehow it has been non-obvious to the four people on our team to the point that we were leaning toward banning its use. I didn’t want to go that far, so figured I better figure out the rules.

I’m referring to the AssertWasCalled extension method in Rhino.Mocks that allows the Arrange/Act/Assert style of testing. We found that sometimes it would behave how we expected, and sometimes it would not, with no apparent explanation. I finally realized that it probably had something to do with how the mock objects were created, so I wrote this simple test:

        1: [TestFixture]

       2: public class AssertCalledTester

       3: {

       4:     private Dictionary<string, IDependency> dependencies;

       5:     private MockRepository mocks;

       6:  

       7:     [SetUp]

       8:     public void Setup()

       9:     {

        mocks = new MockRepository();

        dependencies = new Dictionary<string, IDependency>()

       {

           {"GenerateMock", MockRepository.GenerateMock<IDependency>()},

           {"GenerateStub", MockRepository.GenerateStub<IDependency>()},

           {"StrictMock", this.mocks.StrictMock<IDependency>()},

           {"DynamicMock", this.mocks.DynamicMock<IDependency>()},

           {"Stub", this.mocks.Stub<IDependency>()},

       };

    }

  

    [RowTest]

    [Row("GenerateMock")]

    [Row("GenerateStub")]

    [Row("StrictMock")]

    [Row("DynamicMock")]

    [Row("Stub")]

    public void When_does_it_work(string mockStyle)

    {

        var dependency = dependencies[mockStyle];

        var caller = new Caller(dependency);

        caller.Go();

        dependency.AssertWasCalled(d => d.DoSomething());

    }

}

  

public class Caller

{

    private readonly IDependency dependency;

  

    public Caller(IDependency dependency)

    {

        this.dependency = dependency;

    }

  

    public void Go()

    {

        dependency.DoSomething();

    }

}

  

public interface IDependency

{

    void DoSomething();

}

The results (3 out of 5 tests fail) clearly illustrated the source of our problems: AssertWasCalled only works automatically if you create your mocks using the new static MockRepository methods. Ayende’s initial announcement of the feature used these static methods, but it wasn’t explicit that they were required. They just appeared to be convenience methods. However, reading into the comments, I see that Ayende posted on 6/14/08 that the static methods are indeed special:

The static methods are returning an object that can be used without record/replay. It is already in replay mode, which is needed to use the AAA syntax.

If aren’t using the static methods (or are using something like StructureMap’s RhinoAutoMocker where the mocks are created for you), you can still utilize AssertWasCalled as long as you call ReplayAll() before the “act” phase of your test. For example, in the code above I can add “mocks.ReplayAll()” before line 31 to get 4 out of the 5 tests to pass. The only one that fails is the one that uses a StrictMock, which isn’t too worrisome.

Running jQuery QUnit tests under Continuous Integration

Joshua Flanagan — Thu, 18 Sep 2008 23:23:54 GMT

Setup

This post assumes you are already writing unit tests for your JavaScript code. If not, check out Chad's post on Getting Started with jQuery QUnit. We use jQuery and QUnit at work, so my code examples are geared toward those frameworks. However, the approach should be very easy to adapt to your JavaScript framework of choice.

Overview

A good continuous integration server will let you run your automated tests, fail the build if a test fails, and publish a report of the test results. For all of that to work, your CI tool needs to be able to run the tests from a command-line and harvest the output in a form that it understands. One problem is that JavaScript testing frameworks like QUnit require you to open an HTML page in a browser to run and view the test results. You can certainly launch an HTML page in a browser from a command line ("start mypage.htm"), but you won't be able to feed the results back to the CI server. We can get around this by using a tool like WatiN to control the browser from NUnit (or some other test framework which is supported by your CI server). WatiN allows you to spawn an instance of Internet Explorer, navigate to a URL, inspect the contents of the rendered DOM, and shutdown the browser, all from within an NUnit test.

A simple solution

Our first approach was to modify the QUnit test runner script so that it would create an element named TESTRESULTS that held the number of failed tests. We could then run the HTML page containing our tests and use WatiN to verify that TESTRESULTS contained a 0. An entire page of QUnit tests would be reported by NUnit as a single test. Either all of the QUnit tests passed making the single NUnit test pass, or any QUnit test failed and the NUnit test failed. You can see an example of this approach in my comment on Chad’s post referenced earlier. There are two problems with this approach: your total test count is inaccurate (you could have hundreds of QUnit tests which only show up as a single test in the CI test report), and more importantly, it is not immediately obvious which test failed, since it could have been any one of the many QUnit tests within a page.

Our current approach

A better approach would be to have a single NUnit test for each QUnit test. However, manually writing an NUnit test for each QUnit test sounds like a nightmare. What we need is a way to generate test cases dynamically. MbUnit supports this natively with its Factory attribute. We can get the same behavior in NUnit using the IterativeTest add-in (get the source and compile it against your version of NUnit). It allows you to specify a factory method that supplies a series of values to a single test method. When NUnit loads the class with this test method, it will create a separate test case for each value passed to the test method (if your factory method returns 3 values, you will have 3 test cases).

The full code to our test fixture is at the bottom of this post. The method GetQUnitTestResults takes an HTML page name as input and returns a series of QUnitTest instances. Each QUnitTest instance contains the details of an individual QUnit test run: the filename, the test name, the result, and any related failure message. The method RunQUnitTests is the actual factory method used by the IterativeTest add-in, and allows me to easily add new QUnit HTML pages without having to create new test methods. The gory details of parsing the DOM for the QUnit results are in grabTestResultsFromWebPage. Its not my proudest piece of code, but it gets the job done (for now). This is likely the only method you would need to change if you were using a JavaScript test framework other than QUnit.

Using this approach, we now get an accurate reflection of our test count, as each QUnit test is counted with every other NUnit test. Even better, when a test fails, we get detailed information about exactly which test failed and why. This is the output of a run that has a failing test:

------ Test started: Assembly: DovetailCRM.IntegrationTesting.dll ------

IterativeTest addin loaded.
TestCase 'DovetailCRM.IntegrationTesting.Web.JavaScript.JavaScriptTester.QUnit([tageditortester.htm] Post tag change to server module: should post the tag Name and item Id)'
failed: 
  the item that is tagged expected: 42 actual: 0
  Expected: collection containing "pass"
  But was:  < "fail" >
    D:\code\blue\source\DovetailCRM.IntegrationTesting\Web\JavaScript\JavaScriptTesters.cs(128,0): at DovetailCRM.IntegrationTesting.Web.JavaScript.QUnitTestHelpers.ShouldPass(QUnitTest theTest)
    D:\code\blue\source\DovetailCRM.IntegrationTesting\Web\JavaScript\JavaScriptTesters.cs(29,0): at DovetailCRM.IntegrationTesting.Web.JavaScript.JavaScriptTester.QUnit(Object current)


100 passed, 1 failed, 0 skipped, took 16.48 seconds.

Notes

NUnit add-ins must be compiled against the version of NUnit you are using. Make sure you have the same version of NUnit on your developer desktops and the CI server. We standardized on NUnit 2.4.7 simply because that was the latest version supported by TestDriven.NET.
The IterativeTest add-in must be in the addins sub-folder of the folder that contains your nunit-console.exe. If you want to run the tests via TestDriven.NET, you’ll also want to put a copy of the add-in in <ProgramFiles>\TestDriven.NET 2.0\NUnit\2.4\addins
While there is a way to get the TeamCity custom NUnit runner to load add-ins (using /addin:), I ran into trouble getting it all to work with WatiN as well. I discovered a JetBrains add-in for NUnit that allows nunit-console.exe to report progress to TeamCity in the same way as their custom runner. While it isn’t supposed to be available until TeamCity 4.0, I found the necessary files in this patch (see the Attachments section) which enables NUnit 2.4.7 support to TeamCity 3.1.

The Code

       1: [TestFixture]

       2: public class JavaScriptTester

       3: {

       4:     private IE _ie;

       5:  

       6:     [TestFixtureSetUp]

       7:     public void TestFixtureSetUp()

       8:     {

       9:         _ie = TestBrowser.GetInternetExplorer();

    }

   

    [IterativeTest("RunQUnitTests")]

    public void QUnit(object current)

    {

        ((QUnitTest)current).ShouldPass();

    }

  

    public IEnumerable RunQUnitTests()

    {

        return new[]

                   {

                       "ContextMenuTester.htm",

                       "FilterControlTester.htm",

                       "RepeaterControlTester.htm",

                       "FinderTester.htm",

                       "ConsoleScriptTester.htm",

                       "tageditortester.htm",

                       "CrudFormTester.htm"

                   }.SelectMany(page => GetQUnitTestResults(page));

    }

  

  

    public IEnumerable<QUnitTest> GetQUnitTestResults(string testPage)

    {

        TestFixtureSetUp();

        _ie.GoTo(string.Format("http://localhost/Content/scripts/tests/{0}", testPage));

        _ie.WaitForComplete(5);

  

        return grabTestResultsFromWebPage(testPage);

    }

  

    public IEnumerable<QUnitTest> grabTestResultsFromWebPage(string testPage)

    {

        // BEWARE: This logic is tightly coupled to the structure of the HTML generated by the QUnit testrunner

        // Also, this could probably be greatly simplified with a couple well-crafted xpath expressions

        var testOL = _ie.Elements.Filter(Find.ById("tests"))[0] as ElementsContainer;

        if (testOL == null) yield break;

        var documentRoot = XDocument.Load(new StringReader(makeXHtml(testOL.OuterHtml))).Root;

        if (documentRoot == null) yield break;

  

        foreach (var listItem in documentRoot.Elements())

        {

            var testName = listItem.Elements().First( x => x.Name.Is("strong")).Value;

            var resultClass = listItem.Attributes().First(x => x.Name.Is("class")).Value;

            var failedAssert = String.Empty;

            if (resultClass == "fail")

            {

                var specificAssertFailureListItem = listItem.Elements()

                    .First(x => x.Name.Is("ol")).Elements()

                    .First(x => x.Name.Is("li") && x.Attributes().First(a=> a.Name.Is("class")).Value == "fail");

                if (specificAssertFailureListItem != null)

                {

                    failedAssert = specificAssertFailureListItem.Value;

                }

            }

  

            yield return new QUnitTest

                             {

                                 FileName = testPage,

                                 TestName = removeAssertCounts(testName),

                                 Result = resultClass, Message = failedAssert

                             };

        }

  

    }

  

    private static string makeXHtml(string html)

    {

        return html.Replace("class=pass", "class=\"pass\"")

            .Replace("class=fail", "class=\"fail\"")

            .Replace("id=tests", "id=\"tests\"");

    }

  

  

    private static string removeAssertCounts(string fullTagText)

    {

        if (fullTagText == null) return String.Empty;

        int parenPosition = fullTagText.IndexOf('(');

        if (parenPosition > 0)

        {

            return fullTagText.Substring(0, parenPosition - 1);

        }

        return fullTagText;

    }

}

  

public class QUnitTest

{

    public string FileName { get; set; }

    public string TestName { get; set; }

    public string Result { get; set; }

    public string Message { get; set; }

  

    public override string ToString()

    {

        return string.Format("[{0}] {1}", FileName, TestName);

    }

}

  

public static class QUnitTestHelpers

{

    public static void ShouldPass(this QUnitTest theTest)

    {

        Assert.That(theTest.Result.Split(' '), Has.Member("pass"), theTest.Message);

    }

  

    public static bool Is(this XName xname, string name)

    {

        return xname.ToString().Equals(name, StringComparison.OrdinalIgnoreCase);

    }

}

It makes use of our TestBrowser class, which is just a simple wrapper around WatiN’s IE object.

       1: public static class TestBrowser

       2: {

       3:     public static readonly object LOCK_ROOT = new object();

       4:     public static IE _browser;

       5:  

       6:     public static IE GetInternetExplorer()

       7:     {

       8:         if( _browser == null )

       9:         {

      10:             lock(LOCK_ROOT)

      11:             {

      12:                 if( _browser == null )

      13:                 {

      14:                     IE.Settings.AutoMoveMousePointerToTopLeft = false;

      15:  

      16:                     _browser = (IE)BrowserFactory.Create(BrowserType.InternetExplorer);

      17:                     _browser.ShowWindow(NativeMethods.WindowShowStyle.Hide);

      18:                 }

      19:             }

      20:         }

      21:  

      22:         return _browser;

      23:     }

      24: }

Monkey patching rake for use with TeamCity

Joshua Flanagan — Wed, 10 Sep 2008 23:08:54 GMT

We use Ruby’s rake as our build automation tool. It provides a nice, XML-free way to define logical groupings of steps to perform, with dependency resolution.

We use JetBrain’s TeamCity as our continuous integration server. It has a neat feature that provides real-time reporting of build progress. TeamCity does not have any built-in support for rake*, so we use the simpleRunner build runner which allows you to kick off any process. However, when using the simpleRunner, you do not get any progress updates, since TeamCity doesn’t really know what is going on within your custom process.

Turns out, this is really easy to fix. TeamCity has a very simple API which allows a build script to interact with it. You just have to send specially formatted text messages to standard out (the console). For example, writing the following text to the console from your build script will cause the message “Starting compilation” to appear as the build status (on the project website, or the notification tray):

##teamcity[progressStart 'Starting compilation']

Now, it would be pretty annoying if we had to litter these messages throughout our build script. Fortunately, in ruby, classes are open by default. This means we can add or change behavior of existing classes, even if we do not have control over the source. In this case, I want to change the Execute method on Rake’s Task class so that it sends a message to TeamCity when a task starts and finishes. I use Jay Fields’ example of redefining a method that still needs to call its original implemention (thanks to Steven Harman for the help via twitter). I created a file [TeamCity.rb] with the following contents in the same directory as our RAKEFILE:

module TeamCity
    def teamcity_progress(task)
        teamcity_service_message 'progressStart', task
        yield
        teamcity_service_message 'progressFinish', task
    end

    def teamcity_service_message(message, message_value)
        puts "##teamcity[#{message} '#{message_value}']"
    end
end

class Rake::Task
include TeamCity
old_execute = self.instance_method(:execute)

       define_method(:execute) do |args|
         teamcity_progress("Executing #{name} rake task") { old_execute.bind(self).call(args) }
       end
end

I then conditionally include this patch (so that we do not see all of the annoying TeamCity messages when running the build locally) with the following line in our RAKEFILE:

require 'TeamCity.rb' if ENV["teamcity.dotnet.nunitlauncher"]

I just arbitrarily picked an environment variable that I know is defined when run within TeamCity, that is not defined on our local machines. There is probably a more appropriate/universally applicable environment variable which indicates “running in TeamCity”.

We now can see near real-time which rake task is currently running on the CI server. This can be nice when you are trying to fix a broken build, and you want to know if the current build has successfully passed the step that caused the failure. And yeah, it was also a fun way to explore monkey patching with ruby.

* There is actually an EAP for a Rake Build Runner, which I did not find out about until doing some research for this post. From the description, it sounds more geared toward people actually building ruby applications, which does not include us. All of our code is C#/.NET – only our build script is in ruby. However, its probably worth investigating further.

Hola Los Techies

Joshua Flanagan — Wed, 10 Sep 2008 04:05:29 GMT

I'm very happy to accept an invitation to join the Los Techies community. I've been posting at http://flimflan.com/blog for the past four years, and am looking forward to blogging more among some great company.

Subscribers to my old site should continue to get the new content, assuming they are subscribed via http://feeds.feedburner.com/JoshuaFlanagan. However, if you aren't already subscribed to the Los Techies main feed, I'd highly recommend it.

Since there is no code in this post, I'm going to keep it real short. Thanks again to the Los Techies crew for having me!