puts Blog.new(”nonsense”)

Tests increasingly serve multiple roles in today’s projects. They help us design APIs through test-driven development. They provide confidence that new changes aren’t breaking existing functionality. They offer an executable specification of the application. But can we ever get to a point where we have too much testing?

Enough is Enough

Consider the following test that you might come across in an application with a less-than-ideal test suite. (We’ll pick on some badly-written Rails code for this example, but the ideas we’ll discuss are certainly not unique to just Rails or Ruby or even just to dynamic languages. Unfortunately, these problems are quite universal.)

Whoa! That sure is a lot going on in a single test case. What exactly is it that we’re trying to test here? Let’s take a step back and see if we can figure it out. [1]

• Line 5 - We start off by creating a new product. We give the product a name and a price and call #create. Since we’re in a Rails app, without looking elsewhere in the code, we can make an educated guess that Product is an ActiveRecord subclass and that calling #create will persist the product to the database.
• Line 6 - We send an HTTP GET request to a #show method and pass along the ID of the product we just created. Since we’re in ProductsControllerTest, we can safely infer that we’re calling the #show action in the ProductsController class.
• Line 7 - We assert that the controller action responded with HTTP status code 200 (i.e., success).
• Lines 8-9 - We look for an object stored in the assigns hash (i.e., the hash of variables that will be available to the view template) with a key of :product and we assert that it’s not nil.
• Line 10 - We verify that the product object satisfies all the product validation rules (though we can’t know what those rules are without taking a look at the Product class).
• Lines 11-12 - We assert that the attribute values we provided when creating the product (in line 5) match the attribute values stored in the object we fetched from the assigns hash.

For a class called ProductsControllerTest, it sure feels like we’re doing a fair bit more than just testing the code we’d expect to find in ProductsController (assuming ProductsController conforms to the traditional responsibilities of a controller). Let’s take a look at the controller code to better understand exactly what it is that we’re testing.

It turns out that our controller code is notably simpler than our test case would have led us to believe. The #show action does indeed stick to the expected behavior of a controller, and in this case, it’s able to provide that behavior in a single line of code (despite the 8 lines of test code currently employed above). That single line satisfies all of the expectations of the #show action: look up the product with the given ID and place the product object in an instance variable for use by the view.

If the #show action isn’t responsible for knowing how to successfully save a new product record into the database, why does our test case attempt to provide the data for creating a new product? What will happen to our test case when someone adds a new validation rule requiring that all products also include a quantity attribute? Unfortunately, the test as it’s currently written will break. And while a failing test is often a welcome warning that our changes have inadvertently broken a part of our application, it’s far from desirable for a model-level rule related to creating new records to cause a failure in a controller test related simply to displaying existing records. In short, we’ve given our test case too much responsibility, and in doing so, we’ve made it fragile.

We can see from the #show action above that not only is the controller not responsible for knowing how to create a valid product, it’s also not responsible for ensuring that a product’s attributes are properly populated when the record is read from the database. However, if we take another look at the last few lines of the test case, we might think otherwise. That brings us to the second problem with this particular test: it’s a rotten source of documentation for the code being tested. As we increasingly move toward tests as specifications of our application’s behavior, it’s vital that those specifications clearly communicate the expected behavior. As it’s currently implemented, the overspecification in this test case leaves the reader having to do way too much work to figure out the true expectations of the code being tested.

Communicate Essence

Let’s take another pass at writing a test for the #show action, this time with an eye toward removing the fragility of the previous test case and increasing the value of the test as a specification.

In this implementation, we’ve abstracted away the logic for creating a new product in line 5. We’ve defined a helper method for use by any and all tests in our application that have a need to create a new product. If and when the rules for successfully creating a new product change, we’ll update the #create_product method, and we won’t have to touch the code in ProductsController or ProductsControllerTest at all. [2]

As for the four assertions that appeared at the end of our first attempt at this test case, we’ve replaced those assertions with a single (stronger) assertion. Where we previously asserted that the assigns hash held a non-nil product, that the product was valid, and that its attributes matched the attributes used at the beginning of the test, we now simply verify that the product object in the assigns hash is equal to the product object that we created at the beginning of the test. That single line more accurately and more concisely expresses our expectation: that the product whose ID we provide in the request to the #show action is the same object that’s made available to the view.

By focusing our test case on the essence of the code under test, we’ve ended up with less test code to maintain. And with the extraneous setup and assertions removed, the remaining test code provides a clearer and less fragile specification of the behavior being tested. [3]

May I Take Your Order, Please?

In the previous example, we might have detected the overspecification by the significant mismatch between the lines of test code and the lines of production code, or seeing model-specific assertions in a controller-specific test may have caught our eye. But, overspecification comes in more subtle forms as well. Consider the following method provided by ActiveRecord::Base for fetching the list of columns that hold domain-specific content from a model class in Rails.

To test this method, we’ll need a sample ActiveRecord model class. Let’s use the Topic model, which is backed by the following table.

In our first attempt at testing the #content_columns method, we might come up something similar to this:

Can you spot the overspecification? To give you a hint, compare that test to the actual test employed in the ActiveRecord test suite:

If you’ve ever written a test for something that returns an ordered list of items where the actual order either doesn’t matter or isn’t guaranteed, you’ve probably been bitten by this breed of overspecification, and there’s a good chance your solution matched the solution used in the ActiveRecord test code above. Because the #content_columns method doesn’t guarantee that the columns will be returned in any particular order, it’s inappropriate (and fragile) for our test to specify a certain order.

While the actual ActiveRecord solution above works, it too could be better. First, since we know that the two arrays in line 6 won’t be equal unless they have the same length, we can safely remove the extraneous length assertion in line 5. Second, when we see the calls to #sort on line 6, we’re forced to pause (if even for just a moment) to think about why it might be important to sort the two arrays. We don’t really want to sort the arrays; we’re just using that technique to get around the fact that order doesn’t matter to us, but it does matter when Ruby compares two arrays for equality. Since we really only want to assert that the arrays have the same elements, why not do exactly that?

By using an assertion like Shoulda’s assert_same_elements method, our test can clearly and concisely express the expected behavior.

Use It Wisely

Overspecification comes in many flavors, and the examples above in no way represent a comprehensive list. As developers, we frequently strive to write as little code as possible to accomplish the task at hand. When it comes to writing tests, we should very much keep that goal in mind as well. Good tests communicate the essence of the scenario being tested and nothing more. While I doubt a project will ever suffer from too much testing, it can certainly suffer from tests that specify too much.

Notes

[1] If this test seems absurd to you, good! Your ability to detect this type of overspecification will serve you well.

[2] The actual implementation of the #create_product method isn’t particularly pertinent in this discussion. (In the Rails space, there’s certainly no shortage of options.)

[3] With all the excessive thoroughness that went into overspecifying the the current behavior of the #show method, is it possible that we’ve been too distracted to identify other functionality that’s still missing from that method? In the next post, we’ll take a look at overspecification’s more lethargic cousin: underspecification.

Thanks to Justin Gehtland, Muness Alrubaie, and Glenn Vanderburg for reading drafts of this post.

So you’ve taken your project to 100% code coverage, you’ve configured your continuous integration system to fail the build if that coverage ever drops below 100%, and you’re ready to enjoy the fearless refactoring and the rock solid regression testing suite that your software engineering rigor has now earned you. But are you really covered? What does 100% code coverage mean in your project? Is it enough to know that your test suite encounters every line of code? Or don’t you want to be sure that it exercises every line? If you simply encounter the line without asserting that it produces the correct results, are you any better off? [1]

Failure to Assert

Just achieving 100% code coverage is the easy part. Making it mean something: that’s where the real value kicks in. Consider the ease with which we can get to 100% line coverage on the following code (generated using Rails 2.1 scaffolding).

In order to ensure that the #index method is performing all its proper duties, we’ll define the following “test case.”

We’ll use rcov to assess the results.

And just like that, we have 100% code coverage for the #index method. [2] In this case though, that clearly means nothing more than the fact that we encountered 100% of the lines in the method. When code coverage is that easily achieved, it hardly seems cause for celebration. To get any real value from the test, we need to actually assert that we’re getting the expected results. Until we do so, we have nothing more than incidental coverage. [3]

The test code provided by the Rails scaffolding gets us closer to where we want to be.

The tests pass, and our code coverage is still at 100%. At this point we’ve significantly increased the value of that particular test. No longer does the code have to crash spectacularly in order to yield a test error. Instead, exiting the method with anything other than a success response code will result in a test failure. Similarly, failure to set the @products instance variable will cause the test case to flunk. [4]

100% Covered. 50% Tested.

But while we’ve improved on the initial test case, at best we’re really only halfway to where we want to be. What would our test suite tell us if we were to alter line 9 as follows.

Unfortunately, our test suite still blindly gives a thumbs up, despite the fact that any attempt to access the XML-formatted output would yield an exception. While our test suite includes assertions for how the application should prepare an HTML-bound response, our coverage of the XML-specific logic in line 9 remains merely incidental.

If we want our automated test suite to ensure that the #index method remains capable of producing an XML-formatted response as our codebase changes over time, we’d need to add a test case to exercise that code.

rcov doesn’t reward us with any extra credit for writing this test case, but achieving 100% coverage is not the primary goal of a good test suite. [5] The primary goal is to ensure that the code satisfies the requirements. If the application really is required to provide an XML-formatted list of products, then we should seriously consider defining a test for that functionality in our test suite. [6]

Size Matters

In the examples above, we started off with a 5-line method that had 100% line coverage but lacked any assertions. That scenario provides some insight into other places where we might find a high percentage of incidental coverage. While long methods are a bad idea in general, the simple act of invoking a method may be enough to register it as having 100% coverage. If our test suite results in the execution of a 25-line method with no branches, all 25 lines are considered to be covered, regardless of whether we perform any assertions to verify the results of those 25 lines. Even if we refactor the code into smaller methods, if we don’t unit test those new methods, we’re still left with nothing more than incidental coverage.

Use It Wisely

If your goal is to achieve 100% coverage, then incidental coverage is your friend, but your test suite will fall far short of its full potential. You’ll run the risk of acquiring a false sense of security, and the ability to safely and fearlessly refactor is out the window. Changes or enhancements to your application will occur without the safety net of a full regression suite.

If your goal is instead to develop a comprehensive test suite to A) validate your application’s functionality and B) ensure that you build just enough software, then test-driven development (TDD) is your friend (as are peer reviews and/or pair programming). And subsequently, you’ll enjoy the nice side effect of 100% code coverage, because you’ll only build the code necessary to satisfy your tests.

Notes

[1] Sure, you know that the line executes without causing the application to crash (at least for this set of inputs), but does a lack of crashing really constitute success? Not likely.

[2] While we have 100% line coverage for the #index method, we don’t have 100% coverage for the ProductsController class as a whole. As the full coverage report shows, the tests provided with the Rails scaffolding do not cover the exception cases in the #create and #update methods.

[3] Incidental coverage is just as valuable as that guy that puts in “face time” at the office. He’s physically present at least 8 hours every day. People see him there. He’s at the office. He must be doing something. Right?

[4] At this point we’ve gone from covering nothing to covering something. That’s good, but we should ask ourselves whether we’re performing the most appropriate assertions for this test case. For example, is it enough to assert that assigns(:products) is not nil, or should we be making a stronger assertion about its value? Do we want to ensure that we’re rendering the intended view templates? What about verifying the content of the HTML (or XML) that gets rendered? Tackling issues specific to testing Rails controllers would take away from the focus of this post, but the strength of assertions and the layers at which we test is surely fodder for future posts.

[5] However, with anything less than 100% coverage, fearless refactoring and full regression testing simply isn’t something your test suite can provide. In Software Testing Techniques (1996), Boris Beizer takes the hardline on anyone that would consider developing a codebase without at least 100% line coverage: “[Line coverage] is the weakest measure in the family [of structural coverage criteria]: testing less than this for new software is unconscionable …”

[6] You could argue that this new test case is too similar to the original test case, that it needs stronger assertions, or that the current functionality is too simple for this new test case to offer sufficient benefit, but it’s up to you to assess how important automated testing is of that code. If you opt to not test your application’s ability to provide an XML-formatted list of products, you must not allow your 100% line coverage to give you a false sense of security about your code. That code may be covered, but it’s not tested.

Thanks to Stuart Halloway for reading drafts of this post.

In any discussion of code coverage, it’s important to understand the type of coverage that’s being measured. In the same way that you wouldn’t plan a trip to the beach without knowing whether it’s going to be 30 degrees Celsius (perfect!) or 30 degrees Fahrenheit (stay home), we can’t have a truly meaningful discourse on code coverage without first identifying the analysis type.

Lining Up

Many popular code coverage analysis tools currently report what’s known as line coverage (also commonly referred to as statement coverage). Line coverage analysis (as the name implies) identifies which lines were encountered as a result of your tests.

To better illustrate what we can expect from line coverage analysis, let’s first consider the following Ruby module and a possible test case for that module.

Read the rest of this entry »

Rails Security Audit is an straight-talking introduction to the techniques for keeping your Rails app secure. Over the course of 47 pages, Aaron Bedra tackles the security audit process, common security-related bugs, essential server lockdown tactics, and an approach for assessing the severity of the issues you find.

Rails offers sound solutions for keeping your application safe from the likes of SQL injection, Cross-Site Scripting (XSS), Cross-Site Request Forgery (CSRF), and the perils of mass assignment, but are you sure you’re using those solutions? Without a proper understanding of these exploits, it’s unlikely that a developer would safely navigate past those issues entirely. Aaron provides easy-to-follow examples of each exploit, shows the consequences of ignoring them, and introduces the tools and techniques for identifying and avoiding those problems.

Going beyond Rails-specific security, Aaron covers a handful of guidelines for ensuring good server hygiene. As a developer (read: not a system administrator), I personally couldn’t tell you which ports should be open on a server and which ones are an invitation for trouble. Aaron remedies that situation by demonstrating the tools for determining how well your server is currently secured and providing scripts and instructions for getting you to where you need to be.

The book closes out the discussion on server security with additional recommendations for restricting access to your servers, but it stops short of providing any direction for doing so. While a quick round of Googling would likely locate a decent tutorial, I’d rather have seen the information included in the book (even if it just took the form of pointers to recommended tutorials).

Once you’ve identified the security issues in your app, Aaron offers honest, no-nonsense advice regarding the next step: “I guarantee at some point you will audit an application and find something in the form of a security hole. Your job as an auditor is to present the vulnerability, period.” Translation: security audits aren’t about sugar-coating your findings or pulling punches. He then pragmatically explains that anything other than the quick fixes will likely warrant further exploration of the actual risk posed by the issue, and provides a step-by-step walkthrough of a sample risk analysis effort.

If you’re even the least bit uncertain about any of the topics discussed above, you owe it to yourself to check out Aaron’s highly-approachable guide to assessing the state of security in your Rails app.

Full disclosure: Aaron and I work together at Relevance, and his potential for hacking into my laptop inspires great fear in me. Nevertheless, I stand by the comments above even in the face of that adversity.

InfoQ recently posted the video of my presentation on Grails from QCon San Francisco. If 50 slides in 50 minutes sounds a tad formulaic and tired to you, then you’re in luck. Instead, you’ll see 50 slides in about 5 minutes, followed immediately by 50 minutes of no-nonsense live coding goodness.

In what could perhaps be described as a series of 12 back-to-back lightning talks, you can see what it takes to go from a blank slate to a deployable Grails app including…

• defining domain classes,
• setting up relationships,
• hooking up a database,
• establishing constraints and validation error messages,
• enjoying sexy dynamic finders,
• applying custom URL mappings,
• working with tag libraries,
• encapsulating business logic in services,
• integrating with existing Java code,
• sending e-mail,
• finding and installing plugins, and
• locking down the app with secure authentication and authorization

There’s some good Q & A in there as well. Unfortunately not all of the questions came through on the audio, but in most cases you can pick up the context from the reply.

You’ll also hear me reference Charles Nutter’s JRuby talk a few times over the course of the presentation, and I recommend checking out that video as well.

Muness blogged a photographic introduction to the Relevance mothership, and Glenn Vanderburg went all meta on us and asked what we could learn from comparing the books in active use on the desks to the less fortunate books relegated to use as monitor stands.^§

Wired? Erlang.
Expired? RMI.

Read the rest of this entry »

With all the recent fuss about the game-changing advantages of Git and distributed version control in general, it would be easy to overlook what Git does for deciding whether (and when) to use version control for a given task. Sure, Git makes non-linear development a breeze, it manages large projects with uncanny efficiency, and we probably can’t even fathom yet just how transforming github is going to be for open source. But, if you look closely, there’s something worth noting way before you create your first branch, before your project is even thirty minutes old, and well before you’re ready to share it with the community: git init is so pleasantly simple, you’ll never again think twice about “whether it’s worth it” to throw something into version control.

As someone who had the, um, “joy,” of working with CVS, SourceSafe, ClearCase, and other SCM “solutions” that made you wish you were instead just using NTFS, I definitely appreciate what SVN did for the state of version control systems. Nevertheless, there were countless prototypes, drafts, experiments, etc. that I talked myself out of storing in SVN. The conversation typically went something like so: “Do I really want to create a new repository just for this experiment? Should it go in my local repo, or does it belong up on the shared repo? Should I bother setting up the standard dirs for trunk, tags, and branches? Maybe I should just add it to a grab-bag repo for now? Nah. Forget it. It’s not worth the trouble yet.”

Git removes many of those decisions altogether, and (in true agile fashion) it allows me to defer the others until they actually matter. I can “execute, build momentum, and move on.” Let’s say I’m halfway through a blog post, and I decide that I want to try taking it in a different direction. No problem. Drop it in Git, and experiment away…

So while you’ll continue to hear people (myself included) champion Git’s importance as a solution for team-based or community-based development, its ability to give you instant, no-questions-asked version control is enough to earn a place for Git on your system, even if you’re the only one who will ever see your work.

And Git’s agility doesn’t stop there. From branching on a dime, to the oh-so-beautiful stash, to the ability to rework past commits, Git reminds me that decisions are temporary. Or, to quote Ryan Tomayko, “Git means never having to say, ‘you should have.’”

–

Be sure to check out Rob’s post for a whole host of Git-infused goodness.

Rob dared me to fire up my favorite shell and jump into the game. Imagine my disappointment when I was greeted with this bummer of an error message.

Hmm. No dice. OK, on to my second choice.

The result? A few well-known friends and some that likely deserve a bit of elaboration.

• m is an alias for the mate shell command for TextMate
• mman is that handy TextMate man page viewer we talked about recently
• The one that’s bleeped out (i.e., **) is an alias that takes me straight to the source dir for my current client project
• cap1 is, um, a reminder that deprec2 is gonna be a nice upgrade

• Geertjan Wielenga suggests the first (and only) Java closure solution that makes any sense.

• Wait for it. Wait for it. Are you serious?! Damn, that’s cool!

• As always, Alan Hoffler offers some welcome advice “and stuff like that” on using words that add nothing to your message.

• Rails goes Git. Sweet! Who’s next?

• Multi-touch is so 1984.

• “Working for oneself, or at least for a small group, is the natural way for programmers to live.” Well said. And if you disagree with Paul Graham, watch out; you’re being graded.

Andres Almiray interviewed me this week for the Groovy Zone. We cover a breadth of topics, including:

• Just how far Grails has come in the past two years
• Why the GORM DSL likely obviates previous mapping techniques
• Groovy as a gateway drug to more and better developer testing
• Why Grails testing infrastructure improvements deserve top billing in Grails 1.1
• Something called Rails
• New testing-related developments in the Groovy ecosystem

For all that and more, check out the interview at Groovy Zone, a new(ish) and hoppin’ community for Groovy and Grails news.

(Did I mention that we discuss testing?)

Many thanks to Andres and DZone for the interview.