Testing tips

I’ve done a fair amount of data heavy testing recently and here’s some tricks that make things a little easier.


Fixtures

For this example let’s imagine we have a website selling content subscriptions. We have 3 levels of membership:

enum Membership: Int, Comparable {
    case free
    case basic
    case pro
}

func < (lhs: Membership, rhs: Membership) -> Bool {
    return lhs.rawValue < rhs.rawValue
}

Our User type contains a membership value and a function to determine if a user can access some content:

struct User {
    let membership: Membership

    func canView(content: Content) -> Bool {
        return content.eligibleMembership <= membership  
    }
}

Our content is then marked with an eligibleMembership level:

struct Content {
    let eligibleMembership: Membership
}

To exhaustively test this function we need to write 9 test examples. This is calculated by looking at the inputs to this function.

NB because this is a member function we also count the member variable membership.

User.membership            - has 3 cases (free|basic|pro)
Content.eligibleMembership - has 3 cases (free|basic|pro)

3 cases x 3 cases = 9 test examples

Writing 9 separate tests is going to have a lot of duplication and quite frankly I struggle to name 1 test let alone 9 similar tests that vary only by input.

This is where I would use fixtures and some funky formatting to help readability.

func testCanViewContent() {
    let fixtures: [(Bool, Membership, Membership)] = [
        // expected  |  userMembership  | eligibleMembership
            (true,         .free,              .free),
            (true,         .basic,             .free),
            (true,         .pro,               .free),
            (false,        .free,              .basic),
            (true,         .basic,             .basic),
            (true,         .pro,               .basic),
            (false,        .free,              .pro),
            (false,        .basic,             .pro),
            (true,         .pro,               .pro),
    ]

    for (expected, userMembership, eligibleMembership) in fixtures {
        let result = User(membership: userMembership).canView(content: .init(eligibleMembership: eligibleMembership))
        XCTAssertEqual(expected, result)
    }
}

The basic idea here is to loop over an array of tuples that contain the inputs for all of our examples. In the body of the loop we do the usual arrange/act/assert dance for unit testing.

What I like about the above is:

What’s bad:

I think we can do better and solve the first point…


Use #line

To get better feed back #line is your friend. It adds a little more boilerplate to each fixture but my goal for testing is to make it super easy to read the tests and figure out when things go wrong. This is why I’m happy to use an odd tabular format and add a sprinkling of boilerplate.

Here’s the above example with #line added to show how it looks:

func testCanViewContent() {
    let fixtures: [(Bool, Membership, Membership, UInt)] = [
        // expected  |  userMembership  | eligibleMembership  |  line
        (true,         .free,              .free,               #line),
        (true,         .basic,             .free,               #line),
        (true,         .pro,               .free,               #line),
        (false,        .free,              .basic,              #line),
        (true,         .basic,             .basic,              #line),
        (true,         .pro,               .basic,              #line),
        (false,        .free,              .pro,                #line),
        (false,        .basic,             .pro,                #line),
        (true,         .pro,               .pro,                #line),
    ]

    for (expected, userMembership, eligibleMembership, line) in fixtures {
        let result = User(membership: userMembership).canView(content: .init(eligibleMembership: eligibleMembership))
        XCTAssertEqual(expected, result, line: line)
    }
}

The key changes are:

With these changes in place Xcode will now highlight the actual fixture that is failing.


Write Comments

Looping back to an earlier point - a goal when writing good tests is that they be easy to read and simple to debug. That’s why it’s really important that you add comments when it makes sense. I often find myself falling into the trap of trying to contort a test function name to reveal all intent but it often ends up in an unreadable mess.

A good comment for a test would explain why some set up is being done. I also find that it’s sometimes really useful to add // Given, // When and // Then comments throughout a test to help delineate the different phases of the function.

Applying these suggestions to the above example might result in this change:

/// Membership levels are ordered `free < basic < pro`.
/// These tests run all permutations to ensure that a user can only
/// view content that is less than or equal to their membership level.
func testCanViewContent() {
    let fixtures: [(Bool, Membership, Membership, UInt)] = [
        // expected  |  userMembership  | eligibleMembership  |  line
        (true,         .free,              .free,               #line),
        (true,         .basic,             .free,               #line),
        (true,         .pro,               .free,               #line),
        (false,        .free,              .basic,              #line),
        (true,         .basic,             .basic,              #line),
        (true,         .pro,               .basic,              #line),
        (false,        .free,              .pro,                #line),
        (false,        .basic,             .pro,                #line),
        (true,         .pro,               .pro,                #line),
    ]

    for (expected, userMembership, eligibleMembership, line) in fixtures {
        // Given
        let user    = User(membership: userMembership)
        let content = Content(eligibleMembership: eligibleMembership)

        // When
        let result = user.canView(content: content)

        // Then
        XCTAssertEqual(expected, result, line: line)
    }
}

Add .make() Extensions

If you are writing a lot of tests and you have types that are awkward to create then you’d benefit from creating factory helpers.

Let’s take the following type:

struct User {
    let id: String
    let username: String
    let membership: Membership
}

It only has a couple of properties but it will already be a pain to create lots of users especially if for example we was writing tests that only cared about membership.

We can simplify things by creating an extension only within our test target that provides defaults for all values:

extension User {
    static func make(id: String = "_", username: String = "_", membership: Membership = .free) -> User {
        return .init(id: id, username: username, membership: membership)
    }
}

With this helper in place we can create new users really easily and just provide the values we care about:

greet(user: User.make(username: "Paul"))

// We can take advantage of type inference and drop the type as well
greet(user: .make(username: "Paul"))

Conclusion

Some of these tips are only useful in certain circumstances but I’ve found it really helpful having these techniques in my tool bag.