Testing tips

I’ve done a fair amount of data heavy testing recently and here’s some tricks that make things a little easier.


Fixtures

For this example let’s imagine we have a website selling content subscriptions. We have 3 levels of membership:

enum Membership: Int, Comparable {
    case free
    case basic
    case pro
}

func < (lhs: Membership, rhs: Membership) -> Bool {
    return lhs.rawValue < rhs.rawValue
}

Our User type contains a membership value and a function to determine if a user can access some content:

struct User {
    let membership: Membership

    func canView(content: Content) -> Bool {
        return content.eligibleMembership <= membership  
    }
}

Our content is then marked with an eligibleMembership level:

struct Content {
    let eligibleMembership: Membership
}

To exhaustively test this function we need to write 9 test examples. This is calculated by looking at the inputs to this function.

NB because this is a member function we also count the member variable membership.

User.membership            - has 3 cases (free|basic|pro)
Content.eligibleMembership - has 3 cases (free|basic|pro)

3 cases x 3 cases = 9 test examples

Writing 9 separate tests is going to have a lot of duplication and quite frankly I struggle to name 1 test let alone 9 similar tests that vary only by input.

This is where I would use fixtures and some funky formatting to help readability.

func testCanViewContent() {
    let fixtures: [(Bool, Membership, Membership)] = [
        // expected  |  userMembership  | eligibleMembership
            (true,         .free,              .free),
            (true,         .basic,             .free),
            (true,         .pro,               .free),
            (false,        .free,              .basic),
            (true,         .basic,             .basic),
            (true,         .pro,               .basic),
            (false,        .free,              .pro),
            (false,        .basic,             .pro),
            (true,         .pro,               .pro),
    ]

    for (expected, userMembership, eligibleMembership) in fixtures {
        let result = User(membership: userMembership).canView(content: .init(eligibleMembership: eligibleMembership))
        XCTAssertEqual(expected, result)
    }
}

The basic idea here is to loop over an array of tuples that contain the inputs for all of our examples. In the body of the loop we do the usual arrange/act/assert dance for unit testing.

What I like about the above is:

  • It’s compact and all cases can be in one test.
  • Being in a tabular layout makes it easier to read.
  • I only have to name the test function once.

What’s bad:

  • Performing assertions in a loop makes it hard to see which case is failing.
    • Xcode will just highlight the XCTAssert* line, which is not going to help when we run multiple examples.
  • It takes a little practice and getting used to.

I think we can do better and solve the first point…


Use #line

To get better feed back #line is your friend. It adds a little more boilerplate to each fixture but my goal for testing is to make it super easy to read the tests and figure out when things go wrong. This is why I’m happy to use an odd tabular format and add a sprinkling of boilerplate.

Here’s the above example with #line added to show how it looks:

func testCanViewContent() {
    let fixtures: [(Bool, Membership, Membership, UInt)] = [
        // expected  |  userMembership  | eligibleMembership  |  line
        (true,         .free,              .free,               #line),
        (true,         .basic,             .free,               #line),
        (true,         .pro,               .free,               #line),
        (false,        .free,              .basic,              #line),
        (true,         .basic,             .basic,              #line),
        (true,         .pro,               .basic,              #line),
        (false,        .free,              .pro,                #line),
        (false,        .basic,             .pro,                #line),
        (true,         .pro,               .pro,                #line),
    ]

    for (expected, userMembership, eligibleMembership, line) in fixtures {
        let result = User(membership: userMembership).canView(content: .init(eligibleMembership: eligibleMembership))
        XCTAssertEqual(expected, result, line: line)
    }
}

The key changes are:

  • Update every case to take the #line.
  • Update the call to XCTAssert* to take #line.

With these changes in place Xcode will now highlight the actual fixture that is failing.


Write Comments

Looping back to an earlier point - a goal when writing good tests is that they be easy to read and simple to debug. That’s why it’s really important that you add comments when it makes sense. I often find myself falling into the trap of trying to contort a test function name to reveal all intent but it often ends up in an unreadable mess.

A good comment for a test would explain why some set up is being done. I also find that it’s sometimes really useful to add // Given, // When and // Then comments throughout a test to help delineate the different phases of the function.

Applying these suggestions to the above example might result in this change:

/// Membership levels are ordered `free < basic < pro`.
/// These tests run all permutations to ensure that a user can only
/// view content that is less than or equal to their membership level.
func testCanViewContent() {
    let fixtures: [(Bool, Membership, Membership, UInt)] = [
        // expected  |  userMembership  | eligibleMembership  |  line
        (true,         .free,              .free,               #line),
        (true,         .basic,             .free,               #line),
        (true,         .pro,               .free,               #line),
        (false,        .free,              .basic,              #line),
        (true,         .basic,             .basic,              #line),
        (true,         .pro,               .basic,              #line),
        (false,        .free,              .pro,                #line),
        (false,        .basic,             .pro,                #line),
        (true,         .pro,               .pro,                #line),
    ]

    for (expected, userMembership, eligibleMembership, line) in fixtures {
        // Given
        let user    = User(membership: userMembership)
        let content = Content(eligibleMembership: eligibleMembership)

        // When
        let result = user.canView(content: content)

        // Then
        XCTAssertEqual(expected, result, line: line)
    }
}

Add .make() Extensions

If you are writing a lot of tests and you have types that are awkward to create then you’d benefit from creating factory helpers.

Let’s take the following type:

struct User {
    let id: String
    let username: String
    let membership: Membership
}

It only has a couple of properties but it will already be a pain to create lots of users especially if for example we was writing tests that only cared about membership.

We can simplify things by creating an extension only within our test target that provides defaults for all values:

extension User {
    static func make(id: String = "_", username: String = "_", membership: Membership = .free) -> User {
        return .init(id: id, username: username, membership: membership)
    }
}

With this helper in place we can create new users really easily and just provide the values we care about:

greet(user: User.make(username: "Paul"))

// We can take advantage of type inference and drop the type as well
greet(user: .make(username: "Paul"))

Conclusion

Some of these tips are only useful in certain circumstances but I’ve found it really helpful having these techniques in my tool bag.

Tying things together

This post is an attempt to tie together a lot of related thoughts I’ve had and written about for a long time. The high level idea is to take our clean well modularised code and reuse it in interesting ways.

I take a networking client written in a reusable way that enables us to build an iOS application, a command line interface (CLI) and a website. In this post the CLI and website are not for production use - they are for debugging and pro-active monitoring, to support the iOS application.


Overview

In the linked Github repo I’ve built the following structure.

Overview

There is a shared component and then 3 front ends built on top of it. The high level pattern in each front end is:

  • Utilise the Networking Client to fetch [Person] instances.
  • Transform [Person] instances into a representation that is appropriate for each front end
    • iOS uses a table view
    • cli uses pretty printed JSON
    • web uses HTML

Why bother?

Before diving into any code it’s probably worth looking a bit more at why we would want to do this.

Why make a website?

Imagine getting a call out because your app is not displaying all the data it should from the backend. To debug you might need to do at least these two things:

1) Check the backend data manually
2) Run the app in the debugger to explore the data

The first of these can actually be fairly tricky to do. You might be a master with curl but not all APIs are easy to inspect; perhaps there is encryption, authentication, specific headers are required or there is a lot of link traversal.

This brings you to the second type of debugging, which can be slow and painful. Depending on how you have programmed your decoding it may not be easy to figure out why data is being discarded. This topic is something I touched on in Handling bad input with decodable.

Now imagine if you could instead just navigate to a website that uses your production code and displays all errors. How much time would that save?

Why make a CLI?

Giving your code a CLI gives you the ability to programmatically interact with it from a lot of different contexts. Imagine writing a tool that will pro-actively monitor the usage of your code and reports automatically when there are issues. At this point you can find issues before your customers do.

Pushing further upstream - you could provide your CLI to the team who manages the backend so that they can add it to their CI pipelines. By adding your production code to their CI pipelines you can be assured that any releases they do will now never break your application.


The above just scratches the surface of what you can do but hopefully it provides enough motivation to read on.


The shared client

In order to keep the examples on Github as simple as possible I’ve put all of the shared code into a single module.

The networking client has the following interface:

public protocol JSONPlaceholderClient {
    public typealias Cancellation = () -> Void

    public func fetch(completion: @escaping (Result<Decoded<[Person]>>) -> Void) -> Cancellation
}

This provides a single function that will fetch some people from jsonplaceholder.typicode.com.

The result makes use of an odd looking type called Decoded that I briefly covered at the bottom of Handling bad input with decodable. As a recap it looks like this:

  enum Decoded<T: Decodable>: Decodable {
    case debug(value: T, raw: AnyCodable, errors: [Error])
    case prod(value: T)
}

The reason for this type is that I ideally want my debugging tools to use the production code. This type allows you provide a single API (the fetch function) that can return just the data you need in production and richer debug data when you ask for it. I’ve toyed with the idea of just having two functions called fetch that have different completion handlers with the data required but it’s not a generic solution and it’s easy for the two methods to get out of sync.

In order to change whether you get debug or prod data you configure the client when you create it using a factory function:

func makeClient(debugEnabled: Bool = false) -> JSONPlaceholderClient

If you want to poke around the shared framework some more you can find it here on Github.


Basic CLI

My project on Github is structured like this:

.
├── app
├── cli
├── shared
└── web

I’ve created the CLI in the cli folder and then used Swift Package Manager’s ability to reference a package by path to import the Shared module.

The simplest Package.swift to get started looks like this:

// swift-tools-version:4.2

import PackageDescription

let package = Package(
    name: "cli",
    dependencies: [
         .package(path: "../Shared"),
    ],
    targets: [
        .target(name: "cli", dependencies: [ "Shared" ]),
    ]
)

My networking client only has one method so I don’t need a super complicated CLI. The following does the job quite simply:

 1 import Foundation
 2 import Shared
 3 
 4 _ = makeClient(debugEnabled: true).fetch { result in
 5     let encoded = result.flatMap { value -> Result<String> in
 6         do {
 7             let encoder = JSONEncoder()
 8             encoder.outputFormatting = .prettyPrinted
 9 
10             return .success(String(data: try encoder.encode(value), encoding: .utf8) ?? "")
11         } catch {
12             return .failure(error)
13         }
14     }
15 
16     switch encoded {
17     case .success(let value):
18         print(value)
19         exit(EXIT_SUCCESS)
20     case .failure(let error):
21         print(error)
22         exit(EXIT_FAILURE)
23     }
24 }
25 
26 dispatchMain()

Line 4 creates my client with debugEnabled: true - this is really important as we want as much data as possible in order to debug issues.

Lines 5-14 are creating a pretty printed JSON string representation of the data, this will be the simplest format for interoperating with other tools. Imagine piping this to the input of other tools on the command line.

Line 26 is what keeps this app running so that we can perform our asynchronous networking.

Lines 16-23 perform the duty of printing our output and exiting with the correct status code.


That’s it for a basic CLI - this was extremely painless to create but now opens up a lot of possibilities. We can access the data in the exact same way that out iOS application does using the same networking and serialisation code.

Running this tool looks something like this (this JSON is heavily edited as it’s long):

➜  cli swift build
➜  cli .build/debug/cli
{
  "value" : [
    {
      "id" : 8,
      "name" : "Nicholas Runolfsdottir V",
      "website" : "jacynthe.com"
    },
    ...
  ],
  "raw" : [
    {
      "id" : 8,
      "name" : "Nicholas Runolfsdottir V",
      "website" : "jacynthe.com",
      ...
    },
    ...
  ],
  "errors" : [
    "Invalid website 'hildegard.org' found at [_JSONKey(stringValue: \"Index 0\", intValue: 0)].",
    ...
  ]
}

The interesting thing to note about this JSON is that there are 3 top level keys value, raw and errors. These correspond to the app’s decoded representation, the raw JSON representation and any decoding errors respectively.


Basic website

In this example I use Vapor - the idea is that I can provide a slightly more human friendly user interface that can be deployed in a docker container. The website will again leverage the exact same production code used within the iOS application but it will be in debug mode.

Here’s what the end result looks like:

Debugging Website

It’s not pretty but it’s functional.

There are 3 sections:

  • On the top we have the errors that were output during serialisation
  • On the left we have the raw input data
  • On the right we have the decoded data as the app sees it

In the screenshot above we can see that a lot of users are not being parsed due to some website error. On the left hand side we could now inspect the raw data and compare against the items that were parsed. This should make debugging much much simpler.


I’m not going to lie the building of the website is more involved than building the CLI. The principle is the same as the CLI in that we are reusing our client and essentially mapping from the client’s representation of a result to a HTML representation.

To do this I’ve registered a single route that will run a single function called Index.action.

routes.swift

import HtmlVaporSupport
import Shared
import Vapor

public func routes(_ router: Router) throws {
    router.get(use: Index.action)
}

The Index.action function does the conversion from our shared libraries completion handler to Vapor’s promises:

IndexAction.swift

 1 import HtmlVaporSupport
 2 import Shared
 3 import Vapor
 4 
 5 enum Index {
 6     static func action(_ request: Request) -> Future<Node> {
 7         let promise = request.eventLoop.newPromise(Node.self)
 8 
 9         _ = makeClient(debugEnabled: true).fetch { result in
10             promise.fulfil(result.flatMap(prepare).map(render))
11         }
12 
13         return promise.futureResult
14     }
15 }

This function actually looks simpler than the CLI but that is purely because I have moved prepare and render out to different files.

Due to the added complexity of rendering for the web I’ve made prepare act like a presenter that is just preparing data to be shown. Then render is where the prepared data is essentially pushed into the HTML template.


Conclusion

This is one example use case or creating new tools that utilise your code in new and interesting ways. I’ve put up a Github repo that demonstrates all of the things mentioned, so anyone should be able to clone and play around with the concept to see how it all hangs together.

Handling bad input with Decodable

When consuming data from an external service we can’t always guarantee that the source data will be structured how we expect. This makes our apps potentially brittle when using Decodable because it is so strict on structure. In general everything needs to decode successfully to get results back, if any object fails to decode then you get no results at all. This is not ideal for our end users as we should fail gracefully by showing the content that our app can parse and not just show empty data screens.


Problem outline

In the following code we have a data type called Person that has two non optional properties. We also have some input data in the form of a JSON array that contains two objects. The first object has both of the required properties whereas the second does not.

struct Person: Codable {
    let name: String
    let favouriteAnimal: String
}

let data = Data("""
[
    {
        "name" : "Paul Samuels",
        "favouriteAnimal" : "Dog"
    },
    {
        "name" : "Elliot Samuels"
    }
]
""".utf8)

do {
    _ = try JSONDecoder().decode([Person].self, from: data)
} catch {
    print(error)
}

When executing the above code we do not get an array with one valid Person instance instead we end up in the catch statement with a keyNotFound error, which is caused by the second object in the JSON array not having both of the required properties.


How do we fix this?

This is actually not as easy as you might think (at least I didn’t think it was).

The first problem to solve is that the default behaviour when decoding a collection is that it will throw if decoding any of the collection’s children throws. If we decode a collection manually we can avoid this behaviour but then we encounter the issue that the decoder’s currentIndex will only progress forwards when a decode of a child object is successful. This basically means that when trying to decode the collection’s children if any of the decodes fail we won’t be able to continue iterating through the collection.

There are a few strategies that we could take:

1) Loosen the strictness of our type
This seems like a terrible idea to me. Our types should model our domain accurately e.g. if our app can’t handle a property not being present then we shouldn’t model it as an optional. I try to get rid of optionality as soon as possible or you find that you get optional handling code leaking throughout your entire codebase.

2) Create a private type that is less strict
This is an improvement on the above because we are restricting the scope of the more permissive type. Internally we might parse a private _Person type that has the optionality and then optionally convert this back to our Person type.

3) Create a wrapper to swallow any exceptions during parsing
This is fairly similar to option 2 in that we are creating a wrapper but it’s more generic as we won’t need to manually hand roll these private types.


Option 2

I’m not bothering with option 1 as it’s a weak option so I’m jumping straight to option 2.

Let’s start by creating the more permissive variant of Person where the properties are now optional:

struct _Person: Decodable {
    let name: String?
    let favouriteAnimal: String?
}

When using the [_Person].self type the decoding will no longer throw with our input data. Next we need to convert _Person instances into our stronger Person type - we’ll add a new init to do the optional conversion:

extension Person {
    init?(person: _Person) {
        guard
            let name = person.name,
            let favouriteAnimal = person.favouriteAnimal else {
                return nil
        }

        self.name            = name
        self.favouriteAnimal = favouriteAnimal
    }
}

With this scaffolding in place our call to decode the JSON now looks like this:

try JSONDecoder().decode([_Person].self, from: data).compactMap(Person.init(person:))

It’s not particularly pretty but it’s now gracefully parsing as much of the input data as possible and discarding the bits that are not usable.


Option 3

This option comes from this answer on stackoverflow. The high level idea is to wrap the type we want to decode with something that will successfully decode regardless of whether the wrapped type succeeds or not.

The code to do this looks like the following:

struct FailableDecodable<Base: Decodable>: Decodable {
    let base: Base?

    init(from decoder: Decoder) throws {
        let container = try decoder.singleValueContainer()
        self.base     = try? container.decode(Base.self)
    }
}

With this in place our decoding changes slightly to:

- try JSONDecoder().decode([_Person].self, from: data).compactMap(Person.init(person:))
+ try JSONDecoder().decode([FailableDecodable<Person>].self, from: data).compactMap { $0.base }

I think Option 3 is the stronger option here because it requires less code and less duplication of types.


How do we debug?

We’ve made our app more resilient to bad input but we’ve introduced an issue. How on earth do we debug this now when stuff goes wrong?

In general if the backend feed is providing 10 objects then our app should be able to parse and handle all 10 of those objects. If the app is silently discarding data then this could be seen as good and bad

  • Good because the user is seeing some data.
  • Bad because the app is silently throwing data away and not telling anyone.

The silencing of errors is occurring due to the try? being used in FailableDecodable. What we need to do is capture the errors rather than just discarding them.


Again this is another case where it’s not immediately obvious how we can resolve the problem. This is still thinking in progress but here’s one potential way:

  • Add some context to our Decoder for capturing errors
  • Teach FailableDecodable how to use the new context.
  • Make both of the above optional so we don’t waste resource storing/processing data that we don’t need.

Let’s start by adding context to our Decoder. A Decoder has a userInfo property that seems like it will be ideal. We’ll add a class at a known key that can hold our errors collection (it needs to be a class so that the errors collection can be mutated).

The following code performs the above but in a slightly more involved way to remove the stringly typed aspect of storing things in a Dictionary:

class DecoderDebugContext {
    var errors = [Error]()
}

private let decoderDebugContextKey = CodingUserInfoKey(rawValue: "com.paul-samuels.decoder-debug-context")!

extension JSONDecoder {
    var debugContext: DecoderDebugContext? {
        return userInfo[decoderDebugContextKey] as? DecoderDebugContext
    }

    var debugContextEnabled: Bool {
        get { return debugContext != nil }
        set {
            if newValue {
                userInfo[decoderDebugContextKey] = debugContext ?? DecoderDebugContext()
            } else {
                userInfo[decoderDebugContextKey] = nil
            }
        }
    }
}

extension Decoder {
    var debugContext: DecoderDebugContext? {
        return userInfo[decoderDebugContextKey] as? DecoderDebugContext
    }

    var debugContextEnabled: Bool {
        return debugContext != nil
    }
}

Now that we have this scaffolding in place we need to teach FailableDecodable how to use this stuff. This is essentially removing the try? and expanding it into a full try/catch and using the error:

struct FailableDecodable<Base: Decodable>: Decodable {
    let base: Base?

    init(from decoder: Decoder) throws {
        let container = try decoder.singleValueContainer()

        do {
            self.base = try container.decode(Base.self)
        } catch {
            decoder.debugContext?.errors.append(error)
            self.base = nil
        }
    }
}

Pulling all of this together we can now parse our code in a fail safe way and get both the results and errors (when we want them). To get the errors we need to configure the decoder before using it like this:

let decoder = JSONDecoder()
decoder.debugContextEnabled = true
let result = try decoder.decode([FailableDecodable<Person>].self, from: data).compactMap { $0.base }

print(result)
print(decoder.debugContext?.errors ?? [])

Conclusion

In this post we’ve looked at some motivations for wanting to parse data in a more permissive way and how you can achieve it using Decodable. Some of these things are not very obvious but once you get your head around how it works you can start to see further ways to improve debugging.


Bonus debugging fun

Getting both the results and any errors that were generated whilst creating the results is great but wouldn’t it be even better if we could also capture the raw data. This sounds a little daft because we should have access to the raw data if we are decoding it - this is true but in my experience I see people tend to chuck raw data into a decode call and leave it at that. By doing this we are losing the potential context of the raw data because we don’t parse/log it anywhere.

We can fix this problem by creating a new generic type that will decode what we want but also optionally grab the raw data as well. This example makes use of AnyCodable from Flight School.

enum DebugDecodable<T: Decodable>: Decodable {
    case debug(AnyDecodable, T)
    case simple(T)

    init(from decoder: Decoder) throws {
        let base = try decoder.singleValueContainer().decode(T.self)
        if decoder.debugContextEnabled {
            self = .debug(try decoder.singleValueContainer().decode(AnyDecodable.self), base)
        } else {
            self = .simple(base)
        }
    }
}

This is now super powerful as we have access to the raw data, the parsed data and any errors that occurred when throwing data away.

let decoder = JSONDecoder()
decoder.debugContextEnabled = true
let result = try  decoder.decode(DebugDecodable<[FailableDecodable<Person>]>.self, from: data)

print(result)
print(decoder.debugContext?.errors ?? [])