Accidentally Synchronous Tests

09 Jan 2019

Asynchronous code is hard and it’s very easy to make subtle mistakes. I look at a real life influenced example where we accidentally made our unit tests synchronous, therefore reducing the tests effectiveness at catching bugs.

Background

It’s worth repeating that asynchronous code is hard and it’s very easy to make subtle mistakes.

The real life issue manifested in considerably more interesting ways than what follows but the example here is just to highlight the root cause of the problem. One more disclaimer - the code that caused the issue is pretty poorly designed and is a hang over from an old Objective-C code base, which means the following code is in no way idiomatic and one could argue that more modern coding styles would never have allowed this bug.

Apologies out of the way…

We have two functions:

populate(random:) - is a function that will take a Random (a reference type class) and destructively set its number value.
process() - is a synchronous function that creates a Random, feeds it to populate(random:) and then returns the result.

(Yes I know this is a terrible example but please stick with me)

The current tests for this set up look something like this:

class Random {
    var number = Int(0)
}

class RandomNumberGenerator {
    func populate(random: Random) {
        random.number = 42
    }
}

func process(randomNumberGenerator: RandomNumberGenerator = .init()) -> Int {
    let random = Random()
    randomNumberGenerator.populate(random: random)
    return random.number
}

final class Step1: XCTestCase {
    func testProcess() {
        XCTAssertEqual(42, process())
    }
}

Everything is working great and we continue working through our backlog. A year later a new requirement comes in that we should use a fancy new random number generating web service. We integrate this new service and add the infrastructure to make it possible to stub it out in our tests. The end result is not too dissimilar. First we start by updating the RandomNumberGenerator to utilise our new service (in this case I’m just going to call the completion after a short wait).

class RandomNumberGenerator {
    var remoteNext: (@escaping (Int) -> Void) -> Void = { completion in
        DispatchQueue.main.asyncAfter(deadline: DispatchTime.now() + 3) {
            completion(42)
        }
    }

    func populate(random: Random) {
        remoteNext { number in
            random.number = number
        }
    }
}

The above provides a default implementation of remoteNext but allows us to override it in our tests because it’s just a var. The next thing to do is update our tests to provide a stub implementation of remoteNext that we control for testing.

final class Step2: XCTestCase {
    func testProcess() {
        let randomNumberGenerator = RandomNumberGenerator()

        randomNumberGenerator.remoteNext = { completion in
            completion(42)
        }

        XCTAssertEqual(42, process(randomNumberGenerator: randomNumberGenerator))
    }
}

The test passed so we move on to our next task.

NB: Unit tests are not acceptance tests and in real life we should run the automated/manual acceptance tests before moving on.

Where did it all go wrong?

Some people will have been internally screaming whilst reading the above changes. We’ve made the error of not taking a step back from our implementation and sense checking it. The process function is synchronous but the RandomNumberGenerator.populate(random:) function was updated to be asynchronous - this is not going to work. The issue has been completely masked by the fact that our unit tests were accidentally making the asynchronous RandomNumberGenerator.populate(random:) synchronous.

What does it mean to make “the asynchronous RandomNumberGenerator.populate(random:) synchronous”? Let’s demonstrate by changing the tests to be truly async again:

1
2
3
4
5
6
7
8
9
10
11
12
13
final class Step3: XCTestCase {
    func testProcess() {
        let randomNumberGenerator = RandomNumberGenerator()

        randomNumberGenerator.remoteNext = { completion in
            DispatchQueue.main.async {
                completion(42)
            }
        }

        XCTAssertEqual(42, process(randomNumberGenerator: randomNumberGenerator))
    }
}

In lines 6-8 I’m executing the completion within a DispatchQueue.async instead of calling it immediately. With this change we now get a failing test XCTAssertEqual failed: ("42") is not equal to ("0") -, which is what we expect.

Now that we have a failing test to guide us we have a few options to resolve this. Let’s update the test first to what we believe the api should now be in a fully asynchronous world:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
final class Step4: XCTestCase {
    func testProcess() {
        let randomNumberGenerator = RandomNumberGenerator()

        randomNumberGenerator.remoteNext = { completion in
            DispatchQueue.main.async {
                completion(42)
            }
        }

        weak var completionCalled = expectation(description: "Completion was called.")

        process(randomNumberGenerator: randomNumberGenerator) { number in
            XCTAssertEqual(42, number)
            completionCalled?.fulfill()
        }

        wait(for: [ completionCalled! ], timeout: 0.1)
    }
}

Lines 11, 15 and 18 are added to take advantage of the asynchronous testing features of XCTest.
Line 13 represents how the new API will look when it has been made asynchronous.

Finally we need to update the production code to make this test pass:

class RandomNumberGenerator {
    var remoteNext: (@escaping (Int) -> Void) -> Void = { completion in
        DispatchQueue.main.asyncAfter(deadline: DispatchTime.now() + 3) {
            completion(42)
        }
    }

    func populate(random: Random, completion: @escaping (Random) -> Void) {
        remoteNext { number in
            random.number = number
            completion(random)
        }
    }
}

func process(randomNumberGenerator: RandomNumberGenerator = .init(), completion: @escaping (Int) -> Void) {
    let random = Random()
    randomNumberGenerator.populate(random: random) { random in
        completion(random.number)
    }
}

The source transformation here boils down to adding completion handlers to RandomNumberGenerator.populate and process and wiring everything up. With this in place we have passing unit tests.

Conclusion

By accidentally making our asynchronous code synchronous in our tests we are not validating our code in a realistic situation. If remoteNext is performing networking then these tests are actually testing an impossible situation as we would never expect an immediate response.

In the example above we changed the implicit contract of our implementation by accident, which lead to incorrect behaviour. It could have been equally possible that we deliberately made our tests synchronous to make them easier to write. Whilst not adding the expectation boilerplate is nicer it also leads to potentially fragile tests or weakened acceptance criteria. A DispatchQueue.async may not model the production code exactly but it’s better than changing the functions implicit contract and will not add much overhead to a test run.

Swift Codable Testing

07 Jan 2019

What do you test when it comes to Swift’s Codable? I start with some thoughts on stuff that might be useful to test and then dive into TDD’ing an example of a custom Decodable implementation.

Test Candidates

Deciding what we could/should test is worth spending some good time thinking about. To start I’d consider the following:

1) How are required properties handled?
I expect that my type will decode successfully if all the required properties are present and any missing optional properties should have no effect.

2) Are source values being used?
I expect that if my source had the value Paul for the key name then I would end up with a parsed Swift type that reflects these values.

3) Can the data round trip through Codable?
If I am conforming to Codable and not just Decodable or Encodable individually then I expect my type to be able to go through both decoding/encoding without losing any data.

Off the shelf `Codable`

If you are just conforming to Codable without providing a custom init(from:) or encode(to:) then you probably don’t need to add any of your own unit tests to validate the types decoding/encoding directly. Looking at the list above we can feel pretty confident that the compiler is generating code that handles all of these cases. This is a great place to be - if I have the following type:

struct Square: Codable {
    let height: Double  
    let width: Double
}

Then I have great documentation that states that both height and width are required and attempting to decode from a source that is missing either will behave in a well defined way. Due to the fact that the implementations of both encoding/decoding are being generated by the compiler I can also be confident that it will generate the correct code to create instances that contain the values from the source.

Custom `Codable`

For cases where I am defining my own custom implementation of Codable I need to test all the scenarios above. Let’s take the example of wanting to be able to decode the following types:

struct Rectangle: Decodable, Equatable {
    let length: Double  
}

struct Square: Decodable, Equatable {
    let height: Double  
    let width: Double  
}

enum Shape: Decodable, Equatable {
    case rectangle(Rectangle)  
    case square(Square)  

    init(from decoder: Decoder) throws {
      fatalError("Not implemented")
    }
}

from JSON that looks like this:

[
  {
    "type" : "square",
    "attributes" : {
      "length" : 200
    }
  },
  {
    "type" : "rectangle",
    "attributes" : {
      "height" : 200,
      "width" : 100
    }
  }
]

First up notice that Rectangle and Square are both using the off the shelf Codable implementations provided by the compiler so it’s only Shape that we need to write tests for.

Let’s test drive this

Looking at the JSON we can see that there are keys for type and attributes that are not in our native type. Both of these keys are required in order to parse a Shape so we need to write tests to cover this (don’t worry I simplify the following monstrosity a bit further on):

final class ShapesTests: XCTestCase {
    func testDecoding_whenMissingType_itThrows() {
        XCTAssertThrowsError(try JSONDecoder().decode(Shape.self, from: fixtureMissingType)) { error in
            if case .keyNotFound(let key, _)? = error as? DecodingError {
                XCTAssertEqual("type", key.stringValue)
            } else {
                XCTFail("Expected '.keyNotFound' but got \(error)")
            }
        }
    }

    func testDecoding_whenMissingAttributes_itThrows() {
        XCTAssertThrowsError(try JSONDecoder().decode(Shape.self, from: fixtureMissingAttributes)) { error in
            if case .keyNotFound(let key, _)? = error as? DecodingError {
                XCTAssertEqual("attributes", key.stringValue)
            } else {
                XCTFail("Expected '.keyNotFound' but got \(error)")
            }
        }
    }
}

private let fixtureMissingType = Data("""
{
  "attributes" : {
    "height" : 200,
    "width" : 400
  }
}
""".utf8)

private let fixtureMissingAttributes = Data("""
{
  "type" : "square"
}
""".utf8)

These two tests are verifying that both those properties are required.

NB: I’ve not placed the fixtures within the main body of ShapesTests on purpose to make it so the main body is all about the tests. I’ve used private let outside the scope of the unit test class instead of in an extension ShapesTests because I find it tends to be easier to maintain. If I nest inside an extension I end up more often than not breaking tests if I decide to rename the test class name and don’t remember to update all the extensions.

To get these tests to pass I need to write a little bit of implementation. The simplest thing I can do is ensure that both of the keys exist without using their values and then create a default instance of Shape.rectangle (because I need to set self to something). I can do this by decode‘ing from the keyed container and assigning to _:

enum Shape: Decodable, Equatable {
    case rectangle(Rectangle)
    case square(Square)

    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)

        _ = try container.decode(String.self, forKey: .type)
        _ = try container.decode(Square.self, forKey: .attributes)

        self = .rectangle(.init(length: 200))
    }

    private enum CodingKeys: String, CodingKey {
        case attributes
        case type
    }
}

This implementation is pure nonsense but it makes the tests pass.

I’m not happy with the current tests as they are pretty hard to read and are not doing a great job of describing my intent.

Refactoring tests

I want to refactor these tests to make it really clear what their intent is. Tests that are difficult to read/understand are often just thrown away at a later date, in an effort to avoid this I’d rather spend a little bit of time making things clean.

1) Extract a helper for the assertion
That assertion is pretty difficult to read - at its core it is just checking that decoding will throw a key missing error. I create a helper by copying the original implementation into a new function and making the external interface simpler, which as you’ll see below tidies up the final callsite.

func AssertThrowsKeyNotFound<T: Decodable>(_ expectedKey: String, decoding: T.Type, from data: Data, file: StaticString = #file, line: UInt = #line) {
    XCTAssertThrowsError(try JSONDecoder().decode(decoding, from: data), file: file, line: line) { error in
        if case .keyNotFound(let key, _)? = error as? DecodingError {
            XCTAssertEqual(expectedKey, key.stringValue, "Expected missing key '\(key.stringValue)' to equal '\(expectedKey)'.", file: file, line: line)
        } else {
            XCTFail("Expected '.keyNotFound(\(expectedKey))' but got \(error)", file: file, line: line)
        }
    }
}

The tests now become:

final class ShapesTests: XCTestCase {
    func testDecoding_whenMissingType_itThrows() {
        AssertThrowsKeyNotFound("type", decoding: Shape.self, from: fixtureMissingType)
    }

    func testDecoding_whenMissingAttributes_itThrows() {
        AssertThrowsKeyNotFound("attributes", decoding: Shape.self, from: fixtureMissingAttributes)
    }
}

2) Extract helper for munging JSON

The next thing I would consider refactoring is the fixture duplication. For both of the tests above I am essentially taking a working JSON object and stripping away keys one at a time and verifying that the correct error is thrown. I can leverage some good old fashion Key Value Coding to make this simple helper:

extension Data {
    func json(deletingKeyPaths keyPaths: String...) throws -> Data {
        let decoded = try JSONSerialization.jsonObject(with: self, options: .mutableContainers) as AnyObject

        for keyPath in keyPaths {
            decoded.setValue(nil, forKeyPath: keyPath)
        }

        return try JSONSerialization.data(withJSONObject: decoded)
    }
}

With the above refactorings the entire test file now looks like this (assuming the helpers were placed in new files):

final class ShapesTests: XCTestCase {
    func testDecoding_whenMissingType_itThrows() throws {
        AssertThrowsKeyNotFound("type", decoding: Shape.self, from: try fixture.json(deletingKeyPaths: "type"))
    }

    func testDecoding_whenMissingAttributes_itThrows() throws {
        AssertThrowsKeyNotFound("attributes", decoding: Shape.self, from: try fixture.json(deletingKeyPaths: "attributes"))
    }
}

private let fixture = Data("""
{
  "type" : "square",
  "attributes" : {
    "height" : 200,
    "width" : 400
  }
}
""".utf8)

Now that the tests are looking cleaner lets add some more to force the next bit of production code to be written. I’m going to chose to just verify that type is utilised correctly. I have a feeling this test will be temporary as a later test will make it redundant but let’s write it to keep things moving:

func testDecoding_whenSquare_returnsASquare() throws {
    let result = try JSONDecoder().decode(Shape.self, from: fixture)

    if case .square = result {
        XCTAssertTrue(true)
    } else {
        XCTFail("Expected to parse a `square` but got \(result)")
    }
}

func testDecoding_whenRectangle_returnsARectangle() throws {
    let result = try JSONDecoder().decode(Shape.self, from: fixtureRectangle)

    if case .rectangle = result {
        XCTAssertTrue(true)
    } else {
        XCTFail("Expected to parse a `rectangle` but got \(result)")
    }
}

...

private let fixtureRectangle = Data("""
{
  "type" : "rectangle",
  "attributes" : {
    "height" : 200,
    "width" : 400
  }
}
""".utf8)

These tests are pretty permissive and will allow anything as long as it’s the correct enum case. The simplest thing I can write to make this pass is to hardcode some random shapes:

init(from decoder: Decoder) throws {
    let container = try decoder.container(keyedBy: CodingKeys.self)

    _ = try container.decode(Square.self, forKey: .attributes)

    switch try container.decode(String.self, forKey: .type) {
    case "rectangle": self = .rectangle(.init(length: 200))
    case "square":    self = .square(.init(height: 200, width: 400))
    default: fatalError("Unhandled type")
    }
}

At this point my tests are now a bit rubbish. I’ve had to provide a new fixtureRectangle which isn’t actually representing a valid source anymore. It makes sense to remove these tests and write some more reasonable assertions. I’ll start by addressing squares first:

func testDecoding_whenSquare_returnsASquare() throws {
    XCTAssertEqual(.square(square), try JSONDecoder().decode(Shape.self, from: fixture))
}

...

private let square = Square(height: 200, width: 400)

The tests pass and I didn’t change the current implementation of init(from:), which is slightly worrying as I know I hardcoded values. The best thing to do is to write another assertion with different data:

func testDecoding_whenSquare_returnsASquare() throws {
    XCTAssertEqual(.square(square), try JSONDecoder().decode(Shape.self, from: fixture))
    XCTAssertEqual(.square(square2), try JSONDecoder().decode(Shape.self, from: fixture2))
}

...

private let square2 = Square(height: 100, width: 200)

private let fixture2 = Data("""
{
  "type" : "square",
  "attributes" : {
    "height" : 100,
    "width" : 200
  }
}
""".utf8)

This gets us back to a broken test as the hardcoded values I return no longer match those I expect with different fixture data.

Making the parsing code work is just a case of updating the case "square" logic in the init(from:) func:

init(from decoder: Decoder) throws {
    let container = try decoder.container(keyedBy: CodingKeys.self)

    _ = try container.decode(Square.self, forKey: .attributes)

    switch try container.decode(String.self, forKey: .type) {
    case "rectangle": self = .rectangle(.init(length: 200))
    case "square":    self = .square(try container.decode(Square.self, forKey: .attributes))
    default: fatalError("Unhandled type")
    }
}

Let’s repeat the above to get Rectangles working as well.

Add some tests:

func testDecoding_whenRectangle_returnsARectangle() throws {
    XCTAssertEqual(.rectangle(rectangle), try JSONDecoder().decode(Shape.self, from: fixture3))
    XCTAssertEqual(.rectangle(rectangle2), try JSONDecoder().decode(Shape.self, from: fixture4))
}

...

private let rectangle = Rectangle(length: 100)
private let rectangle2 = Rectangle(length: 200)

private let fixture3 = Data("""
{
  "type" : "rectangle",
  "attributes" : {
    "length" : 100
  }
}
""".utf8)

private let fixture4 = Data("""
{
  "type" : "rectangle",
  "attributes" : {
    "length" : 200
  }
}
""".utf8)

Then updating the parsing code becomes:

init(from decoder: Decoder) throws {
    let container = try decoder.container(keyedBy: CodingKeys.self)

    switch try container.decode(String.self, forKey: .type) {
    case "rectangle": self = .rectangle(try container.decode(Rectangle.self, forKey: .attributes))
    case "square":    self = .square(try container.decode(Square.self, forKey: .attributes))
    default: fatalError("Unhandled type")
    }
}

Recap

We’ve now written tests that cover the requirements at the top (for the Decodable half of the Codable story). We’ve verified that required keys are in fact required and checked that when you do use a decoder your newly created types take the values from the source.

Looking at this new code (listed below) I can see that adding all these new tests has gotten really ugly:

final class ShapesTests: XCTestCase {
    func testDecoding_whenMissingType_itThrows() throws {
        AssertThrowsKeyNotFound("type", decoding: Shape.self, from: try fixture.json(deletingKeyPaths: "type"))
    }

    func testDecoding_whenMissingAttributes_itThrows() throws {
        AssertThrowsKeyNotFound("attributes", decoding: Shape.self, from: try fixture.json(deletingKeyPaths: "attributes"))
    }

    func testDecoding_whenSquare_returnsASquare() throws {
        XCTAssertEqual(.square(square), try JSONDecoder().decode(Shape.self, from: fixture))
        XCTAssertEqual(.square(square2), try JSONDecoder().decode(Shape.self, from: fixture2))
    }

    func testDecoding_whenRectangle_returnsARectangle() throws {
        XCTAssertEqual(.rectangle(rectangle), try JSONDecoder().decode(Shape.self, from: fixture3))
        XCTAssertEqual(.rectangle(rectangle2), try JSONDecoder().decode(Shape.self, from: fixture4))
    }
}

private let square = Square(height: 200, width: 400)
private let square2 = Square(height: 100, width: 200)

private let rectangle = Rectangle(length: 100)
private let rectangle2 = Rectangle(length: 200)

private let fixture = Data("""
{
  "type" : "square",
  "attributes" : {
    "height" : 200,
    "width" : 400
  }
}
""".utf8)

private let fixture2 = Data("""
{
  "type" : "square",
  "attributes" : {
    "height" : 100,
    "width" : 200
  }
}
""".utf8)

private let fixture3 = Data("""
{
  "type" : "rectangle",
  "attributes" : {
    "length" : 100
  }
}
""".utf8)

private let fixture4 = Data("""
{
  "type" : "rectangle",
  "attributes" : {
    "length" : 200
  }
}
""".utf8)

My concerns here are primarily based around adding lots of new fixtures that are poorly named. I could improve the naming but then I feel like the data definition and usage is really far apart. I’d most likely still have to go hunting to look at the fixtures regardless of the name.

I can start to tidy this up by inlining square, square2, rectangle and rectangle2 and deleting the constants:

func testDecoding_whenSquare_returnsASquare() throws {
    XCTAssertEqual(.square(.init(height: 200, width: 400)), try JSONDecoder().decode(Shape.self, from: fixture))
    XCTAssertEqual(.square(.init(height: 100, width: 200)), try JSONDecoder().decode(Shape.self, from: fixture2))
}

func testDecoding_whenRectangle_returnsARectangle() throws {
    XCTAssertEqual(.rectangle(.init(length: 100)), try JSONDecoder().decode(Shape.self, from: fixture3))
    XCTAssertEqual(.rectangle(.init(length: 200)), try JSONDecoder().decode(Shape.self, from: fixture4))
}

A further enhancement would be to bring the fixture data into the body of the test as well. We could use a similar idea from earlier where we just munge some existing data inside the test body where it is used so everything is kept local to the test. Here’s the helper function, which again leverages the power of Key Value Coding:

extension Data {
    func json(updatingKeyPaths keyPaths: (String, Any)...) throws -> Data {
        let decoded = try JSONSerialization.jsonObject(with: self, options: .mutableContainers) as AnyObject

        for (keyPath, value) in keyPaths {
            decoded.setValue(value, forKeyPath: keyPath)
        }

        return try JSONSerialization.data(withJSONObject: decoded)
    }
}

Using the new helper gives the following tests. The code isn’t as short anymore but all the data is local to the tests making it easier for future readers to figure out what data is changing to drive the various conditions:

func testDecoding_whenSquare_returnsASquare() throws {
    XCTAssertEqual(.square(.init(height: 200, width: 400)), try JSONDecoder().decode(Shape.self, from: fixture))
    XCTAssertEqual(.square(.init(height: 100, width: 200)), try JSONDecoder().decode(Shape.self, from: fixture.json(updatingKeyPaths: ("attributes", [ "height" : 100, "width" : 200 ]))))
}

func testDecoding_whenRectangle_returnsARectangle() throws {
    let rectangle = { try fixture.json(updatingKeyPaths: ("type", "rectangle"), ("attributes", [ "length" : $0 ])) }

    XCTAssertEqual(.rectangle(.init(length: 100)), try JSONDecoder().decode(Shape.self, from: rectangle(100)))
    XCTAssertEqual(.rectangle(.init(length: 200)), try JSONDecoder().decode(Shape.self, from: rectangle(200)))
}

For reference the full code listing after all of the above is:

Shapes.swift

struct Rectangle: Decodable, Equatable {
    let length: Double
}

struct Square: Decodable, Equatable {
    let height: Double
    let width: Double
}

enum Shape: Decodable, Equatable {
    case rectangle(Rectangle)
    case square(Square)

    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)

        switch try container.decode(String.self, forKey: .type) {
        case "rectangle": self = .rectangle(try container.decode(Rectangle.self, forKey: .attributes))
        case "square":    self = .square(try container.decode(Square.self, forKey: .attributes))
        default: fatalError("Unhandled type")
        }
    }

    private enum CodingKeys: String, CodingKey {
        case attributes
        case type
    }
}

ShapesTests.swift

final class ShapesTests: XCTestCase {
    func testDecoding_whenMissingType_itThrows() throws {
        AssertThrowsKeyNotFound("type", decoding: Shape.self, from: try fixture.json(deletingKeyPaths: "type"))
    }

    func testDecoding_whenMissingAttributes_itThrows() throws {
        AssertThrowsKeyNotFound("attributes", decoding: Shape.self, from: try fixture.json(deletingKeyPaths: "attributes"))
    }

    func testDecoding_whenSquare_returnsASquare() throws {
        XCTAssertEqual(.square(.init(height: 200, width: 400)), try JSONDecoder().decode(Shape.self, from: fixture))
        XCTAssertEqual(.square(.init(height: 100, width: 200)), try JSONDecoder().decode(Shape.self, from: fixture.json(updatingKeyPaths: ("attributes", [ "height" : 100, "width" : 200 ]))))
    }

    func testDecoding_whenRectangle_returnsARectangle() throws {
        let rectangle = { try fixture.json(updatingKeyPaths: ("type", "rectangle"), ("attributes", [ "length" : $0 ])) }

        XCTAssertEqual(.rectangle(.init(length: 100)), try JSONDecoder().decode(Shape.self, from: rectangle(100)))
        XCTAssertEqual(.rectangle(.init(length: 200)), try JSONDecoder().decode(Shape.self, from: rectangle(200)))
    }
}

private let fixture = Data("""
{
  "type" : "square",
  "attributes" : {
    "height" : 200,
    "width" : 400
  }
}
""".utf8)

Conclusion

Testing Codable implementations isn’t particularly hard but the boilerplate code required can get out of hand pretty quickly. I thought I’d run through a TDD process to get to the final solution as I find this stuff personally interesting and hopefully someone else might to. Hopefully I’ve highlighted some basic stuff to test when looking at custom Decodable implementations and shown that it’s useful to refactor not only the production code but the test code as well.

Swift Heterogeneous Codable Array

02 Jan 2019

Quite the mouthful of a title but nevertheless it’s a typical problem. Receiving data from a remote service is super common but it’s not always obvious how to represent our data in a strongly typed language like Swift.

Problem outline

Let’s imagine an example where we are using a remote service that returns a collection of shapes. We have structs within our app that represent the various shapes and we want to parse the JSON objects into these native types.

Here’s the struct definitions:

struct Square: Codable {
    let length: Double

    var area: Double {
        return length * length
    }
}

struct Rectangle: Codable {
    let width: Double
    let height: Double

    var area: Double {
        return width * height
    }
}

and our JSON feed looks like this:

{
  "shapes" : [
    {
      "type" : "square",
      "attributes" : {
        "length" : 200
      }
    },
    {
      "type" : "rectangle",
      "attributes" : {
        "width" : 200,
        "height" : 300
      }
    }
  ]
}

A first attempt at a solution

Our initial attempt to parse this might end up creating a new type called FeedShape that has optional attributes for every possible shape. We can use JSONDecoder to parse the feed. Then as a second step we can map the shapes into our native types. That might look like this:

struct Feed: Decodable {
    let shapes: [FeedShape]

    struct FeedShape: Decodable {
        let type: String
        let attributes: Attributes

        struct Attributes: Decodable {
            let width: Double?
            let height: Double?
            let length: Double?
        }
    }
}

let feedShapes = try JSONDecoder().decode(Feed.self, from: json).shapes

var squares    = [Square]()
var rectangles = [Rectangle]()

for feedShape in feedShapes {
    if feedShape.type == "square", let length = feedShape.attributes.length {
        squares.append(.init(length: length))
    } else if feedShape.type == "rectangle", let width = feedShape.attributes.width, let height = feedShape.attributes.height {
        rectangles.append(.init(width: width, height: height))
    }
}

Whilst this will work it’s really not pleasant to write/maintain or use.

There are many issues with the above:

1) Optionals everywhere

Every time a new type is added that we can support within the app our Attributes struct will grow. It’s a code smell for there to be a type where most of its properties will be nil.

2) Manually checking requirements before creating objects

In order to create the concrete types we have to manually check the type property and that all the other required properties have been decoded. The code to do this is not easy to read, this fact is painful because this code ultimately is the source of truth for how to decode these objects. Looking at the current Attributes type we can see that all it’s properties are Double? - it could be quite easy to copy and paste the property checking logic and end up trying to use the wrong key across multiple types.

3) Stringly typed code

To create the concrete types we need to check the type property against a String. Having repeated strings scattered throughout a codebase is generally bad form just asking for typos and refactoring issues.

4) We’ve lost the order

Due to the way the above is modelled there is no current way to keep track of the order in which the concrete types should actually appear.

5) It’s not taking advantage of our Codable types

Our Square and Rectangle types already conform to Codable so it would be beneficial to make use of this rather than manually creating our types. Using Codable also resolves the poor documentation issue raised in 2 because for simple types the compiler will generate the Codable implementation just from the type declaration.

Can we do better?

To make an improvement that addresses 2, 4 and 5 we can deserialise our collection to an [Any] type. This requires a custom implementation of Decodable in which we loop over the items and delegate the decoding to the Shape/Rectangle decodable implementations. The code looks like the following:

struct Feed: Decodable {
    let shapes: [Any]
    
    init(from decoder: Decoder) throws {
        var container = try decoder.container(keyedBy: CodingKeys.self).nestedUnkeyedContainer(forKey: .shapes)
        
        var shapes = [Any]()
        
        while !container.isAtEnd {
            let itemContainer = try container.nestedContainer(keyedBy: CodingKeys.self)
            
            switch try itemContainer.decode(String.self, forKey: .type) {
            case "square":    shapes.append(try itemContainer.decode(Square.self, forKey: .attributes))
            case "rectangle": shapes.append(try itemContainer.decode(Rectangle.self, forKey: .attributes))
            default: fatalError("Unknown type")
            }
        }

        self.shapes = shapes
    }
    
    private enum CodingKeys: String, CodingKey {
        case attributes
        case shapes
        case type
    }
}

Although this is an improvement we still have stringly typed code and we’ve introduced another issue. Now we have an [Any] type. The use of Any can be a smell that we are not modelling things as well as we can do. This can be seen when we come to use the collection later on - we’ll be forced to do lot’s of type checking at run time. Type checking at run time is less desirable than at compile time because it means our app might crash in the wild as opposed to simply not compiling. There is also the issue that there is nothing at compile time that forces us to handle all cases e.g. I could very easily write code like this

shapes.forEach { item in
    if let square = item as? Square {
        // do square stuff
    }
    
    // Ooops I forgot to handle Rectangle's or any other new type we add
}

Can we do better still?

The issues above can all be addressed.

In order to resolve 5 we need to create an array that can contain one type or another. Enums are the mechanism for creating the sum type we need, which gives us:

enum Content {
    case square(Square)
    case rectangle(Rectangle)
}

Issues 1, 2 and 5 can all be resolved by taking advantage of the fact that our types are already Codable. If we make our new Content type Decodable we can check the type we are dealing with and then delegate the decoding to the appropriate Square/Rectangle decodable implementation.

NB: This is probably the trickiest transformation to follow, especially if you’ve not worked with custom decoding before. Just google any API you don’t recognise.

enum Content: Decodable {
    case square(Square)
    case rectangle(Rectangle)

    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)

        switch try container.decode(String.self, forKey: .type) {
        case "square":    self = .square(try container.decode(Square.self, forKey: .attributes))
        case "rectangle": self = .rectangle(try container.decode(Rectangle.self, forKey: .attributes))
        default: fatalError("Unknown type")
        }
    }

    private enum CodingKeys: String, CodingKey {
        case attributes
        case type
    }
}

Finally to resolve 3 we can leverage the exhaustive checking of switch statements on enums.

enum Content: Decodable {
    case square(Square)
    case rectangle(Rectangle)

    var unassociated: Unassociated {
        switch self {
        case .square:    return .square
        case .rectangle: return .rectangle
        }
    }

    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)

        switch try container.decode(String.self, forKey: .type) {
        case Unassociated.square.rawValue:    self = .square(try container.decode(Square.self, forKey: .attributes))
        case Unassociated.rectangle.rawValue: self = .rectangle(try container.decode(Rectangle.self, forKey: .attributes))
        default: fatalError("Unknown type")
        }
    }

    enum Unassociated: String {
        case square
        case rectangle
    }

    private enum CodingKeys: String, CodingKey {
        case attributes
        case type
    }
}

By reifying the type property from a String to a real Swift type we convert run time bugs into compile time issues, which is always a great goal to aim for.

NB: The Unassociated enum might look a little odd but it helps us model the types in one concrete place rather than having strings scattered throughout our callsites. It’s also quite useful in situations where you want to check the type of something without resorting to case syntax e.g. if we want to filter our collection to only Squares then this is one line with our new Unassociated type:

shapes.filter { $0.unassociated == .square }

without the unassociated type this ends up being something like

shapes.filter {
    if case .square = $0 {
        return true
    } else {
        return false
    }
}

// or

shapes.filter {
    switch $0 {
    case .square: return true
    default:      return false
    }
}

Conclusion

The two key takeaways here are

If you need to represent a collection that can have multiple types then you’ll need some form of wrapper and enums can perform that duty well when it makes sense.
Swift’s Codable is really powerful and helped remove a heap of issues that arise from manually parsing/creating objects.

Removing optionality, reifying types and using compiler generated code are great ways of simplifying our code. In some cases this also helps move runtime crashes into compile time issues, which is generally making our code safer. The benefits here are great and it shows that it’s really worth taking time to model your data correctly and then use tools like Codable to munge between representations.

The title was a little bit of a lie as I only walked through the Decodable part of Codable (see the listing below for the Encodable implementation).

Full code listing

The full code to throw into a playground ends up looking like this:

//: Playground - noun: a place where people can play

import Foundation

let json = Data("""
{
  "shapes" : [
    {
      "type" : "square",
      "attributes" : {
        "length" : 200
      }
    },
    {
      "type" : "rectangle",
      "attributes" : {
        "width" : 200,
        "height" : 300
      }
    }
  ]
}
""".utf8)

struct Square: Codable {
    let length: Double

    var area: Double {
        return length * length
    }
}

struct Rectangle: Codable {
    let width: Double
    let height: Double

    var area: Double {
        return width * height
    }
}

struct Feed: Codable {
    let shapes: [Content]

    enum Content: Codable {
        case square(Square)
        case rectangle(Rectangle)

        var unassociated: Unassociated {
            switch self {
            case .square:    return .square
            case .rectangle: return .rectangle
            }
        }

        init(from decoder: Decoder) throws {
            let container = try decoder.container(keyedBy: CodingKeys.self)

            switch try container.decode(String.self, forKey: .type) {
            case Unassociated.square.rawValue:    self = .square(try container.decode(Square.self, forKey: .attributes))
            case Unassociated.rectangle.rawValue: self = .rectangle(try container.decode(Rectangle.self, forKey: .attributes))
            default: fatalError("Unknown type")
            }
        }

        func encode(to encoder: Encoder) throws {
            var container = encoder.container(keyedBy: CodingKeys.self)

            switch self {
            case .square(let square):       try container.encode(square, forKey: .attributes)
            case .rectangle(let rectangle): try container.encode(rectangle, forKey: .attributes)
            }

            try container.encode(unassociated.rawValue, forKey: .type)
        }

        enum Unassociated: String {
            case square
            case rectangle
        }

        private enum CodingKeys: String, CodingKey {
            case attributes
            case type
        }
    }
}

let feed = try JSONDecoder().decode(Feed.self, from: json)
print(feed)

let jsonEncoder = JSONEncoder()
jsonEncoder.outputFormatting = .prettyPrinted
print(String(data: try jsonEncoder.encode(feed), encoding: .utf8)!)

Older Newer

paul-samuels.com

Accidentally Synchronous Tests

Background

Where did it all go wrong?

Conclusion

Swift Codable Testing

Test Candidates

Off the shelf Codable

Custom Codable

Let’s test drive this

Refactoring tests

Recap

Conclusion

Swift Heterogeneous Codable Array

Problem outline

A first attempt at a solution

Can we do better?

Can we do better still?

Conclusion

Full code listing

Off the shelf `Codable`

Custom `Codable`