Swift Heterogeneous Codable Array

Quite the mouthful of a title but nevertheless it’s a typical problem. Receiving data from a remote service is super common but it’s not always obvious how to represent our data in a strongly typed language like Swift.

Problem outline

Let’s imagine an example where we are using a remote service that returns a collection of shapes. We have structs within our app that represent the various shapes and we want to parse the JSON objects into these native types.

Here’s the struct definitions:

struct Square: Codable {
    let length: Double

    var area: Double {
        return length * length
    }
}

struct Rectangle: Codable {
    let width: Double
    let height: Double

    var area: Double {
        return width * height
    }
}

and our JSON feed looks like this:

{
  "shapes" : [
    {
      "type" : "square",
      "attributes" : {
        "length" : 200
      }
    },
    {
      "type" : "rectangle",
      "attributes" : {
        "width" : 200,
        "height" : 300
      }
    }
  ]
}

A first attempt at a solution

Our initial attempt to parse this might end up creating a new type called FeedShape that has optional attributes for every possible shape. We can use JSONDecoder to parse the feed. Then as a second step we can map the shapes into our native types. That might look like this:

struct Feed: Decodable {
    let shapes: [FeedShape]

    struct FeedShape: Decodable {
        let type: String
        let attributes: Attributes

        struct Attributes: Decodable {
            let width: Double?
            let height: Double?
            let length: Double?
        }
    }
}

let feedShapes = try JSONDecoder().decode(Feed.self, from: json).shapes

var squares    = [Square]()
var rectangles = [Rectangle]()

for feedShape in feedShapes {
    if feedShape.type == "square", let length = feedShape.attributes.length {
        squares.append(.init(length: length))
    } else if feedShape.type == "rectangle", let width = feedShape.attributes.width, let height = feedShape.attributes.height {
        rectangles.append(.init(width: width, height: height))
    }
}

Whilst this will work it’s really not pleasant to write/maintain or use.

There are many issues with the above:

1) Optionals everywhere

Every time a new type is added that we can support within the app our Attributes struct will grow. It’s a code smell for there to be a type where most of its properties will be nil.

2) Manually checking requirements before creating objects

In order to create the concrete types we have to manually check the type property and that all the other required properties have been decoded. The code to do this is not easy to read, this fact is painful because this code ultimately is the source of truth for how to decode these objects. Looking at the current Attributes type we can see that all it’s properties are Double? - it could be quite easy to copy and paste the property checking logic and end up trying to use the wrong key across multiple types.

3) Stringly typed code

To create the concrete types we need to check the type property against a String. Having repeated strings scattered throughout a codebase is generally bad form just asking for typos and refactoring issues.

4) We’ve lost the order

Due to the way the above is modelled there is no current way to keep track of the order in which the concrete types should actually appear.

5) It’s not taking advantage of our Codable types

Our Square and Rectangle types already conform to Codable so it would be beneficial to make use of this rather than manually creating our types. Using Codable also resolves the poor documentation issue raised in 2 because for simple types the compiler will generate the Codable implementation just from the type declaration.


Can we do better?

To make an improvement that addresses 2, 4 and 5 we can deserialise our collection to an [Any] type. This requires a custom implementation of Decodable in which we loop over the items and delegate the decoding to the Shape/Rectangle decodable implementations. The code looks like the following:

struct Feed: Decodable {
    let shapes: [Any]
    
    init(from decoder: Decoder) throws {
        var container = try decoder.container(keyedBy: CodingKeys.self).nestedUnkeyedContainer(forKey: .shapes)
        
        var shapes = [Any]()
        
        while !container.isAtEnd {
            let itemContainer = try container.nestedContainer(keyedBy: CodingKeys.self)
            
            switch try itemContainer.decode(String.self, forKey: .type) {
            case "square":    shapes.append(try itemContainer.decode(Square.self, forKey: .attributes))
            case "rectangle": shapes.append(try itemContainer.decode(Rectangle.self, forKey: .attributes))
            default: fatalError("Unknown type")
            }
        }

        self.shapes = shapes
    }
    
    private enum CodingKeys: String, CodingKey {
        case attributes
        case shapes
        case type
    }
}

Although this is an improvement we still have stringly typed code and we’ve introduced another issue. Now we have an [Any] type. The use of Any can be a smell that we are not modelling things as well as we can do. This can be seen when we come to use the collection later on - we’ll be forced to do lot’s of type checking at run time. Type checking at run time is less desirable than at compile time because it means our app might crash in the wild as opposed to simply not compiling. There is also the issue that there is nothing at compile time that forces us to handle all cases e.g. I could very easily write code like this

shapes.forEach { item in
    if let square = item as? Square {
        // do square stuff
    }
    
    // Ooops I forgot to handle Rectangle's or any other new type we add
}

Can we do better still?

The issues above can all be addressed.

In order to resolve 5 we need to create an array that can contain one type or another. Enums are the mechanism for creating the sum type we need, which gives us:

enum Content {
    case square(Square)
    case rectangle(Rectangle)
}

Issues 1, 2 and 5 can all be resolved by taking advantage of the fact that our types are already Codable. If we make our new Content type Decodable we can check the type we are dealing with and then delegate the decoding to the appropriate Square/Rectangle decodable implementation.

NB: This is probably the trickiest transformation to follow, especially if you’ve not worked with custom decoding before. Just google any API you don’t recognise.

enum Content: Decodable {
    case square(Square)
    case rectangle(Rectangle)

    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)

        switch try container.decode(String.self, forKey: .type) {
        case "square":    self = .square(try container.decode(Square.self, forKey: .attributes))
        case "rectangle": self = .rectangle(try container.decode(Rectangle.self, forKey: .attributes))
        default: fatalError("Unknown type")
        }
    }

    private enum CodingKeys: String, CodingKey {
        case attributes
        case type
    }
}

Finally to resolve 3 we can leverage the exhaustive checking of switch statements on enums.

enum Content: Decodable {
    case square(Square)
    case rectangle(Rectangle)

    var unassociated: Unassociated {
        switch self {
        case .square:    return .square
        case .rectangle: return .rectangle
        }
    }

    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)

        switch try container.decode(String.self, forKey: .type) {
        case Unassociated.square.rawValue:    self = .square(try container.decode(Square.self, forKey: .attributes))
        case Unassociated.rectangle.rawValue: self = .rectangle(try container.decode(Rectangle.self, forKey: .attributes))
        default: fatalError("Unknown type")
        }
    }

    enum Unassociated: String {
        case square
        case rectangle
    }

    private enum CodingKeys: String, CodingKey {
        case attributes
        case type
    }
}

By reifying the type property from a String to a real Swift type we convert run time bugs into compile time issues, which is always a great goal to aim for.

NB: The Unassociated enum might look a little odd but it helps us model the types in one concrete place rather than having strings scattered throughout our callsites. It’s also quite useful in situations where you want to check the type of something without resorting to case syntax e.g. if we want to filter our collection to only Squares then this is one line with our new Unassociated type:

shapes.filter { $0.unassociated == .square }

without the unassociated type this ends up being something like

shapes.filter {
    if case .square = $0 {
        return true
    } else {
        return false
    }
}

// or

shapes.filter {
    switch $0 {
    case .square: return true
    default:      return false
    }
}

Conclusion

The two key takeaways here are

  • If you need to represent a collection that can have multiple types then you’ll need some form of wrapper and enums can perform that duty well when it makes sense.

  • Swift’s Codable is really powerful and helped remove a heap of issues that arise from manually parsing/creating objects.

Removing optionality, reifying types and using compiler generated code are great ways of simplifying our code. In some cases this also helps move runtime crashes into compile time issues, which is generally making our code safer. The benefits here are great and it shows that it’s really worth taking time to model your data correctly and then use tools like Codable to munge between representations.

The title was a little bit of a lie as I only walked through the Decodable part of Codable (see the listing below for the Encodable implementation).


Full code listing

The full code to throw into a playground ends up looking like this:

//: Playground - noun: a place where people can play

import Foundation

let json = Data("""
{
  "shapes" : [
    {
      "type" : "square",
      "attributes" : {
        "length" : 200
      }
    },
    {
      "type" : "rectangle",
      "attributes" : {
        "width" : 200,
        "height" : 300
      }
    }
  ]
}
""".utf8)

struct Square: Codable {
    let length: Double

    var area: Double {
        return length * length
    }
}

struct Rectangle: Codable {
    let width: Double
    let height: Double

    var area: Double {
        return width * height
    }
}

struct Feed: Codable {
    let shapes: [Content]

    enum Content: Codable {
        case square(Square)
        case rectangle(Rectangle)

        var unassociated: Unassociated {
            switch self {
            case .square:    return .square
            case .rectangle: return .rectangle
            }
        }

        init(from decoder: Decoder) throws {
            let container = try decoder.container(keyedBy: CodingKeys.self)

            switch try container.decode(String.self, forKey: .type) {
            case Unassociated.square.rawValue:    self = .square(try container.decode(Square.self, forKey: .attributes))
            case Unassociated.rectangle.rawValue: self = .rectangle(try container.decode(Rectangle.self, forKey: .attributes))
            default: fatalError("Unknown type")
            }
        }

        func encode(to encoder: Encoder) throws {
            var container = encoder.container(keyedBy: CodingKeys.self)

            switch self {
            case .square(let square):       try container.encode(square, forKey: .attributes)
            case .rectangle(let rectangle): try container.encode(rectangle, forKey: .attributes)
            }

            try container.encode(unassociated.rawValue, forKey: .type)
        }

        enum Unassociated: String {
            case square
            case rectangle
        }

        private enum CodingKeys: String, CodingKey {
            case attributes
            case type
        }
    }
}

let feed = try JSONDecoder().decode(Feed.self, from: json)
print(feed)

let jsonEncoder = JSONEncoder()
jsonEncoder.outputFormatting = .prettyPrinted
print(String(data: try jsonEncoder.encode(feed), encoding: .utf8)!)

Unit testing retain cycles

We all write retain cycles from time to time regardless of experience. Retain cycles are not always obvious to spot and can result in hours of debugging. There are of course great tools like the memory graph debugger available but debugging retain cycles can still be a painful and time consuming task.

The key thing to fixing retain cycles is detecting them. This post looks at some code you can incorporate into your unit tests to help with the discovery of retain cycles.


The basic idea

The ownership rules are fairly simple with ARC. An object will only be kept alive for as long as there is at least one strong reference to it.

class NoisyDeinit {
    deinit {
        print("I'm in deinit")
    }
}

var example: NoisyDeinit? = .init()

// Setting `example` (the only reference to the `NoisyDeinit` instance) to `nil`
// causes the instance to be deallocated and it's `deinit` will be invoked.
example = nil

Equally we know that a weak reference to an object will be nil‘d out when the last strong reference is released.

var example: NoisyDeinit? = .init()
weak var weakReference = example

assert(weakReference != nil)

// Setting `example` (the only reference to the `NoisyDeinit` instance) to `nil`
// also causes `weakReference` to be set to `nil`.
example = nil
assert(weakReference == nil)

Knowing the above we can write our tests in such a way that we have both a strong and weak reference to our object under test. After we have finished exercising our object we can set the strong reference to nil and then verify that the weak reference is also nil. If the weak reference is not nil at this point then we have to figure out what is causing the object to stay alive (this could be a retain cycle).

Let’s see how this would look. Here is a unit test without cycle checking:

final class SampleTests: XCTestCase {    
    func testGreeting() {
        let sut = Greeter()
        XCTAssertEqual(sut.greet("Paul"), "Hello Paul")
    }
}

In order to add this new checking we need to add three lines per object we want to check and make our original reference both var and Optional:

With this done the code looks like the below:

 1 final class SampleTests: XCTestCase {
 2     func testGreeting() {
 3         var sut: Greeter? = .init()
 4         weak var weakSut = sut
 5 
 6         XCTAssertEqual(sut?.greet("Paul"), "Hello Paul")
 7 
 8         sut = nil
 9         XCTAssertNil(weakSut)
10     }
11 }
  1. A new weak var to hold the object who’s lifecycle we want to verify (line 4)
  2. nil‘ing out the strong reference (line 8)
  3. The assertion that the new variable does become nil (line 9)

Can we simplify this?

Adding 3 lines per object is a little tedious and error prone. For example you may accidentally forget any one of these steps and the validation will no longer work.

We can write a couple of helper functions that we can add as an extension on XCTestCase that allow us to get this down to just one line per object who’s lifecycle we want to validate.

First let’s add a function that allows us to validate that an object is correctly deallocated after we execute an arbitrary block of caller provided code. This will be useful for scenarios where you have an instance property that is holding onto your object.

 1 extension XCTestCase {
 2     func assertNil(_ subject: AnyObject?, after: @escaping () -> Void, file: StaticString = #file, line: UInt = #line) {
 3         guard let value = subject else {
 4             return XCTFail("Argument must not be nil", file: file, line: line)
 5         }
 6 
 7         addTeardownBlock { [weak value] in
 8             after()
 9             XCTAssert(value == nil, "Expected subject to be nil after test! Retain cycle?", file: file, line: line)
10         }
11     }
12 }
  • Lines 3-5 perform a little bit of validation. It’s programmer error to set the assertion up when the object is already nil so we guard against that scenario
  • Lines 7-9 are enqueuing a closure to be invoked after the test has been run
    • Line 7 is where our weak reference to our object is created
    • Line 8 is where we execute our arbitrary closure
    • Line 9 is where we perform the assertion that our weak reference is nil‘d out

When using our helper function our unit test above becomes:

final class SampleTests: XCTestCase {
    var sut: Greeter!

    override func setUp() {
        super.setUp()
        sut = Greeter()
        assertNil(sut, after: { self.sut = nil })
    }

    func testGreeting() {
        XCTAssertEqual(sut.greet("Paul"), "Hello Paul")
    }
}

In scenarios where we don’t have an instance property holding onto our object we can provide a simpler function. We’ll write it so that it just calls through to our helper above:

extension XCTestCase {
    func assertNilAfterTest(_ subject: AnyObject?, file: StaticString = #file, line: UInt = #line) {
        assertNil(subject, after: {}, file: file, line: line)
    }
}

The above works because if there is nothing holding onto our subject outside the scope of the test function it should be naturally cleaned up by the fact that the only strong reference has gone out of scope. This allows for an even simpler test:

final class SampleTests: XCTestCase {
    func testGreeting() {
        let sut = Greeter()
        assertNilAfterTest(sut)
        XCTAssertEqual(sut.greet("Paul"), "Hello Paul")
    }
}

Conclusion

The two helper functions above make for a simple API that should hopefully be useful for helping detect those painful retain cycles before they become a real problem. The code is hopefully simple enough to understand and doesn’t require modifying existing tests too heavily (no subclassing etc).

For usage I am tempted in my own projects to litter this cycle checking throughout most tests and not make contrived tests that just create an object and check it gets deallocated. By putting this logic on most tests I can get a level of comfort that I am not creating retain cycles whilst exercising the various functions of an object.

Creating Swift Package Manager tools from your existing codebase

The Swift Package Manager (SPM) is perfect for writing quick tools and you can even bring along your existing code from your production apps. The trick is realising that you can symlink a folder into the SPM project, which means with some work you can create a command line tool that wraps parts of your production code.

Why would you want to do this?

It’s very project dependent but a common use case would be for creating support/debugging/CI validation tools. For example a lot of apps work with remote data - in order to carry out it’s function the app will need to convert the remote data into custom types and use business rules to do useful things with this data. There are multiple failure points in this flow that will manifest as either an app crash or incorrect app behaviour. The way to debug this would be to fire up the app with the debugger attached and start exploring, this is where it would be nice to have tools to help explore problems and potentially prevent them.

Caveats

You can not use code that imports UIKit which means that this technique is only going to work for Foundation based code. This sounds limiting but ideally business logic and data manipulation code shouldn’t know about UIKit.

Having dependencies makes this technique harder. You can still get this to work but it will require more configuration in Package.swift.

How do you do it?

This depends on how your project is structured. I’ve got an example project here. This project is a small iOS app that displays a list of blog posts (don’t look at the iOS project itself it’s not really important for this). The blog posts come from a fake JSON feed that doesn’t have a particularly nice structure, so the app needs to do custom decoding. In order to keep this light I’m going to build the simplest wrapper possible - it will:

  • Read from standard in
  • Use the production parsing code
  • Print the decoded results or an error

You can go wild and add a lot more features to this but this simple tool will give us really quick feedback on whether some JSON will be accepted by the production code or show any errors that might occur, all without firing up a simulator.

The basic structure of this example project looks like this:

.
└── SymlinkedSPMExample
    ├── AppDelegate.swift
    ├── Base.lproj
    │   └── LaunchScreen.storyboard
    ├── Info.plist
    ├── ViewController.swift
    └── WebService
        ├── Server.swift
        └── Types
            ├── BlogPost.swift
            └── BlogPostsRequest.swift

I have deliberately created a Types directory that contains only the code I want to reuse.

To create a command line tool that makes use of this production code I can perform the following:

mkdir -p tools/web-api
cd tools/web-api
swift package init --type executable

This has scaffolded a project that we can now manipulate. First let’s get our production source symlinked:

cd Sources
ln -s ../../../SymlinkedSPMExample/WebService/Types WebService
cd ..

You’ll want to use a relative path for the symlink or it will break when moving between machines

The project structure now looks like this:

.
├── SymlinkedSPMExample
│   ├── AppDelegate.swift
│   ├── Base.lproj
│   │   └── LaunchScreen.storyboard
│   ├── Info.plist
│   ├── ViewController.swift
│   └── WebService
│       ├── Server.swift
│       └── Types
│           ├── BlogPost.swift
│           └── BlogPostsRequest.swift
└── tools
    └── web-api
        ├── Package.swift
        ├── README.md
        ├── Sources
        │   ├── WebServer -> ../../../SymlinkedSPMExample/WebService/Types/
        │   └── web-api
        │       └── main.swift
        └── Tests

Now I need to update the Package.swift file to create a new target for this code and to add a dependency so that the web-api executable can utilise the production code.

Package.swift

// swift-tools-version:4.0

import PackageDescription

let package = Package(
    name: "web-api",
    targets: [
        .target(name: "web-api", dependencies: [ "WebService" ]),
        .target(name: "WebService"),
    ]
)

Now that SPM knows how to build the project let’s write the code mentioned above to use the production parsing code.

main.swift

import Foundation
import WebService

do {
  print(try JSONDecoder().decode(BlogPostsRequest.self, from: FileHandle.standardInput.readDataToEndOfFile()).posts)
} catch {
  print(error)
}

With this in place we can now start to run JSON through the tool and see if the production code would handle it or not:

Here’s what it looks like when we try and send valid JSON through the tool:

$ echo '{ "posts" : [] }' | swift run web-api
[]

$ echo '{ "posts" : [ { "title" : "Some post", "tags" : [] } ] }' | swift run web-api
[WebService.BlogPost(title: "Some post", tags: [])]

$ echo '{ "posts" : [ { "title" : "Some post", "tags" : [ { "value" : "cool" } ] } ] }' | swift run web-api
[WebService.BlogPost(title: "Some post", tags: ["cool"])]

Here’s an example of the error messages we get with invalid JSON:

$ echo '{}' | swift run web-api
keyNotFound(CodingKeys(stringValue: "posts", intValue: nil), Swift.DecodingError.Context(codingPath: [], debugDescription: "No value associated with key CodingKeys(stringValue: \"posts\", intValue: nil) (\"posts\").", underlyingError: nil))

$ echo '{ "posts" : [ { } ] }' | swift run web-api
keyNotFound(CodingKeys(stringValue: "title", intValue: nil), Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "posts", intValue: nil), _JSONKey(stringValue: "Index 0", intValue: 0)], debugDescription: "No value associated with key CodingKeys(stringValue: \"title\", intValue: nil) (\"title\").", underlyingError: nil))

$ echo '{ "posts" : [ { "title" : "Some post" } ] }' | swift run web-api
keyNotFound(CodingKeys(stringValue: "tags", intValue: nil), Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "posts", intValue: nil), _JSONKey(stringValue: "Index 0", intValue: 0)], debugDescription: "No value associated with key CodingKeys(stringValue: \"tags\", intValue: nil) (\"tags\").", underlyingError: nil))
  • The first example is erroring as there is no key posts
  • The second example is erroring because a post does not have a title key
  • The third example is erroring because a post does not have a tags key

In real life I would be piping the output of curling a live/staging endpoint not hand crafting JSON.

This is really cool because I can see that the production code does not parse some of these examples and I get the error messages that explain why. If I didn’t have this tool I would need to run the app manually and figure out a way to get the different JSON payloads to run through the parsing logic.

Conclusion

This post covers the basic technique of using SPM to create tools using your production code. You can really run with this and create some beautiful workflows like:

  • Add the tool as a step in the CI pipeline of the web-api to ensure no deployments could take place that break the mobile clients.
  • Expand the tool to also apply the business rules (from the production code) to see if errors are introduced at the level of the feed, parsing or business rules.

I’ve started using the idea in my own projects and I’m excited about how it going to help me and potentially other members of my team.