When consuming data from an external service we can’t always guarantee that the source data will be structured how we expect.
This makes our apps potentially brittle when using Decodable because it is so strict on structure.
In general everything needs to decode successfully to get results back, if any object fails to decode then you get no results at all.
This is not ideal for our end users as we should fail gracefully by showing the content that our app can parse and not just show empty data screens.
Problem outline
In the following code we have a data type called Person that has two non optional properties.
We also have some input data in the form of a JSON array that contains two objects.
The first object has both of the required properties whereas the second does not.
When executing the above code we do not get an array with one valid Person instance instead we end up in the catch statement with a keyNotFound error, which is caused by the second object in the JSON array not having both of the required properties.
How do we fix this?
This is actually not as easy as you might think (at least I didn’t think it was).
The first problem to solve is that the default behaviour when decoding a collection is that it will throw if decoding any of the collection’s children throws.
If we decode a collection manually we can avoid this behaviour but then we encounter the issue that the decoder’s currentIndex will only progress forwards when a decode of a child object is successful.
This basically means that when trying to decode the collection’s children if any of the decodes fail we won’t be able to continue iterating through the collection.
There are a few strategies that we could take:
1) Loosen the strictness of our type
This seems like a terrible idea to me.
Our types should model our domain accurately e.g. if our app can’t handle a property not being present then we shouldn’t model it as an optional.
I try to get rid of optionality as soon as possible or you find that you get optional handling code leaking throughout your entire codebase.
2) Create a private type that is less strict
This is an improvement on the above because we are restricting the scope of the more permissive type.
Internally we might parse a private _Person type that has the optionality and then optionally convert this back to our Person type.
3) Create a wrapper to swallow any exceptions during parsing
This is fairly similar to option 2 in that we are creating a wrapper but it’s more generic as we won’t need to manually hand roll these private types.
Option 2
I’m not bothering with option 1 as it’s a weak option so I’m jumping straight to option 2.
Let’s start by creating the more permissive variant of Person where the properties are now optional:
When using the [_Person].self type the decoding will no longer throw with our input data.
Next we need to convert _Person instances into our stronger Person type - we’ll add a new init to do the optional conversion:
It’s not particularly pretty but it’s now gracefully parsing as much of the input data as possible and discarding the bits that are not usable.
Option 3
This option comes from this answer on stackoverflow.
The high level idea is to wrap the type we want to decode with something that will successfully decode regardless of whether the wrapped type succeeds or not.
I think Option 3 is the stronger option here because it requires less code and less duplication of types.
How do we debug?
We’ve made our app more resilient to bad input but we’ve introduced an issue.
How on earth do we debug this now when stuff goes wrong?
In general if the backend feed is providing 10 objects then our app should be able to parse and handle all 10 of those objects.
If the app is silently discarding data then this could be seen as good and bad
Good because the user is seeing some data.
Bad because the app is silently throwing data away and not telling anyone.
The silencing of errors is occurring due to the try? being used in FailableDecodable.
What we need to do is capture the errors rather than just discarding them.
Again this is another case where it’s not immediately obvious how we can resolve the problem.
This is still thinking in progress but here’s one potential way:
Add some context to our Decoder for capturing errors
Teach FailableDecodable how to use the new context.
Make both of the above optional so we don’t waste resource storing/processing data that we don’t need.
Let’s start by adding context to our Decoder.
A Decoder has a userInfo property that seems like it will be ideal.
We’ll add a class at a known key that can hold our errors collection (it needs to be a class so that the errors collection can be mutated).
The following code performs the above but in a slightly more involved way to remove the stringly typed aspect of storing things in a Dictionary:
Now that we have this scaffolding in place we need to teach FailableDecodable how to use this stuff.
This is essentially removing the try? and expanding it into a full try/catch and using the error:
Pulling all of this together we can now parse our code in a fail safe way and get both the results and errors (when we want them).
To get the errors we need to configure the decoder before using it like this:
In this post we’ve looked at some motivations for wanting to parse data in a more permissive way and how you can achieve it using Decodable.
Some of these things are not very obvious but once you get your head around how it works you can start to see further ways to improve debugging.
Bonus debugging fun
Getting both the results and any errors that were generated whilst creating the results is great but wouldn’t it be even better if we could also capture the raw data.
This sounds a little daft because we should have access to the raw data if we are decoding it - this is true but in my experience I see people tend to chuck raw data into a decode call and leave it at that.
By doing this we are losing the potential context of the raw data because we don’t parse/log it anywhere.
We can fix this problem by creating a new generic type that will decode what we want but also optionally grab the raw data as well. This example makes use of AnyCodable from Flight School.
Testing is fairly common practice but there are plenty of rough edges that catch people out. One area that causes trouble is testing across module boundaries. In this post I’m going to walk through the evolution of an API and explain how to approach testing at each step, which often requires evolving the API slightly.
Understanding how to test different scenarios is important because it empowers us to craft the boundaries we want without having to compromise on testing or aesthetics.
There are a few key approaches that I’m going to cover:
The relationship between these types is PersonRepository --> TransientStore, which is to say that the PersonRepository has a strong dependency on TransientStore and knows the type by name.
What do we want to test?
Before we dive into analysing the current structure I think it’s important to highlight exactly what I feel is important to test here for this blog post.
From within my main application I want to test the collaboration between PersonRepository and TransientStore - this is the collaboration across the module boundary. In more concrete terms I want to be able to write tests like:
If I call PersonRepository.fetch(id:) it should:
Invoke TransientStore.get(key:) with the id value that was passed to the original function
If data is returned it should attempt to JSON decode it
If I call PersonRepository.store(id:person:) it should:
Attempt to JSON encode the person passed to the original function
Invoke TransientStore.set(key:value:) with the id from the original function and the encoded person
The above are the high level collaborations, in reality there would be many permutations of these tests to validate what happens for the unhappy paths like invalid input etc.
What I am not interested in for the sake of this blog is testing the behaviour of TransientStore. In a real project I would expect that TransientStore is well tested to ensure that it honours the public contract that it provides.
Subclass and Substitute
With this first iteration I can test this collaboration by subclassing TransientStore and overriding it’s various functions to create a test double. Here’s an implementation of this test double:
To show how this would be used - here’s the two test cases I mentioned above:
finalclassPersonRepositoryTests:XCTestCase{vartransientStoreMock:TransientStoreMock!varsut:PersonRepository!overridefuncsetUp(){super.setUp()transientStoreMock=TransientStoreMock()sut=PersonRepository(cache:transientStoreMock)}overridefunctearDown(){transientStoreMock=nilsut=nilsuper.tearDown()}functestFetch_collaboratesWithTransientStore(){letexpectedID="12345"transientStoreMock.onFetchCalled={XCTAssertEqual(expectedID,$0)returnself.encodedPerson}XCTAssertEqual(.fake(),sut.fetch(id:expectedID))}functestStore_collaboratesWithTransientStore(){letexpectedID="12345"varwasCalled=falsetransientStoreMock.onStoreCalled={XCTAssertEqual(expectedID,$0)XCTAssertEqual(self.encodedPerson,$1)wasCalled=true}sut.store(id:expectedID,person:.fake())XCTAssertTrue(wasCalled)}varencodedPerson:Data{// Happy to use a force cast here because if this fails there is something really wrong.returntry!JSONEncoder().encode(Person.fake())}}extensionPerson{staticfuncfake(name:String="Elliot")->Person{return.init(name:name)}}
This works but there are a few things I’m not keen on:
To actually make this work I need to update TransientStore to be open so that it can be subclassed externally. This is not a great change to be making just to enable tests. The mere addition of the open access control modifier may suggest to an API user that this type is intended to be subclassed.
This only works for class types so we need a different solution for struct and enum.
There is a burden on me as a test writer to know what to override in our TransientStore subclass. If I don’t override the right stuff then my tests will not be isolated and could be causing all kinds of side effects.
Before moving on… if the above technique fits your needs and you don’t share my concerns then by all means use it - there really is no right and wrong if stuff works for your requirements.
Use an Interface
We can resolve all 3 of the issues above by making PersonRepository depend on an interface that we’ll call Store and then make TransientStore depend on the same interface.
This has the effect of inverting the direction of the dependency. Doing this would give us the following (notice how the arrows all point away from the concrete details):
PersonRepository --> Store (protocol) <-- TransientStore
Let’t take a look at the changes required to get this working. We’ll update the Storage module first:
+ public protocol Store {
+ func get(key: String) -> Data?
+ func set(key: String, value: Data?)
+ }
- public class TransientStore {
+ public class TransientStore: Store {
public init()
public func get(key: String) -> Data?
public func set(key: String, value: Data?)
}
Above I’ve added Store as a protocol. TransientStore is almost identical to our first implementation except we are able to remove the open modifier and we conform to Store.
With this change in place we can update the PersonRepository to the following:
final class PersonRepository {
- let cache: TransientStore
+ let cache: Store
- init(cache: TransientStore = .init()) {
+ init(cache: Store = TransientStore()) {
self.cache = cache
}
func fetch(id: String) -> Person? {
return cache.get(key: id).flatMap { try? JSONDecoder().decode(Person.self, from: $0) }
}
func store(id: String, person: Person) {
cache.set(key: id, value: try? JSONEncoder().encode(person))
}
}
The only difference here is that all references to TransientStore have been replaced with Store except for the default argument instantiation in the initialiser.
With this the body of the tests can remain identical but we need to update the test double to conform to a protocol rather than subclassing:
- class TransientStoreMock: TransientStore {
+ class StoreMock: Store {
var onFetchCalled: (String) -> Data? = { _ in nil }
var onStoreCalled: (String, Data?) -> Void = { _, _ in }
func get(key: String) -> Data? {
return onFetchCalled(key)
}
func set(key: String, value: Data?) {
onStoreCalled(key, value)
}
}
As promised this resolves all 3 issues mentioned above and it didn’t really require many changes. The first two are resolved because we have removed the inheritance aspect. The third issue is resolved because if we modify the protocol to add a new requirement then our tests will no longer compile.
This gets me to my happy place where I am doing compiler driven development, which means I just fix all the things the compiler complains about.
I do have a gripe with the above solution and it’s that we’ve hidden some details because we are using a protocol but I still had to reference the TransientStore type by name within the PersonRepository, which highlights that TransientStore is still publicly visible. If we look at the public header for our Storage module again we can see that it leaks implementation details:
As a consumer of the module I might assume that it would be sensible to use TransientStore directly as it’s freely provided in the public API.
Hiding Details
We can resolve the above issue by hiding the concrete TransientStore type entirely. The way to do this is to provide a factory function that will create a TransientStore but it won’t externally reference the TransientStore type. We can then set everything on TransientStore to have internal visibility:
+ public func makeTransientStore() -> Store {
+ return TransientStore()
+ }
- public class TransientStore: Store {
- public init()
- public func get(key: String) -> Data?
- public func set(key: String, value: Data?)
- }
+ class TransientStore: Store {
+ init()
+ func get(key: String) -> Data?
+ func set(key: String, value: Data?)
+ }
It may not seem like we did anything there apart from change some visibility but the end result is the public interface for the Storage module is now much simpler:
public protocol Store {
public func get(key: String) -> Data?
public func set(key: String, value: Data?)
}
+ public func makeTransientStore() -> Store
- public class TransientStore: Store {
- public init()
- public func get(key: String) -> Data?
- public func set(key: String, value: Data?)
- }
As you can see there is no mention of the actual type TransientStore. The function name does include the name but this is just a label it’s not the actual type itself being leaked.
At this point we have a nice seam that allows us to provide alternate Store implementations into our code base, whether that be in tests or in production code.
Type Erasure
Type erasure can be pretty daunting but it’s really useful when you know when it can be utilised. I don’t think I’ll ever get to the point where I use it often enough that I remember how to do it without googling - maybe I’ll end up back on this post in the not too distant future.
Continuing with our example above we might wonder if we can make our API more generic and use any Hashable type as the key.
To achieve this in Swift we need to add an associatedtype to the Store protocol and use the new type where we was previously hardcoding the String type:
public protocol Store {
+ associatedtype Key: Hashable
- public func get(key: String) -> Data?
- public func set(key: String, value: Data?)
+ public func get(key: Key) -> Data?
+ public func set(key: Key, value: Data?)
}
Updating the TransientStore to conform to this interface requires that we make the class generic:
- class TransientStore: Store {
- func get(key: String) -> Data?
- func set(key: String, value: Data?)
- }
+ class TransientStore<Key: Hashable>: Store {
+ func get(key: Key) -> Data?
+ func set(key: Key, value: Data?)
+ }
The changes so far are valid but the compiler starts getting very unhappy with our factory function for creating a TransientStore
publicfuncmakeTransientStore()->Store{//=> Protocol 'Store' can only be used as a generic constraint because it has Self or associated type requirementsreturnTransientStore()//=> Return expression of type 'TransientStore<_>' does not conform to 'Store'}
This isn’t going to work because the associatedtype means that we can’t use Store in the following places:
As the return type of this function.
As the type of cache variable in PersonRepository.
We have two options to get around this restriction.
1) Forget about the interface approach and go back to using the concrete type directly - just like in the problem statement. 2) Create a type eraser that acts as a wrapper over our concrete types.
As you can tell from the less than positive wording of 1 I’m not going to go that route in this post. Again if this is the right solution for your code base then go ahead and use it.
The mechanics of what we will do is:
A) Create a concrete type which follows the naming convention of adding Any to the beginning of our type e.g. AnyStore. B) The AnyStore will be generic over the key’s type where Key: Hashable. C) When instantiating an AnyStore<Key> you will need to provide an instance to wrap, which will need to conform to Store. D) Replace references to Store within function return types or variable declarations with our new AnyStore<Key> type.
Line 1 is defining our new type and stating that it’s generic over a Key type that must be Hashable.
Lines 5-8 is where most of the heavy lifting is done. We are taking in another concrete type that conforms to Store and grabbing all of it’s functions and placing them into some variables 2-3.
By doing this it means that we can implement the Store interface get(key:) and set(key:value:) and then delegate to the functions that we captured.
With this in place we move onto updating any place where Store was mentioned as a return type or a variable’s type and change to use our new type eraser.
- public func makeTransientStore() -> Store {
- return TransientStore()
- }
+ public func makeTransientStore<Key>() -> AnyStore<Key> {
+ return AnyStore(TransientStore())
+ }
final class PersonRepository {
- let cache: Store
+ let cache: AnyStore<String>
- init(cache: Store = makeTransientStore()) {
+ init(cache: AnyStore<String> = makeTransientStore()) {
self.cache = cache
}
func fetch(id: String) -> Person? {
return cache.get(key: id).flatMap { try? JSONDecoder().decode(Person.self, from: $0) }
}
func store(id: String, person: Person) {
cache.set(key: id, value: try? JSONEncoder().encode(person))
}
}
There were surprisingly few changes required to get this to work.
What did we just do?
Let’s look at the public interface for the Storage module:
public protocol Store {
- public func get(key: String) -> Data?
- public func set(key: String, value: Data?)
+ associatedtype Key : Hashable
+ public func get(key: Key) -> Data?
+ public func set(key: Key, value: Data?)
}
+ public class AnyStore<Key: Hashable>: Store {
+ public init<Concrete: Store>(_ store: Concrete) where Key == Concrete.Key
+ public func get(key: Key) -> Data?
+ public func set(key: Key, value: Data?)
+ }
- public func makeTransientStore() -> Store
+ public func makeTransientStore<Key: Hashable>() -> AnyStore<Key>
We’ve had to expose a new concrete type AnyStore in order to accommodate the fact that we wanted Store to be generic. Exposing a new concrete type may seem at odds with the idea of relying on abstractions over concretions but I tend to think of this kind of type erasure as a fairly abstract wrapper that exists solely to hide concrete implementations.
Expanding our Type Erasure
To really ground our understanding let’s make our Store abstraction more powerful and make it work for any value that is Codable instead of just working with Data.
The current method of working with Data directly pushes complexity onto the clients of our Store API as they have to handle marshalling to and from Data.
First let’s see how this change will actually simplify our API usage:
To make the above work here’s the modifications required to add the new generic to the Store protocol and feed it through our AnyStore type eraser:
- public class AnyStore<Key: Hashable>: Store {
- public init<Concrete: Store>(_ store: Concrete) where Key == Concrete.Key
- public func get(key: Key) -> Data?
- public func set(key: Key, value: Data?)
- }
+ public class AnyStore<Key: Hashable, Value: Codable>: Store {
+ public init<Concrete: Store>(_ store: Concrete) where Key == Concrete.Key, Value == Concrete.Value
+ public func get(key: Key) -> Value?
+ public func set(key: Key, value: Value?)
+ }
public protocol Store {
associatedtype Key: Hashable
+ associatedtype Value: Codable
- public func get(key: Key) -> Data?
- public func set(key: Key, value: Data?)
+ public func get(key: Key) -> Value?
+ public func set(key: Key, value: Value?)
}
- public func makeTransientStore<Key: Hashable>() -> AnyStore<Key>
+ public func makeTransientStore<Key: Hashable, Value: Codable>() -> AnyStore<Key, Value>
Conclusion
That was a lot to go through and it got pretty difficult at the end. I covered a few different methods for testing that have various tradeoffs but are all useful for helping to test across boundaries to ensure that objects are collaborating correctly.
Hopefully the above will demonstrate some of the techniques that can be used to design clean boundaries without compromising because we couldn’t figure out a way to test things.
A real strength of statically compiled languages is that you always have a really clever friend (the compiler) looking over your shoulder checking for various types of errors.
Just like real friendships you’ll have times where you seem to disagree on really stupid things but at the end of the day you know they have your back.
Here’s some tips on ways that you can lean on your friend to really take advantage of their expertise.
Disclaimer everything below is a suggestion and your milage may vary. As with anything in software there are no silver bullets.
Avoid force unwrapping (even when you know it’s safe)
I’ve seen code that will do a nil check to guard against having an optional but the code then continues to use force unwraps for all subsequent uses. Consider the following:
The change itself is trivial and it may be difficult to see all the advantages. The biggest win for me is that the compiler is now primed to have my back during future refactorings.
What do I mean by the compiler is now primed to have my back? Let’s first look at some of the problems from the above.
With the first listing the biggest maintenance headache is that we have added a line of code that cannot be safely relocated. Relocating lines of code can come about for many different reasons:
1) Copy and Pasta
A developer may copy and paste the working line of code (line 5) without the guard statement and place it somewhere else. Because this line has a force unwrap the compiler won’t force us to explicitly handle a nil check. In a worst case scenario we may not exercise the use case where shape is nil very often, which would result in run times crashes that are rarely reached.
2) Refactoring
Refactoring is a dangerous game unless you have really good test coverage (quality coverage covering all branches, not just a high number from executing everything). Imagine someone removed lines 1-3 in an attempt to tidy things up - we’d be back to the point of crashes that we may or may not reproduce. This seems a little laughable with the example above but it would be really easily done if the guard statement was more complicated to begin and we was careless with the tidy up.
3) Bad Merge
There is always a chance in a multi-author system that people’s work may not merge as cleanly as we would like, which could result in the guard being taken out.
How is the second listing better?
I’m glad you asked. With the second listing all three scenarios above are just non existent. If I take the line update(with: shape) and place it anywhere that does not have an unwrapped shape around then the compiler will shout. This shouting will be at build time so I don’t need to spend potentially hours tracking down a crash, I get an immediate red flag that I need to handle the potential that this reference could be nil.
Avoid non-exhaustive switch statements (even when it’s painful)
Switch statements need to be exhaustive in order to compile but I often see code that does not exhaustively list out cases, instead opting to use default. Consider the following:
Like the first example it may seem like this is a trivial change with no immediate advantage but it’s the future maintenance that I think is important here.
With the second listing the compiler will be ready to shout at us if we expand our shapes to include a case rectangle - this will force us to consider the function and provide the right answer.
In the first listing the compiler will not notice any issues and worse the code will now incorrectly report that a .rectangle does not have four sides.
In this case I would argue that this is a trickier bug than a crash because it’s non fatal and relies on us checking logic correctly either in an automated test or via manual testing.
Create real types instead of leaning on primitives
Creating real types gives the compiler even more scope to help you. Consider the following:
With the above API it’s actually impossible to tell without looking at some documentation or viewing the source whether you should pass a User.id or a User.bestFriendID to the bestFriendLookup(id:) function.
If we was using more specific types instead of String for the id the function might look more like this:
funcbestFriendLookup(id:BestFriendID)->User{...}
Note that I mean a real type e.g. struct BestFriendID { ... } not just a typealias which would have no benefit
I’m not going to list a solution here because you may as well just check out https://github.com/pointfreeco/swift-tagged repo/README for how you can easily use real types to solve this problem.
Avoid Any
There are absolutely times that we have to use Any but I’m willing to bet that most of the times I encounter it in code there could have been a way to rephrase things to keep the type knowledge.
A common example, that I have definitely done myself, occurs when crossing a module boundary. If I write a new module that I call into from my main app I may want to to pass a type declared in my main app to the module as userData: Any.
In this instance the new module has to take Any as I don’t want it to know anything about the main app.
This userData: Any is another potential bug just waiting to be encountered because the compiler can’t validate that I didn’t get my types mixed up. The fix for this is to provide a module type that uses generics. The best example of this is going to be the collection types in the standard library - they don’t have an interface that works with Any instead they are generic over a type.
Conclusion
I’ve outlined a few cases where leaning on the compiler can eliminate potential future maintenance issues. There is no one size fits all solution so I encourage people to keep sweating the details and maybe some of the above suggestions will be appropriate for your code bases.
The compiler isn’t going to solve all of our coding problems but it can certainly help reduce pain if we code in a compiler aware way - I like to ask the question “can I rephrase this code so that it won’t compile if the surrounding context is removed”.