Sync is one of the notoriously hard problems in computing. There are all kinds of edge cases that have to be accounted for lest they throw the data into an ambiguous state. It comes in multiple flavors and with a variety of needs, but all of them eventually boil down to one goal: ensuring the local copy of the data matches the remote copy.
Relatively speaking, Portfolio gets off easy in this; it only needs to support syncing in one direction. It’s still a hard problem, though. There are a huge number of potential failure points that must be handled correctly, and while data loss is not the consideration that it is in two-way sync, the time loss that can happen because of a botched sync is. It takes a long time to sync up potentially multiple gigabytes of data over the internet.
Background
Portfolio’s original sync support was introduced in version 3.0, which was released in February 2013 when iOS 6 was at the bleeding edge. I had limited past experience with developing anything that synced, and the implementation definitely showed it.
The original syncing system supported only two of the loading sources: File Sharing and Dropbox. File Sharing was relatively easy since iTunes didn’t support copying file hierarchies (i.e., no folders) to an iPad and still doesn’t to this day. Dropbox was a bit more difficult because it did, and ensuring a full file hierarchy matched was much more complex.
Because of their differences, both of the original sync loading source options were implemented with mostly separate code paths. Unfortunately, this made the future extensions I wanted to do much more cumbersome, and it ultimately ended up as enough of a hurdle that I only ever added one more loading source option: SFTP.
Sync Means Many Things
Aside from the importance of a unified code path for the syncing system, the most important lesson learned was one completely unexpected and thankfully much easier to fix. The interface that I thought was very clear turned out not to be.
Casual users do not have the same precision in terminology that is a necessity for developers, and a huge influx of support requests started coming in due to the resulting confusion. They were assuming that the sync setting in the add gallery panel was the sole way to load content in.
It’s a futile exercise to try and correct users’ perceptions of an interface through the support line. It needs to be done, but at that point you’ve already failed. You’re treating the symptom instead of fixing the problem, and it’s just going to continue until fixed.
The solution ultimately ended up being rather simple: move the sync configuration. Sync is a power user feature — most users will never want to use it and will usually be happier not even knowing it exists as an option. What was once accessible from three distinct spots in the interface got consolidated to a single button at the top of the gallery list. Power users could still find it quickly and easily, and users without any need for it could ignore it completely.
Rewriting
Because of how syncing was originally implemented (a mix of NSConditionLock and - performSelectorInBackground:withObject:), I didn’t feel I could salvage much of it. There was too much potential for race conditions that could easily corrupt the sync operation and lead to a lot of wasted time. It needed rewritten, and I knew exactly the tool for the job: GCD or more specifically dispatch_group_t.
I started by working up the top level steps that each sync operation would go through. Since this was the second writing of it, I had a very thorough understanding of what exactly needed to happen and the best approach for it. If only it was possible to be that certain the first time working through a problem!
Process Outline
Show progress if sync is manually started
- Gather sync items that should be checked.
- Iterate through each sync item:
- Generate sync record for the configured target.
- Generate sync record for the existing state.
- Check the two sync records for equality.
- Return the next action (i.e., skip or sync)
Show progress if sync is automatically started and at least one item needs synced
- Iterate through each item marked to sync:
- Generate difference list.
- Process removal list.
- Process additions list.
- Mark complete.
- Mark operation complete.
1. Gathering
Why not start with the easiest part? Gathering the items that need synced can be done synchronously, which makes for much less risky code.
The operation first checks to see if it was passed an explicit list of syncable items to use. If not, it continues on and checks if the whole library is configured to sync, using it if so. Lastly, it iterates over every gallery and folder in the library. Anything with sync enabled and with a mode matching how the sync was triggered (e.g., manual or automatic) is added to the check list.
2. Checking
Checking for changes involves two main parts: (a) generating the local and remote states; and (b) comparing the two. Previously this was the start of the separate code paths, so it was important to unify the process as much as possible to make extension as easy as possible.
I ended up abstracting the remote state generation into a new class collection: BPSyncSource. It would be responsible for any interfacing with the configured source, and there would be exactly one subclass per loading method. The sync operation itself then would not need to understand anything about the actual interfacing — the same code path could be followed for everything.
3. Syncing
Once an item has been marked as needing synced, this stage of the process goes beyond just checking for sameness and generates a full difference list for the local and remote states. What it does with this difference list was one of the major deficiencies with the first incarnation of the syncing system and often was the source of the problems that reached my support inbox.
Portfolio development started with the first iOS SDK targeting the iPad, iOS 3.2. At the time of developing the first syncing system, it still supported as far back as iOS 4 (I later dropped support for iOS 4 with Portfolio 3.0, but the design decisions had already been made). This meant that I was unable to use one of the most important additions ever to make it into Core Data, - performBlock:, and that all of the persistent store interfacing was shunted onto the main thread. Fast forward a couple years and users are attempting to sync libraries much, much larger than I ever anticipated, which proved too much for the existing syncing system.
The revised approach to performing the sync leverages a dedicated background NSManagedObjectContext to perform just about every action. It starts by first processing the deletion list, moves on to create any missing nodes in the revised hierarchy, and finishes by queuing up load operations for each media file added.
After everything is queued up, it starts the load operation queue and just waits for it to finish. This stage is also enhanced with a more informative loading indicator to better update the user on where in the process it is and how long it still has to go.
4. Finishing Up
The sync operation finishes once the last sync has finished and exited the overall dispatch group. At this point it clears the background task it set up in case the user closed the app and re-enables the idle timer.
Conclusions
Writing the new syncing engine was fairly painless. With almost three years to think about the problem, the mental model I had built up proved very effective for this particular application. Debugging was surprisingly (and thankfully) light, the only problems being in the code rather than the approach taken.
While not always practical to do, writing something for the second time almost always results in code that’s more concise, clearer, and just generally better all around. Portfolio’s new syncing engine could not have ended up better, and I’m optimistic that the support requests related to syncing will virtually disappear as people update.