Development – iOS, Mac OS X, and more

Why I’m Waiting on Swift

The promise of Swift looks great. Faster performance¹, better error handling, and better compile-time bug detection sound like very welcome additions to the iOS/OS X development ecosystem. The thing is, though, there are problems, problems that make it impractical to put time into mastering the language.

Swift is being actively developed as a language, so much so that Apple is making changes to it that often break existing code. I believe this is necessary in the early stages of the language to avoid the significant cruft that can accumulate by maintaining backwards compatibility, but I can’t afford to regularly rewrite existing code because something in the language changed. Until the language reaches a stability point where existing syntax can be reasonably guaranteed to still be the same in the near future, I believe it wasteful to use it in a production environment where that time could be put to better use enhancing the product.

Swift’s immaturity also shows in its design. Many patterns we’ve embraced in object oriented programming have not yet made it into Swift. For example, objects in an array conforming to a child protocol cannot be indirectly cast as conforming to the parent protocol. Instead, the workaround requires fully copying the source array and doing an explicit cast in a map. Things like this add up. To be most productive it’s important that the language get out of the developer’s way – proven design patterns should be supported without having to resort to programming a workaround.

Ultimately I believe Swift will mature and meet its promise, but I don’t know when that will be. I have no special attachment to Objective-C – it’s just the best tool for the job currently. Until Swift can dethrone Objective-C, I’ll stick with it.

Despite its claims, it isn’t faster. ↩

iOS

The Evolution of Portfolio

Portfolio began as an offshoot of a more specific app I made for our photography business. While we no longer have a need for that original app, Portfolio has been in continuous development for five years now.

Over its lifespan it has had almost 80 updates, and in that time its interface has evolved significantly to meet the needs of its users. It has gained numerous features and customization points with the constant battle to minimize complexity and keep it usable – the core flow has never changed for that reason. Still, I have said no to far more feature requests than have ever made it in.

This animation shows the evolution of Portfolio’s viewer from version 1.0 in August 2010 to the current version.

Home

The original home screen for Portfolio was pretty limited compared to its current incarnation. Customization was limited to choosing an image for landscape orientation and one for portrait. Today you can create almost any layout and fit the look you want.

Portfolio 1.0

Portfolio 4.0

Appearance

The appearance customization panel has probably seen some of the most visible changes. In its first version the only customizations allowed were the landscape and portrait home screen images; by 2.0 it had gained some color customizations; 3.0 brought user-creatable themes and options to tweak different behaviors in the viewer; and 4.0, this year’s major update, brought further refinements and a new interface.

Portfolio 1.0

Portfolio 2.0

Portfolio 3.0

Portfolio 4.0

Content

Content is key to making Portfolio actually useful. Without it, the viewer is just a bunch of buttons. The original content manager was a two-column panel: one for adding, removing, and editing galleries (just galleries at this point – no folders), and one for interacting with the files loaded into a gallery. Since then it has evolved significantly more functionality while keeping the same basic structure.

Portfolio 1.0 only supported image-based content. This was on iOS 3.2, which lagged behind the iPhone on iOS 4.0 until the two were unified around November/December 2010. The more recent additions to iOS that later made other types of content more practical to support didn’t exist then.

That first version also supported only two loading sources: the Camera Roll and a list of URLs. Dropbox’s SDK hadn’t been released yet, though it did become the first addition to the loading source options.

Portfolio 1.0

Portfolio 2.0

Portfolio 3.0

Portfolio 4.0

Viewer

Most of the viewer’s changes have been far more subtle than the other portions of the app. Its initial behavior was modeled on how the native Photos app behaves, which has become the standard for photo browsing on iOS.

Version 1.4 brought the first revision of a compare mode.

This lasted until version 4.0, which introduced a completely new viewing engine making it possible to select even more.

1.3 revealed the first major additions that led to many photographers using Portfolio as a client proofing tool.

Over time these have been adjusted and refined to where they sit in 4.0 today.

Beyond these aesthetic changes the viewer has also had significant development time devoted to both improving image loading performance as well as its quality.

Icon

Portfolio has seen only three significant versions of its icon. The second revision was introduced to match the new shape iOS 7 brought, and the third was done for Portfolio 4.0.

Other Changes

Other changes have often spanned multiple areas of the app. Each of Portfolio’s version groupings has had one or two central focuses. Some have addressed feature requests, others bugs, and some still iOS updates.

1.1 Dropbox

1.1 introduced support for loading from Dropbox and started the first of many steps to providing help directly in the app with a quick start panel.

1.2 Multi-Screen

This version brought the first multi-screen support and the ability to import and filter by IPTC keywords. While it’s common now, it’s important not to forget that the original iPad could not display on a second screen unless apps were specifically written to detect and use it.

1.3 Client Proofing

Client proofing support began with the additions in 1.3, providing support for individual ratings and notes for each image. It also included the first user manual.

1.4 Videos

Videos were the second supported content type to be added and came with the 1.4 update. This update also enhanced the proofing support in 1.3 by adding the ability to compare two photos side-by-side.

1.5 Metadata, PDFs, and More

Version 1.5 was one of the most significant updates in Portfolio’s history. It was built at the point where the iPad and iPhone SDKs joined with iOS 4.2, so it was finally able to take advantage of the many additions since iOS 3.2.

The ImageIO framework in iOS 4 made a lot of things available. Loading speed could be improved with access to the new APIs, it became possible to extract and display even more metadata, and PDFs were now easy to render and display.

Two other highlights of this release were the addition of a Mac-hosted loading program, which enabled bulk loading without an internet connection, and a way to backup and restore the library without having to do a full device restore through iTunes.

2.0 Folders and Polish

Folders were a long-requested feature to make it easier to build a content hierarchy. 2.0 made the changes necessary to the underlying storage system to make it possible. I also spent the time implementing a number of other requests that added to the overall polish of the app.

2.1 Exporting and Filtering

One of the most surprising requests I’ve received (on more than one occasion) is to get files back out of Portfolio since the originals were lost. It happened enough that it became one of 2.1’s key features. 2.1 also brought enhancements to the viewer, mainly along the lines of filtering the current view to help with photo proofing.

3.0 Themes and Synchronization

The two focuses for 3.0 were to address long-requested items. Users wanted more appearance customization and they also wanted automatic synchronization with a service like Dropbox. In addition to each of these, 3.0 also brought a large number of fixes and improvements to bring it up to speed with iOS updates.

3.1 iOS 7

iOS 7 is notable for bringing the largest number of UI changes in the life of iOS. It introduced so much that it needed an entire line of updates focused just on the changes in it.

3.2 Performance

When iPads first got their Retina display it meant that Portfolio suddenly had to load images four times the previous size in the same timeframe. 3.2 was an update to focus on improving the performance and responsiveness of the interface, particularly when viewing.

3.5 SFTP

Many companies use Portfolio, often for confidential materials. SFTP support provided for a loading option that avoids any 3rd party services.

4.0 Overhaul

For any project worked on for the length of time Portfolio has been, it develops a certain amount of cruft, both in the code itself and in the interface. 4.0’s sole goal was to remove as much of this as possible. Almost every component was revisited to remove the old and unused bits that had accumulated as well as to bring the UI up to current standards.

iOS

Building a Better Cache

For my app Portfolio, image loading performance is arguably the most important consideration, especially given that the typical image a user loads is in the 2048-4096 pixel range. These aren’t small images from Instagram. With the upcoming 4.1.0 update I have been putting significant work into maximizing the loading performance to get the most responsiveness and best framerate possible.

Portfolio pre-caches a couple different sizes when images are first loaded in, but due to the various ways someone might interact with the interface, it can have the need to generate new sizes on the fly to conserve memory (e.g., pinching to zoom in).

Existing Caching Options

Some time ago I had added SDWebImage as an intermediate layer to cache these ad-hoc representations to help improve performance – as long as there is disk space available, why not use some of it to prevent unnecessary image resizing? It worked, but it has always shown up when doing performance profiling in Instruments. While that’s not necessarily indicative of anything, it provided a good place to start looking for improvements.

SDWebImage saves cached images using UIImagePNGRepresentation. This is sensible for a basic, general purpose cache implementation, but I don’t need lossless caching in Portfolio so the performance hit from doing so is unnecessary.

I have looked at most of the notable iOS caching options:

While each would work to some degree, they’re not exactly ideal for the size of images Portfolio deals with: SDWebImage is generally I/O bounds, FastImageCache works best when all images are the same size, TMCache uses too much memory, and Haneke is also I/O bound. Based on that and my own testing of the image reading and writing options in iOS, I decided the best solution was to build a cache more suitable for Portfolio.

Building a Caching Solution

With Portfolio, image loading performance is far more important than image saving. Saving can be deferred and performed on a background thread, but image loading needs to be done as fast as possible when requested.

For some time Portfolio has used CGImageSourceCreateThumbnailAtIndex to generate the correct-sized image from the source, but this has proved only barely fast enough. It does this to generate non-full-resolution images that conserve on memory usage since iOS devices have always been severely constrained on memory, particularly when trying to work with large images. The warning issued by the system about using too much memory is generally too late to do anything useful, which then leads to being killed by the memory watchdog process and looking identical to a crash by the user.

I have been considering this problem since long before the version I decided to finally tackle it. Over that time I have been building on an initial idea of only loosely matching items in the cache. When Portfolio needs to load a full screen image, it doesn’t necessarily need that exact size – it could be a bit larger and still work fine as long as it isn’t so large to create memory problems. The approach I ultimately settled on still uses CGImageSourceCreateThumbnailAtIndex – it’s faster and of better image quality than resizing by drawing into a smaller bitmap context – but it’s now used only as a last resort. When Portfolio queries the cache it gives it a threshold value (right now this defaults to 20%). If there’s an existing image size within 20% of the size requested, it returns that instead of generating a new size.

When a cache request is made, the cache first checks its in-memory cache (just a standard NSCache instance) for an image within the threshold. Since an NSCache instance has no public method to return the keys that currently exist in it, a separate record is kept for all keys added to the cache (some of these might have been purged but it doesn’t matter since checking is relatively cheap operation).

If no image is found within the in-memory cache, it then checks the disk cache. The cache controller uses GCD to monitor the cache location for any changes and keeps the current file list in memory. This means that checking for any possible matches is not I/O bound at all – it only accesses the disk when attempting to load in the image.

On-disk images are stored as raw BGRA data (full credit to FastImageCache for this idea). These take up significantly more space than JPEGs or PNGs but make up for it by loading significantly faster. They also have the benefit of already being byte-aligned for Core Animation, which Portfolio uses extensively for displaying.

If the cache has not found a match in either the in-memory or disk caches Portfolio uses up its final optimization: it checks to see if the desired size is within 20% of the size of its pre-generated sizes (currently a thumbnail size and a full-screen size). If that matches it loads and uses that file. If not it creates the size it needs. Either way it adds that to the cache for future loading operations.

Conclusions

Caches are not a one-size-fits-all optimization. Most of the caching options available on Github are built around the more common use case of lots and lots of unique small images (e.g., social media thumbnails) rather than multiple versions of the same file. Portfolio’s image browsing performance is now back in line with how it was on the second generation iPad before it had to cope with four times the image data in the same timeframe. When performance matters enough, and there’s a way to leverage the unique needs of an app to improve performance, it’s worthwhile spending the time to do it.