I’m back for real.

I’ve been very busy the past three years with my latest job, and with my family life, and other things that eat time, so I’ve let this blog twist in the wind. I’ve decided to bring it back to life and to start writing pieces not about technology or programming but more about some of the experiences I’ve had in the past, specifically what’s going on with companies that I’ve used to be involved with, either as an employee or as a very involved party. Companies like IBM, NorTel, and Palm. I’ve been in this business for almost 33 years and I’ve had a lot of experiences, some good and some bad. I’ve also had time to look back and see what’s going on in the programming world in general, and there are lessons to be learned from some of the things I’ve witnessed.

And funny things happened, too. I hope to share some of those as well.

I do not plan to talk about anything relating to my current employer, Apple, so if you want any juicy tidbits about there, go somewhere else.

Why so few older developers?

I just read through The Developer’s Dystopian Future (along with both Marco Arment’s and  Matt Gemmell’s commentaries on this). I have been a developer for quite a long time (over 32 years professionally, add a few years for my college and pre-college days), and I’ve seen this phenomenon happen over and over. Very few of my peers from when I started are still developers.

I think about this a lot. Technology has changed so much over this time period, so much that things that happened well after I started working are now grist for historically-inspired dramas (Halt and Catch Fire anyone?) (and it pains me to think of anything that has happened in my lifetime as historical just because that’s what makes you feel old). I’ve seen a lot of technologies come and go, and have mastered more than my share of them. I’ve worked at companies that I thought would stand the test of time only to watch them disappear due to incompetence, market changes, or both (DEC, NorTel). The world is a fluid place, change is the only thing I’ve come to count on happening in my life.

One thing that used to scare me was the future of software development as a job (call it software engineering, programming, whatever). There is an intense desire to make it something that can be done by just about anyone (and is done by everyone). When’s the last time you saw someone who didn’t know how to program in Star Trek? And there were no characters who identified as programmers there. I remember reading one of the early Java books that claimed the language would finally be the one that broke through and enabled everyone to program (and put me out of a job). And there are technologies such as programming by example or diagrammatic programming that promised to take the programmer out of programming. The best counterpoint to this is writing a story. Anybody can write, and a lot try to make a living by writing. Few are able to sell their stories, and fewer are able to make enough from their writing to live.

Getting back to these articles, I know the fear that they speak about. I think they get one thing wrong. They look fondly at the past few years, with the indie developer gold rush, and how it’s changed, but I think they draw the wrong conclusion. I think we just lived through the indie developer bubble, and the bubble’s burst. It’s not that it’s now harder than ever to succeed as an indie developer – it’s that there was a very short time where it was easier than normal to succeed as an indie developer, but now it’s over. I think it’s just as easy to be an indie developer as it was in the 1980’s – in fact, it might be easier than then. Distribution is so much better, it’s never been easier to get your software onto a potential customer’s device, and there are a lot more customers.

I also know the fatigue they talk about. I’ve had to keep learning new technologies and techniques over my decades in development. It’s a lot of work, and it’s also a bit of a gamble, and sometimes you guess wrong. For example, a few years ago I started to learn Flex development. Flex is a set of API’s that work with the Adobe Flash Player that were designed to enable users to build web apps with a lot more interactivity than was possible. I drank the kool-aid on this, to the point of actually defending it (against Microsoft Silverlight and HTML5) at a conference at IBM (to show you how it went, the audience ended up siding with the Silverlight guy because, you know, Microsoft always wins, but everyone wanted HTML5 to win – including me – but I was “pragmatic”). Well, we know how that went. Thankfully, I was at a place where I could transition on to other things with little risk, but I spent a lot of time and effort learning Flex (including going to the 360|Flex conferences, which was a very worth-it thing to do as it led me to find the 360|iDev conference).

After Flex I focused on Python-based development and MySQL at work and iOS development on the side at home. I was going in two technological directions at the same time. This was also during the Great Recession, so my main fear was being able to keep employed. I was able to ride it out

I’m always thinking about what is the next thing I want to do. What keeps me in programming after all this time is that I still do enjoy it. I like where I’m working, I like the people I work with (and I’ve had times where these were not true). I know that what I’ll be doing 5-10 years from now might bear little resemblance to what I’m doing now. I’m fine with that.

Still Alive…

I’m still alive, but life has taken a big change (for the better).  A few months ago I started working for Apple as a software engineer.  It’s been a very hectic time for me (a new job and moving to a new city take a lot of time).  Not much more to say but I’m very happy how things are working out.

I’m trying to figure out what to do with this blog.  I’m not going to be able to post any more code things for the time being, that’s obvious (and that means that my iCloud tutorial is not going to be finished).  Likewise, I’m not about to make any comments about the tech industry, Apple or its competitors.  I don’t want to drop this blog entirely, though.

I love this industry, and I’m very excited about what’s going on.  It’s still the early days, everyone…

Acorn: Filter to save icons for Dropbox usage

I’m adding Dropbox support to my storyboarding application, and I use Acorn from Flying Meat Software to do most of my graphic work.  A while back Gus Mueller (Mr. Flying Meat) posted a plugin to save Acorn native images in both retina display format and in “regular” format.  To submit an app to Dropbox you should provide them with 16×16, 64×64, and 128×128 icons.  What I did was take Gus’s plugin and created one to save an Acorn native image in these formats for Dropbox.

As Gus shared his code, so I share my little swizzle on his work: (file: Save@Dropbox.jstalk)

// This script requires Acorn 3.0 or later, and goes in your
// ~/Library/Application Support/Acorn/Plug-Ins/ folder

function main(ciimage, doc, layer) {
  // make sure the file is saved, and make sure it's saved
  // as an Acorn file.
  if (![doc fileURL] || !([[doc fileURL] pathExtension] == "acorn")) {
    return;
  }

  var path1 = [[[doc fileURL] path] stringByDeletingPathExtension] + "-16.png"
  var path2 = [[[doc fileURL] path] stringByDeletingPathExtension] + "-64.png"
  var path3 = [[[doc fileURL] path] stringByDeletingPathExtension] + "-128.png"

  // save our 128/128 first.
  [doc scaleImageToWidth:128];
  var opts = {'uti': 'public.png', 'file': path3};
  [doc webExportWithOptions:opts];

  // scale our image to 64x64 size and save it.
  [doc scaleImageToWidth:64]
  var opts = {'uti': 'public.png', 'file': path2};
  [doc webExportWithOptions:opts];

  // Now save our 16x16 image.
  [doc scaleImageToWidth:16]
  var opts = {'uti': 'public.png', 'file': path1};
  [doc webExportWithOptions:opts];

  // undo the scale
  [doc undo];
}

Another point on “The Church of Market Share”

If you haven’t read Gruber’s post titled The Church of Market Share you should now before I continue.

There.  Finished? Good.

I agree with everything that Gruber wrote in this piece.  In fact, I’d like to dig a little into history to reinforce what he said.

Consider mobile app development in the early 2000’s.  The leading platform in terms of market share was Symbian (Nokia mostly, although some Ericsson and later Sony/Ericsson).  They were selling like hotcakes. In fact, the only competition that they had were a brick-sized and shaped phone from Qualcomm running Palm OS, another Palm OS phone from Samsung, the phone add-in from Handspring (again, Palm OS), and maybe the earliest Windows Mobile phones from a couple of vendors. Again, none of these could hold a candle to Symbian in terms of market share. In fact, the only competition really was between Symbian phones and non-phone Personal Digital Assistants (PDA’s), such as the Palm V, Handspring Visor and the Compaq iPaq, but even there, the device sales numbers were roundoff error compared to Symbian phones.  (Blackberry was just starting to take off so they don’t factor into this.)

So where did the developers migrate to?  Palm OS.  Pocket PC.  We weren’t against Symbian development, but the few hardy souls that ventured into those waters were very disappointed with respect to the number of sales.

Now realize that this was a pre-App Store environment.  Developers would either try to distribute their apps themselves, but, more often, we would use Electronic Sales Distributors (ESD’s) like PalmGear and Handango.  Users would have to know about these sites so that they could go and buy apps.  Buying apps was also a painful process, with an email going to the developer from the ESD and the developer sending the customer an activation code that they would have to enter by hand onto the screen.  Sometimes the email from the ESD to the developer got lost; sometimes the email from the developer to the customer; sometimes the customer had trouble getting the code in (if they could find the registration screen in the first place).

Getting back on point, most developers were building for Palm (especially for consumer-oriented apps) and Pocket PC/Windows CE (mostly for enterprise).  Some were making good money, but most were just scratching out a living.  Periodically, some new developer would ask one of us if it was worth developing for Symbian, but it never was a long discussion.

(I’m sure there were a few developers who had successful Symbian applications, but it wasn’t anywhere near the numbers that Palm and Pocket PC/Windows CE developers were seeing.)

Food for thought when people start talking about market share.

Sorry for the iCloud tutorial delay

I became very busy with life and work after my presentation at CocoaHeads NY, so I set this aside.  I’m now getting back into the swing of things, and I’m now getting back to this tutorial

Sorry for the delay.

(BTW, if you had noticed, the source code for the initial non-iCloud version of my tutorial program had disappeared.  During building the iCloud version I refactored the code significantly – to the point that the non-iCloud version of NotePlus was a different program.  I’ve now retrofitted the non-iCloud version so that it is in line with my updated code.  It’s better, trust me.)

iCloud 3: Data and Metadata, and an introduction to NotePlus

This is part 3 in my iCloud tutorial series – the previous article is iCloud 2: Setting up an application for iCloud

Before I start digging into the code, I thought I’d write about how devices communicate with iCloud – the types of data and metadata that are shared bet we the iCloud infrastructure and the various platforms.  There’s not a lot of information on this, so I thought it would be helpful to include this into the tutorial.

The types of data that are stored in iCloud

There are two types of data that you store in iCloud

  • Key-value objects (property lists)
  • Documents – files and directories

The following figure illustrates these types of data.

iCloud synchronization (Source: Apple.com)

Key-value storage

Key-value storage is designed for holding small amounts of data which you access through key-value bindings.  Think of this as a property list or dictionary – you ask for an object using a key and get the object (or nil if no object for that key exists).  A good use of this is for storing application settings for sharing between devices.

This is obviously not meant for storing a lot of data.  In fact, Apple limits it to 64 KB per application.  That’s kilobytes, not gigabytes.

Document Storage

Document storage is where most of the interesting storage is going on in iCloud.  Typically you store files that your user generates in the application.  For example, in NotePlus, this is the notes and audio files that are created when the user creates a note.

Let’s consider a single file that is in iCloud.  There are two components associated with this file:

  • Metadata: information about the file, such as its name, versioning information and the number of bytes of data in it;
  • Data: the actual contents of the file

The idea is that metadata is very small in relation to the overall data for the file, and that an application can easily contain all the metadata for all the files stored in iCloud.  Any changes to the metadata of a file on a device are pushed back to the iCloud servers. When you create a file on a device, iCloud will also pull that file to the server.

Conversely, on a device pulling a file is managed by the operating system and the application in tandem. You should assume that your application will request a file that is not on the device and then it will be pulled down (there are scenarios where the OS will pull down files, but this has not been documented by Apple). This is good in terms of managing the amount of space you need for an application; the cost is that now opening a file is an asynchronous operation with a noticeable lag.  This latency may be long enough that you cannot just put everything on hold until you have finished the operation – for example, if you are doing a file open and read in the middle of a multi-player game the lag might be long enough that your character gets killed while waiting.  Not good. The good thing is that your application will be able to determine if a file is local or not, and you, as the developer, can handle this accordingly.

Data and efficiency

iCloud initially uploads the entire file from a device.  When the user makes a change to the file, then iCloud sends only the changes back to the server to create a new version of the document.  The following illustrate this action:

 

The device wants to push a new version of a file to iCloud, ...

 

Changes in the file are identified

... Changes in the file are identified, ...

 

A new version of the file is created

... the changes are uploaded and a new version of the file is created.

 

NotePlus

(Note: The source code for the non-iCloud version of NotePlus is available by clicking this link.)

I’m not going to change gears and talk a little more about the NotePlus application. When I wrote NotePlus I tried to follow the Model-View-Controller (MVC) design pattern as best I could.  I have several data objects that represent the text of a note (NpNoteText) and the audio associated with the note (NpNoteAudio).  A class, NpNoteManager, provides an interface to get a list of note names and to manage notes by providing interfaces to add and remove them from the system.

The master (note list) screen for NotePlus (NpMasterViewController)

The NotePlus app is a Universal Master-Detail application.  On the iPad this is represented as a split-view controller; on the iPhone this is done by relying on the navigation controller mechanism to navigate to a note.  The two main screen objects are the master controller (NpMasterViewController) which displays the list of all the notes, and NpDetailViewController, which provides the interface for editing a note and recording and playing the audio associated with the note.  On the iPad one or both controller may be visible.

NotePlus Note Screen

Editing a note in NotePlus (NpDetailViewController)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

On NpMasterViewController’s navigation bar are buttons to create a new note and to edit the notes (which only supports deleting notes).  A user may also delete a note by swiping from left to right on the note in the list.  The advantage of metadata is obvious in an application like NotePlus.  Populating the list inNpMasterViewController only needs the metadata, not the full note information.

NpDetailController is the note editor.  This class contains a UITextView for editing the text of the note and a UIToolbar that has buttons for recording and playing audio.  It supports both landscape and portrait orientations.

One more data class I have is NpNoteConfiguration.  This is a simple class that keeps track of the last note the user opened. This is used mostly for remembering the user’s state when the user quits the application and returns.

Where do we go from here

Now that I have introduced the NotePlus application, I will start to update it to incorporate iCloud into the application.  The steps I will need to take are:

  • Determine if iCloud is available or not;
  • Access the metadata for all the files (for the master screen);
  • Load a note from iCloud storage (including resolving conflicts);
  • Save a note and synchronize it with iCloud.

Again, the source code for the initial version of NotePlus is available here.

Until next time!

Next: Determining if iCloud is available.

iCloud 2: Setting up an application for iCloud

This is part 2 in my iCloud tutorial series – the previous article is iCloud 1: An introduction to iCloud.

In this installment I’ll discuss how to set up an application to use iCloud. In this post I discuss the steps you need to do to set up iCloud support for your application (I do create an application in this posting, it just doesn’t do any of the “iCloud goodness” yet).

The things you need to do for setting up an application to use iCloud are:

  • Setting up an app ID in the Apple Provisioning Portal and configuring it to support iCloud;
  • Setting up the iCloud entitlements in your application;
  • Obtaining a provisioning profile for the application.

What is confusing is that these steps take place in different locations – the first is done using the provisioning portal on the developer web site using a web browser, the other is done in XCode (Note: I am using XCode 4.2.1 for this tutorial – you need to be using XCode 4.2 or later due to the need to support iOS 5 for iCloud).

In this tutorial I am going to focus on iOS devices.  You can build and test a lot of your application there, but Apple currently does not allow the simulator to support iCloud.  This means the only way you can test iCloud functionality is to run it on a real device, which means the device needs to be registered with the provisioning portal and enabled for development use.  I will not go over how to use the provisioning portal – if you need help on this the best place to go is the Apple iOS Dev Center at http://developer.apple.com/ios.

For this tutorial I will use an application that I have developed calle NotePlus. This is a simple note taking application that also allows you to record an audio file to go along with each note.  I have made the source code for the initial version of NotePlus available here (Note that this is a ZIP archive that contains all the source code for this application).

NotePlus is a universal application.  I’m doing this because the easiest to see if iCloud is working by testing with two devices, and I assume that most people will either have an iPad and an iPhone or iPod Touch tied to the same Apple ID or they’ll have an iPhone and an iPod touch tied to the same Apple ID.

The importance of the bundle identifier

Make sure that you note the value of the Bundle Identifier for NotePlus – this is the suffix for the App ID we use to identify this appellation (Note that my bundle identifier is com.beret.NotePlus – you will use a different bundle identifier based on your personal or company Apple Developer Portal setup). This is what we will use to enable iCloud for this application in the provisioning portal.

Setting up NotePlus for iCloud in the Provisioning Portal

iCloud is enabled on an app-by-app basis in the provisioning portal.  The current default value is that it is not configured, which makes sense when you think about it. I will configure the NotePlus application in the provisioning portal and configure iCloud during the process.

The steps are:

  • Go to the App ID section of the provisioning portal and click on the “New App ID” button;
  • Fill out the form for the new App ID, ensuring that you put the correct bundle identifier in the right place on the form, then click the “Submit” button.
New Application ID

New App ID form for the NotePlus app

  • Once you click submit you will see the App ID has been added to the list of App ID’s that you have created.  It should look like something in the file below.  You are not done, however.  If you look at the column for iCloud you will see that there is a yellow ball with the text “Configurable” next to it. Click on the blue Configure link to the right of the App ID entry so that we can configure this.
NotePlus App ID created in the provisioning profile

NotePlus App ID in the provisioning profile

  • When the configuration dialog opens in the browser, click on the “Enable for iCloud” checkbox (Note that you will get a dialog box that will inform you that all provisioning profiles that support this application will support iCloud).  Once you click through the information dialog click on the “Done” button.
Configure the NotePlus app ID for iCloud

Configuring the NotePlus App ID for iCloud

  • Now you are back at the list of App ID’s in the provisioning profile.  Note now that the NotePlus application has a green ball and the text “Enabled” in the iCloud column. Our job is done here, so we can go back to XCode and do the second part.
NotePlus is now iCloud enabled

NotePlus is now iCloud enabled

Getting a Provisioning Profile for NotePlus

Now that you have set up NotePlus for iCloud, you need to get a provisioning profile to use.  You can either generate one using XCode (in the Organizer) or you can create a new profile in the Provisioning Portal.  All this is done using the standard techniques for obtaining a provisioning profile.

Setting up entitlements for NotePlus

Once you have completed the above we need to set up the entitlements for NotePlus to allow it to use iCloud. Apple uses this as part of mapping the application to the iCloud storage, and it also allows the developer to manage the type of data that the developer wants to synchronize.  The developer can specify one or more directories/folders to enable for iCloud storage and also enable key-value storage.

Why would you want to manage this from an application standpoint?

  • iCloud storage is not an infinite resource.  Users get 5 GB of storage for free but this storage is shared between all their applications (excluding storage for items bought through the iTunes music storage).  This means that your application and other applications are trying to carve out a piece of this storage space for themselves.  Since this is obviously not coordinated, at some point in the future the user may run out of space in the cloud.  The options the user has are to either buy more storage space from Apple or to free up some space.  As an application developer you need to be mindful of what you store in iCloud.
  • iCloud does allow you to share data between applications (just not between users). You can specify the bundle ID’s for other applications you wish to use to share data.

Once you’ve unpacked the NotePlus application open it in XCode.  Once it has loaded select “NotePlus” in the “Targets” portion of the middle area of the screen.  You should see something like the following:

XCode project summary

NotePlus target summary information in XCode

Be sure to change the value of the “Identifier” field in the right-most section from com.beret.NotePlus to whatever bundle ID you have defined for your application in the provisioning portal.

Now scroll down the right-most section to the “Entitlements” section.  This is where you enable your entitlements.  You should see the following on your screen:

The Entitlements section for the NotePlus application before enabling iCloud

Now click on the checkbox at the top of the Entitlements section (the one labelled Enable Entitlements).  You should now see the following (only with your bundle ID in the various sections instead of com.beret.NotePlus – you did change it, didn’t you?):

After enabling the Entitlements for NotePlus

Notice that the act of checking the Enable Entitlement checkbox fills in the information for iCloud, including the iCloud key-value store information and the iCloud containers.  You can use the iCloud containers to add access to shared containers (where you can share data with other applications).  For this tutorial series I’m going to keep it simple and just use the defaults.

At this point, we’ve done all the infrastructure setup to enable NotePlus to use iCloud.  Of course, just enabling it doesn’t make data sharing happen – we need to add logic to the code to actually access the iCloud data.  That will happen in the next several tutorial postings.  ‘Til then…

Next: Data and Metadata

iCloud 1: An introduction to iCloud

Before talking about how to use iCloud, it helps to understand what it is.  Let’s start with a description from Apple’s iCloud web site:

“iCloud stores your music, photos, documents, and more and wirelessly pushes them to all your devices. Automatic, effortless, and seamless — it just works.”

To make it simpler: iCloud is synchronization.  It syncs your data between your iPhone and the cloud.  It syncs your data between your iPhone and your Mac (there is also some limited support for Windows). It is incorporated into iOS and OS X Lion in such a way that, when combined with other enhancements (such as document versioning) you can get access to the latest, most up-to-date version of your data.  The image below illustrates this flow.

iCloud

iCloud Synchronization (Source: Apple.com)

Note that in the above description I emphasized the word “your”.  As it is currently implemented, it is only data that you create or consume on your devices. Data access and storage is based on your Apple ID.  There is no mechanism for associating multiple Apple ID’s with your data.

Before iCloud, applications normally accessed data objects (files/documents/databases) that were stored locally on the device.  Sharing data between devices was typically done through an explicit user action, such as copy the file to and from a file server or web site.

What iCloud does is provide a series of interfaces that developers can use to integrate synchronizing this data with Apple’s iCloud servers.  The synchronization is a part of the operating system (OS) that developers can access using these application programming interfaces (API’s).  Some of the features of these interfaces include:

  • File upload/download;
  • Data streaming (for database objects);
  • Conflict resolution (there are changes in both the document on the server and the document locally that need to be reconciled).

Apple has done a really good job at integrating iCloud in such a way as to minimize the amount of change you need to make to your code, but there are some challenges that developers need to keep in mind.  For example, opening a document that is stored locally on the file system is fairly simple – you can easily search the file system for the file, and opening it is so fast that you can typically wait for the file to open.  With iCloud, you now have to deal with network latency issues.  Opening a file now might entail downloading the document from the iCloud servers.  You now are at the mercy of the speed of the network and the size of the document.

iCloud versus Dropbox

A lot of developers are incorporating services like Dropbox into their applications to enable synchronization and sharing of information (there are other services, such as Box.net, that some use as well, but Dropbox seems to be the most popular as of this writing, so that is where I will focus).

Dropbox provides file-level syncing between devices and the Dropbox cloud. Applications can take advantage of the API that Dropbox provides for uploading/downloading files.  There is no intelligent cache management – it just downloads the files to the device and stores it if there is enough room.  On some systems Dropbox also hooks into the operating system to provide an intelligent folder that mirrors the layout of data in the Dropbox cloud.

Another advantage of Dropbox – it supports multiple operating systems (OS X, iOS, Windows, Linux) and provides API’s for several different languages (Objective-C, Ruby, Python).  There are also some third-party API’s that provide support for C#/.Net and Node.js (among others).

Finally, Dropbox supports sharing files and folders between users.  This is done using the Dropbox login ID.  There are no limits to the number of users with whom you can share data.

The cost of Dropbox is that you need to learn another API, and the amount of storage that you get for free is 2 GB.  iCloud gives you 5 GB of storage, over and above any purchases you make in iTunes (any music/apps/books that you buy via iTunes do not count against your 5 GB storage limit).  Both services allow you to purchase additional storage for a fee.

Any data shared between Dropbox users counts against the quota of each Dropbox users.  For example, if user A shares a folder with users B and C, and C puts 2 GB of data in that folder, then both A and B are at their 2 GB limit, even if they haven’t put a single byte of data into their Dropbox storage.

Which to use?  It all depends.  If you have an existing appellation that stores data in local storage and want to take advantage of cloud backup and synchronization, then iCloud is the simplest way to get this.  If you need to share data with multiple users, then you should use Dropbox (or one of its competitors).

Next: Enabling an XCode application to use iCloud

Announcing an iCloud Tutorial

I’m going to be speaking at CocoaHeads NY next month (December 5th) (as one of a couple of speakers).  I’m planning on doing an introduction to iCloud for developers.  What I’d like to do is work on pieces of this presentation here so that I could get some feedback during this.

I am currently in the finishing up stage of an application that I am building – a storyboarding application for the iPad (StoryBuild).  I am incorporating iCloud into this app partly as a learning experience and partly as a mechanism to sync the storyboards between different iPads (eventually I plan to port this application to other devices, but the iPad was my inspiration for this app).  Some of the topics I plan on covering in this tutorial are (in no particular order):

  • How to set up an application to use iCloud
  • Using NSFileManager and friends with iCloud
  • UIDocument and iCloud
  • Core Data and iCloud
  • Gotchas and failing gracefully

I only have a 30 minute presentation slot, and this is an ambitious list.  While I plan on referencing code I’m using in StoryBuild, I plan on having a separate project that will be available to download for you to try.  As to the format, I’m planning on following the same format I used in my Introduction to Cocos2D presentation from 360|iDev Austin in November, 2010.