Monday, October 15, 2012

Bundling Python files into a stand-alone executable


One of the problems with building a medium to large sized program in Python (or similar scripting languages) is distributing it to users. When a Python script grows beyond a couple hundred lines, most programmers prefer to split that single script file into multiple Python modules and packages. For an individual developer, modules and packages are primarily an aid in mental organization, though they also ease navigating around the project. For a large Python program being developed by a team, modules and packages are an important way to communicate the structure and intent of the code.

Unfortunately, distributing a multi-module Python program has a number of problems. First, you must carefully assemble all your program's dependencies in a single directory tree. Second, you need to make a zip or tarball of the directory tree for distribution. Third, you need to instruct your end users on how to unpack the zipped or tarballed program and how to correctly set their PYTHONPATH and which Python file or shell script in the directory tree to invoke to run your program.

Python has long included the distutils module to help developers distribute Python code. Distutils is focused on distributing Python modules and packages for use by other Python developers and is great for its intended purpose; it can also install shell scripts in the standard operating system command directory (such as /usr/local/bin on most UNIX-derived systems). It has a big problem though: Python libraries installed by distutils are made available to all Python code unless special care is taken. If you include any common third-party libraries in your program, you run the risk that your end user may have a different, possibly incompatible version of that library already on their system. You risk breaking other Python programs, and being broken in turn if you share libraries. Windows users have long dealt with DLL Hell, a similar problem where different Windows applications would install incompatible versions of shared libraries.

Today even the computer in your pocket has dozens of gigabytes of storage so modern development has moved away from sharing library code between programs. For Python developers, virtualenv allows you to quickly and easily create separate virtual Python installations on a single computer, each one isolated from the others and from the "real" Python installation. You can install Python modules and packages in one virtualenv without affecting the others. Used along with the pip package manager, it's easy to document and recreate a virtualenv Python environment, which is a boon to Python web developers.

Virtualenv is still overkill for end users, technical or not, who simply want to run your program in order to get their work done. Fortunately, Python quietly added a new feature in 2.5 that makes it possible to bundle up a directory full of Python code into a single executable file. I say "quietly" because Python 2.5 was released in 2006 and I only heard about this feature now in 2012, six years later. (Okay, it's possible I wasn't paying close attention. :-) Typical of Python, the feature isn't pretty but it has a certain elegance to it: the __main__.py file.

How to use a __main__.py file
The Python documentation for the __main__.py file explains its purpose succinctly but barely hints at the possibilities. I'll try to do a better job. Lets start by creating a directory for our Python application named app:
$ mkdir app
Now open your favorite text editor and create the file app/__main__.py. Add the following code to it:
# file app/__main__.py

def main():
  print('The rain in Spain falls mainly in the plain.')

if __name__ == '__main__':
  main()
If you've done some Python programming, you'll recognize the __name__ == '__main__' idiom used to determine if a python module is being executed directly rather than imported as a module. When it's executed directly, the example simply calls the main() function, which prints "The rain in Spain falls mainly in the plain." to standard out.

Now let's run this program. Instead of calling __main__.py directly, we can treat the app directory as our Python program:
$ python app
The Python interpreter sees that app is a directory and checks for a __main__.py file inside it. Note that Python only checks the top level of the directory; it doesn't search subdirectories. Since there is a __main__.py directly in app, the interpreter runs it and the output is:
The rain in Spain falls mainly in the plain.

In addition, the Python interpreter will add the directory to the start of sys.path so that all imports will check the that directory first. By placing all of the modules and packages that our program depends on in the directory, we can stay isolated from whatever versions the end user may have installed as well as keep our dependencies isolated from the end user's system.

Zip it up
Python has supported loading modules and packages out of a zip file since 2.3. Just as it now looks in a directory for __main__.py, Python will also look in a zip file for __main__.py. Let's zip up the app directory and test this.

Note that the __main__.py file needs to be at the top level in the zip container, not in a subdirectory. This makes creating the zip file a little tricky. We want to recursively zip up everything in our app directory, but not include the app directory itself. (Windows users will need a command line zip program to follow along.)
$ cd app
$ zip -r ../app.zip *
$ cd ..
(Use *.* instead of * on Windows.)

To test that you've zipped things up correctly, run your Python program directly from the zip file:
$ python app.zip
You should see the expected output:
The rain in Spain falls mainly in the plain.

Python will place the zip file first on sys.path just as it does for a directory; all modules and packages imports will search the zip file first. Be sure to place your modules and packages at the top level in your directory along side the __main__.py file.

Load a resource
If you've put all your Python code in the right place using this scheme, everything pretty much just works as you expect it to. But some programs depends on resources aside from Python code, and need to load various data files that come bundled with the program. The easiest way to find and load a program bundle like this is to use the pkg_resources module. The pkg_resources module does a lot of things, but you'll want to look first at the ResourceManager API which has the most common functions for finding and loading resource files.

Let's add a resource file to our little app and load it using the pkg_resources.resource_string function. Create a subdirectory under app called resources.
$ mkdir app/resources
Using your favorite text editor again, create the file app/resources/inFrance.txt and add some text to it:
But the ants in France are mainly in your pants.
Now edit app/__main__.py so that it looks like this:
# file app/__main__.py

import pkg_resources

def main():
  print('The rain in Spain falls mainly in the plain.')
  print(pkg_resources.resource_string('resources', 'inFrance.txt'))

if __name__ == '__main__':
  main()
You may already have pkg_resources.py installed on your system. If you don't, you'll find it's part of the distribute package. Download the latest version of distribute, unpack the tarball and find pkg_resources.py inside. Copy pkg_resources.py to app/pkg_resources.py. (Even if you already have the pkg_resources module on your system, if you use it in your program, you should add it to your bundle before distributing it to others.)

Now when you run the program:
$ python app
You should see this output:
The rain in Spain falls mainly in the plain.
But the ants in France are mainly in your pants.

Make it executable
Finally, you can turn your zipped program bundle into a stand-along executable on UNIX-like systems using a couple of commands. Zip up the latest version of the program in the app directory and name it app2.zip.
$ cd app
$ zip -r ../app2.zip *
$ cd ..
Now use a bit of UNIX magic to turn app2.zip into an executable.
$ echo '#!/usr/bin/env python' | cat - app2.zip > app2
$ chmod +x app2
The first command inserts a UNIX shebang at the start of the zip file and writes it to a new file called simply app2. The zip file format is designed to allow a small executable program to be inserted at the front (that's how self-extracting zip files are created), so this is kosher and doesn't corrupt the zip file. The second command sets the executable bits on app2.

Now you can simply run app2 like any executable.
$ ./app2
And you should see the expected output.
The rain in Spain falls mainly in the plain.
But the ants in France are mainly in your pants.

Wednesday, October 10, 2012

BSD-style license for code

We have published a fair amount of source code on the Able Pear Software blog over the past few years, but we neglected to specify any kind of software license to go along with it. We occasionally get asked about an open source license (the UrlEncoding category for NSDictionary is particularly popular).

All the code we publish on the Able Pear Blog is free for you to use under a BSD-style license, as is our Autoindigestion tool. We like the simplicity of unrestrictive open source licenses like the BSD and MIT licenses. If you decide to incorporate some of our code into your project or product, we'd love to know. Modern software is so complex, it's simply not practical to write everything from scratch. Many parts of OS X and iOS have their roots in the FreeBSD project, which itself is a descendant of BSD UNIX. Even Microsoft has incorporated some BSD code in Windows.

Here is the Able Pear Software blog open source code license. Adjust the copyright year to match the date on the blog post. (Please note that the full text of the blog is not included in this license, only the source code.)
Copyright (c) 2012, Able Pear Software Inc.
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

- Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

- Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Monday, October 8, 2012

Smart App Banners

The new version of Mobile Safari that ships with iOS 6 has a great new feature for app publishers: Smart App Banners.
When a user visits your site in Mobile Safari on iOS, you can now add a pop-up banner to promote your iOS app, which includes a direct link to the app in the App Store and optionally your iTunes affiliate information. David Smith has a great overview on his blog, and you can find all the details of Smart App Banners in Apple's Safari Web Content Guide on the Apple developer site.

Though I doubt that this will do away with all those annoying full-page "get our iPad app" pop-ups, I hope that at least some sites will start to use this instead. Thanks Apple for providing something to help sites promote their apps in a less annoying way.

Monday, July 30, 2012

Adding a Retina-ready icon to your Mac app in three easy steps

I've been doing more OS X development recently, and coming from the iOS world, there's a lot that's familiar but I still stumble over many things. Creating an app icon is one of them. After some poking around, I've discovered what you need to do to create a custom icon for your modern Mac app.

Step 1: Create the icon images
Modern Mac icons pack five different resolutions in one .icns file, from 16x16 to 512x512:
  • icon_16x16.png
  • icon_32x32.png
  • icon_128x128.png
  • icon_256x256.png
  • icon_512x512.png
The dimensions are actually "points" rather than "pixels". On standard resolution displays, 1 point == 1 pixel; iOS developers are already familiar with handling "retina" resolution displays where 1 point == 2 pixels. These double resolution "retina" resources get "@2x" added to the base filename, so in addition to the five standard resolution icon images, you now need five double resolution ones:
  • icon_16x16@2x.png
  • icon_32x32@2x.png
  • icon_128x128@2x.png
  • icon_256x256@2x.png
  • icon_512x512@2x.png
The names describe the icon size in points, so icon_16x16@2x.png is 32 by 32 pixels and icon_512x512@2x.png is 1024 by 1024 pixels. Some of these images have equivalent pixel sizes: icon_32x32.png and icon_16x16@2x.png are both 32 by 32 pixels. You may be able to get away with using the same bitmap in many cases, but you (or your icon designer) may want to tweak each version to look best on their respective display types.

Step 2: Add an .iconset to the Xcode project
In the olden days, you would use the Icon Composer app to build your .icns file, but it's no longer being updated by Apple and doesn't support double resolution images. Today, Xcode will build your .icns file automatically for you. (Alternately you can use the iconutil command line utility to build .icns with high res images or to extract images from .icns files; see Apple's High Resolution Guidelines for OS X for details.)

The trick to making Xcode build your .icns automatically is to put your set of icon images in a folder named <ICON_NAME>.iconset, where <ICON_NAME> becomes the base name for your .icns file. Add this folder to your Xcode project and Xcode will then create <ICON_NAME>.icns when you build, and automatically copy it to your app bundle.

Note that the images in the .iconset directory need to have the names given above: "icon_16x16.png" through "icon_512x512@2x.png" or you will see a warning in Xcode when building your project and the mis-named images won't be included.

Step 3: Add the icon to the Info.plist
To set your app's main icon, add the "Icon File" key to your app's Info.plist file and set the value to <ICON_NAME> (without the .icns extension). If you used the standard Xcode template to create your app, your Info.plist will be named "<APP_NAME>-Info.plist", where <APP_NAME> is your app or project name. For those of you who like to edit your .plist files as XML, the key name is CFBundleIconFile.

You can also drag the .iconset directory from the finder or the Xcode project navigator to the "App Icon" pane of the Summary tab for your app's target (pictured above). If you use this approach, Xcode will insist on copying the .iconset into the root folder of your project, even if you've already added the .iconset somewhere else in your project's directory tree (<sarcasm>another great example of Apple's attention to detail</sarcasm>.) If you're not fussy about your project organization, you can simply let Xcode have its way; otherwise you can remove the copy Xcode makes and add the .iconset in a more appropriate place, like the Resources folder; just make sure to leave the "Icon File" key in your Info.plist and Xcode will show the icon in the "App Icon" pane.

Easy as 1-2-3
Not hard once you figure it out, and a nicer workflow than having to wrestle with a half-baked special purpose app like Icon Composer.

Monday, July 9, 2012

Download iTunes Connect sales reports with Autoindigestion

When you start selling your first app in the app store, it's very exciting to check you sales every day in iTunes Connect. After the initial excitement wears off and your app sales settle into a steady state, it's easy to get involved in your next project and forget to log into iTunes Connect periodically to download your daily or weekly sales reports. Apple only makes the last 14 days of daily reports and the last 13 weeks of weekly reports available; if you're not diligent, time can fly by and you will lose important sales records.

Apple does provide a tool to automate the retrieval of iTunes Connect sales reports: the Auto-Ingest tool, a small Java command line app. While it's good that Apple provides an officially supported tool, it's a very minimal one. The Auto-Ingest tool only downloads one report at a time and doesn't have any intelligence for determining which reports have already been downloaded, nor can it handle downloading multiple report types or multiple iTunes Connect vendor accounts without some scripting help. Enter Autoindigestion.

Autoindigestion is a command line utility for Mac OS X Lion that uses the Auto-Ingest tool to automatically download daily and weekly sales reports. The first time it runs, Autoindigestion will grab all available reports; on subsequent runs it will download new reports based on the current date and the dates of previously downloaded reports. If you miss a day because your system is off or iTunes Connect reports are delayed, Autoindigestion will catch up the next time it runs. Autoindigestion can also be configured to handle multiple iTunes Connect vendors, which is great if you handle sales reporting for clients.

Autoindigestion is open source and available under a BSD style license. It's currently available as source from the Autoindigestion GitHub page. It is a native OS X command line program written in Objective-C and requires Xcode to build. Complete instructions for installing and configuring Autoindigestion are included on the GitHub page. I enjoyed creating Autoindigestion and hope you find it useful.

Wednesday, June 20, 2012

WWDC 2012 Videos

Apple has posted the videos and slides from the technical sessions of this year's Worldwide Developers Conference yesterday. The videos are available for free, but you need to be a registered Apple developer to watch them. (You can register at https://developer.apple.com/.)

As in previous years, there's a lot of good stuff here for iOS and OS X developers, as well as some sessions on other parts of the Apple ecosystem like Safari and iAds. Also featured this year are several sessions related to authoring content for the iBookstore using both EPUB 3 and Apple's proprietary iBooks Author formats. There are also a good number of sessions focused on getting the most out of Xcode and the Apple toolchain.

I've watched many of the videos from previous years and they are quite useful. If you are an experienced iOS developer, these sessions are a great way to help you get up to speed on the latest stuff. If you are new to iOS, the WWDC sessions, especially from previous years, contain some great overviews that will help you get up to speed fast. I highly recommend them.

Monday, March 5, 2012

Instance Variables in the Implementation File

Xcode
Recent versions of iOS and the Apple LLVM 2.1 compiler use an improved version of libobjc, the core library that powers Objective-C features like classes and method calls. This "modern runtime" library enables a lot of new, cool flexibility in creating Objective-C classes. One of my favorite new features is Instance Variables in the Implementation File, which allows you to move your private instance variables out of your .h files. iOS developer Tips shows you how.

Friday, February 24, 2012

Something wonderful: new Objective-C literal syntax

mountain lion cub going rawrrrrrWith the announcement of OS X 10.8 Mountain Lion, Apple made the beta version of Xcode 4.4 and the Mountain Lion SDK available to developers enrolled in the Mac Developer Program (the US$99 annual membership required to submit apps to the Mac App Store). The new OS X version has created a lot of developer buzz around Gatekeeper, the upcoming requirement that all apps in the Mac App Store run in a restrictive sandbox similar to iOS apps. This is a big problem for makers of development tools and other specialized apps that need low level access to the system; Apple has effectively banned many of them from the App Store. On the other hand, strong argument can be made that Gatekeeper and app sandboxing will be a big win for most non-technical Mac users.

The news about Mountain Lion isn't all bad for developers. Apple keeps their beta SDKs under a non-disclosure agreement, but with developer interest high for both iOS and OS X, details tend to leak out. CocoaHeads recently revealed that the Apple LLVM 4.0 compiler in the Xcode 4.4 beta includes support for an Objective-C literal syntax for NSDictionary, NSArray and NSNumber objects.

If you've done any Objective-C programming, you've seen the NSString literal syntax:
// NSString literal
NSString *name1 = @"Lana Kane";

// creating NSString from a C string literal
NSString *name2 = [NSString stringWithCString:"Sterling Archer"
encoding:NSUTF8StringEncoding];

An NSString literal is like a plain old C string literal, but prefixed with the '@' character. This has made using NSString objects easy, such as adding them to a NSArray. Using numbers, however, meant lots of tedious calls to [NSNumber numberWithXxx:...]. Like NSString literals, the new NSNumber literals are simply the plain old C number literals prefixed with '@'. This included character literals (since characters are simply small integers of type char) as well as the BOOL values YES and NO (though technically YES and NO are #define'd constants, not literals).
// new NSNumber literals
// and old style NSNumber creation
NSNumber *intNumber1 = @42;
NSNumber *intNumber2 = [NSNumber numberWithInt:42];

NSNumber *doubleNumber1 = @3.1415926;
NSNumber *doubleNumber2 = [NSNumber numberWithDouble:3.1415926];

NSNumber *charNumber1 = @'A';
NSNumber *charNumber2 = [NSNumber numberWithChar:'A'];

NSNumber *boolNumber1 = @YES;
NSNumber *boolNumber2 = [NSNumber numberWithBool:YES];

The new syntax supports the usual number suffixes like 'f' for float and 'u' for unsigned number types.
NSNumber *unsignedIntNumber1 = @256u;
NSNumber *unsignedIntNumber2 = [NSNumber numberWithUnsignedInt:256u];

NSNumber *floatNumber1 = @2.718f;
NSNumber *floatNumber2 = [NSNumber numberWithFloat:2.718f];

Having a compact NSNumber literal makes using numbers in collections much nicer. Here's a NSArray with some strings and numbers:
// an array with string and number literals
NSArray *array1 = [NSArray arrayWithObjects:@"foo", @42, @"bar", @3.14, nil];

// and the old way
NSArray *array2 = [NSArray arrayWithObjects:@"foo",
[NSNumber numberWithInt:42],
@"bar",
[NSNumber numberWithDouble:3.14],
nil];

This would be nice all by itself, but really shines when combined with the new NSArray and NSDictionary literals. Here is the array from above using the new NSArray literal syntax:
// an array literal
NSArray *array1 = @[@"foo", @42, @"bar", @3.14];

Like many popular languages today, NSArray literals use the square bracket characters '[' and ']' to delimit the array literal; the only twist here is you prefix it with '@' symbol. And thankfully, you no longer need to remember to put nil at the end!

And there's a similar notation for NSDictionary literals:
// a dictionary literal
NSDictionary *dictionary1 = @{ @1: @"red", @2: @"green", @3: @"blue" };

// old style
NSDictionary *dictionary2 = [NSDictionary dictionaryWithObjectsAndKeys:@"red", @1,
@"green", @2,
@"blue", @3,
nil];

The curly brace characters '{' and '}' delimit the dictionary literal and the colon ':' character to separate keys and values, similar to widely used languages like Python and JavaScript, but of course prefixed with '@'. I was never a fan of the backwards "ObjectsAndKeys" way of creating a NSDictionary, and I'm glad to have a much more compact alternative.

This new syntax is basically what Ole Begemann proposed in his blog in 2010 (EDIT: and Stig Brautaset proposed in his blog in 2008, and by others even earlier) and it's great to see that Apple is actively eventually listening to their developer community and finding real ways to make Objective-C a nicer language. Along with the @property syntax and ARC, this goes a long way toward reducing boilerplate code. Objective-C's dynamic object model, inspired by Smalltalk, has always made it more akin to dynamic languages like Python and Ruby than static ones like C++ and Java. ARC and compact container literals are big steps that take Objective-C further into dynamic language territory, yet the low level power of C is always available when you need it. I can't wait until this is released.

Update: Apple has committed a patch with the new syntax to the LLVM project.

Thursday, February 16, 2012

Add "Social Networking" with the Twitter & Accounts Frameworks in iOS 5

Recently, on the iOS Developer Network (iOS DevNet) on LinkedIn, I posted a discussion on integrating "social media hubs" into your iOS 5 app and provided some references.  Here is the discussion:

There are several ways to integrate your app with Social Media hubs like Facebook and Twitter. Now with the new Accounts and Twitter frameworks in iOS 5 Apple makes it even easier.

Peter Friese has a good post on his blog covering Accounts and Twitter integration.

Lasse Bunk shows another example using JSON and Storyboards in Xcode 4.2.

More information on Working with JSON and iOS 5 can be found in Marin Todorov's post on Ray Wenderlich's blog.

You can, and should, review Apple's document on on the Twitter Framework and the Accounts Framework to become familiar with the new APIs.

In addition, and if you like the feel of paper, or use the Kindle app, Amazon is now caring Apress's Beginning iOS Apps with Facebook and Twitter APIs: for iPhone, iPad and iPod touch book.

Cheers and good luck on your app's social integration.

Monday, January 9, 2012

The year ahead for Apple: the iPad dominates (and one more thing...)

iPad
With 2012 barely a week old (and possibly the last year ever), I thought it would be a good time to make some prognostications for the tech industry in the year ahead. As an iOS developer, I pay a lot of attention to Apple, so I'll concentrate on what I think is in store for the Cupertino company. Love them or hate them, they're certain to continue to make their presence felt in 2012.

iPad continues to dominate
Apple will announce the iPad 3 in March, to begin selling sometime in late May. It will be an iteration of the existing iPad design, but a little thinner, sleeker and sporting a 2048x1536 Retina display. Tech pundits will give it mixed reviews, repeatedly pointing out that there is no LTE version, but consumers will embrace the device and Apple will again have problems meeting strong demand. In addition, Apple will keep the iPad 2 around, slashing the price of the base model to US$299. With Apple's BOM costs for the iPad 2 now falling under US$200, they maintain their profit margins and whittle away at the value proposition of competing tablets, most notably the Kindle Fire.

This isn't a big deal for Amazon; the Kindle Fire is their razor handle and Amazon's digital books, music, movies and apps are the blades. Amazon will continue to improve their Kindle line, updating the Kindle Fire software and introducing a follow-on color tablet device in a larger form factor that's more suitable for viewing movies, magazines and textbooks. The Kindle doesn't need to appeal to everyone, only to Amazon's best customers, and ensure that Amazon has a path to their digital customers outside of Apple and Google. Amazon is happy to be the Schick to Apple's Gillette in the tablet market.

For the Android tablet makers, that large sucking sound you hear is Apple continuing to vacuum up all the mobile device profits in developed countries. With Apple using their billions in cash to lock up the best values in the supply chain, Android tablet makers will be hard pressed to match the iPad's juicy margins. Making it up in volume won't be possible: with Android lacking any compelling value proposition for the general consumer, Android tablets will remain the province of open source zealots, tinkerers and Apple-haters, and sales will be anemic in developed countries. The story will be a bit different in emerging economies where US$100 is too large a differential for most consumers, but razor thin profit margins will make Android tablets a money loser for all but the most nimble device makers. The story will be brighter for Android in the phone market, where Android's "openness" aligns closely with the mobile carriers' desire for total control of their customers, but here too Apple will continue to soak up the lion's share of profits on mobile phone sales.

The sad passing of Steve Jobs won't slow Apple down this year; if anything, his spirit will continue to haunt One Infinite Loop and energize the company. Though Tim Cook is no Steve Jobs, he doesn't have to be; ultimately Apple's products will speak for themselves, as they always have. And while we will all miss watching Jobs unveil Apple's latest and greatest, there will still be one more thing...the new Apple TV.

Apple TV
Television, the Apple way
With all the rumor and speculation swirling around the forthcoming Apple TV device, from Jobs' cryptic comment in his biography that "I finally cracked it" to talk of a 50-inch Apple branded television, it's hard to see where this one might go. I'd be surprised if they announce this early in the year; Apple needs to line up enough content owners behind the new device for it to be compelling to consumers and seem inevitable to the rest of the industry. The movie studios are scared of change and will draw out negotiations. I would look for a big announcement in the late summer or early fall.

A lot of speculation focuses on what kind of UI the device might have, with talk of Siri-style voice control, but I don't think that's the most interesting thing here: I'm sure the UI will be butter smooth and easy to use, whatever form it takes. When Jobs said "I finally cracked it", I believe he was thinking at a higher level. Apple has had the technology to build a nice TV set for a long time, but they haven't because there's no money to be made there, and simply building some pretty on-screen menus isn't interesting enough. Changing the way people perceive television and building a new business model behind that is what's interesting to Apple.

How many of you remember the first Motorola ROKR? That was the first "Apple phone", a Motorola mobile phone with a built-in iPod, released in September 2005 to widespread yawns and poor sales. It's no coincidence that Apple and Motorola parted ways not long after, with Apple announcing the iPhone in January 2007. The ROKR was a learning experiment on Apple's part, a way for them to get an understanding of mobile phone technology and the business model behind it. The genius of the iPhone was in seeing what a mobile phone could be. In 2006, a mobile phone was a commoditized product that let you make phone calls, painfully type out text messages on a cramped telephone keypad and take tiny, fuzzy pictures. Most consumers just took the cheapest device they could get from their mobile carrier. In 2007, a mobile phone became your portable connection to the internet, people lined up at the Apple store to buy it and happily switched carriers to make it work. Apple makes handsome margins on the device and a little pocket change on media sales through iTunes, the carrier attracts deep pocket customers who gladly pony up US$80 a month or more for an expensive data plan, and consumers get a clean, easy to use device that gives them Internet access on the go. Making phone calls becomes almost an afterthought. While the iPhone may not be in the carrier's best interests long term, the prospect of all those free-spending Apple customers in the short term was enough to sign up one carrier in every big market.

In 2011, teevee consists is a giant flat panel with a rat's nest of wires to a craptastic cable box, DVD player and game console, along with four inscrutable 57-button remotes lying on the coffee table. People pay US$100 or more per month to their monopoly cable company for access to hundreds of streams of random programming interspersed with twenty minutes of 100 decibel commercials per hour. Consumers pay attention to exactly 3.5 hours of programming a week; the rest of the time they flick channels endlessly, searching for something good. The flat panel screen is a commodity, as are the cable box and DVD or Blu-ray player; the game console is a barely differentiated loss-leader. Apple has been playing around the periphery of this miserable, steaming technology stack for several years with their "hobby", the little Apple TV box, that lets you watch movies and TV bought on iTunes and stream Netflix. They're learning about building UIs for giant flat panels and figuring out how to negotiate with the studios to get compelling content.

Apple has a lot of interesting pieces in place. iOS devices are already eating away at the hand-held game market. Sales and rentals of videos though iTunes have value to both the content owners and consumers with disposable income: the studios get flexible pricing and detailed sales metrics, consumers get their favorite programs without the commercial, movie previews or FBI warnings and the can watch at home or on their mobile device. Content distributors like MLB and Netflix can create custom apps for both iOS and Apple TV that give them greater control over how their content is distributed. In many ways, apps are the channels of the future. Facetime video chat is a great app, especially on iPad. The Airplay feature of iOS devices is a hidden gem here, effectively turning that big flat panel into a flexible video terminal for your iPhone, iPad or iPod touch, allowing you to project your favorite movie, TV show, game or app onto the big screen for all to enjoy. Apple TV sales figures may be modest when compared to other iOS devices, but the couple of million little Apple TV boxes sold in 2011 could be just the tip of the iceberg. The killer app for Apple TV is the ability to effortlessly bore friends and neighbors with all those bad vacation photos you took.

Apple could take the Apple TV platform in a number of directions. While I would love for them to do something that would allow me to ditch my collection of 57-button remotes, more likely Apple views the DVD/Blu-ray player, cable box and game console as legacy -- they will all go away on their own soon enough. Imagine Steve Jobs watching television, cross-legged on the polished hardwood floor of a nearly empty room with just a Tiffany lamp and a sleek, 50-inch Apple branded flat screen, the slim, silver seven-button Apple remote in his hand. You wouldn't need or want anything else. If Apple does produce their own television set, simplicity will be it's main value proposition to consumers. It will be designed to stand alone rather than integrate with all those legacy devices. You won't need to hire Geek Squad to set it up, but it won't appeal to people who are heavily invested in all those legacy devices. The tech press will complain about it being closed and limited, but it will strike a chord with certain buyers, particularly the 9% of US consumers that have dropped their cable television service. I'm sure Apple has a mature prototype television set sitting in Sir Jony Ive's office in Cupertino; whether Apple thinks the world is ready for it is the real question.

For the rest of us, Apple will continue to make the inexpensive little Apple TV box. They may update the hardware this year to add Siri-style voice control, but I'm expecting the real innovation will be on how content is bought and delivered. The rumor that Apple plans to bid for rights to broadcast English Premier League soccer is intriguing. The Apple TV already comes with apps for NBA, NHL and MLB as well as the Wall Street Journal. What would your television viewing experience be like if all your cable channels became apps on Apple TV, with in-app purchase or subscriptions for premium content? How would you divvy up that US$100 a month you currently spend on cable TV among all those apps? If enough of your favorite shows were available on Apple TV, canceling your cable TV subscription would be a no-brainer. I'll bet that many of the content owners would be happy to cut out the cable companies and have better control over their relationship with viewers.

Whatever Apple does around teevee this year, you can bet that critics will deride it as too limited, closed or incompatible, just as critics derided the first iPhone and iPad. Meanwhile, Apple will be laser-focused on making their customers happy and earning generous profits in the process.

Thursday, January 5, 2012

Avoid agile dogma: recommendations not rules

Bozo the Clown
Years ago as a developer working mostly solo, I got interested in automated testing and unit testing, which in turn lead me to extreme programming (XP) and agile software methodologies. I've had the wonderful good fortune of working for Pivotal Labs a number of times in recent years and been a part of a number of successful agile projects. I know that an agile approach to project management can work well in many situations and I think I have a good understanding of how and why. But I'm not an agile zealot -- agile isn't appropriate for every type of project or organization and a good team can usually succeed using any process. Still, I think working on an XP project is more fun and rewarding for everyone involved.

I've talked with many developers who range from agile skeptic to agile detractor. They often react negatively to a particular XP or agile practice, with pair programming and unit testing being the two most common ones. Some developers are drawn to programming because they get great satisfaction from grappling with interesting intellectual challenges in solitude; agile doesn't have much to offer an individual developer when this is their top priority. If you're this kind of developer and you've found a place that gives you the freedom to work solo on cool stuff, more power to you!

More often, I talk to developers who recoil from the dogmatic pronouncements of agile proponents.

Software developers are a funny bunch. We are prone to seeing the world in very black or white, zero or one terms, and we generally have finely tuned bullshit filters. We prefer a world that is rational, measurable and repeatable to one where truth is determined by politics, personality and political correctness. When confronted with admonishments to pair program, write unit tests, use Pivotal Tracker or pat your head while rubbing your belly, we ask, "Why?" All too often, agilistas respond by implying that you can't possibly succeed in software development without doing all these things, and more: this is just padawan stuff, just wait 'till you become a full-fledged Jedi! At this point, the bullshit filter is engaged and anything that smacks of "agile" is permanently tagged with bozo, to be called up later when a target for ridicule is needed.

That's a shame, because agile methodologies like XP and Scrum really aren't about writing software, but about managing complex projects where the goal is to build something unique and novel. And it's a rare software project that doesn't have uncertain requirements, technical risk, deadlines and limited budget. Software project planning and management is far too often dominated by the politics, personality and political correctness that software developers eschew. The core idea of agile methodologies is to turn project planning and management into something more rational, measurable and repeatable. Blanket prescriptions that you must write code a certain way, use a particular tool or hold certain kinds of meetings aren't helpful; in fact, they can be actively harmful to a functioning development team.

Mike Cohn (the author of one of my favorite books, Agile Estimating and Planning) made a New Year's resolution to "Make recommendations not rules". Mike lays out his view of the core rules that make a software development team "agile" in his eyes. It's a short list, just five points, but it gets at the heart of good project management, whether you call it agile or not. "Beyond that, it’s much more about recommendations," Mike states, and I agree.

If you're an agile proponent, try to keep in mind that each team's pain is unique, and there are many different paths to success. User stories or planning poker may be the shiny new candy to you, but most people just want to get their work done. Pushing strange new techniques on people and organizations that are unready or unreceptive is ineffective at best and counterproductive at worst. Making sure the right people are having the right conversations at the right time is at the heart of any successful project. Agile methodologies like XP and Scrum are frameworks for making this happen, but they're not the only way. Avoid agile purity tests, adapt to local circumstances and prefer recommendations over rules.