Monday, October 15, 2012

Bundling Python files into a stand-alone executable


One of the problems with building a medium to large sized program in Python (or similar scripting languages) is distributing it to users. When a Python script grows beyond a couple hundred lines, most programmers prefer to split that single script file into multiple Python modules and packages. For an individual developer, modules and packages are primarily an aid in mental organization, though they also ease navigating around the project. For a large Python program being developed by a team, modules and packages are an important way to communicate the structure and intent of the code.

Unfortunately, distributing a multi-module Python program has a number of problems. First, you must carefully assemble all your program's dependencies in a single directory tree. Second, you need to make a zip or tarball of the directory tree for distribution. Third, you need to instruct your end users on how to unpack the zipped or tarballed program and how to correctly set their PYTHONPATH and which Python file or shell script in the directory tree to invoke to run your program.

Python has long included the distutils module to help developers distribute Python code. Distutils is focused on distributing Python modules and packages for use by other Python developers and is great for its intended purpose; it can also install shell scripts in the standard operating system command directory (such as /usr/local/bin on most UNIX-derived systems). It has a big problem though: Python libraries installed by distutils are made available to all Python code unless special care is taken. If you include any common third-party libraries in your program, you run the risk that your end user may have a different, possibly incompatible version of that library already on their system. You risk breaking other Python programs, and being broken in turn if you share libraries. Windows users have long dealt with DLL Hell, a similar problem where different Windows applications would install incompatible versions of shared libraries.

Today even the computer in your pocket has dozens of gigabytes of storage so modern development has moved away from sharing library code between programs. For Python developers, virtualenv allows you to quickly and easily create separate virtual Python installations on a single computer, each one isolated from the others and from the "real" Python installation. You can install Python modules and packages in one virtualenv without affecting the others. Used along with the pip package manager, it's easy to document and recreate a virtualenv Python environment, which is a boon to Python web developers.

Virtualenv is still overkill for end users, technical or not, who simply want to run your program in order to get their work done. Fortunately, Python quietly added a new feature in 2.5 that makes it possible to bundle up a directory full of Python code into a single executable file. I say "quietly" because Python 2.5 was released in 2006 and I only heard about this feature now in 2012, six years later. (Okay, it's possible I wasn't paying close attention. :-) Typical of Python, the feature isn't pretty but it has a certain elegance to it: the __main__.py file.

How to use a __main__.py file
The Python documentation for the __main__.py file explains its purpose succinctly but barely hints at the possibilities. I'll try to do a better job. Lets start by creating a directory for our Python application named app:
$ mkdir app
Now open your favorite text editor and create the file app/__main__.py. Add the following code to it:
# file app/__main__.py

def main():
  print('The rain in Spain falls mainly in the plain.')

if __name__ == '__main__':
  main()
If you've done some Python programming, you'll recognize the __name__ == '__main__' idiom used to determine if a python module is being executed directly rather than imported as a module. When it's executed directly, the example simply calls the main() function, which prints "The rain in Spain falls mainly in the plain." to standard out.

Now let's run this program. Instead of calling __main__.py directly, we can treat the app directory as our Python program:
$ python app
The Python interpreter sees that app is a directory and checks for a __main__.py file inside it. Note that Python only checks the top level of the directory; it doesn't search subdirectories. Since there is a __main__.py directly in app, the interpreter runs it and the output is:
The rain in Spain falls mainly in the plain.

In addition, the Python interpreter will add the directory to the start of sys.path so that all imports will check the that directory first. By placing all of the modules and packages that our program depends on in the directory, we can stay isolated from whatever versions the end user may have installed as well as keep our dependencies isolated from the end user's system.

Zip it up
Python has supported loading modules and packages out of a zip file since 2.3. Just as it now looks in a directory for __main__.py, Python will also look in a zip file for __main__.py. Let's zip up the app directory and test this.

Note that the __main__.py file needs to be at the top level in the zip container, not in a subdirectory. This makes creating the zip file a little tricky. We want to recursively zip up everything in our app directory, but not include the app directory itself. (Windows users will need a command line zip program to follow along.)
$ cd app
$ zip -r ../app.zip *
$ cd ..
(Use *.* instead of * on Windows.)

To test that you've zipped things up correctly, run your Python program directly from the zip file:
$ python app.zip
You should see the expected output:
The rain in Spain falls mainly in the plain.

Python will place the zip file first on sys.path just as it does for a directory; all modules and packages imports will search the zip file first. Be sure to place your modules and packages at the top level in your directory along side the __main__.py file.

Load a resource
If you've put all your Python code in the right place using this scheme, everything pretty much just works as you expect it to. But some programs depends on resources aside from Python code, and need to load various data files that come bundled with the program. The easiest way to find and load a program bundle like this is to use the pkg_resources module. The pkg_resources module does a lot of things, but you'll want to look first at the ResourceManager API which has the most common functions for finding and loading resource files.

Let's add a resource file to our little app and load it using the pkg_resources.resource_string function. Create a subdirectory under app called resources.
$ mkdir app/resources
Using your favorite text editor again, create the file app/resources/inFrance.txt and add some text to it:
But the ants in France are mainly in your pants.
Now edit app/__main__.py so that it looks like this:
# file app/__main__.py

import pkg_resources

def main():
  print('The rain in Spain falls mainly in the plain.')
  print(pkg_resources.resource_string('resources', 'inFrance.txt'))

if __name__ == '__main__':
  main()
You may already have pkg_resources.py installed on your system. If you don't, you'll find it's part of the distribute package. Download the latest version of distribute, unpack the tarball and find pkg_resources.py inside. Copy pkg_resources.py to app/pkg_resources.py. (Even if you already have the pkg_resources module on your system, if you use it in your program, you should add it to your bundle before distributing it to others.)

Now when you run the program:
$ python app
You should see this output:
The rain in Spain falls mainly in the plain.
But the ants in France are mainly in your pants.

Make it executable
Finally, you can turn your zipped program bundle into a stand-along executable on UNIX-like systems using a couple of commands. Zip up the latest version of the program in the app directory and name it app2.zip.
$ cd app
$ zip -r ../app2.zip *
$ cd ..
Now use a bit of UNIX magic to turn app2.zip into an executable.
$ echo '#!/usr/bin/env python' | cat - app2.zip > app2
$ chmod +x app2
The first command inserts a UNIX shebang at the start of the zip file and writes it to a new file called simply app2. The zip file format is designed to allow a small executable program to be inserted at the front (that's how self-extracting zip files are created), so this is kosher and doesn't corrupt the zip file. The second command sets the executable bits on app2.

Now you can simply run app2 like any executable.
$ ./app2
And you should see the expected output.
The rain in Spain falls mainly in the plain.
But the ants in France are mainly in your pants.

Wednesday, October 10, 2012

BSD-style license for code

We have published a fair amount of source code on the Able Pear Software blog over the past few years, but we neglected to specify any kind of software license to go along with it. We occasionally get asked about an open source license (the UrlEncoding category for NSDictionary is particularly popular).

All the code we publish on the Able Pear Blog is free for you to use under a BSD-style license, as is our Autoindigestion tool. We like the simplicity of unrestrictive open source licenses like the BSD and MIT licenses. If you decide to incorporate some of our code into your project or product, we'd love to know. Modern software is so complex, it's simply not practical to write everything from scratch. Many parts of OS X and iOS have their roots in the FreeBSD project, which itself is a descendant of BSD UNIX. Even Microsoft has incorporated some BSD code in Windows.

Here is the Able Pear Software blog open source code license. Adjust the copyright year to match the date on the blog post. (Please note that the full text of the blog is not included in this license, only the source code.)
Copyright (c) 2012, Able Pear Software Inc.
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

- Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

- Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Monday, October 8, 2012

Smart App Banners

The new version of Mobile Safari that ships with iOS 6 has a great new feature for app publishers: Smart App Banners.
When a user visits your site in Mobile Safari on iOS, you can now add a pop-up banner to promote your iOS app, which includes a direct link to the app in the App Store and optionally your iTunes affiliate information. David Smith has a great overview on his blog, and you can find all the details of Smart App Banners in Apple's Safari Web Content Guide on the Apple developer site.

Though I doubt that this will do away with all those annoying full-page "get our iPad app" pop-ups, I hope that at least some sites will start to use this instead. Thanks Apple for providing something to help sites promote their apps in a less annoying way.

Monday, July 30, 2012

Adding a Retina-ready icon to your Mac app in three easy steps

I've been doing more OS X development recently, and coming from the iOS world, there's a lot that's familiar but I still stumble over many things. Creating an app icon is one of them. After some poking around, I've discovered what you need to do to create a custom icon for your modern Mac app.

Step 1: Create the icon images
Modern Mac icons pack five different resolutions in one .icns file, from 16x16 to 512x512:
  • icon_16x16.png
  • icon_32x32.png
  • icon_128x128.png
  • icon_256x256.png
  • icon_512x512.png
The dimensions are actually "points" rather than "pixels". On standard resolution displays, 1 point == 1 pixel; iOS developers are already familiar with handling "retina" resolution displays where 1 point == 2 pixels. These double resolution "retina" resources get "@2x" added to the base filename, so in addition to the five standard resolution icon images, you now need five double resolution ones:
  • icon_16x16@2x.png
  • icon_32x32@2x.png
  • icon_128x128@2x.png
  • icon_256x256@2x.png
  • icon_512x512@2x.png
The names describe the icon size in points, so icon_16x16@2x.png is 32 by 32 pixels and icon_512x512@2x.png is 1024 by 1024 pixels. Some of these images have equivalent pixel sizes: icon_32x32.png and icon_16x16@2x.png are both 32 by 32 pixels. You may be able to get away with using the same bitmap in many cases, but you (or your icon designer) may want to tweak each version to look best on their respective display types.

Step 2: Add an .iconset to the Xcode project
In the olden days, you would use the Icon Composer app to build your .icns file, but it's no longer being updated by Apple and doesn't support double resolution images. Today, Xcode will build your .icns file automatically for you. (Alternately you can use the iconutil command line utility to build .icns with high res images or to extract images from .icns files; see Apple's High Resolution Guidelines for OS X for details.)

The trick to making Xcode build your .icns automatically is to put your set of icon images in a folder named <ICON_NAME>.iconset, where <ICON_NAME> becomes the base name for your .icns file. Add this folder to your Xcode project and Xcode will then create <ICON_NAME>.icns when you build, and automatically copy it to your app bundle.

Note that the images in the .iconset directory need to have the names given above: "icon_16x16.png" through "icon_512x512@2x.png" or you will see a warning in Xcode when building your project and the mis-named images won't be included.

Step 3: Add the icon to the Info.plist
To set your app's main icon, add the "Icon File" key to your app's Info.plist file and set the value to <ICON_NAME> (without the .icns extension). If you used the standard Xcode template to create your app, your Info.plist will be named "<APP_NAME>-Info.plist", where <APP_NAME> is your app or project name. For those of you who like to edit your .plist files as XML, the key name is CFBundleIconFile.

You can also drag the .iconset directory from the finder or the Xcode project navigator to the "App Icon" pane of the Summary tab for your app's target (pictured above). If you use this approach, Xcode will insist on copying the .iconset into the root folder of your project, even if you've already added the .iconset somewhere else in your project's directory tree (<sarcasm>another great example of Apple's attention to detail</sarcasm>.) If you're not fussy about your project organization, you can simply let Xcode have its way; otherwise you can remove the copy Xcode makes and add the .iconset in a more appropriate place, like the Resources folder; just make sure to leave the "Icon File" key in your Info.plist and Xcode will show the icon in the "App Icon" pane.

Easy as 1-2-3
Not hard once you figure it out, and a nicer workflow than having to wrestle with a half-baked special purpose app like Icon Composer.

Monday, July 9, 2012

Download iTunes Connect sales reports with Autoindigestion

When you start selling your first app in the app store, it's very exciting to check you sales every day in iTunes Connect. After the initial excitement wears off and your app sales settle into a steady state, it's easy to get involved in your next project and forget to log into iTunes Connect periodically to download your daily or weekly sales reports. Apple only makes the last 14 days of daily reports and the last 13 weeks of weekly reports available; if you're not diligent, time can fly by and you will lose important sales records.

Apple does provide a tool to automate the retrieval of iTunes Connect sales reports: the Auto-Ingest tool, a small Java command line app. While it's good that Apple provides an officially supported tool, it's a very minimal one. The Auto-Ingest tool only downloads one report at a time and doesn't have any intelligence for determining which reports have already been downloaded, nor can it handle downloading multiple report types or multiple iTunes Connect vendor accounts without some scripting help. Enter Autoindigestion.

Autoindigestion is a command line utility for Mac OS X Lion that uses the Auto-Ingest tool to automatically download daily and weekly sales reports. The first time it runs, Autoindigestion will grab all available reports; on subsequent runs it will download new reports based on the current date and the dates of previously downloaded reports. If you miss a day because your system is off or iTunes Connect reports are delayed, Autoindigestion will catch up the next time it runs. Autoindigestion can also be configured to handle multiple iTunes Connect vendors, which is great if you handle sales reporting for clients.

Autoindigestion is open source and available under a BSD style license. It's currently available as source from the Autoindigestion GitHub page. It is a native OS X command line program written in Objective-C and requires Xcode to build. Complete instructions for installing and configuring Autoindigestion are included on the GitHub page. I enjoyed creating Autoindigestion and hope you find it useful.

Wednesday, June 20, 2012

WWDC 2012 Videos

Apple has posted the videos and slides from the technical sessions of this year's Worldwide Developers Conference yesterday. The videos are available for free, but you need to be a registered Apple developer to watch them. (You can register at https://developer.apple.com/.)

As in previous years, there's a lot of good stuff here for iOS and OS X developers, as well as some sessions on other parts of the Apple ecosystem like Safari and iAds. Also featured this year are several sessions related to authoring content for the iBookstore using both EPUB 3 and Apple's proprietary iBooks Author formats. There are also a good number of sessions focused on getting the most out of Xcode and the Apple toolchain.

I've watched many of the videos from previous years and they are quite useful. If you are an experienced iOS developer, these sessions are a great way to help you get up to speed on the latest stuff. If you are new to iOS, the WWDC sessions, especially from previous years, contain some great overviews that will help you get up to speed fast. I highly recommend them.

Monday, March 5, 2012

Instance Variables in the Implementation File

Xcode
Recent versions of iOS and the Apple LLVM 2.1 compiler use an improved version of libobjc, the core library that powers Objective-C features like classes and method calls. This "modern runtime" library enables a lot of new, cool flexibility in creating Objective-C classes. One of my favorite new features is Instance Variables in the Implementation File, which allows you to move your private instance variables out of your .h files. iOS developer Tips shows you how.