One of the main problems when developing Scala on Android is the fairly heavy runtime - Scala adds a few megabytes which need to be converted to Dalvik bytecode during the build process (slow!). The common workaround is to use tools like proguard or treeshaker to shrink the code before dexing it.

However if this is still to slow there is another option which works on rooted devices and the Android emulator: predex the Scala libraries and add them to the Android runtime, on the device itself.

Android has the concept of a boot classpath (similar to the JVM) which contains all the system framework classes. It is set up in the file /init.rc on the phone:

$ adb -d shell 'grep BOOTCLASSPATH init.rc'
export BOOTCLASSPATH /system/framework/core.jar:
 /system/framework/bouncycastle.jar:
 /system/framework/ext.jar:
 /system/framework/framework.jar:
 /system/framework/android.policy.jar:
 /system/framework/services.jar:
 /system/framework/core-junit.jar

So "all" we need to do is to predexed the Scala library and append it to this path.

Preparing the emulator

Predexing the Scala libraries

Stéphane Micheloud has done some work to make this step easy - he created a bunch of scripts to dex the libraries and create custom RAM images for the Android emulator, described in Tweaking the Android emulator:

$ git clone git://github.com/jberkel/android-sdk-scala.git
$ cd android-sdk-scala
$ ./bin/createdexlibs
Using Scala 2.8.1.final in /usr/local/Cellar/scala/2.8.1/libexec
Generating scala-library.jar...
Generating scala-collection.jar...
Generating scala-immutable.jar...
Generating scala-mutable.jar...
Generating scala-actors.jar...
Converting scala-library.jar into a dex file...
Converting scala-collection.jar into a dex file...
Converting scala-immutable.jar into a dex file...
Converting scala-mutable.jar into a dex file...
Converting scala-actors.jar into a dex file...
Dex files were successfully generated (configs/framework)

After a successful run the processed libraries can be found in configs/framework.

Creating the emulator ramdisks

To create the ramdisks for an emulator you need to run another script:

$ bin/createramdisks

Boot up the emulator with the modified ram disk and copy the predexed libraries:

$ emulator -avd ... -ramdisk /path/to/custom.img
$ adb shell mkdir -p /data/framework
$ for i in configs/framework/*.jar; do adb push $i /data/framework/; done

Reboot again and make sure the new BOOTCLASSPATH is active:

$ adb shell echo '$BOOTCLASSPATH'

You should now be able to install and run Scala apks directly on the device without shipping the whole runtime with each install.

Patching a real device

If you want to do the same thing on a real device things get a little bit trickier - you cannot just edit /init.rc and reboot the phone since the init scripts are included in a filesystem in the boot image, so you need to extract your current boot image to patch it, reassemble a new image and flash it.

General instructions are available in HOWTO: Unpack, Edit, and Re-Pack Boot Images.

Extract your current boot image

$ adb shell 'cat /proc/mtd'
dev:    size   erasesize  name
mtd0: 000e0000 00020000 "misc"
mtd1: 00500000 00020000 "recovery"
mtd2: 00280000 00020000 "boot"
mtd3: 09100000 00020000 "system"
mtd4: 05f00000 00020000 "cache"
mtd5: 0c440000 00020000 "userdata"

$ adb pull /dev/mtd/mtd2 boot.img

The boot.img file uses a custom layout which contains a header, kernel and the ramdisk. These parts can be extracted using a Perl script (split_bootimg.pl):

$ curl https://raw.github.com/gist/1087743/5be96af0e1c1346678379b0c0f0330b71df51f25/split_bootimg.pl
 > split_bootimg.pl
$ perl split_bootimg.pl boot.img
Page size: 2048 (0x00000800)
Kernel size: 1835568 (0x001c0230)
Ramdisk size: 143802 (0x000231ba)
Second size: 0 (0x00000000)
Board name:
Command line: no_console_suspend=1 wire.search_count=5
Writing boot.img-kernel ... complete.
Writing boot.img-ramdisk.gz ... complete.

Now the ramdisk needs to be unpacked:

$ mkdir ramdisk
$ cd ramdisk
$ gzip -dc ../boot.img-ramdisk.gz | cpio -i

You can now make changes to ramdisk/init.rc and add the Scala libraries to BOOTCLASSPATH, it should look similar to this (without line breaks):

export BOOTCLASSPATH /system/framework/core.jar:
  /system/framework/bouncycastle.jar:
  /system/framework/ext.jar:
  /system/framework/framework.jar:
  /system/framework/android.policy.jar:
  /system/framework/services.jar:
  /system/framework/core-junit.jar:
  /data/framework/scala-actors.jar:
  /data/framework/scala-collection.jar:
  /data/framework/scala-immutable.jar:
  /data/framework/scala-library.jar:
  /data/framework/scala-mutable.jar

After you have made that change you need to reassemble the boot image. You will need two command line tools from the Android source code to do that (which need to be compiled first, here a quick HOWTO: compile mkbootimg/mkbootfs).

$ mkbootfs ramdisk/ | gzip > newramdisk.gz
$ mkbootimg  --cmdline 'no_console_suspend=1 wire.search_count=5' \
    --kernel boot.img-kernel \
    --ramdisk newramdisk.gz \
    --base 0x20000000 # Only needed for Nexus One \
    -o boot-new.img

Make sure to adjust the cmdline parameter to reflect the output of the split_bootimg command, and to leave out the base parameter if the device is not a Nexus One.

Install the new boot image

$ adb push boot-new.img /sdcard/
$ adb shell

$ cat /dev/zero > /dev/mtd/mtd2
write: No space left on device [this is ok, you can ignore]

$ flash_image boot /sdcard/boot-new.img
flashing boot from /sdcard/boot-new.img
mtd: successfully wrote block at 0
mtd: successfully wrote block at 20000
mtd: successfully wrote block at 40000
mtd: successfully wrote block at 60000
mtd: successfully wrote block at 80000
mtd: successfully wrote block at a0000
mtd: successfully wrote block at c0000

Don't forget to copy the Scala jars to /data/framework (same step as emulator install above), then reboot the phone. It also a good idea to have a recovery update image on the sdcard in case anything goes wrong.

If everything worked you should see the new BOOTCLASSPATH active after reboot, test with:

$ adb shell 'echo $BOOTCLASSPATH'

Deploy Scala straight to your phone

Now you can enjoy reasonably fast development cycles on your device, when using the sbt-android-plugin just add the line

override def skipProguard = true  # sbt 0.7.x
useProguard in Android := false   # sbt 0.1x

to your project config (example). This currently only works with the development version (0.5.2-SNAPSHOT).

For comparison, a "hello world" Scala project, with proguard enabled:

$ time sbt clean package-debug
real    0m40.514s
user    0m50.632s
sys 0m2.345s

and without:

$ time sbt clean package-debug
real    0m16.507s
user    0m22.199s
sys 0m1.873s

When you get to the point of releasing your code you can just re-enable proguard to produce a distributable package.

It's been a while since I wrote about Scala on Android - I'm still maintaining and working on the sbt-android-plugin but I haven't written any full-blown apps with Scala. I started a project (gigjet) but abandoned it after a while (there were some Java-Scala interop problems, and a general lack of motivation).

I checked and the initial commit on that project was almost two years ago - a lot of things have changed since then, so I wanted to get an idea of the current situation by writing a small but useful application with the goal of actually shipping some code this time (= publishing it to the Android market).

The idea is simple - write an app to push gists (snippets of text) using the recently released version of the github API (which added write support for the first time).

As I started to work on it I realised that a lot of things have changed - here is a quick summary:

Better tool support

When I first started with Scala it didn't have good IDE support - I use Intellij IDEA for Java projects but their Scala support was very preliminary to say the least. Eclipse supposedly had a bit more to offer but is still, well, Eclipse so not really an option for me. I ended up using vim which worked OK but didn't feel that productive.

In the meantime the Scala core team has realised that tool support is crucial for the wider adoption of the language and Martin Odersky (Scala's designer) is working on (and dogfooding) the Eclipse plugin. I've read somewhere that he was mostly using Emacs before - not something you can necessarily assume from average Java Joe.

The Intellij Scala plugin is very functional and behaves mostly as you would expect - it has some nice features like highlighting of implicit parameters which is very useful. It also works well with the built-in Android support / facet - there is support for logcat dumps, emulator integration, source code level debugging etc.

The only downside is that compilation and syntax checking is very slow - the syntax checker routinely lags a few seconds after making a couple of edits, which can be very irritating. Compilation and packaging is also very slow - sbt with its powerful console mode is the better tool for that. I tend to use IDEA for editing and refactorings and then just hit compile or package in the sbt console.

If you're doing TDD both IDEA and sbt support most Scala test frameworks - there is even an IDEA plugin for the (relatively) new specs2.

Shaken, not stirred

One of the biggest drawbacks of using Scala on Android is the additional required treeshake / proguard step. This is necessary to keep the app size down since Scala has quite a big runtime library. Depending on the size of the application and dependencies this can easily add an extra 30 seconds on top of the normal build process which slows down the development cycle drastically.

However there is a way around that: you can tweak the emulator to include predexed versions of the Scala runtime, which means you can completely skip the proguard step during the (development) build. Together with sbt's incremental compilation this makes for really fast build times, not much slower compared to a standard Java build. Needless to say this makes a huge productivity difference because you can iterate a lot faster.

Growing up

Not only the tools but also Scala itself has matured over the last two years - a few influential companies have started to adopt it, there are more libraries and open source projects around to use and learn from. Many users regard the release of version 2.8 as Scala's "coming of age", it was often claimed that it should really have been called 3.0.

Most importantly it felt like a good fit for the project I was working on - at the core of the app is the API integration with github which is naturally very async and callback heavy - and Scala made that part easy.

As an example take the way asynchronous tasks are normally implemented in Android - you are expected to subclass (!) AsyncTask and override some methods in order to provide callbacks:

  
  class ApiTask extends AsyncTask<Request, Void, HttpResponse> {
      @Override public void onPreExecute() { ... }
      @Override public HttpResponse doInBackground(Request... args) {
        // actual work done here
      }
      @Override public void onPostExecute() { ... }
  }

  new ApiTask().execute(...);
   

With Scala you can just pass your callbacks directly as functions which is a lot more elegant and less boilerplaty, here a simplified example taken from the gist-it app:

  
  def executeApi(call: Request => HttpResponse, req: Request, expected: Int)
                (success: HttpResponse => Any)
                (error: Either[IOException,HttpResponse] => Any) {
    //...
  }
  

This defines a method which takes as parameters

• a function which takes a request object and returns a HTTP response (call)
• a request (req)
• an expected HTTP status code (expected)
• a success callback which gets passed the HttpResponse object (success)
• an error callback which gets passed either an IOException or an HttpResponse object (error)

You can see that this does a lot more than the AsyncTask - it basically abstracts a common API request (do this, expect this, return value, etc). The client code just passes in all the necessary callbacks.

Another Scala feature which is very useful on Android are traits. I use traits in a similar way Ruby modules can be used - to mix behaviour into existing classes. The reason this is useful on Android is that the framework forces you to inherit in a lot of places - your activities need to subclass Activity, your views View and so on. This makes adding extra functionality a bit cumbersome - you can add intermediate classes or use delegation.

With Scala you can encapsulate common behaviour in traits and then mix them into your activities or views - you end up with smaller and more reusable pieces of code, potentially even across different projects.

Again as a quick example a trait to perform the lookup of the current token using Android's AccountManager API:

  
  case class Token(access: String)

  trait TokenHolder extends Context {
    lazy val accountType = getString(R.string.account_type)
    def account: Option[Account] =
      AccountManager.get(this).getAccountsByType(accountType).headOption

    def token: Option[Token] = account.map(a =>
      Token(AccountManager.get(this).getPassword(a)))
  }

  class MyActivity extends Activity with TokenHolder { ... }
  

The trait extends Context so you can use it with any Android context (usually activities or your application instance). It also means the trait itself has access to all methods provided by Context.

The gist (sorry...)

It's been fun to use Scala for this project - after some initial time spent on getting everything set up I was immediately very productive. The app got finished and released and has already some users.

You can find it in the Android Market and the code on github. Fork away if you have any ideas for new features!

So, what are Scala's chances in the wider Android ecosystem? There's no official word from Google yet but Tim Bray (Android developer advocate) is very keen on supporting non-Java languages on the platform. More and more developers are interested in Android but not everybody wants to use Java - Scala is an excellent alternative. A few success stories here and there, a high profile Android app implemented in Scala would be a good start.

There is now also a dedicated mailing list (scala-on-android) - still too small to be called community but hopefully growing faster.

Updated 17/06/2011:

I'll add some links to relevant new blog posts as I find them:

• Scala: A Better Java for Android
• Scala’s Popularity on the Rise in Boston: sbt for Android
• How How we use Scala in Bump for Android

ACRA is a great little Android open source library which posts error reports (stack traces) from your application in the wild to a Google spreadsheet. It is an important tool especially given the limitation of Google's built-in error reporting (Android market apps / 2.2+ only).

So with ACRA in place you can easily check all error reports, inspect stack traces and so on.

What's missing however is the big picture - are there more crashes on Android 1.x than on 2.x ? Has the latest upgrade of your app improved the situation? Has the Motorola DROID really the buggiest Android firmware on this planet?

Because ACRA uses a Google spreadsheets to store the data it is very easy to run queries against it with the Google Visualization API. The API has two layers - a data access layer with a powerful query language (similar to SQL) and a lot of different components to visualise the data.

The interesting parts of the query language for this project are the GROUP BY and PIVOT clauses. Getting crash counts for different application versions, grouped by different device models is very easy:

  
    SELECT model, COUNT(A) GROUP BY model PIVOT version
  

The results can now directly be used to create a new dynamic table or chart (check out the live demo).

To use this for your own application take a look at acra-analysis.html and change the Spreadsheet link. You will also need to set the sharing settings of the spreadsheet to 'Anyone with the link' or 'public'. If this project evolves further I'll probably move it to a separate project.

Rant

These analytic features should really be part of the Google Android dev console - they own all the pieces to build it but then leave it to somebody else. The fact that an open source project (ACRA) is superior to the built-in crash monitoring (which arrived really late) says it all.

“Louis, I think this is the beginning of a beautiful friendship”

The movie database IMDB is a great source of information, but there is one aspect of it most people don't know about: most films have a "Movie connections" section, which lists all references to other films (or TV series). What can be considered a reference is pretty open: it could be music, a film poster visible in a scene, or one of the characters uttering a well-known line from another film, like Humphrey Bogart's words "I think this is the beginning of a beautiful friendship" from Casablanca's famous ending.

The current IMDB dataset contains around one million of those references, far too many to visualise or comprehend easily. So how could this data be filtered, to get to the interesting films and references?

Rank it!

Now, if you look at film references as edges between nodes (films) in a graph you can apply existing algorithms like PageRank to obtain a score for each film, similar to one of the ways Google determines importance of web pages. An intentional reference to another film is similar to citation in a book, or a link to another web page.

I have had the idea to use graph algorithms to analyse the IMDB movie connections for a while now but only started building something until quite recently. Then, after a few days of coding and just before publishing this blog article I saw that somebody else had done more or less the same work, a few months earlier: in the blog post "The most important movies of all time", Thore Husfeldt presents a list of important movies as determined by his FilmRank algorithm which is (as the name suggest) pretty similar to PageRank.

The tl;dnr version of the blog post: PageRank on a movie reference graph produces lists very similar to top film lists compiled by critics. The list he presents is also similar to mine, so I will not repeat it here.

As always, it is very hard to come up with novel ideas (or even just applications thereof) - there is a very high probability that somebody has already implemented and published "your" idea somewhere on the internet. However there are two things his article is lacking: a graphical representation of the results and the source code to obtain them from the raw IMDB data set.

Pretty graphs and weird numbers

Interestingly, a look at the Top 250 (Google doc / CSV) list reveals a lot of well-known films but also some more obscure ones. How did they get on this list? It's not really obvious from the list data itself, you need to look at the connections to find out why.

One of the properties of PageRank is that important nodes distribute their importance to all nodes they link to. Now, if a very significant node (let's say Star Wars) links to a few other nodes, these nodes will be considered important as well. So in order to understand these rankings better we need to look at the links between nodes.

I created a few graphs with the top N films and all the connections in between them, filtering out unconnected nodes. In the graph on the side, I was curious about the film on the left, 21-87. I had never heard of it, but it is obviously ranked highly because Star Wars and THX 1138 refer to it. Both these films were directed by George Lucas (THX 1138 was his first film), so there had to be be some sort of connection.

A quick look at the movie connections page revealed two numerical references (Leia's Death Star cell number and a date), and a search for "21-87 George Lucas" finds the article "21-87: How Arthur Lipsett influenced George Lucas’s career" which goes on to explain how Lipsett's experimental short films influenced the sound design of George Lucas' later films.

I used graphviz to show connections between top ranked films. Graphviz can output SVG, so it was pretty easy to create high quality vector documents which render in most modern browsers (all except Internet Explorer < 9, Android WebKit).

There are a few different versions since more nodes means more edges and therefore less readability (I did not apply any sort of edge pruning):

• Top 50 (SVG)
• Top 100 (SVG)
• Top 250 (SVG)

And the raw data:

• Top 250 (CSV)

Films from the same decade are grouped vertically, starting from the 1900s on the left. Tooltips show more information about each film, and each node links directly to the IMDB connections page. Films with bold titles indicate a IMDB Top 250 title. Darker coloured nodes are TV series. Since the SVG standard does not yet have zoom controls you will need to use your browser's zoom controls.

As a side note, the algorithm used to produce these rankings does not make a distinction between movies or TV series - all links are considered to be of equal importance. This has the interesting side effect that TV series make it into the ranking - almost all top charts (IMDB Top 250 included) do not contain TV productions, although some of them had a big influence on cinema. You might wonder then why the Simpsons (a true reference generating machine) are not part of the graph - although they rank 61st the IMDB data only contains incoming links for them, presumably because they already have their own reference database, the episode guidebook.

Also there is no link dampening applied (downweighting references from a recent film to an old film).

Show me the code

I have released the source code to produce this rankings, which makes it reasonably easy it to create your own experiments and graphs. It is basically just a glue Rakefile with some Python code to do the analysis part (using the excellent NetworkX library). Other requirements are Python (>= 2.6) with IMDbPy, Ruby (>= 1.9) and graphviz.

 $ git clone git://github.com/jberkel/imdb-movie-links.git
 $ cd imdb-movie-links
 $ easy_install networkx imdbpy
 $ brew install graphviz wget   # OSX/homebrew
 $ bundle install
 $ rake rank                    # CSV export ranking
 $ rake graph.svg MAX=50        # create a graph, max. 50 nodes

Happy ranking!

What next?

It would be interesting to do this kind of analysis with different media - for example music (look at sample references in a track) or even literature.

The problem is to get a good dataset - the movie example works so well because the IMDB data has been collected by a lot of people, over the course of almost 20 years now.

Note: this article hasn't been updated in a while and is out of date. In doubt refer to the README of the sbt-android-plugin.

If you're thinking about developing Android apps in Scala there are not that many different options for building your project - I started out with a hack based on Rake (source), then there are some ways to get Eclipse to build your project, as documented on the Novoda blog. I personally don't use an IDE, Scala is in contrast to Java perfectly usable in a normal text editor, mainly because it requires a lot less typing and boiler plate code.

My solution based on Rake worked well but had a couple of shortcomings - it added another language to the codebase, and not everyone is necessarily familiar with Ruby, additionally your build time gets longer as the Scala compiler will always perform a full rebuild of the project. A Scala source file typically produces more than just one single class file, so it's not possible to perform a simple time stamp-based check and just recompile changed files. Unfortunately the compiler itself is not able to figure out which files need rebuilding (coming in Scala 2.8). The Scala based build system sbt (simple build tool) implements partial recompilation which means that the code will get built a lot faster. Besides speed improvements it also gives you a complete build tool with external dependency management using Apache Ivy (Maven dependencies are supported as well).

Sbt also follows Rake's approach of using the language itself for the configuration of the build which gives you additional flexibility and expressiveness over static XML files. It also makes the reuse of build steps a lot simpler because you can use the host language's natural abstraction mechanisms. So when I decided to reimplement my build system with sbt it turned out that someone had already written an Android plugin for it. Mark Harrah, the author of sbt, extracted it from someone else's Android project and released it on github, where I found it. I had to make a couple of modifications to get it to build my project but it quickly became obvious that sbt was a better fit for Android Scala projects than any other ad hoc solution. Mark merged my changes back into his repository and eventually asked whether I wanted to take over maintenance of the plugin, hence this introductory blog post. Let's get started!

Installing sbt

First you need to install sbt itself (assuming that Scala is already installed). Detailed instructions are available here, but in a nutshell:

  
  $ cd ~/bin # or any other directory in your path
  $ wget http://simple-build-tool.googlecode.com/files/sbt-launch-0.7.4.jar
  $ echo 'java -Xmx512M -jar `dirname $0`/sbt-launch-0.7.4.jar "$@"' > sbt
  $ chmod u+x sbt
  

If you're on OS X and use homebrew you can take a shortcut with brew install sbt.

Generating a new project

To get started with new projects sbt already ships with a generator which will set up the initial directory structure (run sbt without arguments). Unfortunately, when using plugins the situation is a bit more complicated as the plugin itself needs to be set up. To make things simpler I created a Scala script to bootstrap a full Android project with sbt. You simply call it with the name of your project and package:

  
  $ git clone git://github.com/jberkel/android-plugin.git
  $ cd android-plugin
  $ script/create_project foo com.example.android
  

This will create a fully usable Android project called Foo with one activity (showing Hello World).

The generated directory layout follows Maven conventions:

  
    |-- project
    |   |-- build
    |   |   `-- Foo.scala
    |   |-- build.properties
    |   `-- plugins
    |       `-- Plugins.scala
    |-- src
    |   |-- main
    |   |   |-- AndroidManifest.xml
    |   |   |-- assets
    |   |   |-- java
    |   |   |-- res
    |   |   |   |-- drawable
    |   |   |   |-- layout
    |   |   |   |-- values
    |   |   |   `-- xml
    |   |   `-- scala
    |   |       `-- Activity.scala
    |   `-- test
    |       `-- scala
    |           `-- Specs.scala
    `-- tests
  

The sbt build configuration is in the project directory, source code and unit tests in src, tests is used for Android integration testing (more on that later).

Build configuration

The main project build information is contained in the file project/build/Foo.scala:

  
  import sbt._

  trait Defaults {
    def androidPlatformName = "android-1.6"
  }
  class Foo(info: ProjectInfo) extends ParentProject(info) {
    override def shouldCheckOutputDirectories = false
    override def updateAction = task { None }

    lazy val main  = project(".", "foo", new MainProject(_))
    lazy val tests = project("tests",  "tests", new TestProject(_), main)

    class MainProject(info: ProjectInfo) extends AndroidProject(info) with Defaults {
      val scalatest = "org.scalatest" % "scalatest" % "1.0" % "test"
    }

    class TestProject(info: ProjectInfo) extends AndroidTestProject(info)
                                         with Defaults
  }
  

The project definition is contained in the class Foo which extends ParentProject, a special sbt construct for supporting multiple projects in one single file (sub project documentation). This is necessary because Android integration tests have to be built and installed as a separate apk package. Note the use of Scala's traits to mix in default settings for both projects. If you don't need integration tests in your application you can use a simpler project definition which could look like this:

  
  import sbt.__
  class Foo(info: ProjectInfo) extends AndroidProject(info) {
    override def androidPlatformName = "android-1.6"
  }
  

Dependency management

The main advantage of using sbt/Ivy as a dependency manager is that you declare your dependencies programmatically instead of shipping all the jar files with your project, sbt will then figure out on build time which libraries to download. In sbt, an external dependency is declared as follows:

  
   val scalatest = "org.scalatest" % "scalatest" % "1.0" % "test->default"
  

This declares a dependency to the module "scalatest" (a Scala test framework), using version 1.0 in the test configuration. A configuration in Ivy works similar to scopes in Maven, you usually use them to separate build and runtime dependencies (sbt will automatically exclude build dependencies from the package).

Hello Android

Besides creating the necessary directory structure for building Scala projects the generator script will also create a simple "Hello World"-style activity, which can be found in src/main/scala/Activity.scala.

  
  class MainActivity extends Activity {
    override def onCreate(savedInstanceState: Bundle) {
      super.onCreate(savedInstanceState)
      setContentView(new TextView(this) {
        setText("hello, world")
      })
    }
  }
  

In order to build the full package, use the package-debug sbt action:

  
  $ export ANDROID_SDK_HOME=path/to/android_sdk
  $ sbt update             # download dependencies
  $ sbt package-debug      # build packages
  $ sbt reinstall-emulator # (re)install in emulator
  

This will download all dependencies, then compile and build the two packages (target/foo-0.1.apk, tests/target/tests-0.1.apk). It is worth noting that sbt has two different modes of operation: interactive and command-line. If invoked without action arguments, the interactive session will be launched, which saves start-up time and is quite useful in general.

Sbt will automatically recompile the project definition (even when in interactive mode) which makes it almost as easy to use as other script-based build systems.

1,2,3 testing

Sbt has built-in support for most common Scala testing frameworks (ScalaCheck, specs, ScalaTest). It tries to automatically detect the framework you are using in your project, usually "sbt test" will do the Right Thing. The example project set up by the script defaults to ScalaTest which has excellent BDD support with a DSL similar to rspec's.

Writing Android specs using a Scala framework like ScalaTest is a very convenient way to test generic functionality, but does not replace running tests on the device / emulator itself. The DalvikVM is very different from a normal Java VM (smaller stack sizes etc.), so you should always test on device as well. On Android this is quite painful: you have to create a separate project, get the dependencies right, install the package in the emulator and run a command. With sbt's multiproject support it becomes quite easy to handle integration tests, you can subclass AndroidTestProject to get some additional actions for running tests (test-emulator and test-device). Sbt will also track dependencies between your main and test project.

Another nifty feature is continuous compilation / testing. When you prefix any action with a tilde (~), sbt will automatically trigger it as soon as one of the source files changes, which is very useful when doing test-driven development.

Demo

I went to the first Android-focused barcamp (droidcamp) which took place in Berlin-Dahlem this week. It was organised in conjunction with the more formal, pay-for droidcon, which happened the next day, same location. From what I gathered the entrance fees for droidcon partially subsidised the barcamp, an interesting concept. In the end lots of people attended both conferences (I didn't).

There was a great interest in droidcamp beforehand, most of the tickets were snapped up in the first hour after registration opened. Luckily another 50 tickets were made available shortly afterwards, resulting in a total number of attendees of around 200.

I was expecting the barcamp to be a mostly German-dominated event but was proven wrong - the introduction round was done in English and there were indeed quite a few people travelling from abroad to attend. Consequently most sessions were done in English, with the odd exception.

Sessions

The quality of sessions (overview) was quite high - a mix of code and app marketing related talks, except for two pitch sessions. The first talk I went to see was "Music creation for Android" by Alex Shaw, who demonstrated his synthesiser built using the NDK (native development kit). Compared to the iPhone, interesting Android music apps are still hard to find, so it was good to see someone working on that.

Next one up was "OpenStreetMap on Android" which gave an overview of the different libraries and apps using OSM data. For example there is osmdroid which aims to provide a free replacement for Google's MapView class. I was particularly interested in the offline capabilities, a huge advantage over Google's solution which always requires an Internet connection.

The web widget development talk (Jo Ritter) made obvious in what a mess the widget standardisation process is in at the moment – there are at least three different "standards" being worked on, all in different states of completion. It's probably best to wait a while before any sort of merging happens.

Carl Harroch presented RESTProvider, an interesting framework to consume REST services using Android's ContentProvider API.

Afterwards Stefan Alund gave an early look at the DroidPush API (slides), currently being developed at Ericsson.

My talk was next, with the aim to convince attendees of Scala's merits for Android development (slides). I think I managed to confuse some people with a mostly code snippet based presentation, but still had a few interested people asking questions afterwards. It seems that developers are mostly concerned about tool support (i.e. full integration in Eclipse) and performance / memory consumption issues, so it would be good to do some research and profiling of a real Scala Android app (there are hardly any at the moment). The DalvikVM is quite a different beast, so some optimisations in Scala geared towards the JVM might not be applicable there. Regardless I'm still convinced that Scala is a good fit for doing development on Android, so hopefully more people will get interested and start experimenting.

The last session I attended was an open discussion about the state of open source projects on Android, hosted by Friedger Müffke. Google's open source credibility has suffered a bit recently when they sent a "cease and desist" letter to the developer of cyanogen, a popular firmware replacement (it included some closed source apps). Apparently the community is now working on open source replacements for some of these components.

Unfortunately I missed two interesting looking sessions, both gaming related: urban golf and Mister X goes Android, a location-based game which got played and talked a lot about during the barcamp.

Summary

It was great barcamp, very well-run and with lots of interesting attendees and sessions. There definitely is a big interest in Android right now, and a common talking point amongst attendees was the notion that Android will "really take off" in 2010. Let's hope it happens, most people have been waiting for quite a while now :). Here's another list of "hot topics" at droidcamp.

It's been a while since I released spotify-api, a homegrown implementation of a restful API to the Spotify music service. Now that the API is getting to a usable state I decided to build some applications on top of it.

What music do your friends love?

My goal was to automatically create some Spotify playlists with interesting or personalised content. On Last.fm there is a concept of loved tracks, where users can tag songs they like. So, if you share a similar music taste with your friends on last.fm you can easily create a good playlist by taking all the recently loved tracks from your friends and convert them into a Spotify playlist.

What you get is a "socially filtered" playlist, hopefully with some good music friends would recommend to you anyway. Another option would be to take recently loved tracks by your last.fm neighbours, but that relies more on algorithms used by Last.fm than trust (but which could still produce interesting results).

Here's the code for code for generating those playlists: lastfm2spotify_loved_tracks, you'll need a Last.fm API key as well as a premium Spotify account to use it.

What music do people listen to in other cities?

I really like the citysounds.fm mashup, which uses music and data from soundcloud to give you an idea what styles of music get produced in different cities across the world.

Last.fm have recently added some functionality to their API which let you grab music charts by location (New Geo Services: Metro Charts).

The difference here is that citysounds.fm provides you with music produced in different cities, whereas the Last.fm data is music consumed in those cities.

Here are some Spotify playlists produced from this data:

• Berlin
• London
• Paris
• Barcelona
• Rome
• Istanbul
• Oslo
• Montreal
• Auckland

I used the getMetroUniqueTrackChart API method to obtain the tracks, which is meant to return tracks uniquely popular in a city. This kind of works, although there are still some tracks present in most of the playlists (The xx seem to be everywhere!) it generates some unique and local tracks. The Berlin playlist has a good share of techno (Paul Kalkbrenner), Paris is quite heavy on French productions such Air and Vitalic, and Auckland has a lot of Ladyhawke and Fat Freddy's Drop, both hailing from Wellington.

The Rome and Barcelona playlists both have quite a few Italian and Spanish songs in them, in fact the Rome playlist is mostly Italian. Oslo has a lot of Kings of Convenience and Röyksopp, and so on. I'm not sure how Last.fm generate this data and how representative it is for a given city, it probably depends on the number of users in a location.

If you want to have a play yourself, the code to generate these playlists is in the spotify-api repo (lastfm2spotify_metrochart). You might experience occasional timeouts using the API, it's not rock solid yet. Also make sure to check the Spotify service status to verify that all services are working as expected.

A couple of months ago I realised that I got a bit bored with web development and decided to try something new. Lots of interesting stuff seemed to happen in the mobile development sector, mainly driven by the huge success of Apple's iPhone platform. At a London hackspace meetup I finally got to play with one of the first Android phones, a G1 (also known as HTC Dream) and was impressed: it certainly lacked the elegance and polish of an Apple product, but the platform looked promising. I decided to get started on Android mainly for two reasons: firstly, it's an open platform built on open source technology which makes it a lot easier to hack on. This applies to the application distribution channel (Android Market) as well, where applications don't need to be approved before they can be installed. In fact it's even possible to bypass the Market completely and install apps directly from a website. Compare that to Apple's mysterious and not very transparent approval "policies".

Secondly, as the platform is still quite young there's a potential for early adopters to build interesting things as the market is not completely saturated.

What follows is a mini-review, highlighting the good & bad points from my perspective.

The bad stuff

Usability

While Android's inner workings might be well engineered, Google sucks at user experience design. For example, in the Android Market you can browse through a list of featured applications by sliding your finger over the display. The name of the focused application is located under the row of icons, exactly where you're finger is normally placed to scroll through the list, making it very cumbersome to use. Why not just put it above the icons so you can read it while you browse ?

The Android Market, the main place to install new software and therefore quite an important application is pretty crap. Apps are listed in completely useless or arbitrary categories (Health, Lifestyle, Productivity) and don't provide any screen shots. The text-based search interface is rudimentary (no spelling suggestion, only exact queries will match), quite embarrassing for a company like Google which should know how to implement good search functionality.

The most recent version of the SDK (1.6, still in developer beta) addresses at least some of these issues: Android Market Updates in 1.6.

Performance

Android has its own custom JVM implementation called Dalvik, which is heavily optimised for mobile devices (low memory footprint, process isolation). Unfortunately it doesn't support JIT or AOT compilation at the moment (although they have stated "We do plan to include JIT and/or AOT compilation in a future release").

This might not be a problem for simple apps like TODO lists etc. but don't expect anything CPU intensive to run smoothly on this device. I tried to port jsidplay (a Java library which emulates SID, the sound chip used in the old Commodore 64) to Android and quickly gave up after listening to the first results, it was just too slow and produced choppy playback.

Similarly the JavaGB project (open source Java Game Boy emulator) abandoned the Android version because of the missing JIT compilation.

So in the meantime the only way to get reasonable performance out of Android is to go native (C/C++) using the Android NDK, which ultimately means trading ease of development for speed.

Building everything from source

I've already mentioned that the whole platform being released as open source (Apache Software License) was one of the reasons why I picked Android over other systems. So one of the first things I tried after getting my Google dev phone was to replace the pre-installed firmware with a new version entirely compiled from source, following the instructions ("Building For Dream"). It turns out that it's actually very difficult to produce a usable image which is comparable to the one shipped with the device, binary drivers need to be extracted from the phone and the build process itself is very complicated.

No Multitouch

By default Android doesn't support multitouch gestures. Apparently there is support for it in the codebase but it got disabled on Apple's request (legal/patent issues?). There are some unofficial firmwares floating around which re-enable it, but I haven't actually tried them out.

Gdata api integration

While applications shipped with Android phones integrate well with Google's different services (GMail, GCalendar) the situation looks bad from a developer's perspective: the SDK doesn't contain Java APIs to access Google's services programmatically, you have to roll your own.

The good stuff

polyglot programming

Although Dalvik uses its own bytecode format and has no support for JIT yet, it still opens up a lot of possibilities for developing Android applications in languages other than Java. I'm planning to use Scala for my first Android project and a proof of concept looks promising so far: Hello World in Scala on Android.

JRuby project lead Charles Nutter is working on getting JRuby fully supported on Android (project Ruboto) and some people even tinker with Clojure, a modern implementation of Lisp on the JVM (github.com/remvee/clojure).

Not JVM based, but still interesting is the Android Scripting Environment (ASE), a project aiming to make scripting languages available for application development. At the moment it supports Python, Perl, Lua and BeanShell.

api/architecture design

Overall the API is reasonably well designed. One part of it stands out: the Intent/Activity model. The idea behind intents is to make some parts of applications callable from other programs, often to perform specific tasks. The barcode scanner application ZXing is a good example. Say you wanted to integrate some barcode reading functionality in your own application, instead of linking and bundling it with your code you can request the scan from an already installed application as follows:

  
    Intent intent = new Intent("com.google.zxing.client.android.SCAN");
    intent.putExtra("SCAN_MODE", "QR_CODE_MODE");
    startActivityForResult(intent, 0);
  

This will launch the barcode application in scan mode (if not already running), and transfer control back to your handler method after the scan is completed:

  
    public void onActivityResult(int requestCode, int resultCode, Intent intent) {
      if (requestCode == 0) {
          if (resultCode == RESULT_OK) {
              String contents = intent.getStringExtra("SCAN_RESULT");
              String format = intent.getStringExtra("SCAN_RESULT_FORMAT");
              // Handle successful scan
          } else if (resultCode == RESULT_CANCELED) {
              // Handle cancel
          }
      }
    }
  

It's a move away from monolithic, isolated applications, and a big step ahead of the current iPhone programming model. The Open Intents project aims to provide a registry for reusable intents which can be used by other applications.

Good tool support

The SDK ships with a lot of tools, my favourite is the Dalvik Debug Monitor Service (DDMS), see screenshot below.

It lets you monitor pretty much everything happening on a running device (or emulator), CPU load, memory allocation, logging output etc.

Android projects are by default built using ant, but it's very easy to use Rake instead (Andrake, hello world with rake). If you prefer developing in an IDE, there are plugins available for the most common ones (Intellij IDEA, Eclipse, Netbeans).

Community

There is a growing hacker community around Android and Google now tries to encourage developers to contribute to the codebase. CyanogenMod is an example of a custom firmware with an optimised kernel and various other enhancements. I'm hopeful that the open nature of Android itself motivates more developers to build a rich infrastructure of open source projects around the platform, similar to Linux. At the moment most applications in the Android Market are available for free (but not necessarily open source), although that might change as the user base grows.

Conclusion

With more Android phones arriving in the next few months there is a good chance that the platform will become the second player after Apple in the mobile market, but there are obviously quite a few rough edges which need sorting out. The potential is clearly there.

Last month I attended the first music hack day, held at the Guardian offices in London. It was a great event, even though it was very frustrating that I couldn't finish my hack in time (tip: don't use unfamiliar languages/technologies if you just want to get stuff done, doh). One the web site of the event it says:

European music sites are revolutionising the music industry, not least with their eagerness to open up their data with APIs.

The idea of taking a bunch of existing (music) services to create something entirely new is great and only possible because web APIs have become the norm. We just expect services we entrust with our data to give us access to their API. For me it has become an important box to tick before signing up to a new service (does it have an API? is it useful? etc).

The companies involved at music hack day all offer APIs. My favourite moment of the weekend: I asked about a particular feature in Songkick's API in the #musichackday irc channel. A couple of minutes later Phil Cowan (Songkick CTO) stands next to me and asks what data I needed to get out of their API. I explain it to him, he walks off and an hour later the requested geodata was made available in their API!

One notable European music company missing at hack day was Spotify. Since the Swedish company launched their ad-supported streaming service end of last year it has at times been labeled as "game-changer" of the music business or otherwise as "iTunes killer". A slick interface and fast P2P streaming makes accessing remote music on-demand indistinguishable from a locally stored music library.

Spotify is in many ways a very untypical, almost unfashionable application: it sits on the desktop, there is no "social" angle to it, no "friends" to follow, the only feature which comes close are collaborative playlists which can be shared with other listeners. A couple of websites have been created to fill in the gaps, for example Share My Playlists where users can post and discuss these playlists. Because Spotify doesn't offer a web API (there only is libspotify, which is sadly next to useless) users of these sites need to manually copy&paste their playlist into a web form. You want to create a dynamic playlist, say by consuming data from another music service API? Sorry. Impossible.

From a technical perspective Spotify's decisions are understandable: a proprietary desktop client offers more control, especially given that their business model is based on advertising which needs to be displayed to the user. Of course, this didn't stop some members of #hack.se from reverse-engineering the P2P-protocol. So earlier this year despotify was released, an open-source (C-based) library which made it possible to access Spotify's service without the official client. Spotify reacted by blocking free (ad-supported) despotify users. However, the success of despotify encouraged more people to get involved in reverse-engineering - there are now at least three different active projects.

While writing my last blog post (100 records...) I spent most of the time manually searching Spotify URLs for the albums and artists in the playlists, a boring job a program could have done in a matter of seconds.

So I took another look at the existing open-source libraries and found jotify, which is currently the most advanced implementation (it supports the feature I was looking for, creation of playlists). It is written in Java, so it was very easy to use integrate in JRuby. I decided to use sinatra to give it a rest-based API so it can be used from other services. The result is spotify-api, a JSON-speaking API for spotify. Like despotify it needs a premium account to work, but I'm happy to pay for a service without ads and an API.

The API is not complete and it's the first release, but it's all there is until the "official" API arrives (if it does at all, it has been announced/promised for quite a while now...the delay is most likely due to political, not technical reasons).

Update (04/05/2010)

At the last music hack day in Amsterdam Spotify (which were present this time) announced the new playlist API and ran a session to get feedback from potential users. Nice! You can find their current progress on github: playlist-api. So it looks like this hack won't be needed for much longer.

Update (16/12/2010)

Sadly, Spotify still has no API. There is some discussion on the issue tracker.

Update (16/06/2011)

Guess what? Still no API. However I got pointed to a new Spotify API server project (spotify-api-server) which wraps libspotify and has playlist support.

From the Wire #175, September 1998: 100 Records That Set The World On Fire (scans).

"In The Wire 175, we polled our writers to nominate records that should have ignited the world's imagination - but somehow got forgotten along the way."

List obtained from different places (1, 2). Where the album was available on Spotify I linked the album, otherwise the artist. If you need more structured data, check the Google docs spreadsheet. I'm in the process of autogenerating and annotating this list with information from different sources, but it's not quite ready yet.

Enjoy!

• Charles Ives - Symphony No. 4 (1910-16) (Grammophon, 1988)
• Blind Willie Johnson - Dark Was The Night, Cold Was The Ground (Columbia, 1929)
• Bob Graettinger - City Of Glass/This Modern World (Capitol, 1953)
• Louis & Bebe Barron - Forbidden Planet OST (Small Planet, 1956)
• Esquivel And His Orchestra - Other Worlds Other Sounds (RCA, 1958)
• The Blue Men - I Hear A New World (RGM White Label/RPM, 1960)
• Joe Harriott - Abstract (Columbia/Capitol, 1961)
• Son House - The Original Delta Blues (Columbia/Legacy, 1964)
• William S. Burroughs - Call Me Burroughs (ESP, 1965)
• Steve Reich - Early Works: Come Out/It's Gonna Rain (Elektra, 1965)
• Albert Ayler - In Greenwich Village (Impulse!, 1967)
• Bill Dixon Orchestra - Intents And Purposes (RCA, 1967)
• Gottfried Michael Koenig - Terminus II/Funktion Gruen (Deutsche Grammophon, 1967)
• Sun Ra - Strange Strings (Saturn, 1967)
• Blue Cheer - Vincebus Eruptum (Philips, 1968)
• Dr. John the Night Tripper - Gris-Gris (Atco, 1968)
• Pearls Before Swine - Balaklava (ESP, 1968)
• Spontaneous Music Ensemble - Karyobin (Chronoscope, 1968)
• The United States Of America - The United States Of America (CBS, 1968)
• El Camaron De La Isla & Paco De Lucia - Al Verte Las Floras Lloran (Philips, 1969)
• Ram John Holder - Black London Blues (Beacon, 1969)
• Phil Ochs - Rehearsals For Retirement (A&M, 1969)
• Buffy Sainte-Marie - Illuminations (Vanguard, 1969)
• Sonny Sharrock - Black Woman (Vortex, 1969)
• Silver Apples - Contact (Kapp Records, 1969)
• Alexander 'Skip' Spence - Oar (Columbia, 1969)
• Kevin Ayers & The Whole World - Shooting At The Moon (Harvest, 1970)
• Comus - First Utterance (BGO, 1970)
• Michael Gibbs - Michael Gibbs (Deram, 1970)
• Alvin Lucier - I Am Sitting In A Room (Lovely Music, 1970)
• Cluster - Cluster 71 (Philips/Sky, 1971)
• The Last Poets - This is Madness (Douglas Music, 1971)
• The Master Musicians Of Jajouka - Brian Jones Presents The Pipes Of Pan At Jajouka (Rolling Stones, 1971)
• John Cale - Paris 1919 (Reprise, 1972)
• Alice Coltrane - Universal Consciousness (Impulse!, 1972)
• Miles Davis - On The Corner (Columbia, 1972)
• Hugh Hopper - 1984 (CBS/Cuneiform, 1972)
• Modern Lovers - The Original Modern Lovers (Mohawk, 1972)
• Annette Peacock - I'm The One (RCA, 1972)
• Pierre Akendengue - Nandipo (Saravah, 1973)
• Faust - The Faust Tapes (Virgin) 1973
• Herbie Hancock - Sextant (Columbia, 1973)
• Larry Young - Lawrence Of Newark (Perception, 1973)
• Betty Davis - They Say I'm Different (Vinyl Experience, 1974)
• Furry Lewis - In His Prime 1927-1928 (A&M, 1975)
• Pere Ubu - 30 Seconds Over Tokyo (Hearthan, 1975)
• Lee Perry - Revolution Dub (Cactus, 1975)
• Lou Reed - Metal Machine Music (RCA, 1975)
• The Electric Eels - Cyclotraon/Agitated 7" (Rough Trade, 1977)
• Captain Beefheart & The Magic Band - Bat Chain Puller (Unreleased, 1976)
• Henry Cow - Concerts (Recommended, 1976)
• The Residents - Satisfaction (Ralph, 1976)
• Johnny 'Guitar' Watson - Ain't That A Bitch (DJM, 1976)
• Ornette Coleman - Dancing In Your Head (A&M, 1977)
• Glenn Gould - The Solitude Trilogy (CBS, 1967-77)
• Al Green - The Belle Album (Motown, 1977)
• Ron 'Pate's Debonairs featuring Rev Fred Lane - Raudeluna's 'Pataphysical Revue (Say Day Bew, 1977)
• Iggy Pop & James Williamson - Kill City (Bomp, 1977)
• Tim Souster - Swit Drimz (Transatlantic, 1977)
• The Human League - Being Boiled (Fast Product, 1978)
• The Walker Brothers - Nite Flights (GTO Records, 1978)
• Chrome - Half Machine Lip Moves (Siren/Beggars Banquet, 1979)
• Lol Coxhill - Digswell Duets (Random Radar, 1979)
• Robert Fripp - Exposure (EG/Polydor, 1979)
• Nurse With Wound - Chance Meeting On A Dissecting Table Of A Sewing Machine And An Umbrella (United Dairies, 1979)
• Family Fodder - Monkey Banana Kitchen (Fresh, 1980)
• Fire Engines - Get Up And Use Me (Pop: Aural, 1980)
• La Nimba De N'Zerekore - Gon Bia Bia (Syliphone, 1980)
• Nancy Sesay & The Melodaires - C'est Fab 7" (It's War Boys, 1980)
• Monoton - Monotonprodukt 07 (Monotonprodukt, 1981)
• Derek Bailey - Aida (Incus/Dexter's Cigar, 1982)
• Bad Brains - Bad Brains (ROIR, 1982)
• Kip Hanrahan - Desire Develops An Edge (American Clave, 1983)
• Youssou N'Dour - Djamil (Senegalese Cassette) 1983
• Mark Stewart & The Maffia - Learning To Cope With Cowardice (On-U Sound, 1983)
• Jonathan Harvey - Bhakti (NMC, 1984)
• The Homosexuals - The Homosexuals (Recommended, 1984)
• Cheba Fadela & Cheb Sahraoui - N'Sel Fik (Factory/Mango, 1985)
• Christian Marclay - Record Without A Cover (Recycled, 1985)
• Arthur Russell - World of Echo (Upside/Rough Trade, 1986)
• Fingers Inc - Another Side (Trax, 1988)
• Conlon Nancarrow - Studies For Player Piano (Wergo, 1988)
• Dead C - Trapdoor Fucking Exit (Siltbreeze, 1990)
• Royal Trux - Twin Infinitives (Drag City, 1990)
• Fushitsusha - DBL Live (PSF, 1991)
• Public Enemy - Apocalypse 91 ... The Enemy Strikes Black (Def Jam, 1991)
• Galina Ustvolskaya - No. 1 (Hat Art, 1991)
• Steven Jesse Bernstein - Prison (Sub Pop, 1992)
• Bally Sagoo - Wham Bam 2: The Second Massacre (Oriental Star Agencies, 1992)
• Luke Skywalker - I Wanna Rock 12" (Luke Records, 1992)
• Bernhard Gunter - Un Peu De Neige Salie (Selektion/Table of the Elements, 1993)
• Ken Ishii - Garden On The Palm (R&S, 1993)
• Jean C Roche - A Nocturne Of Nightingales (Sittele, 1993)
• Jeff Mills - X-103 Atlantis (Axis/Tresor, 1993)
• Paul Dolden - L'Ivresse De La Vitesse (Empreintes Digitales, 1994)
• 4 Hero - Parallele Universe (Reinforced, 1994)
• Joey Beltram - Places (Tresor, 1995)
• Oval - 94 Diskont (Mille Plateaux, 1995)
• Tony Conrad - Four Violins (Table Of The Elements, 1997)
• Cathy Lane - Nesting Stones (Unknown Public, 1998)

The 30 others that did not make it into the original article:

• King Sunny Ade - Ju-Ju Music (Island, 1982)
• Arcane Device - Engines Of Myth (1988)
• The Art Ensemble Of Chicago - Fanfare For The Warriors (Atlantic, 1974)
• Baby Ford - Ford Trax (Rhythm King, 1988)
• Ray Charles - The Spirit Of Christmas (CBS, 1985)
• Vinicius Cantuaria - Sol Na Cara (Gramavision, 1997)
• Charles Brown Superstar - Days Of Our Drive/Sweet Piece Of Ass (Win, 1995)
• Lowell Davidson - Lowell Davidson Trio (ESP Disk, 1965)
• Dead Can Dance - The Serpent's Egg (4AD CD, 1988)
• Eric Dolphy - Out To Lunch (Blue Note, 1964)
• Bob Dylan - Live At The Manchester Free Trade Hall (Columbia, 1966)
• The 49 Americans - We Know Nonsense (Quartz LP, 1982)
• Jimmy Giuffre - Free Fall (Columbia, 1962)
• Golden Gate Jubilee Quartet - Golden Gate Gospel Train (Bluebird, 1937)
• Hot Gossip - The Hollywood Jungle (DinDisc unreleased, 1981)
• Howlin Wolf - The Howlin' Wolf Album (Chess/Cadet, 1968)
• Lee Konitz - Motion (Verve, 1961)
• Labradford - A Stable Reference (Kranky/Flying Nun, 1995)
• Last Exit - Last Exit (Enemy, 1986)
• JB Lenoir - Alabama Blues (Bellaphon CD, 1965)
• Derrick May - Debut LP (Transmat unreleased)
• Rachel's - Music For Egon Schiele (Quarterstick CD, 1996)
• The Soul Stirrers (featuring Sam Cooke) - Jesus Gave Me Water (1951)
• The Staple Singers - Uncloudy Day (Vee Jay Records, 1959)
• Cecil Taylor - Looking Ahead! (Original Jazz Classics, 1958)
• Willie Mae Thornton - Hound Dog (Vogue, 1953)
• Lennie Tristano - I Can't Get Started With You (Keystone, 1946)
• Various Artists - Ice Cream And Suckers (Mercury, 1963)
• David S Ware - Third Ear Recitation (DIW, 1993)
• Marva Whitney - It's My Thing (King, 1969)

The 12th of June 2009 marked the end for analog television in the Unites States. On that day all American TV stations switched their analog signals off. The Federal Communications Commission (FCC) had decided that analog's time was over in what could be called a forced upgrade. They are now able to sell the unused frequencies in the analog spectrum, most likely for digital transmissions protocols like mobile broadband.

Can this switch be called historic? I don't know, but some more relicts of the analog area are going to disappear. After the test card (when TV broadcasting moved to a 24/7 schedule) the next thing from the 20th century to go will be the white screen of static.

The sky above the port was the color of television, tuned to a dead channel.

This famous opening paragraph from the science-fiction novel "Neuromancer" by William Gibson sounds very out of date now. Glitchy and chopped up playback (as seen with damaged DVDs) have replaced the grainy static snow.

The analog to digital switchover hasn't really made news - most people I asked were not aware that it happened. I read about it on rhizome.org, a web site dedicated to new media art which commented on the use of YouTube to document the switchover:

There is something curiously surreal about these grainy videos of television screens switching to static, taped in people's homes on cell phones and digital cameras, only to be posted on YouTube moments later. The novelty of their circulation itself - a historic transition from analog to digital television captured on digital video and then transmitted online - speaks to the media environment we inhabit with accidental precision.

(Ceci Moss, Analog to Digital)

It's interesting to see how the last moments of analog TV were featured by the stations: some invited engineers from the "olden days", some gave background information, others just put a banner and cut in the middle of an advertising block.

I've created a little mini-app with 12 YouTube videos showing the last minute of analog TV in the US: endofanalog. You can play clips individually or play them all at once. They're synchronised so they should all reach the moment of the switchover at the same time, but network delays make this very unreliable. Oh, and a word of warning, playing all clips at the same time looks great but will slow down your system: we've come a long way from analog to digital but it's still impossible to have a dozen clips play at the same time without your computer falling over.