Making HTML5 audio actually work on mobile

This project on GitHub:
https://github.com/pupunzi/jquery.mb.audio

Demos:
http://pupunzi.github.com/jquery.mb.audio/demo.html
http://pupunzi.github.com/jquery.mb.audio/demo_queue.html

The game: Bust the Dust

This story starts when we had to provide audio to an HTML5 game we were building.

intro

The game is for mobile browsers, actually a limited and updated range of mobile browsers: Safari for iOS 6+, and the stock browser (i.e. the default browser) for Android 4+. With the well known scaling trick (CSS “transform: scale(…)”) which we built in our mobile app, we got it to run quite nicely in all resolutions, so also on desktop browsers. Hence we would want our audio solution to work also on desktops.

This looks like a peachy start, as the browsers for which support is required are on mature platforms supporting (in theory) a wide range of HTML5 features.

Our audio feature requests are very limited: we want a background track for the game, changing only at level change, and a few sound effects on events. The background audio is not responsive with respect to game events.

Being a 3-match game, events are at a slow pace so effect sounds would be mostly played one at a time, at most two having a partial overlap.

There is no development platform built in the last 20 years that does not support such minimal requirements for audio (and much, much more, see e.g. Java, Flash, Unity…) so we started in a quite confident mood.

We also got further confidence by consulting sites such as http://caniuse.com/audio where it shows the audio element as fully supported since several major versions.

table

Big, big mistake.

The first approach: keep it simple

Given our limited intellectual means and minimal requirements, we first checked available HTML5 audio libraries in order to include them and so we didn’t have to reinvent the wheel. While the ones tried were mostly working on desktop browsers (meaning Firefox and Webkit family), first test with multi audio on Android mobile weren’t working. Some demos didn’t even load pages correctly, like http://www.createjs.com/#!/SoundJS/demos/testSuite, others worked but were “audio player” oriented, like http://kolber.github.com/audiojs.

Moreover we didn’t want the Flash fallback.

So we had the choice of either looking inside these third party libraries to see what was the problem, or do something ourselves. Now no library was minimalistic in code, and as what we needed seemed so simple, we decided to try to just use the <audio> element and play that. Simple enough.

As our application is pure JavaScript, and as backgrounds sounds and effects change during gameplay, we couldn’t just “print” the HTML audio elements in the page, we created a simple function to inject such element in the DOM when needed, and to play, pause, stop the audio. The idea was to create a very simple jQuery component to be added to Pupunzi’s jQuery components library. In case you didn’t want the audio as DOM elements, you could use the instantiate audio objects via JavaScript this way:

but it doesn’t really make any difference.

Problems

On desktop, everything was working fine. On iOS (6, on iPhone 4/5), the results were erratic: having multiple audio elements on the page, started from code or by click / touch events interfered heavily with the DOM updates of the game engine. Sound should naturally be played asynchronously by the browser engine while the JavaScript runtime proceeds with game play, but this was not so. And there were the same problems on Android – plus more.

If you build a simple demo page with just two buttons launching play of two distinct <audio> elements, playing the second button after the first will stop the first sound, which will never be played again. The first sound is dead, and there is no way to get it playing again. This is fine if instead you create new Audio instances – but this is only a secondary problem.

One interesting discovery is this: “One of the biggest limitations imposed by mobile Safari is that only a single audio stream can be played at one time.”

We had heard this so many times that we believed it, and were surprised in discovering that it is false for iOS 6: iOS 6 can play multiple audio streams at the same time (just don’t launch them in the same millisecond); its Android mobile, even the latest version , that can’t!

Not only iOS 6+ allows for multiple audio streams, it also allows starting sound from JavaScript, differently from Android where a launching touch event is required (for details see the dedicated section below).

This is what happens also in the case of the available JavaScript libraries we tried: no multiple sounds on Android’s stock browser, not even on the latest version. So ok, we downgraded the play experience for Android players, removing background audio: only effects. Still and even on Samsung latest phone, launching sounds (short, light mp3 files) on touch events severely impacted game performance.

This is the core problem: launching new sounds (even preloaded sounds – but you can’t do actually that on mobile) on touch events severely impacts performance, on both Android 4+ and on iOS6+. All the DOM updates and effects in your game will get impaired.

Reality is actually worse: launching sounds on touch events, with the possibility of concurrence, actually often crashes the browser (and sometimes even the operating system). It seems that browser implementers simply haven’t dealt with the problem of concurrent sounds in the browser.

So the “advantage” of HTML5 delegation, just putting simple <audio> tags on the page and leaving implementers to take care of all the rest, which should bring optimized user experiences, actually doesn’t work.

Autoplay

Do sounds – or better, even a sad, single sound – autoplay on loading a page on mobile?
Mostly they do not.

The happy guys at Apple, always ready for a joke at the expense of web developers, made autoplay behavior different in case you are browsing the web from the browser or browsing from a fixed starting page through the “add to home” trick. N.b. you can’t do that on Android devices.

The three way of launching autoplay audio are:

  1. Simply having an <audio> element in the page HTML
  2. Adding an <audio> element to the DOM via jQuery
  3. Creating audio with  new Audio() .
Can you start audio on page load, without a touch event? iOS6 Android4
Browser + DOM <audio> with attribute AUTOPLAY no yes
Browser + jQuery <audio> with attribute  AUTOPLAY no yes
Browser + instantiate via new Audio() no no
Launching the browser as standalone app (after “add to desktop”) iOS6 Android4
Home + DOM <audio> with attribute  AUTOPLAY yes *
Home + jQuery <audio> with attribute  AUTOPLAY yes *
Home + instantiate via new Audio() yes *

* Currently you can’t “add to home” on Android.

Sound queue

Considering that the core of our application is the game play mechanics, we fixed the requirement:

We need game play not to be compromised by poor audio support.

As we want to decently support Android, and sound can be played only one at a time, we had the problem that launching a new effect immediately truncated the previous one, without letting it finish. For our game a slight sound asynchrony feels better than sounds being truncated (and we suspect that this is often true), so we decided to no longer launch sounds on touch events: we created a sound queue, and when some game play consequence required playing a sound, we just added it to the sound queue.

This made sound effects terminate when complete, also on Android, as we started a sound on the completion callback of the previous one.

But we still had performance problems, as launching several sounds even in a queue impacts the overall browser performance, on both iOS and Android. Our game engine relies on requestAnimationFrame (of course), but even having the sound queue on a completely different handler, using a simple setTimeout did not solve the problem.

Sound sprites

So we had to revert to the old technique of audio sprites. If you know what a CSS sprite or animation sprite is, an audio sprite is the same thing, just with audio. We load a single audio file, and we can load it on the first game touch event, so even older Androids will load it. We keep the sound queue, only it will just determine playing subsequent parts of the same sound file.

So problems over? No, not yet. On iOS, if the sound has not been played, it is not seekable. Funny eh? In order to make the sound seekable we have to issue the command:

 player.play();

immediately followed by

 player.pause();

From there on, the sound is seekable and everything works.

The HTML5 audio API says that for this you should listen events and there check media properties like “readyState” on audio, but these are simply not fired by the target browsers.

Seeking on Android

Another problem that drove us almost crazy is that with the mp3 audio file, seeking from Android devices suddenly started to give erratic results: you tell it to start playing at second say 20.5, and it actually plays 5 seconds earlier – sometimes. Luckily we found this issue which helped us find the fix:

http://code.google.com/p/android/issues/detail?id=27424

An mp3 file saved at low bitrate (which works perfectly on iOS and desktops) is not seeked correctly on Android devices. Just saving the file at a higher bitrate solved the problem. Yes you get a heavier file, but it’s a hard life anyway.

Wrapping up

What we obtained at the end is a simple library that works with iOS6+ and Android4+ using sound sprites and queues, so that sound more or less works and does not dramatically impact mobile browser performance.

Our audio queue engine supports having multiple queues, so what we do is we load an audio sprite for effects and a different one for background music – the latter is loaded on iOS but not on Android. Having just two does not disrupt browser performance.

We have sedimented our hacking in a simple library which you can freely use – it’s actually public on GitHub. If you are building a simple web game for latest Android and iOS it may shorten your development times.

This project on GitHub:
https://github.com/pupunzi/jquery.mb.audio

Demos:
http://pupunzi.github.com/jquery.mb.audio/demo.html
http://pupunzi.github.com/jquery.mb.audio/demo_queue.html

The game: Bust the Dust

There are browser-specific improvements on audio in html e.g. using web audio API like in Webkitaudiocontext, but of course we couldn’t use them for this solution as they are not standard, and should for the moment be called Only Webkit Audio API and even there maybe :-D

Notice that in our demos on desktop audio pauses if you lose focus ;-).

Still not working

There is a case where everything goes wrong, and it seems an implementation bug (easy excuse :-D ): on iOS, make your app a “add to home” app, if you close the app while sound is playing and quickly reopen it, it freezes all the phone functionalities. But the game still works ;-)

References and contacts

We wish to thank “the Marks” from jPlayer, Mark Panaghiston and Mark Boas for comments on an earlier version of this post. Responsibility for any damage, offence, nuclear war or botanophobia caused directly or indirectly by this post is of course entirely theirs.

Contact the authors:
Matteo BicocchiPietro Polsinelli

Some references:
http://jplayer.org/
http://stackoverflow.com/questions/1933969/sound-effects-in-javascript-html5
http://www.ibm.com/developerworks/library/wa-ioshtml5/index.html
http://remysharp.com/2010/12/23/audio-sprites/