Making HTML5 audio actually work on mobile
This project on GitHub:
This story starts when we had to provide audio to an HTML5 game we were building.
The game is for mobile browsers, actually a limited and updated range of mobile browsers: Safari for iOS 6+, and the stock browser (i.e. the default browser) for Android 4+. With the well known scaling trick (CSS “transform: scale(…)”) which we built in our mobile app, we got it to run quite nicely in all resolutions, so also on desktop browsers. Hence we would want our audio solution to work also on desktops.
This looks like a peachy start, as the browsers for which support is required are on mature platforms supporting (in theory) a wide range of HTML5 features.
Our audio feature requests are very limited: we want a background track for the game, changing only at level change, and a few sound effects on events. The background audio is not responsive with respect to game events.
Being a 3-match game, events are at a slow pace so effect sounds would be mostly played one at a time, at most two having a partial overlap.
There is no development platform built in the last 20 years that does not support such minimal requirements for audio (and much, much more, see e.g. Java, Flash, Unity…) so we started in a quite confident mood.
We also got further confidence by consulting sites such as http://caniuse.com/audio where it shows the audio element as fully supported since several major versions.
Big, big mistake.
The first approach: keep it simple
Given our limited intellectual means and minimal requirements, we first checked available HTML5 audio libraries in order to include them and so we didn’t have to reinvent the wheel. While the ones tried were mostly working on desktop browsers (meaning Firefox and Webkit family), first test with multi audio on Android mobile weren’t working. Some demos didn’t even load pages correctly, like http://www.createjs.com/#!/SoundJS/demos/testSuite, others worked but were “audio player” oriented, like http://kolber.github.com/audiojs.
Moreover we didn’t want the Flash fallback.
So we had the choice of either looking inside these third party libraries to see what was the problem, or do something ourselves. Now no library was minimalistic in code, and as what we needed seemed so simple, we decided to try to just use the <audio> element and play that. Simple enough.
var audio = new Audio(“my file”);
but it doesn’t really make any difference.
If you build a simple demo page with just two buttons launching play of two distinct <audio> elements, playing the second button after the first will stop the first sound, which will never be played again. The first sound is dead, and there is no way to get it playing again. This is fine if instead you create new Audio instances – but this is only a secondary problem.
One interesting discovery is this: “One of the biggest limitations imposed by mobile Safari is that only a single audio stream can be played at one time.”
We had heard this so many times that we believed it, and were surprised in discovering that it is false for iOS 6: iOS 6 can play multiple audio streams at the same time (just don’t launch them in the same millisecond); its Android mobile, even the latest version , that can’t!
This is the core problem: launching new sounds (even preloaded sounds – but you can’t do actually that on mobile) on touch events severely impacts performance, on both Android 4+ and on iOS6+. All the DOM updates and effects in your game will get impaired.
Reality is actually worse: launching sounds on touch events, with the possibility of concurrence, actually often crashes the browser (and sometimes even the operating system). It seems that browser implementers simply haven’t dealt with the problem of concurrent sounds in the browser.
So the “advantage” of HTML5 delegation, just putting simple <audio> tags on the page and leaving implementers to take care of all the rest, which should bring optimized user experiences, actually doesn’t work.
Do sounds – or better, even a sad, single sound – autoplay on loading a page on mobile?
Mostly they do not.
The happy guys at Apple, always ready for a joke at the expense of web developers, made autoplay behavior different in case you are browsing the web from the browser or browsing from a fixed starting page through the “add to home” trick. N.b. you can’t do that on Android devices.
The three way of launching autoplay audio are:
- Simply having an <audio> element in the page HTML
- Adding an <audio> element to the DOM via jQuery
- Creating audio with new Audio() .
|Can you start audio on page load, without a touch event?||iOS6||Android4|
|Browser + DOM <audio> with attribute AUTOPLAY||no||yes|
|Browser + jQuery <audio> with attribute AUTOPLAY||no||yes|
|Browser + instantiate via new Audio()||no||no|
|Launching the browser as standalone app (after “add to desktop”)||iOS6||Android4|
|Home + DOM <audio> with attribute AUTOPLAY||yes||*|
|Home + jQuery <audio> with attribute AUTOPLAY||yes||*|
|Home + instantiate via new Audio()||yes||*|
* Currently you can’t “add to home” on Android.
Considering that the core of our application is the game play mechanics, we fixed the requirement:
We need game play not to be compromised by poor audio support.
As we want to decently support Android, and sound can be played only one at a time, we had the problem that launching a new effect immediately truncated the previous one, without letting it finish. For our game a slight sound asynchrony feels better than sounds being truncated (and we suspect that this is often true), so we decided to no longer launch sounds on touch events: we created a sound queue, and when some game play consequence required playing a sound, we just added it to the sound queue.
This made sound effects terminate when complete, also on Android, as we started a sound on the completion callback of the previous one.
But we still had performance problems, as launching several sounds even in a queue impacts the overall browser performance, on both iOS and Android. Our game engine relies on requestAnimationFrame (of course), but even having the sound queue on a completely different handler, using a simple setTimeout did not solve the problem.
So we had to revert to the old technique of audio sprites. If you know what a CSS sprite or animation sprite is, an audio sprite is the same thing, just with audio. We load a single audio file, and we can load it on the first game touch event, so even older Androids will load it. We keep the sound queue, only it will just determine playing subsequent parts of the same sound file.
So problems over? No, not yet. On iOS, if the sound has not been played, it is not seekable. Funny eh? In order to make the sound seekable we have to issue the command:
immediately followed by
From there on, the sound is seekable and everything works.
The HTML5 audio API says that for this you should listen events and there check media properties like “readyState” on audio, but these are simply not fired by the target browsers.
Seeking on Android
Another problem that drove us almost crazy is that with the mp3 audio file, seeking from Android devices suddenly started to give erratic results: you tell it to start playing at second say 20.5, and it actually plays 5 seconds earlier – sometimes. Luckily we found this issue which helped us find the fix:
An mp3 file saved at low bitrate (which works perfectly on iOS and desktops) is not seeked correctly on Android devices. Just saving the file at a higher bitrate solved the problem. Yes you get a heavier file, but it’s a hard life anyway.
What we obtained at the end is a simple library that works with iOS6+ and Android4+ using sound sprites and queues, so that sound more or less works and does not dramatically impact mobile browser performance.
Our audio queue engine supports having multiple queues, so what we do is we load an audio sprite for effects and a different one for background music – the latter is loaded on iOS but not on Android. Having just two does not disrupt browser performance.
We have sedimented our hacking in a simple library which you can freely use – it’s actually public on GitHub. If you are building a simple web game for latest Android and iOS it may shorten your development times.
This project on GitHub:
There are browser-specific improvements on audio in html e.g. using web audio API like in Webkitaudiocontext, but of course we couldn’t use them for this solution as they are not standard, and should for the moment be called Only Webkit Audio API and even there maybe
Notice that in our demos on desktop audio pauses if you lose focus ;-).
Still not working
There is a case where everything goes wrong, and it seems an implementation bug (easy excuse ): on iOS, make your app a “add to home” app, if you close the app while sound is playing and quickly reopen it, it freezes all the phone functionalities. But the game still works
References and contacts
We wish to thank “the Marks” from jPlayer, Mark Panaghiston and Mark Boas for comments on an earlier version of this post. Responsibility for any damage, offence, nuclear war or botanophobia caused directly or indirectly by this post is of course entirely theirs.