Making HTML5 audio actually work on mobile
This project on GitHub:
https://github.com/pupunzi/jquery.mb.audio
Demos:
http://pupunzi.github.com/jquery.mb.audio/demo.html
http://pupunzi.github.com/jquery.mb.audio/demo_queue.html
The game: Bust the Dust
This story starts when we had to provide audio to an HTML5 game we were building.
The game is for mobile browsers, actually a limited and updated range of mobile browsers: Safari for iOS 6+, and the stock browser (i.e. the default browser) for Android 4+. With the well known scaling trick (CSS “transform: scale(…)”) which we built in our mobile app, we got it to run quite nicely in all resolutions, so also on desktop browsers. Hence we would want our audio solution to work also on desktops.
This looks like a peachy start, as the browsers for which support is required are on mature platforms supporting (in theory) a wide range of HTML5 features.
Our audio feature requests are very limited: we want a background track for the game, changing only at level change, and a few sound effects on events. The background audio is not responsive with respect to game events.
Being a 3-match game, events are at a slow pace so effect sounds would be mostly played one at a time, at most two having a partial overlap.
There is no development platform built in the last 20 years that does not support such minimal requirements for audio (and much, much more, see e.g. Java, Flash, Unity…) so we started in a quite confident mood.
We also got further confidence by consulting sites such as http://caniuse.com/audio where it shows the audio element as fully supported since several major versions.
Big, big mistake.
The first approach: keep it simple
Given our limited intellectual means and minimal requirements, we first checked available HTML5 audio libraries in order to include them and so we didn’t have to reinvent the wheel. While the ones tried were mostly working on desktop browsers (meaning Firefox and Webkit family), first test with multi audio on Android mobile weren’t working. Some demos didn’t even load pages correctly, like http://www.createjs.com/#!/SoundJS/demos/testSuite, others worked but were “audio player” oriented, like http://kolber.github.com/audiojs.
Moreover we didn’t want the Flash fallback.
So we had the choice of either looking inside these third party libraries to see what was the problem, or do something ourselves. Now no library was minimalistic in code, and as what we needed seemed so simple, we decided to try to just use the <audio> element and play that. Simple enough.
As our application is pure JavaScript, and as backgrounds sounds and effects change during gameplay, we couldn’t just “print” the HTML audio elements in the page, we created a simple function to inject such element in the DOM when needed, and to play, pause, stop the audio. The idea was to create a very simple jQuery component to be added to Pupunzi’s jQuery components library. In case you didn’t want the audio as DOM elements, you could use the instantiate audio objects via JavaScript this way:
var audio = new Audio(“my file”); audio.play();
but it doesn’t really make any difference.
Problems
On desktop, everything was working fine. On iOS (6, on iPhone 4/5), the results were erratic: having multiple audio elements on the page, started from code or by click / touch events interfered heavily with the DOM updates of the game engine. Sound should naturally be played asynchronously by the browser engine while the JavaScript runtime proceeds with game play, but this was not so. And there were the same problems on Android – plus more.
If you build a simple demo page with just two buttons launching play of two distinct <audio> elements, playing the second button after the first will stop the first sound, which will never be played again. The first sound is dead, and there is no way to get it playing again. This is fine if instead you create new Audio instances – but this is only a secondary problem.
One interesting discovery is this: “One of the biggest limitations imposed by mobile Safari is that only a single audio stream can be played at one time.”
We had heard this so many times that we believed it, and were surprised in discovering that it is false for iOS 6: iOS 6 can play multiple audio streams at the same time (just don’t launch them in the same millisecond); its Android mobile, even the latest version , that can’t!
Not only iOS 6+ allows for multiple audio streams, it also allows starting sound from JavaScript, differently from Android where a launching touch event is required (for details see the dedicated section below).
This is what happens also in the case of the available JavaScript libraries we tried: no multiple sounds on Android’s stock browser, not even on the latest version. So ok, we downgraded the play experience for Android players, removing background audio: only effects. Still and even on Samsung latest phone, launching sounds (short, light mp3 files) on touch events severely impacted game performance.
This is the core problem: launching new sounds (even preloaded sounds – but you can’t do actually that on mobile) on touch events severely impacts performance, on both Android 4+ and on iOS6+. All the DOM updates and effects in your game will get impaired.
Reality is actually worse: launching sounds on touch events, with the possibility of concurrence, actually often crashes the browser (and sometimes even the operating system). It seems that browser implementers simply haven’t dealt with the problem of concurrent sounds in the browser.
So the “advantage” of HTML5 delegation, just putting simple <audio> tags on the page and leaving implementers to take care of all the rest, which should bring optimized user experiences, actually doesn’t work.
Autoplay
Do sounds – or better, even a sad, single sound – autoplay on loading a page on mobile?
Mostly they do not.
The happy guys at Apple, always ready for a joke at the expense of web developers, made autoplay behavior different in case you are browsing the web from the browser or browsing from a fixed starting page through the “add to home” trick. N.b. you can’t do that on Android devices.
The three way of launching autoplay audio are:
- Simply having an <audio> element in the page HTML
- Adding an <audio> element to the DOM via jQuery
- Creating audio with new Audio() .
Can you start audio on page load, without a touch event? | iOS6 | Android4 |
Browser + DOM <audio> with attribute AUTOPLAY | no | yes |
Browser + jQuery <audio> with attribute AUTOPLAY | no | yes |
Browser + instantiate via new Audio() | no | no |
Launching the browser as standalone app (after “add to desktop”) | iOS6 | Android4 |
Home + DOM <audio> with attribute AUTOPLAY | yes | * |
Home + jQuery <audio> with attribute AUTOPLAY | yes | * |
Home + instantiate via new Audio() | yes | * |
* Currently you can’t “add to home” on Android.
Sound queue
Considering that the core of our application is the game play mechanics, we fixed the requirement:
We need game play not to be compromised by poor audio support.
As we want to decently support Android, and sound can be played only one at a time, we had the problem that launching a new effect immediately truncated the previous one, without letting it finish. For our game a slight sound asynchrony feels better than sounds being truncated (and we suspect that this is often true), so we decided to no longer launch sounds on touch events: we created a sound queue, and when some game play consequence required playing a sound, we just added it to the sound queue.
This made sound effects terminate when complete, also on Android, as we started a sound on the completion callback of the previous one.
But we still had performance problems, as launching several sounds even in a queue impacts the overall browser performance, on both iOS and Android. Our game engine relies on requestAnimationFrame (of course), but even having the sound queue on a completely different handler, using a simple setTimeout did not solve the problem.
Sound sprites
So we had to revert to the old technique of audio sprites. If you know what a CSS sprite or animation sprite is, an audio sprite is the same thing, just with audio. We load a single audio file, and we can load it on the first game touch event, so even older Androids will load it. We keep the sound queue, only it will just determine playing subsequent parts of the same sound file.
So problems over? No, not yet. On iOS, if the sound has not been played, it is not seekable. Funny eh? In order to make the sound seekable we have to issue the command:
player.play();
immediately followed by
player.pause();
From there on, the sound is seekable and everything works.
The HTML5 audio API says that for this you should listen events and there check media properties like “readyState” on audio, but these are simply not fired by the target browsers.
Seeking on Android
Another problem that drove us almost crazy is that with the mp3 audio file, seeking from Android devices suddenly started to give erratic results: you tell it to start playing at second say 20.5, and it actually plays 5 seconds earlier – sometimes. Luckily we found this issue which helped us find the fix:
http://code.google.com/p/android/issues/detail?id=27424
An mp3 file saved at low bitrate (which works perfectly on iOS and desktops) is not seeked correctly on Android devices. Just saving the file at a higher bitrate solved the problem. Yes you get a heavier file, but it’s a hard life anyway.
Wrapping up
What we obtained at the end is a simple library that works with iOS6+ and Android4+ using sound sprites and queues, so that sound more or less works and does not dramatically impact mobile browser performance.
Our audio queue engine supports having multiple queues, so what we do is we load an audio sprite for effects and a different one for background music – the latter is loaded on iOS but not on Android. Having just two does not disrupt browser performance.
We have sedimented our hacking in a simple library which you can freely use – it’s actually public on GitHub. If you are building a simple web game for latest Android and iOS it may shorten your development times.
This project on GitHub:
https://github.com/pupunzi/jquery.mb.audio
Demos:
http://pupunzi.github.com/jquery.mb.audio/demo.html
http://pupunzi.github.com/jquery.mb.audio/demo_queue.html
The game: Bust the Dust
There are browser-specific improvements on audio in html e.g. using web audio API like in Webkitaudiocontext, but of course we couldn’t use them for this solution as they are not standard, and should for the moment be called Only Webkit Audio API and even there maybe 😀
Notice that in our demos on desktop audio pauses if you lose focus ;-).
Still not working
There is a case where everything goes wrong, and it seems an implementation bug (easy excuse 😀 ): on iOS, make your app a “add to home” app, if you close the app while sound is playing and quickly reopen it, it freezes all the phone functionalities. But the game still works 😉
References and contacts
We wish to thank “the Marks” from jPlayer, Mark Panaghiston and Mark Boas for comments on an earlier version of this post. Responsibility for any damage, offence, nuclear war or botanophobia caused directly or indirectly by this post is of course entirely theirs.
Contact the authors:
Matteo Bicocchi, Pietro Polsinelli
Some references:
http://jplayer.org/
http://stackoverflow.com/questions/1933969/sound-effects-in-javascript-html5
http://www.ibm.com/developerworks/library/wa-ioshtml5/index.html
http://remysharp.com/2010/12/23/audio-sprites/
[…] So I’m using this js library: http://pupunzi.open-lab.com/2013/03/13/making-html5-audio-actually-work-on-mobile […]
[…] is a big pit, but fortunately someone stepped on it.The jumping pit process is here:http://pupunzi.open-lab.com/2013/03/13/making-html5-audio-actually-work-on-mobile/Of course, there are still some achievements:https://github.com/pupunzi/jquery.mb.audioHere are some […]
[…] Overcoming iOS HTML5 audio limitations And Making HTML5 audio actually work on mobile, for some notes on […]
Hi,
This API is not anymore supported, sorry,
Matteo
This is not working for me in iOS 8.2.
Any ideas? (Your demo page will not work.)
Sorry to keep posting replies on this, but hopefully this is useful. The only thing that makes a difference is turning wifi on! I tried to make sure there was no odd connection to Google Play Music app, so I disabled for now its streaming/wifi requirement, but no difference.
Merely turning wifi on will help me. My audio file is very small (48kb total, with 24kbps rate), 8 bit, mono, like 8s long. I’ve tried also to sample it as 32-bit stereo / 128kbps, and no difference. In fact, with wifi, do to the other timing issue on low fidelity recordings, I get much better tracking to my sprite.
I stuck to whole seconds to keep it simple. Just more info, though, hope this helps anyone else tracing similar work! If this is the wrong place to continue this discussion, I can take it to the github, but since I don’t know if this is an issue with the “lib” per se, feels more natural here for now.
It appears it was not an upgrade that fixed, probably just the nice clean restart state. I believe I am facing a “thread” issue here in Chrome. Your tip will possibly help, thanks!
Hi,
unfortunately (or fortunately :-)) the audio API is changing continuously causing discontinuity on the library behavior.
There are few specific conditional on Chrome for Android that probably are not needed anymore You could try checking those points and make some tests.
Bye,
Matteo
It appears an update to Android: 4.1.1 has resolved this issue, at least for your demo code. It should fix my problem as well, but wanted to make this update before I have a chance to try.
I’m attempting to use your library and it is working quite well. The only place I’m having an issue today is on Chrome on Android. Firefox on the same system works just fine.
Your demo pages have the same issue as my attempts, no audio, and the controls seem to behave oddly.
In my case, I’m also using observable buffers with an interval, and trying to play the audio within one of these will stop all of my behaviors.
Chrome: 39.0.2171.93
(was an issue before I updated just now, FYI, I didn’t check version before, still broken after today’s update).
Android: 4.0.4; HTC VLE_U
Blink: 537.6 (@185626)
Javascript: V8 3.29.88.17
very strange; the simple play attribute (attention amateur programmer here) works on computer (haha)
but not on androids (mobile) internet browser BUT on mobile firefox on android it does work….
Thank Matteo – yes the situation is crazy. It’s almost as if the makers of certain mobile operating systems don’t want web audio and so web based games to take off! Oh wait …
Yup. Catch up soon!
Hi Mark!
How are you? 🙂
The autoplay is still a mess on mobile, included iOs7. A touch event is needed anyway, the audio otherwise will not start, even if added to home screen. if you test this:
on a iPhone with iOs7 as home screen you can run the background audio by hitting the “lets start” button and then play the audio effects over it on iOs; on Android the background sound will never start… What to say, the audio on HTML is still unusable for mobile devices.
Hope to see you soon,
M
Great article and one I’m drawing from to create a similar piece for MDN.
I’m finding that autoplay in Safari on iOS7 isn’t working even when added to home screen. Human error or regression?
(http://jsbin.com/rufinihi/)
You can now add to home screen on android devices running Chrome 31+. Limited usefulness, but it’s there nonetheless – https://developer.chrome.com/multidevice/android/installtohomescreen
Thnks for the fix 🙂
This lib is great – exactly what I was about to create (honest!)
I did find an issue where play( ) kept resetting the volume to 10. I fixed it by adding the following to setVolume( )
var soundEl = typeof sound == “string” ? $.mbAudio.sounds[sound] : sound;
if (soundEl) {soundEl.volume = player.vol * 10;}
An alternative would have been to init .vol or .volume to -1 and only set it in play( ) if it stayed that way
Thanks again!
C
Very good article. Thanks.
Also very funny:
“The happy guys at Apple, always ready for a joke at the expense of web developers”
🙂
[…] ) @ Karwar & Dendeli – Video by AnimeshSticky Notes for the Web with HTML5 using JQueryMaking HTML5 audio actually work on mobile body.custom-background { background-image: […]
The SoundJS team has tracked this issue down to the web app entering full screen mode using:
”
A bug was reported to apple (reference #15133492).
Currently we have no work around except not entering full screen mode.
I’m curious to hear more about your experience with SoundJS, as I currently work on it.
In particular, it has been successfully tested with Android Native Browser using the demo you link to. Can you share what did not load correctly for you?
The CDN hosted min file comes in at 30kb, and flash fallback is optional and separately loaded. Does a 30kb footprint create any issues for what you are developing?
Just to confirm, you say you have successfully tested multiple audio tags playing at the same time on iOS 6.1? This would be a nice improvement, so I’m looking forward to testing it out.
Finally, we’ve also encountered the iOS “add to home” problem you describe with web audio, however we cannot reproduce it consistently or with the steps you describe. If you have any further information that would be really helpful.
Thanks
I’ve been struggling with HTML 5 audio in an Android/OUYA game that I have been developing for a couple months now and have tried everything outside of re writing the game in GameClosure, ImpactJS, etc… This looks like it might help me.
Thanks for posting this.
Hi Kris,
Actually the library doesn’t have a resume method for paused sounds. I’ll see to introduce it for the next update.
Bye,
Matteo
HI
Love your library. I went through exactly the same pain as you had and come to the same conclusion. I like your library better than jplayer because it’s much smaller and all I need is just play some sounds in a queue so I don’t need all the heavy lifting that jplayer does. There is only one thing I don’t know how to do. When you pause the audio which is easy enough, how do you resume it form the same paused position?
When I do $.mbAudio.play(‘effectSprite’) it just start form the beginning
[…] cache Issue)Detect mousedown event on canvas elementsDetect mousedown event on canvas elementsMaking HTML5 audio actually work on mobile body.custom-background { background-image: […]
[…] Web, Networking, Sociology, Social MediaCSS menu, 180 best resourcesCSS menu, 180 best resourcesMaking HTML5 audio actually work on mobile /*Is […]
Thanks for publishing these invaluable insights and solution.
I think you saved me a ton of time and saved an otherwise ship wrecked project.
Seriously appreciated!
Hi!
The problem is then I press “play” happens nothing.
I think problem is with my FF. The plugin works on other browser and on FF on other PC.
But I can not understand why it works on your demo site on my FF.
I am going to reinstall my windows software.
Thanks!
Hi,
I’m supposing you are referring to the WordPress component of that plugin.
The first think you can do is looking at the javascript console to see what is the error.
This component on Firefox is using a flash player fall back to play mp3 file as Firefox doesn’t support yet this audio format.
Do you have the latest Flash plug in installed on your computer?
Send me the error you get or give me an URL where I can see what’s going on.
Bye,
Matteo
Hi!
Thank you for your great plugin.
But I have a problem. It works on my IPad but doesn’t work on my PC (Win XP + FireFox).
The demo from http://pupunzi.com/mb.components/mb.miniAudioPlayer/demo/demo.html
works. I tryed copy source code from your demo and it also works on iPad but doesn’t work on PC.
Win XP SP3, FF 21, WP 3.5.1
Thanks!
[…] with CSS 3 and jQueryjQuery Autocomplete widget for CakePHP 2.2jQuery File UploadApple O' ClockMaking HTML5 audio actually work on mobile /*Is […]
Hi,
It seams a well known Safari PC bug (https://groups.google.com/forum/#!msg/jplayer/2t06jGnlGYk/xk1p35aQhuQJ).
Quick Time is needed to play HTML5 media file on Safari.
Bye,
Matteo
Hey! Great project – I am finding your demo page does not work on safari for windows (v 5.1.7)
the error i get in the console is this:
jquery.mb.audio.js:42TypeError: ‘undefined’ is not a function (evaluating ‘myAudio.canPlayType(audioType)’)
any ideas why?
thanks!!
Hi,
First thank you for appreciating this pragmatic library 🙂
For the performance tips on mobile devices you could try using the “queue” method instead of the “play”; The standard usage of this method is asynchronous; next sound is queued and it’ll play once the last in the queue finished; but you can force the next sound to play immediately after the actual playing audio by setting the “hasPriority” parameter to true.
The method to invoke is:
In your case, instead of:
You should write:
I don’t really know if this would solve the performance issue in your case, but you can give it a try.
For the audio preload callback it’s not so easy on mobile devices but I’ll see to find a solution.
Bye,
matteo
This is a great little library. Just what I was looking for.
I couldn’t get a good responstime on the ipad (ios5.0) for a HTML5 piano game I made in Flash first.
No matter what I did. I was using big libraries like createJS and soundManager2 when I started converting. I had allready replaced all that with css, jquery and spritesheets and really anything but canvas, but responstime for audio couldn’t be optimized further within these libraries and was just awfull.
So right when I was ready to isolate the problem and fix it, I came across your library.
No frills and extra’s, cool! It is actually possible to play the piano using a soundsprite on the ipad now. Respons is still a bit long and erratic and I can’t seem to get a clear sound-loaded event, but I’m now convinced the problem is with the i-Pad, not the libraries.
I will continue to use this library, because I believe firmly in performance over convenience, especially when it comes to mobile devices.
If you have any performance tips or tips on how to get a audio-loaded event, they would be quite welcome.
Example:
http://www.snoep.at/labs/html5/piano/
All in all: Well done and thanks for sharing!
[…] Novice To Ninja- Free 151 Page PreviewAjax upload progressBest AJAX Tutorials for FormsMaking HTML5 audio actually work on mobile /*Is […]
Just curious behaviour – on Android device it seems that shorter sounds start playing on click, then after 0.2 -0.4 sec stop and start from the begining. If I hold and release the button it doesn’t happen. Probably there are two click events generated, but can I check if the sound sprite is playing on click? Not the sound file – just the sound sprite.
[…] http://pupunzi.open-lab.com/2013/03/13/making-html5-audio-actually-work-on-mobile/ […]
[…] Making HTML5 audio actually work on mobile […]