Imagine that you’ve got this cool project where you’re building a multimedia scrollstory for the web. There’s some great video and audio content, the designer has made some really nice looking visuals, all you’ve got to do now is to bring it all together in a fantastic interactive extravaganza.

And then suddenly there’s a voice somewhere in the back of your head: wasn’t there something with audio and video on mobile webbrowsers?

The touch revolution has given us wonderful new possibilities, but also a lot of headaches on how to translate the desktop web experience to mobile and tablet. Audio and video might be one of the biggest hurdles.

So, what are those problems? And more importantly: how can we fix them?

Problems

Let’s start with the problems. Basically, there are three major problems with media on mobile.

1. The biggie: on both iOS and Android (this includes both mobile and tablet devices) it is not possible to ‘autoplay’ a media file. What this means, in practice, is that you always need a user action to initiate an audio or video play, such as a button click. For ‘narrative video’, such as a YouTube embed, this is usually not a big problem, but it can be very frustrating when you’re trying to use video as a background layer (example).
2. Only on iOS mobile (hence: iPhone and iPod touch) video will always start in a separate player, outside of the browser. This means that you can’t layer text or images over the video. It also means that the user gets taken out of the experience in the browser. Like the autoplay problem, this is mostly acceptable for ‘narrative video’, but awful for background video. Note that this problem does not apply to audio.
3. On iOS (both tablet and mobile) you can only play one media file at the same time. When you have two videos on a page, you have one of them playing and you hit ‘play’ on the other, the first one will pause.

There are some minor problems too. I won’t delve too deep into these points, it’s just so that you know it when you encounter them:

• On both iOS and Android video files play in another process than the browser. This means that there might be some quirky edge-cases, such as having a different user agent when playing a media file (something like ‘Quicktime’ instead of ‘Safari’) and files that won’t work if they’re located in a folder that is behind basic authentication.
• Support for subtitles (using the WebVTT standard) can be buggy. For example, styling the titles is limited on iOS and might lead to some weird bugs as well, such as videos not playing, even if their paused property says true. Also: the ‘subtitles’ button in the video interface sometimes randomly decides to turn off.

Solutions

I could spend this whole article about why these restrictions exist on mobile, and whether they should be there or not.

I won’t. Let’s just say that there are some good reasons why you don’t want to have autoplay video on mobile (think sleazy advertisements) and some good reasons why you do (that wonderful multimedia project that you want to build). Instead of pointing fingers, let’s look at possible solutions.

First of all: there doesn’t exist one simple and easy solution that will fix all of these problems. There are some stable workarounds and others that are more experimental.

• One solution to the autoplay problem is to only use one single video element and change the source instead of creating a new element. Then, you can ‘save the first click’ and use the event handler to switch between sources. You could even fade the video out, then change the source, then fade in again to have a slightly less clunky transition between two videos. I’ve created an example here (try it on your iPad!). This is a decent solution, but there are a few disadvantages. One of them is that switching still takes quite some time, i’ve timed it to around five seconds or so. Especially if you want to have smooth transitions (such as in a scroll story), this might be unacceptable. Also, this method doesn’t fade audio in and out. Remember that this solution won’t fix problem #2 and #3.
• Another autoplay solution, especially applicable to audio, is using audio sprites. Instead of having different audio files you stitch them all together in one big file and move the current time. Another decent solution, but this takes a lot more work and overhead, and is probably unusable for video because of the file size. Also, you can still only play one sound at the same time.
• Yet another autoplay solution, only applicable for audio, is to use the Web Audio API. Here’s an article about how to use it on iOS. Unfortunately the Web Audio API is not supported on any version of Internet Explorer and the stock Android Browser (Chrome for Android does support it), so you need to use other solutions for those browsers. You could try using something like Howler.js to make it work on all platforms.
• Since iOS 10 there are some new possibilities to play videos inline and with autoplay. Unfortunately, there are some new restrictions as well. Specifically, it’s not possible to play more than one video on the page at the same time.
• A more experimental approach is to use a Javascript video decoder and play the video in a <canvas> tag. This actually works and, in theory, solves all three problems. There are a few big drawbacks though. The audio track doesn’t play at all, the browser needs to download the complete file first and, because it’s such an experimental technology, you might run into problems that are difficult to debug. Also, because the decoding is not hardware accelerated (native video is) this can be too demanding on older devices and result in choppy playback. There’s a decoder for MPEG1 (larger files, but better performance), and one for MPEG4 (smaller files, but buggier with slower performance).
• One method pointed out to me by Daan Louter from the Guardian is crossfading a couple of JPEG files in and out. This works especially well if you don’t have a lot of movement in your ‘video’.
• Don’t rule out the honourable Animated GIF. It tackles all three problems, but there are some obvious downsides. No audio again, and GIF is a very limited format. You can’t have more than 256 colours, and longer files of higher resolution tend to get really huge (think dozens of megabytes) pretty quickly. But for some usecases, that might not be a problem.

Some recent libraries have tried other hacks, especially for iPhone. All these libraries are experimental, so make sure you thoroughly test them out before using  in production.

• The library iphone-inline-video solves the problem by not actually ‘playing’ the video, but by updating the slider 25 times per second. You can see a simplified example of this technique here (iPhone only).
• Canvasvideo.js copies the content to a <canvas> tag and also enables audio.
• I’ve created a bare-bones library that uses the same trick as iphone-inline-video to create an animated gif effect on video files. It’s called gifvideo.

Rethinking the problem

Another solution is to completely rethink the app. Video and audio might be problematic, but why use them at all? Instead of using video to give the user a nice experience, maybe you can get the same thing while using CSS animations or the <canvas> tag.

The elephant in the room is of course going native. All of these problems don’t exist when you’re building an Objective-C / Swift (iOS) or Java (Android) app. But going native brings it own problems, including the need to develop three times if you want to reach the widest audience (iOS, Android and web for desktop), the loss of easy sharing using URLs, the hassle of going through the App / Play store and the fact that mobile developers don’t come cheap.

When all else fails, there’s always the obvious and easiest solution: use click-to-play video when possible, and still images when there’s no other option. But it’s a bit sad that mobile users will get a watered-down version of the desktop experience.

In the end it all comes down to what you really think is important and which tradeoffs you’re willing to take.

Do you have any other solutions or workarounds? Add them in the comments.

Add a comment

3 comments

• Louis-Jean 2015-08-11 on 07:23

  On iOS, there is a way to achieve a form of “spriting” for video tracks: using HTTP Live Streaming. You provide an simple m3u8 file that references all your videos, and the video player won’t skip or block when swapping videos, the different clips being automatically preloaded.

• Mobile Web Weekly No.70 | ENUE 2015-11-28 on 18:15

  […] Tackling video and audio on the mobile web […]

• Frank 2016-01-13 on 11:52

  Great article, helped to have the facts collected like this.

  In my case the videos are only decoration, so showing an image instead on mobile works well.
  If the videos would have had important content, I just would have enabled controls so the user can watch it if he wants to.