Why is there Flash in the HTML5 version of Cut the Rope?

37
JAN 11 12

image

Update: A build that fixes the Flash-related loading problems (see below) in Firefox has gone live. That build also fixed the loading issues with the game on iOS devices and added some (very) basic support for touch events (both iOS and IE10 / Windows 8). Play it at www.cuttherope.ie.

Monday was a big day Pixel Lab. We had the incredible opportunity of developing the HTML5 version of Cut the Rope for ZeptoLab (the brainiac creators of the game) and Monday was the launch. Overall, the feedback has been incredibly positive. It’s been cool to see people get excited about what you can do with standards-based code in a modern browser.

A handful of people (CNET, here)  have noticed that the game appears to require Flash. That’s partially true. The more accurate statement would be: the game will fallback to Flash for audio in some browsers. We’re a little more aggressive about this than we meant to be (more below). More importantly, the Flash decision came about because we were trying to make the game more fun for as many people as possible. Here’s the “behind the scenes” on how we ended up with Flash in an HTML5 game (and a little about HTML5 audio along the way).

HTML5 Audio

First a little bit about sound in HTML5. Right now, there is only one way standards-based way of getting sound into the browser: the <audio> tag. It’s the sound-only equivalent of the video tag. It is, of course, a great addition to the spec, and for most audio—like podcasts and music—it works great in just about every browser.

Sound effects, on the other hand, are a little different. As you can imagine, short, repetitive, time-critical sounds like you find in a game can tax your soundcard in a different way and, frankly, not every browser does a good job in that situation. In fact, the requirements are different enough that there are new standards being proposed to handle this more complex kind of audio. Those standards are still emerging. So, while the future is promising, the present leaves us with a single option: we’re stuck with the <audio> tag for any kind of HTML5 audio be it podcasts or sound effects.

HTML5 Standards in Cut the Rope

Our development goal for the Cut the Rope was to be 100% HTML5 and everyone (us, Microsoft, ZeptoLab) was really committed to this from day one. So, naturally, we built an sound library for the game based on the single HTML5 solution we had available to us: that audio tag.

Unfortunately, the sound was quirky from the beginning. We had great luck in IE (that’s true, not a plug), but as we cycled through different implementation strategies we found quirks in just about every other browser. We tried: recycling audio elements, instantiating audio elements on each play, using audio “sprites” (where we combine sounds into fewer audio files and use offsets to play the sound effects) and using different encoding strategies but nothing reliably fixed the issues.

I don’t think this is news. If you look around, you’ll find numerous discussions about HTML5 games and audio issues (here’s one, here’s another). We’ve had similar issues with other HTML5 apps and my impression is that a lot of people are looking to these forthcoming APIs for a long term solution.

The Conflict

Here’s where the story gets interesting. As we got closer to launch, we realized that the overall experience was really impaired by the unreliable audio. The common solution would be to rely on something else (Flash, Silverlight, etc.) as an audio fallback. Just about every sound library on the web does this, but that conflicted with our goal for a purely standards-based game.

To make matters worse, our fallback criteria were complex. We were up against browser quirks and bugs, not just feature support. In other words, even if a browser supported HTML5 audio we weren’t guaranteed that it would reliably handle the complex sound requirements of the game.

The easy thing would have been to do nothing. The most common side effect would be that audio stops working right around the beginning of the second level. Frankly, Internet Explorer (only coincidentally the sponsor for the project) was the only reliable browser so doing nothing could have been seen as a win for them. To their credit, they didn’t want to do that.

Everyone agreed that we wanted the game to be great in every browser. Microsoft and ZeptoLab were resolute on this point. We were too. Here’s where I think we made the right decision: after days of attempting to get our library working in all browsers, we decided to enable Flash audio for browsers where we had seen audio issues. The only browser where we had seen consistent and low-latency audio in every case was IE. That meant making Flash a fallback option for every other browser.

This may seem a little aggressive, but it had three desirable implications: First, it meant everybody would get audio (the thing that mattered most). Second, it let us stay 100% HTML5 in IE which was, understandably, an important goal for Microsoft. Third, it drastically reduced our testing surface for audio (which we knew to be flaky and quirky and so require a lot of testing).

With that decision, we also took a dependency on the very capable SoundManager2 for our sound effects, relying on their more thoroughly tested Flash implementation. (Not sure if they know that…thanks guys!!)

Loading Issues

We wanted the logic to work like this: if you’re in IE use HTML5 audio, otherwise use Flash… unless Flash isn’t available and then go back to HTML5. This seemed like it would give the most people the best experience. We almost got that right, but after launching we realized that SoundManager2 has a convenient feature that will aggressively force it to use Flash in browsers that don’t have <audio> support for .mp3. It’s a cool feature and perfect for podcasters or musicians who rely on .mp3 (and probably aren’t aware that it’s not always supported). It’s not great for us where want to fallback to HTML5 if Flash isn’t available. We didn’t intend to use that feature, but it’s enabled by default. We accidentally left it on.

The result: if a user comes to the site in an HTML5 browser that doesn’t support .mp3 (mostly Firefox which only supports .ogg) then the library will force Flash audio. So, in Firefox we inadvertently ended up with a hard dependency on Flash. This, understandably, was frustrating to Firefox users. The good news is that we’ve fixed that check and will deploy a build with the fix in (hopefully) the next 48 hours.

Try it Yourself

If you’re curious about the audio bugs, you can force the game to use HTML5 audio (no flash) with this link: http://www.cuttherope.ie/?html5audio=true

The bugs are quirky and not everyone will experience them. In fact, at multiple points during development we thought we had solved the audio bugs only to be greeted by another failure hours or days later. You might need to play a couple of levels before the issues show up… or they may not show up at all.

To the credit of SoundManager2, some of the audio issues seemed to get better after we moved to their library (even in HTML5), but we had a beta team of 200-300 people testing the game and the failures were common enough that we still felt it was important to use the fallback. In fact, we would have had to explicitly disable Flash in SoundManager2 and we thought that seemed purely ideological or could even appear nefarious. The truth is that the Flash decision just the opposite: we wanted the game to be great for as many people as possible, regardless of which browser they were using.

37 Comments

João S. / JAN 11 2012

In this case, the objective was clearly to build this game in HTML5. Now that you’ve acquired plenty of experience with HTML5*, if you had to start another game from scratch and had limited resources and a tight deadline, would you build it in HTML5 or Flash ?
Thanks in advance!

* I assume you’re also experienced with Flash for building games

Joseph Labrecque / JAN 11 2012

The truth is that general users probably have no idea (and don’t care) – they just want to play the game and have fun. You’ve enabled that for them and that is what matters most.

Robert O'Callahan / JAN 11 2012

It would be really useful to us Firefox developers if you could tell us about the Firefox bugs you encountered, with bugs filed on http://bugzilla.mozilla.org if possible :-). Thanks!

João S. / JAN 12 2012

In my last question I was assuming you also have experience in Flash/Flex. In case you haven’t, please indicate so, since I can only expect a fair and honest answer if you’ve had enough experience in both sides of the fence (not that there is a fence, but unfortunately people like to take sides :| ).

Scott Barnes (@MossyBlog) / JAN 12 2012

Nice work guys, you did a great job here and carefactor on plugins vs html, you executed and solved and …SHIPPED..

Kudos
:)

Robby Ingebretsen / JAN 12 2012

@João First, sorry that your name is wrong above. Looks like I got cheap with the international characters in that font.

I haven’t done a lot of Flash game development, so it’s hard to say. I can say that Canvas seems to be incredibly consistent in every browser. I’ve rarely (if ever) hit a browser-specific canvas issue. Media is tough and the UI around the game may have the normal browser issues. Also, we had to deal browser-specific performance issues (mac browsers, for example, really don’t like floating point in drawImage() and weird stuff like that).

@Joseph Exactly. I think it’s the devs who care–we’re all trying to figure out what you can do with the platform.

@Robert Let me try to reset to some older builds and see if I recreate some of the issues. SM2 especially seemed to help in Firefox.

@Scott. Thanks! And for noticing that we cared :)

Judah / JAN 12 2012

I’m glad you got it to work.

I’m a Flash and HTML designer developer and another approach I’m hearing more and more is development teams creating the project first in Flash and requiring that Flash be used in any device or browser that supports it and then creating an HTML5 version to fallback on for devices that don’t support Flash (which is one browser and device – the Safari browser in iPhone and iPad).

The reason to fallback to HTML5 instead of fallback to Flash is because HTML5 is not feature complete (although we all wish it were – even Flash developers like to have the option?).

The other main reason for me at least is that with Flash and ActionScript is that I don’t have to deal with cross browser issues like you mentioned here.

Finally the second part of the first reason is that you can then take your Flash version and export and target other platforms with it with minimal changes (yes there are some minor changes between targets).

Flash Pro, Flash Builder and the free and open source ActionScript compiler and free open source Flex SDK can all be used to export your project to:
• Desktop applications as an EXE or DMG (no runtime is required)
• Mobile devices including iPhones and iPads as iOS .ipa, Android phones and tablest as an .APK and Blackberry Playbook as a .BAR. ALL of these are native apps (does *NOT* require any runtime – yes, in the past a runtime was required)
• Smart TVs including Samsung, LG and Google TV compatible devices
• and again as you know any browser that has the Flash Player installed

If browser vendors want to get rid of these cross browser issues what they really need is to join together to make a single open source plugin for them. Otherwise, to me, it’s like building a house on sand. Each new device or browser / browser version or feature or feature set spec are implemented by separate companies with separate goals.

I really like this line that I hope I’m paraphrasing correctly(?), “The truth is that [the decision to use Flash was because] we wanted the game to be great for as many people as possible, regardless of which browser they were using.”

Really, we are all creators trying to get our creations out there :). Let’s remember that.

Judah / JAN 12 2012

BTW Looks really nice and smooth! :)
Mac – FF8

Robby Ingebretsen / JAN 12 2012

@Judah. Good paraphrasing. We’re in a funny time with all of this. No doubt that there is some tax to developing in HTML5 vs.a proprietary framework like Flash or Silverlight. That’s the cost of an open standard I guess. The good news is that, for the most part, browser issues seem to be much better than they used to be. The tax is smaller than you might expect. Audio was the biggest issue, followed by performance issues (although, oddly, those seemed to be more closely tied to the OS than to the browser).

I wouldn’t blame anybody from choosing Flash for a browser game right now, but I think HTML5 is a great choice too. I wouldn’t have said that even 2 years ago. The fact that it’s a standard also means that it’s likely here for a while so a good bet in terms of developing your skill set.

Vorbis / JAN 12 2012

YOU GUYS SHOULD NOT USE MP3 ONLY. What kind of retarded decision is that to only offer MP3 audio. Use your brains man, Vorbis audio is whats supported in Firefox, Chrome and Opera.

http://en.wikipedia.org/wiki/Vorbis

Stop living in the past, patented audio and video codecs harm the Internet.

Leo / JAN 12 2012

I’ve tried the game on the iPad and seems like its not supported. The worst part is the message that states I have an “old browser”…this is not exactly true.

I’m super glad you took money from Microsoft for making the game, but next time Microsoft finds an excuse to promote HTML5, they should make sure it runs on mobile and tablets, otherwise use goddamn Flash and change the error message to: Sorry, you’ve an iPad. Get out!

Robby Ingebretsen / JAN 12 2012

Why did everyone get so grumpy?

@Vorbis I don’t think you read the post carefully. I didn’t say we don’t support ogg. We do. The .ogg sound files have been their from the beginning. We accidentally kept a default setting in the sound library that forced flash in non-.mp3 browsers. We’ve fixed that, though.

@Leo we also identified the issue keeping iPad from working. It turns out that the iOS version of safari only lets you create a single audio element. Since we create audio to get the browser to download the sound files we need during the loading process and iOS only supports one audio element, loading fails on an iPad. We’ve also got a fix coming for that. If you hang out for a while, eventually the loader will timeout and you can play the game even with the current version.

Scott Schiller / JAN 12 2012

As the guy who makes SoundManager 2, it makes me glad to know people find it useful and ideally, helpful in getting audio to “just work.” :) I’ve been working on the JS/Flash API in one form or another since 2002, so it’s been around for a while.

As you’ve found, HTML5 audio is still pretty rough around the edges when it comes to consistency. We’ll get there eventually, but bugs are still being found and fixed as you mentioned.

I should be more clear about the default behaviour, but yes – soundManager.preferFlash is true by default, and that means that even if HTML5 support exists for MP3/MP4 formats, Flash will be used for playback of those formats given the current (relative) instability of HTML5 audio across popular browsers. e.g., recent Chrome releases randomly stop playback and require a browser re-launch, and/or clip playback of short sounds etc.

Regarding “flash-free” targeting across multiple devices – the current SM2 release includes support for multiple URLs (or, “alternative formats”) on a single sound, so you can specify, say, an OGG and an MP3 file – and whichever is supported first, will be used.

Thus, to get the 100% flash-free experience on Firefox, Opera and other FOSS-ish-type browsers, make MP3 “not required” for start-up (the library assumes most people will want MP3 playback, meaning use flash if needed…)
soundManager.audioFormats.mp3.required = false;

And then after soundManager.onready() you can do things like this:
soundManager.createSound({
id: ‘foo’,
url: ['/path/to/some.ogg', '/path/to/some.mp3']
});

Thus, if Safari (or IE 9) doesn’t do OGG, it’ll use the MP3 – and if others see the OGG, they’ll use that first and so on.

I will make a demo that shows this kind of stuff for the next release, so more people will know about some of the switching techniques possible with this thing. :)

I would think that Safari would do a reasonable job of MP3 playback sans-flash on OS X (and with relatively low latency?), but perhaps your beta tests showed otherwise. In any event, you probably uncovered a few bugs along the way and if bugs haven’t already been filed, they’d probably be appreciated (see RoC’s comment above, etc.)

Finally, http://areweplayingyet.org/ is tangentially-related and might be of interest.

Paul Milham / JAN 12 2012

I have plugins blocked by default in Chrome. So the game just stays at loading 0% indefinitely. Tripped me up for a while until I learned what was happening.

Pete Blois / JAN 12 2012

Great job on the game guys! Agree that sucks for game sound effects but Web Audio could be pretty cool in this with positional sounds and pitch distortion.

Interesting call on using the SM2 fallback for broader platform support- many platform demos take the ‘this is how it should work’ approach and try to shame the lagging browsers into fixing their support. I suppose a good blog post covering the issues can be more productive :)

Robby Ingebretsen / JAN 12 2012

@Scott Thanks for notes (and for the great sound library!). I think that the .mp3 rule is totally sensible as default. The nuance of different audio formats is likely not top of mind for most people dealing with sound on the internet.

Very cool of you to share some tips for getting the settings right for our situation. I think we’ve landed in a good place now and we’ll likely push and updated build in the next 24 hours. It’s in staging/testing right now and seems to work well.

Troubles with HTML5 <audio> | Qtiva / JAN 13 2012

[...] Microsoft funds Pixel Lab to make an HTML5 version of Cut The Rope originally by ZeptoLab . Nerds call it out for using some Flash. Robby Ingebretsen writes really interesting article explaining why . Direct Link to Article — Permalink Troubles with HTML5 is a post from CSS-Tricks Direct Link [...]

John Heston / JAN 13 2012

@Robby Ingebretsen: “The result: if a user comes to the site in an HTML5 browser that doesn’t support .mp3 (mostly Firefox which only supports .ogg)”

Firefox supports .wav files as well. For small audio files (like short sound effects in a game) .wav could be a valid choice.

@Scott Schiller: “I would think that Safari would do a reasonable job of MP3 playback sans-flash on OS X (and with relatively low latency?), but perhaps your beta tests showed otherwise.”

I don’t think it matters. MP3 audio is an impractical choice for HTML5 games due to its licensing problems:

http://www.scirra.com/blog/64/why-you-shouldnt-use-mp3-in-your-html5-games

In general, royalty-bearing formats are all round bad news for the web. I hope Zeptolab and Pixel Lab are lobbying Microsoft for out of the box Ogg Vorbis support in Internet Explorer 10. It doesn’t make much sense at all for Microsoft to exclude it.

David Smith / JAN 13 2012

I like the fact that you opted to use the best tool (or tools in this case) for the job to solve the problems at hand instead of ignoring completely viable options or settling on a damaged user experience.

Hey / JAN 13 2012

Could you mention a single MP3 suit case?

Jozef Mak / JAN 13 2012

It seems there are no problems with ?html5audio=true on Safari 5 (Mac). Is flash audio realy needed in this browser?

Anyway you have done great job, can’t wait for new levels :)

Robby Ingebretsen / JAN 13 2012

@David Thanks!

@Jozef The problems are intermittent. Ironically, Safari on a mac has been one of the trickier browsers for us. Glad it works for you, though!

Eric / JAN 13 2012

LOL one of the bugs in FF is that after you click level 1, the sound goes from your soundcard to your PC speaker.

Links for the Weekend | w00t blog / JAN 13 2012

[...] One of the teams responsible for the HTML5 version wrote a post that unfortunately explained why, so a major global crisis was averted and we can’t have yet another event catfight on the Internet (but don’t worry, there will be another one very soon!). Regardless, read the post that provides good insight into the challenges of HTML5 <audio>. [...]

Mefi / JAN 13 2012

For me this story was interesting because it shows the power and potential of HTML 5 but it also shows that this is the future yet. Soon, a game like Cut the Rope in a browser with 100% pure HTML code will be casual.

Till then, we have awesome guys working on making this future the present.

HTML5 / JAN 14 2012

Played all of Cardboard Box with Firefox 12.0 after the loading issue is fixed and it works fine, no audio issues.

http://nightly.mozilla.org/

I have to wonder if the Flash fallback isn’t insidious, trying to portray other browsers as inferior to IE9 when this clearly isn’t true. IE9 has the worst HTML5 support of all HTML5 browsers.

http://www.html5test.com/results.html

Ahh well, you got your money from MS, good job I guess.

Dennis Lembree / JAN 14 2012

Don’t promote an app/game as HTML5 when it has Flash. Ugh.

Robby Ingebretsen / JAN 15 2012

@HTML5 Hope it’s clear that we weren’t just in this for the paycheck.

@Dennis I don’t think it’s unfair to call this an HTML5 app. Do you really think that’s a mischaracterization? In fact, I think it’s a pretty remarkable statement about the maturity of the standard/industry that Cut the Rope will run in the latest version of every major browser. The audio thing is unfortunate, but it’s a comparatively small part of the game. Frankly, I suspect we could (and will) work around the Flash polyfill with enough time.

Raathigeshan / JAN 19 2012

amazing work… Never saw a html 5 game with this much stability and smoothness.
Keep up the good work.

Some thoughts of the future of Flash | Flash Monkey / JAN 29 2012

[...] audio (for example this from trioptic) whereas HTML5’s audio tag still has some short comings. This article about the recent HTML5 Cut the Rope games explains more. Let’s not forget the HTML5 specification [...]

Steve / FEB 07 2012

There’s a lot of talk lately about troubles with the mp3 format for HTML5 games due to licensing issues (and thus having to pay fees)

Can you clarify how much the fees have been so far that you’ve needed to pay? (when running in IE) or has the team decided to ignore the 800lb elephant in the room?