Making music with JavaScript and Gamepads

Reject.js 2013


What to expect

Web Audio API


Gamepad API


<3 *

* both are drafts, examples only work in Chrome for now

Web Audio API

  • low-level access to all things audio
  • create sounds
  • manipulate sounds (filters)
  • timing sounds precisely
  • tons of other cool stuff (spatial audio, audio analysis)

What is sound?

"Sound is a mechanical wave that is an oscillation of pressure transmitted through some medium, composed of frequencies within the range of hearing."
- Wikipedia

What does sound look like?

a 61 Hertz tone in slow motion

Let's oscillate!

var context = new AudioContext();
var oscillator = context.createOscillator();
oscillator.type = "sine"; // -> sine 
oscillator.frequency.value = 22000;

Let's play sounds from files

1. Load the file

  var xhr = new XMLHttpRequest();'GET', '/get_lucky.mp3', true);

  xhr.responseType = 'arraybuffer';

  xhr.onload = processFile


Let's play sounds from files

2. Process the response

context.decodeAudioData(xhr.response, function(buffer) {

  songBuffer = buffer;

}, function(e) {

  console.log('Whoops, sth. went wrong');


Let's play sounds from files

3. Play it

  var source = context.createBufferSource();

  source.buffer = songBuffer;




  • change the output by manipulating the frequency spectrum
  • most basic filters are already implemented
  • custom filters can be created easily

Basic filters

  var filter = context.createBiquadFilter();

  filter.type = filter.LOWPASS;

  filter.frequency.value = 100;

  // put the filter in between song and speakers

Basic filters

Filters in action


  • Timing is key!
  • before: setTimeout / requestAnimationFrame
  • lots of offset-calculations
  • inaccurate

Timing in Web Audio

  • Timing is relative to the current context!
context.currentTime // -> time bar in seconds

Timing in Web Audio

Let's start a buffer in 2 seconds

	sourceNode.start(context.currentTime + 2);

	sourceNode.start(when, offset, duration);

Timing in Web Audio: Future Value(s)

The API even allows to set values in the future:

  node.gain.setValueAtTime(.2, currentTime + 5);

  node.gain.linearRampToValueAtTime(0, currentTime + 5);


The hello world of Web Audio API - a drum machine


Jam with chrome

Let's talk about Gamepads and Gaming

  • More complex games are written in JS (Canvas, WebGL)
  • Controlling games: W, A, S, D or arrows + mouse + keys
  • Problem: digital vs. analogue values

Wouldn't it be nice to use gamepads in your browser?

Introducing: Gamepad API

Use all your favorite Gamepads in your browser!

Gamepad API 101

	var gamepads = navigator.webkitGetGamepads()

	var firstPad = gamepads[0];

	firstPad.buttons; // -> Array[17]

	firstPad.buttons[0]; // -> 0

Gamepad API 101

	firstPad.axes; // -> Array[4]

	firstPad.axes[0]; // -> 0.38765

But how am I supposed to know which button has which index?

Did you notice something?

  var gamepads = navigator.webkitGetGamepads()

  var firstPad = gamepads[0];

  firstPad.buttons; // -> Array[17]

  firstPad.buttons[0]; // -> 0


It's not event-driven!

  • API is designed for usage in games
  • Games often have an update()-loop where they poll gamepad states
  • We're back at requestAnimationFrame
  • Events may be added in the future

Let's play

A Google doodle that uses the Gamepad API

How do we combine the APIs?

if(gamepad.buttons[0] == 1)

Let me show you how that works


Let's talk about Gamepads again!

What exactly qualifies as a Gamepad?

"If it has a button, it is a Gamepad!"*

* (assuming that your browser knows how to interface it)

Guitar Hero guitar

guitar.js / alleinunterhalter.js

Let's play some guitar

(Don't forget to tune the guitar!)

DJ Hero turntable


Connect your DJ Hero Gamepad and mix it!

(Push the button as if you were Skrillex!)

API problems (Web Audio API)

  • Still unstable: browser may crash when doing fancy stuff
  • No negative playbackRate, yet
  • currentTime of BufferNodes would be aweswome
  • Only one destination for now

API problems (Gamepad API)

  • Unstable API (FF/Chrome incomptatible)
  • Events, please ;)
  • Let me talk to the Gamepad: rumble, LEDs
  • Sometimes unstable and doesn't connect the Gamepads
  • Please add the Guitar Hero drum kit, so we can have a rockband.js

Wrapping it up

  • The Web Audio API is incredibly powerful
  • The Gampad API is awesome and easy to use
  • When combined, they allow to play 'real' instruments in your browser
  • I can see things like Garageband.js with DJ equipment in the future!


THX so much for your attention!