Making music with JavaScript and Gamepads

Reject.js 2013

(inspiration)

What to expect

Web Audio API

+

Gamepad API

=

<3 *

* both are drafts, examples only work in Chrome for now

Web Audio API

  • low-level access to all things audio
  • create sounds
  • manipulate sounds (filters)
  • timing sounds precisely
  • tons of other cool stuff (spatial audio, audio analysis)

What is sound?

"Sound is a mechanical wave that is an oscillation of pressure transmitted through some medium, composed of frequencies within the range of hearing."
- Wikipedia

What does sound look like?

a 61 Hertz tone in slow motion

Let's oscillate!

var context = new AudioContext();
var oscillator = context.createOscillator();
oscillator.type = "sine"; // -> sine 
oscillator.frequency.value = 22000;
oscillator.connect(context.destination);
oscillator.start(0);

Let's play sounds from files

1. Load the file

  var xhr = new XMLHttpRequest();

  xhr.open('GET', '/get_lucky.mp3', true);

  xhr.responseType = 'arraybuffer';

  xhr.onload = processFile

  xhr.send();

Let's play sounds from files

2. Process the response

context.decodeAudioData(xhr.response, function(buffer) {

  songBuffer = buffer;

}, function(e) {

  console.log('Whoops, sth. went wrong');

});

Let's play sounds from files

3. Play it

  var source = context.createBufferSource();

  source.buffer = songBuffer;

  source.connect(context.destination);

  source.start(0);

Filters

  • change the output by manipulating the frequency spectrum
  • most basic filters are already implemented
  • custom filters can be created easily

Basic filters


  var filter = context.createBiquadFilter();

  filter.type = filter.LOWPASS;

  filter.frequency.value = 100;

  // put the filter in between song and speakers
  song.connect(filter);
  filter.connect(context.destination);

Basic filters

Filters in action

Timing

  • Timing is key!
  • before: setTimeout / requestAnimationFrame
  • lots of offset-calculations
  • inaccurate

Timing in Web Audio

  • Timing is relative to the current context!
context.currentTime // -> time bar in seconds

Timing in Web Audio

Let's start a buffer in 2 seconds


	sourceNode.start(context.currentTime + 2);
					

	sourceNode.start(when, offset, duration);

Timing in Web Audio: Future Value(s)

The API even allows to set values in the future:


  node.gain.setValueAtTime(.2, currentTime + 5);
					

  node.gain.linearRampToValueAtTime(0, currentTime + 5);

Demos

The hello world of Web Audio API - a drum machine

Plink!

Jam with chrome

Let's talk about Gamepads and Gaming

  • More complex games are written in JS (Canvas, WebGL)
  • Controlling games: W, A, S, D or arrows + mouse + keys
  • Problem: digital vs. analogue values

Wouldn't it be nice to use gamepads in your browser?

Introducing: Gamepad API

Use all your favorite Gamepads in your browser!

Gamepad API 101


	var gamepads = navigator.webkitGetGamepads()
					

	var firstPad = gamepads[0];
					

	firstPad.buttons; // -> Array[17]
					

	firstPad.buttons[0]; // -> 0

Gamepad API 101


	firstPad.axes; // -> Array[4]
					

	firstPad.axes[0]; // -> 0.38765

But how am I supposed to know which button has which index?

Did you notice something?


  var gamepads = navigator.webkitGetGamepads()

  var firstPad = gamepads[0];

  firstPad.buttons; // -> Array[17]

  firstPad.buttons[0]; // -> 0

It's not event-driven!

  • API is designed for usage in games
  • Games often have an update()-loop where they poll gamepad states
  • We're back at requestAnimationFrame
  • Events may be added in the future

Let's play

A Google doodle that uses the Gamepad API

How do we combine the APIs?

if(gamepad.buttons[0] == 1)
  sound.start(0);

Let me show you how that works

But...

Let's talk about Gamepads again!

What exactly qualifies as a Gamepad?

"If it has a button, it is a Gamepad!™"*

* (assuming that your browser knows how to interface it)

Guitar Hero guitar

guitar.js / alleinunterhalter.js

Let's play some guitar

(Don't forget to tune the guitar!)

DJ Hero turntable

dj.js

Connect your DJ Hero Gamepad and mix it!

(Push the button as if you were Skrillex!)

API problems (Web Audio API)

  • Still unstable: browser may crash when doing fancy stuff
  • No negative playbackRate, yet
  • currentTime of BufferNodes would be aweswome
  • Only one destination for now

API problems (Gamepad API)

  • Unstable API (FF/Chrome incomptatible)
  • Events, please ;)
  • Let me talk to the Gamepad: rumble, LEDs
  • Sometimes unstable and doesn't connect the Gamepads
  • Please add the Guitar Hero drum kit, so we can have a rockband.js

Wrapping it up

  • The Web Audio API is incredibly powerful
  • The Gampad API is awesome and easy to use
  • When combined, they allow to play 'real' instruments in your browser
  • I can see things like Garageband.js with DJ equipment in the future!

whoami

THX so much for your attention!