This article will touch base on every aspect of production: the plan, the server, the sounds, the visuals, and some of the workflow we leverage to design for interactive. Most sections contain code snippets, a demo, and a download. At the end of the article, there is a download link where you can grab them all as one zip file.
We're by no means audio engineers at gskinner.com — but tempt us with a challenge and we'll figure out a plan:
- Users plot tones on a grid, “inspired” by Andre's ToneMatrix
- Tones are wired to sampled instruments, drum kits, or even users' own recordings
- Multiple connected users play on the same grid simultaneously
- …or go into solo mode to explore on their own
- Invitational sessions allow users to organize a band and have an impromptu jam
We offer users an opportunity to explore the Web Audio API by means of a tools panel that applies audio filters and effects on their tones.
- Store users' compositions and effects as data and sync it across clients
- Provide some color options so they can draw cool looking songs
- Offer a gallery so people can listen, love, or even edit, other people's work
We stuck with the familiar grid metaphor, floated it in 3D space, added some lighting, texture and particle effects, housed it in a flexible (or fullscreen) CSS and JS-driven interface.
Instrument, effect, and grid data is consolidated and serialized on the client, then sent off to our custom Node.js backend to resolve for multiple users à la Socket.io. This data is sent back to the client with each players' contributions included, before being dispersed to the relative CSS, WebGL, and WebAudio layers in charge of rendering the UI, samples, and effects during multi-user playback.
We use Node for every aspect of the server. It's a static web server and our socket server all in one. Express is what we ended up using, it's a full web server built entirely on Node. It's super scalable, highly customizable, and handles the low-level server aspects for you (just like Apache or Windows Server would). Then you, as the developer, only have to focus on building your application. Download a more detailed client-server communication diagram
Multi-User Demo (ok, it's really just a screenshot)
This demo requires to be run off a Node server, and since this article isn't one, we've included a screenshot of what the demo looks like after you've installed Node.js, configured your web server, and run it locally. Every time a new user visits your demo installation, a new grid will be added and everyone's work is visible to one another.
Download the multi-user demo. requires Node.js server Be sure to check out the README, it contains info about installing Node.js, configuring your server, and running the demo locally.
Node is easy. Using a combo of Socket.io and custom POST requests, we didn't have to build complex routines for synchronization. Socket.io transparently handles this; JSON gets passed around.
How easy? Watch this.
A few more to tie socket.io in for real-time communication.
Now we just start listening for incoming connections from the HTML page.
A big unknown was the effort entailed with using the Web Audio API. Our initial findings confirmed that Digital Signal Processing (DSP) is very complex, and we were likely in way over our heads. Second realization: Chris Rogers has already done the heavy lifting in the API.
Technitone isn't using any really crazy math or audioholicism; this functionality is easily accessible to interested developers. We really just needed to brush up on some terminology and read the docs. Our advice? Don't skim them. Read them. Start at the top and end at the bottom. They are peppered with diagrams and photos, and it's really cool stuff.
If this is the first you've heard of the Web Audio API, or don't know what it can do, hit up Chris Rogers' demos. Looking for inspiration? You'll definitely find it there.
Web Audio API Demo
Your browser does not support the Web Audio API. Please try running this demo in Chrome.
Here's an overview of the what you'll find in the source:
Load in a sample (sound file)…
…set up modular routing…
…apply a runtime effect (convolution using an impulse response)…
…apply another runtime effect (delay)…
…and then make it audible.
Our approach to playback in Technitone is all about scheduling. Instead of setting a timer interval equal to our tempo to process sounds every beat, we set up a smaller interval that manages and schedules sounds in a queue. This allows the API to perform the up-front labour of resolving audio data and processing filters and effects before we task the CPU with actually making it audible. When that beat finally comes around, it already has all the information it needs to present the net result to the speakers.
Overall, everything needed to be optimized. When we pushed our CPUs too hard, processes were skipped (pop, click, scratch) in order to keep up with the schedule; we put serious effort into halting all the madness if you jump over to another tab in Chrome.
Front and center is our grid and particle tunnel. This is Technitone's WebGL layer.
WebGL offers considerably superior performance than most other approaches to rendering visuals on the web, by tasking the GPU to work in conjunction with the processor. That performance gain comes with the cost of significantly more involved development with a much steeper learning curve. That said, if you're truly passionate about interactive on the web, and want as few performance restraints as possible, WebGL offers a solution comparable to Flash.
Local light position:
X: Y: Z:
Particle count: Particle frequency:
MIN --- X: Y: Z:
MAX --- X: Y: Z:
Use texture normal
The demo introduces shading, textures, animation, particle effects, and interactivity. Each demo builds on the previous concept linearly.Download the WebGL DemoTexture-based demos only work on a server
Note: WebGL abides to a security protocol that prevents it from loading and manipulating textures directly from the local hard-drive.
WebGL content is rendered to a canvas (literally, the HTML5 Canvas) and is comprised of these core building blocks:
- object vertices (geometry)
- position matrices (3D coordinates)
- shaders (a description of geometry appearance, linked directly to the GPU)
- the context (“shortcuts” to the elements that the GPU makes reference to)
- buffers (pipelines for passing context data to the GPU)
- the main code (the business logic specific to the desired interactive)
- the “draw” method (activates the shaders and draws pixels to the canvas)
The basic process to render WebGL content to the screen looks like:
- Set the perspective matrix (adjusts settings for the camera that peers into the 3D space, defining the picture plane).
- Set the position matrix (declare an origin in the 3D coordinates that positions are measured relative to).
- Fill the buffers up with data (vertex position, color, textures…) to pass to the context through the shaders.
- Extract and organize data from the buffers with the shaders and pass it into the GPU.
- Call the draw method to tell the context to activate shaders, run with the data, and update the canvas.
It looks like this in action:
Set the perspective matrix…
…set the position matrix…
…define some geometry and appearance…
…fill the buffers up with data and pass it to the context…
…and call the draw method
Every frame, remember to clear the canvas if you don't want alpha-based visuals to stack up on one another.
From the onset, we decided users should be interacting with the grid as quickly as possible. No splash screen, no instructions, no tutorials, just 'Go.' If the interface is loaded — there should be nothing slowing them down.
This required us to look carefully at how to guide a first-time user through their interactions. We included subtle cues, like having the CSS cursor property change based on the user's mouse position within the WebGL space. If the cursor is over the grid, we switch it to a hand cursor (because they can interact by plotting tones). If it's hovered in the whitespace around the grid, we swap it out for a directional cross cursor (to indicate they can rotate, or explode the grid into layers).
Getting Ready for the Show
LESS (a CSS pre-processor), and CodeKit (web development on steroids) really cut down on the time it took to translate design files into stubbed out HTML/CSS. These let us organize, write, and optimize CSS in a much more versatile fashion — leveraging variables, mix-ins (functions), and even math!
Using CSS3 transitions and backbone.js we created some really simple effects that help bring the application to life and provide users with visual queues that indicate which instrument they are using.
Backbone.js allows us to catch color change events and apply the new color to the appropriate DOM elements. GPU accelerated CSS3 transitions handled the color style changes with little-to-no impact on performance.
Most of the color transitions on interface elements were created by transitioning background colors. On top of this background color, we place background images with strategic areas of transparency to let the background color shine through.
Color Transition Demo — How Many Colors Can You Create?
This demo shows a great example of what can be done with this technique. Pick colors and watch the mad scientist mix them to create a third unique color.
HTML: The Foundation
We needed three color regions for the demo: two user selected color regions and a third mixed color region. We built the simplest DOM structure we could think of that support CSS3 transitions and the fewest HTTP requests for our illustration.
<!-- Basic HTML Setup --> <div class="illo color-mixed"> <div class="illo color-primary"></div> <div class="illo color-secondary"></div> </div>
CSS: Simple Structure with Style
We used absolute positioning to place each region in its correct location and adjusted the background-position property to align the background illustration within each region. This makes all the regions (each with the same background image), look like a single element.
GPU accelerated transitions were applied that listen for color change events. We increased the duration and modified the easing on .color-mixed to create the impression that it took time for the colors to mix.
Assigning colors dynamically is simple. We search the DOM for any element with our color class and set the background-color based on the user's color selections. We apply our transition effect to any element in the DOM by adding a class.
This creates an architecture that is lightweight, flexible and scalable.
Once the primary and secondary colors are selected, we calculate their mixed color value and assign the resulting value to the appropriate DOM element.
Illustrating for HTML/CSS Architecture: Giving Three Color Shifting Boxes Personality
Our goal was to create a fun and realistic lighting effect that maintained its integrity when contrasting colors were placed in adjacent color regions.
A 24-bit PNG allows the background-color of our HTML elements to show through the transparent areas of the image.
The colored boxes create hard edges where different colors meet. This gets in the way of realistic lighting effects and was one of the bigger challenges when designing the illustration.
The solution was to design the illustration so that it never allows the edges of color regions to show through the transparent areas.
Planning for the build was critical. A quick planning session between designer, developer, and illustrator helped the team understand how everything needed to be built so it would work together when assembled.
Check out the Photoshop file as an example of how layer naming can communicate information about CSS construction.
For users without Chrome, we set a goal to distill the essence of the application into a single static image. The grid node became the hero, the background tiles allude to the purpose of the application, and the perspective present in the reflection hints at the immersive 3D environment of the grid.
If you're interested in learning more about Technitone, stay tuned in to our blog.
Thanks for reading, maybe we'll be jamming with you soon!