"In the country of the blind, the one-eyed man is king"

"In the country of the blind, the one-eyed man is king" or how fumbling my way through pixel-based effects made learning WebGL a lot easier Visa-Valtteri “visy” Pimiä

My history with pixel-based visuals • Started doing simple effects with QBASIC • Worked my way to Turbo Pascal and VGA programming • Got sidetracked to NES and Atari 2600 development • Learned OpenGL mostly professionally, then started using GPU-accelerated graphics

But why talk about something whenI can just speedcode here :-) • Processing makes it possible

Coding pixel-based effects • Access the screen through a framebuffer, an array of pixel color values, palette index values or bitplane toggles • Set pixel color values according to position, time, function value etc.

Proprietary “engines” • I started making small games and demo-style little programs • Everything was always coded for one single production, with no mind put into reusability of the tools • This got old pretty quickly and I found myself toying with the idea of a visual editor for demos

First: S.P.L.I.F • TrilobitScriPtableLua Interactive MultiplatForm • Tackles a multitude of problems I had with development: • Inherently multiplatform (OSX, Win32, Linux) because all the GL calls are coded with Lua. • All visual code is reloadable without relaunching or recompiling the program • The demo player is seekable and supports setting sync points • “The demo is the editor” • Always open source

Syncing • Timeline file millisecond syncing of demo parts • FFT beat sync for effects • Seeking for testing

S.P.L.I.F has flaws • Fixed OpenGL version, upgrading would require recompiling the Lua libraries for all the three platforms • No shader support • Terrible hacks for texture handling • The OpenGL / Lua bound version I used was found to be rather incomplete

But it worked well enough • Counter Intelligence Program by Bilotrip

Need for an upgrade • After I heard about WebGL I got the idea of doing something like S.P.L.I.F in the browser with easily reloadable and tweakable visuals • WebGL has shaders, so the natural choice was to enable the editing of shader parameters (in addition for JavaScript variables) realtime

Technology behind DarumaGL • WebGL canvas for the output • Pure JavaScript + HTML5 + CSS3 for the UI • Textures loaded from PNG files • HTML5 audio from OGG file • Timeline.js used for the keyframe editor

Demonstration of the DarumaGL editor • http://www.low.fi/~visy/webgl/iloinen_paiva/

…and it has flaws as well • No real timeline dumping (cut & paste from export windows) • Shader code still requires reloading of the page • HTML5 audio support is still rather flaky cross browsers, and require some time to mature • No cross-browser FFT support for sound analysis • WebGL is still not available for mobile browsers or Internet Explorer

But… • WebGL _is_ a W3 standard, and pretty much all major browser except Internet Explorer have support for it in development roadmaps • No plugins required anymore for GPU accelerated content and complex video/audio data processing and playback • No third party components required

Lessons learned from pixel effects • Pixel shaders are pretty similar to framebuffers in many ways • Can set single resolution independent fragments instead of toggling actual pixels on the screen • Many fragments are calculated concurrently on the GPU, so the shader program is run in parallel.

Some things of note • I tend to avoid calculating complex geometries and like to play around with fragments on a single quad • Complex blending and processing passes possible with the GPU -> enables effects that are just simply impossible to do on a framebuffer with a fast FPS

Demonstration • MKULTRA by Bilotrip • http://www.low.fi/~visy/webgl/mkultra/

Thank you for listening!

"In the country of the blind, the one-eyed man is king"