Pictured: Interactive input and GPU techniques combine to warp live video When I set out to make this blog, I made a list of articles and tutorials I wanted to write. At the top of the list was "FBO ping pong + shaders". It's not a new concept, but it's probably one of the most difficult yet important tools to understand for a creative programmer. The performance and resolution you can achieve from this technique is astounding. Many have ripped their hair out trying to get this going, so I'm hoping this tutorial can help keep a few scalps intact. It's a long read, but worth it if it works out for you. DOWNLOAD SOURCE fboShaderExample.zip CINDER I highly recommend using Cinder for this type of work. I'd originally written this tutorial for openFrameworks , but was disappointed with its performance and compatibility. I'm still a big fan of openFrameworks, but I have to recommend Cinder for this particular type of development. The faster development time and better performance and compatibility is significant. The attached project is meant for Visual C++ 2010 on Windows 7, but the source code should be more or less cross-platform. FRAME BUFFER OBJECTS The key to making fast visuals is getting as much of your data and logic onto your video card as possible. This keeps it close in memory to where it needs to be and frees your computer's CPU and memory to do other things. Frame buffer objects are basically chunks of memory that sit on your video card and hold color data. They are places to render graphics off screen. However, we can interpret this data any way we want by using shaders. FRAGMENT SHADERS OpenGL shaders are applications written in GLSL , a C-like language, which you compile in your C++ application and load onto the video card. A shader application is run against each element it's targeting. A vertex shader is executed against the vertices in your shape. A fragment shader is executed on each, well... fragment. For our purposes, we are drawing a single full screen quad and relying entirely on the fragment shader. A fragment is not the same as a pixel, but we'll be essentially treating them as the same thing. PING PONG The basic idea of the "ping pong" technique is that you have two images. You take the graphics from one image, apply a filter, and draw the result onto the other image. Then you take the image onto which you just drew, pass that through a filter, and draw back onto the first image. Repeat ad infinitum, stopping at a reasonable point. Render the result. CODE To view the full source of this tutorial, download the attached zip file above. This is a pretty small application, but I'm going to leave off maybe 30% of the code which isn't pertinent to understanding this concept. A video card with OpenGL/GLSL support is required to run this application. Let's take a look at our application's header file. ... class fboShaderApp : public AppBasic { public: // Cinder callbacks void draw(); void keyDown(KeyEvent event); void prepareSettings(Settings *settings); void setup(); void update(); private: // Constants static const int FBO_WIDTH = 1024; static const int FBO_HEIGHT = 768; static const int ITERATIONS = 4; // Should always be even // Frame buffer objects int fboPing; int fboPong; gl::Fbo FBOs[2]; // Shaders gl::GlslProg shaderRender; gl::GlslProg shaderProcess; // Background texture gl::Texture texImage; // Randomizer Rand random; // Save frames flag (record output) bool bSaveFrames; // True renders input to screen bool bShowInput; }; ... There are a few declarations to note here. The "FBOs" array contains pointers to two of Cinder's frame buffer objects, which at this moment are basically nothing. To better track the whole "ping pong" thing, I've named the indexes we'll use to reference the FBOs "fboPing" and "fboPong". I also have integer constants for the FBO dimensions and number of shader passes ("ITERATIONS"). The shader pass count must always be even. If you "ping" but don't "pong" you'll get some nasty flickering. The higher this number is, the more drastic changes will be per frame, but the more you'll strain your GPU. Likewise, a larger FBO size will mean more work and possibly a lower frame rate, but you can generally set this pretty high on a decent video card. Moving onto the setup... void fboShaderApp::setup() { // Set flags bSaveFrames = false; bShowInput = false; // Smoothes edges on the random circles glEnable(GL_LINE_SMOOTH); glHint(GL_LINE_SMOOTH_HINT, GL_NICEST); // Load shaders shaderRender = gl::GlslProg(loadResource(RES_PASS_THRU_VERT), loadResource(RES_RENDER_FRAG)); shaderProcess = gl::GlslProg(loadResource(RES_PASS_THRU_VERT), loadResource(RES_PROCESS_FRAG)); // Load image texture and bind at 1 texImage = gl::Texture(loadImage(loadResource(RES_TEXTURE))); texImage.setWrap(GL_REPEAT, GL_REPEAT); texImage.setMinFilter(GL_LINEAR); texImage.setMagFilter(GL_LINEAR); texImage.bind(1); // Create FBO format which will smooth out the image gl::Fbo::Format format; format.enableDepthBuffer(false); format.setSamples(4); format.setCoverageSamples(8); // Set up frame buffer objects fboPing = 0; fboPong = 1; FBOs[fboPing] = gl::Fbo(FBO_WIDTH, FBO_HEIGHT, format); FBOs[fboPong] = gl::Fbo(FBO_WIDTH, FBO_HEIGHT, format); } Loading shaders is piece of cake of Cinder. I have constants pointing to the file locations in my "Resources.h" file. I'm using an overload which points to those, but there is another one which lets you use external files. Below that, I'm loading an image (also defined in "Resources.h") which I'll be refracting with dynamic color data. It's a nice texture-y looking photo I took at the Upper Geyser Basin in Yellowstone . I've got it set to wrap both vertically and horizontally and then I'm binding it at texture unit #1. Depending on your hardware and drivers, you may be able to use up to thirty-two texture units, but we only need two: texture units #0 and #1. To create frame buffer objects in Cinder, you simply initialize them with their dimensions and a format object. Memory will automatically be allocated. The format I'm using to create the FBOs helps smooth out the image, but is a bit expensive, so just use a default format if you need speed over quality. With the FBO array set and the indexes defined, we can move onto Cinder's update callback, which is executed on each frame before drawing. void fboShaderApp::update() { // Set up viewport gl::setMatricesWindow(FBOs[fboPing].getSize(), false); gl::setViewport(FBOs[fboPing].getBounds()); // Loop through iteration count for (int i = 0; i < ITERATIONS; i++) { // Swap FBO indexes fboPing = (fboPing + 1) % 2; fboPong = (fboPong + 1) % 2; // Bind the "ping" FBO so we can draw onto it FBOs[fboPing].bindFramebuffer(); // Bind the "pong" FBO as a texture to // send to the shader FBOs[fboPong].bindTexture(); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT); /****** Begin process shader configuration ******/ // Bind the process shader shaderProcess.bind(); // This was the "ping" FBO we drew onto // in the last iteration shaderProcess.uniform("texture", 0); // This controls the definition of edges in // the shader (higher is softer) shaderProcess.uniform("dampen", 350.0f); // These are used in a mutant variant of the // Gray-Scott reaction-diffusion equation shaderProcess.uniform("ru", 0.33f); shaderProcess.uniform("rv", 0.1f); shaderProcess.uniform("k", 0.06f); shaderProcess.uniform("f", 0.25f); // We pass in the width and height to convert normalized // coordinates to screen coordinates shaderProcess.uniform("width", (float)FBO_WIDTH); shaderProcess.uniform("height", (float)FBO_HEIGHT); /****** End process shader configuration ******/ // Draw the shader output onto the "ping" FBO glColor4f(1.0f, 1.0f, 1.0f, 1.0f); gl::drawSolidRect(FBOs[fboPing].getBounds()); // Stop the shader shaderProcess.unbind(); // Draw a red circle randomly on the screen random.randomize(); RectMapping windowToFBO(getWindowBounds(), FBOs[fboPing].getBounds()); glColor4f(1.0f, 0.0f, 0.0f, 1.0f); gl::drawSolidCircle(windowToFBO.map(Vec2f( random.nextFloat(0.0f, (float)FBO_WIDTH), random.nextFloat(0.0f, (float)FBO_HEIGHT))), random.nextFloat(1.0f, 30.0f), 64); // TO DO: Draw anything you want here in red. // It will refract the image texture. // Unbind the FBO to stop drawing on it FBOs[fboPing].unbindFramebuffer(); } } This is the heart of the application. We start by preparing an OpenGL viewport, but instead of drawing to the screen, we will be doing some offline rendering to our frame buffer objects. The basic process is this: Bind one frame buffer object as a texture to send to the shader Bind the other FBO as the frame buffer, or render target, so we can draw onto it Bind the shader so that it affects any geometry we draw Pass the first FBO as a texture into the shader Render the output of the shader to the other FBO by drawing a full screen quad Unbind the shader Draw any new input, like mouse interaction or in this case, a randomly sized and positioned circle Unbind the frame buffer On the next pass, everything we've drawn will be passed into the shader, further affecting it. Across multiple passes, we can quickly make some big full screen changes. Note that we're only drawing in red. The red channel is used to store input values, while the other channels will be used by the shader to store some other information which we'll use to render the final image. This is the part that can be tough to wrap your brain around, so maybe taking a look at the shaders will clear things up. Vertex Shader void main() { gl_FrontColor = gl_Color; gl_TexCoord[0] = gl_MultiTexCoord0; gl_Position = ftransform(); } Fragment Shader #version 120 #define KERNEL_SIZE 9 // Screen size uniform float width; uniform float height; // Dampening uniform float dampen; // Texture uniform sampler2D texture; // Reaction-Diffusion uniform float ru; uniform float rv; uniform float f; uniform float k; // Offsets vec2 offset[KERNEL_SIZE]; void main(void) { // Get coordinates and pixel sizes vec2 texCoord = gl_TexCoord[0].st; float w = 1.0 / width; float h = 1.0 / height; // Set neighbor locations offset[0] = vec2(-w, -h); offset[1] = vec2(0.0, -h); offset[2] = vec2(w, -h); offset[3] = vec2(-w, 0.0); offset[4] = vec2(0.0, 0.0); offset[5] = vec2(w, 0.0); offset[6] = vec2(-w, h); offset[7] = vec2(0.0, h); offset[8] = vec2(w, h); // Find sum of neighbors and self float sumR = 0.0; float sumB = 0.0; for (int n = 0; n < KERNEL_SIZE; n++) { sumR += texture2D(texture, texCoord + offset[n]).r; sumB += texture2D(texture, texCoord + offset[n]).b; } // Use average to determine new value float u = sumR / float(KERNEL_SIZE); float v = texture2D(texture, texCoord).g; // Tweak values a tad u += u / dampen + 0.2; v += 0.2; // Reaction diffusion float r = texture2D(texture, texCoord).r; float F = f + r * 0.025 - 0.0005; float K = k + r * 0.025 - 0.0005; float uvv = u * v * v; float du = ru * sumR - uvv + F * (1.0 - u); float dv = rv * sumB + uvv - (F + K) * v; u += (du / dampen) * 0.6; v += dv * 0.6; // Clamp values u = 1.0 - clamp(u, 0.0, 1.0); v = 1.0 - clamp(v, 0.0, 1.0); // Set color gl_FragColor = vec4(u, 1.0 - u / v, v, 0.0); } The vertex shader isn't doing much for us, but it's required as the first step in the pipeline to give coordinate and color information to the fragment shader. The fragment shader is doing all the work. This particular shader is an amalgamation of a blur, 2D water , and Gray-Scott reaction-diffusion program. The latter, along with some parts of this tutorial, were borrowed from Robert Hodgin 's RDiffusion Cinder demo. It basically uses neighboring colors to determine a new value, slowly returning to 0.0. Fragments cannot look at their neighbors in memory, but they can look up any part of a texture. Horizontal and vertical coordinates run from 0.0 - 1.0. By passing in the width and height, we can divide to determine the size of one pixel, then take the current coordinate and pull color from neighbors in the texture. Because we're using the last rendering of the screen as a texture, it's almost like getting to work with the entire buffer. If you watch the video below to the 14s mark, you'll get a glimpse into what the shader is doing behind the scenes. Realtime performance is more than double the framerate of this video at 1024 x 768 and even higher. Random circles are drawn and then processed using the FBO ping pong method, producing an image which is used to refract a texture. See what the unrendered input looks like at the 14s mark. The last step is to use this data to against the photograph to make it more interesting. Here is (most of) the draw function. void fboShaderApp::draw() { // Clear screen and set up viewport gl::clear(ColorA(0.0f, 0.0f, 0.0f, 0.0f)); gl::setMatricesWindow(getWindowSize()); gl::setViewport(getWindowBounds()); ... // Bind the FBO we last rendered as a texture FBOs[fboPing].bindTexture(); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT); // Start and configure the render shader shaderRender.bind(); // This is the image we've been drawing shaderRender.uniform("texture", 0); // This is the texture we'll be refracting shaderRender.uniform("srcTexture", 1); // Pass the image size shaderRender.uniform("width", (float)FBO_WIDTH); shaderRender.uniform("height", (float)FBO_HEIGHT); // Draw shader output to screen glColor4f(1.0f, 1.0f, 1.0f, 1.0f); gl::drawSolidRect(FBOs[fboPing].getBounds()); // Stop shader shaderRender.unbind(); ... } To combine the processed input with other imagery we need to bind the last frame we rendered as a texture, then send both that and the image we bound earlier into the shader. Like in the process shader, the width and height are passed in so we can work in pixels. And like the other shader, we render the output to a full screen quad. The difference is that this time we're outputting to the screen. #version 120 // Screen size uniform float width; uniform float height; // Textures uniform sampler2D srcTexture; uniform sampler2D texture; void main(void) { // Get coordinates and pixel sizes vec2 texCoord = gl_TexCoord[0].st; float w = 1.0 / width; float h = 1.0 / height; // Calculate offsets float x = texture2D(texture, texCoord + vec2(0.0, -h)).g - texture2D(texture, texCoord + vec2(w, 0.0)).b; float y = texture2D(texture, texCoord + vec2(0.0, -h)).r - texture2D(texture, texCoord + vec2(0.0, h)).r; // Amplify refraction x *= x * y; // Get color from image texture vec4 srcColor = texture2D(srcTexture, texCoord + vec2(x, y)); // Floor float s = texture2D(texture, texCoord).g * 0.2; // Set color gl_FragColor = vec4(srcColor.r + x - s, srcColor.g + x - s, srcColor.b - y - s, 1.0); } In this shader, I'm using the color data from the texture in the frame buffer object to point to different pixels in the photo, thus warping it. EXTENDING This tutorial focuses on the using the FBO ping pong technique with shaders to manipulate input, then use the resulting color data to do some basic refraction on a JPEG. This is just the beginning. The input here is automated, random circles, but you can draw interactive input from a mouse, multi-touch screen, motion tracking, etc. There is a lot more you can do in GLSL, and you don't just have to render static images. Here are a couple examples which led to the " Manipulating Bob " project. Happy coding! This technique as applied to a multi-touch screen Using optical flow as input, as in the Manipulating Bob project

awesome. can't wait to play.....