Tutorials

Drawing thick outlines in OpenGL

Drawing outlines around 3D shapes is a common feature request, be it for selected items, quest markers, or what have you. In OpenGL there’s a few ways to do it and today I’d like to share with you the two methods I’ve tried – the OpenGL2 offset normals method (left) and the OpenGL3 per-pixel shader (right). Both use a variation on stencils. Stencils are like green screens. By isolating the thing you want to keep from the rest, it’s easy to do all kinds of neat tricks.

For those of you in the know, I’m using a brute force method. Skip to final thoughts for my reasoning.

---

OpenGL stencils

The first method I tried went something like this:

• Draw every mesh
• Draw selected stuff a second time, but this time to the stencil buffers
• Lock the stencil buffer but mask everything – only draw to the non empty parts of the stencil.
• Draw selected stuff a third time in the original buffer, but only edge lines in the selected color.

So let’s assume, for brevity, that there’s a drawAllMeshes method that works as advertised.
Somewhere in the pipeline there’s something like

drawAllMeshes(everything);
outlineSelectedMeshes(probablyLessThanEverything);

then outlineSelectedMeshes does all the stencil magic.

gl3.glEnable(GL3.GL_STENCIL_TEST);  // enable stencil testing  
gl3.glClear(GL3.GL_STENCIL_BUFFER_BIT | GL3.GL_DEPTH_BUFFER_BIT);  // clear the buffer  
gl3.glColorMask(false,false,false,false);  // disable writing to the color buffer  
gl3.glDepthMask(true);  // enable writing to the depth buffer  
gl3.glStencilMask(0xFF);  // enable writing to the stencil buffer  
gl3.glStencilFunc(GL3.GL_ALWAYS,1,0xFF);  // always pass the stencil test and set the value to 1  
gl3.glStencilOp(GL3.GL_KEEP,GL3.GL_KEEP,GL3.GL_REPLACE);  // set the stencil value to 1 when the depth test passes  

drawAllMeshes(gl3, selectedMeshes, camera, originShift);  // draw the selected meshes.  Only now they'll go to the stencil buffer.

gl3.glStencilFunc(GL3.GL_NOTEQUAL,1,0xFF);  // pass the stencil test if the value is not 1  
gl3.glStencilMask(0x00);  // disable writing to the stencil buffer  
gl3.glDepthMask(false);  // disable writing to the depth buffer  
gl3.glColorMask(true,true,true,true);  // enable writing to the color buffer 
gl3.glStencilOp(GL3.GL_KEEP,GL3.GL_KEEP,GL3.GL_KEEP);  // keep the stencil value  
gl3.glDisable(GL3.GL_CULL_FACE);  // draw both sides of the outline

gl3.glLineWidth(outlineThickness);  // set the line thickness
gl3.glPolygonMode(GL3.GL_FRONT_AND_BACK,GL3.GL_LINE);  // draw only lines

// outlineShader is an instance of ShaderProgram which gives me all kinds of convenience methods.
outlineShader.use(gl3);  
outlineShader.setMatrix4d(gl3, "viewMatrix", camera.getViewMatrix());  
outlineShader.setMatrix4d(gl3, "projectionMatrix", camera.getChosenProjectionMatrix(canvasWidth, canvasHeight));  
outlineShader.setColor(gl3, "outlineColor", Color.GREEN);  

// render the selected set with thick lines.
// drawAllMeshes would use the default shader, so I do it manually here
// to control the shader choice.
for(var mesh : selectedMeshes) {  
    outlineShader.setMatrix4d(gl3,"modelMatrix",mesh.getMatrix());  
    mesh.render(gl3);
}

gl3.glPolygonMode(GL3.GL_FRONT_AND_BACK,GL3.GL_FILL);  // reset shapes
gl3.glEnable(GL3.GL_CULL_FACE);  // restore settings
gl3.glDepthMask(true);  // restore settings
gl3.glLineWidth(1);  // restore settings
gl3.glStencilFunc(GL3.GL_ALWAYS,1,0xFF);  // turn off stencil testing  
gl3.glStencilOp(GL3.GL_KEEP, GL3.GL_KEEP, GL3.GL_REPLACE);  
gl3.glDisable(GL3.GL_STENCIL_TEST);

The outline shader v1

outline.vert was doing almost nothing.

#version 330 core  

layout(location = 0) in vec3 aPosition;  
layout(location = 1) in vec3 aNormal;  
layout(location = 2) in vec4 aColor;  
layout(location = 3) in vec2 aTexture;  

uniform mat4 projectionMatrix;  
uniform mat4 viewMatrix;  
uniform mat4 modelMatrix;

void main() {  
    vec3 offsetPosition = aPosition + aNormal * outlineSize;  
    vec4 worldPose = modelMatrix * vec4(offsetPosition, 1.0);  
    gl_Position = projectionMatrix * viewMatrix * worldPose;  
}

and outline.frag was doing truly nothing.

#version 330 core
out vec4 fragColor;  
uniform vec4 outlineColor;  // the outline color

void main() {  
    fragColor = outlineColor;
}

The effect on a basic cube looks like this:

Note that OpenGL also doesn’t give a way to adjust line end caps.

OpenGL 3 per pixel shaders

The OpenGL3 version required more setup but less work at run time. The process is:

• Draw every mesh
• Draw a second time to a stencil buffer FBO
• Use the FBO as a reference texture (not the technical name)
• Draw a rectangle over the whole screen.
• The shader color every pixel in the rectangle near (but not on) the stencil.

Full screen rectangle

I’m sorely tempted to say “eh, I have a Mesh wrapper class, you don’t need these details”… but in the interest of being thorough, here’s the whole rectangle setup.

private final int [] rectangleVAO[1];
private final int [] rectangleVBO[1];

private void generateFullscreenQuad(GL3 gl3) {  
    float v = 1.0f;  // make smaller to check position on screen
    float[] quadVertices = {  
        // positions  // texcoords  
        -v, -v, 0f,   0f, 0f, // bottom-left  
         v, -v, 0f,   1f, 0f, // bottom-right  
        -v,  v, 0f,   0f, 1f, // top-left  
         v,  v, 0f,   1f, 1f  // top-right  
     };

    gl3.glGenVertexArrays(1, rectangleVAO, 0);  // make vao
    gl3.glBindVertexArray(VAO[0]);  // use vao
    gl3.glGenBuffers(2, rectangleVBO, 0);  // make vbos
    gl3.glBindBuffer(GL3.GL_ARRAY_BUFFER, rectangleVBO[0]);    // use vbos
    gl3.glBufferData(GL3.GL_ARRAY_BUFFER, quadVertices.length * Float.BYTES, FloatBuffer.wrap(quadVertices), GL3.GL_STATIC_DRAW);  // fill vbos
    gl3.glBindVertexArray(0);  // stop using vao
}

Frame buffer object

Normally the video card draws into an offscreen back buffer and at the right time swaps the back buffer and the front buffer. This way you don’t see all triangles being drawn one by one. We can create our own Frame buffer object (FBO) as canvas-sized textures. (The canvas is the whole drawing area, which might not be the entire window.) The video card can draw into the FBO of our choosing, and then we can do stuff with it. I couldn’t find an easy way to access the built-in stencil buffer so I did it this way.

// class parameters  
private int outlineThickness = 5;  
private final int [] stencilFBO = new int[1];
private final int [] stencilTexture = new int [1];

// width and height should be the canvas dimensions.
// every time the canvas changes size this should be called again.
private void setupStencilFramebuffer(GL3 gl3, int width, int height) {  
    deleteStencilBuffer(gl3);  
    // Create FBO if not already created  
    gl3.glGenFramebuffers(1, stencilFBO, 0);  
    // Create stencil texture if not created  
    gl3.glGenTextures(1, stencilTexture, 0);  

    gl3.glBindTexture(GL3.GL_TEXTURE_2D, stencilTexture[0]);  
    // create with a single 8-bit red channel
    gl3.glTexImage2D(GL3.GL_TEXTURE_2D, 0, GL3.GL_R8, width, height, 0, GL3.GL_RED, GL3.GL_UNSIGNED_BYTE, null);  
    gl3.glTexParameteri(GL3.GL_TEXTURE_2D, GL3.GL_TEXTURE_MIN_FILTER, GL3.GL_NEAREST);  
    gl3.glTexParameteri(GL3.GL_TEXTURE_2D, GL3.GL_TEXTURE_MAG_FILTER, GL3.GL_NEAREST);  
    gl3.glTexParameteri(GL3.GL_TEXTURE_2D, GL3.GL_TEXTURE_WRAP_S, GL3.GL_CLAMP_TO_EDGE);  
    gl3.glTexParameteri(GL3.GL_TEXTURE_2D, GL3.GL_TEXTURE_WRAP_T, GL3.GL_CLAMP_TO_EDGE);  

    // Bind the FBO and attach the stencil texture  
    gl3.glBindFramebuffer(GL3.GL_FRAMEBUFFER, stencilFBO[0]);  
    gl3.glFramebufferTexture2D(GL3.GL_FRAMEBUFFER, GL3.GL_COLOR_ATTACHMENT0, GL3.GL_TEXTURE_2D, stencilTexture[0], 0);  
    gl3.glDrawBuffer(GL3.GL_COLOR_ATTACHMENT0);  

    // Check FBO status  
    int status = gl3.glCheckFramebufferStatus(GL3.GL_FRAMEBUFFER);  
    if (status != GL3.GL_FRAMEBUFFER_COMPLETE) {  
        throw new RuntimeException("Failed to setup stencil framebuffer: " + status);  
    }  

    // Unbind FBO  
    gl3.glBindFramebuffer(GL3.GL_FRAMEBUFFER, 0);  
}

// called when the canvas changes size or it is disposed.
private void deleteStencilBuffer(GL3 gl3) {  
    if(stencilFBO[0]!=-1) {  
        gl3.glDeleteFramebuffers(1, stencilFBO,0);  
        stencilFBO[0] = -1;  
    }  
    if(stencilTexture[0]!=-1) {  
        gl3.glDeleteTextures(1, stencilTexture,0);  
        stencilTexture[0] = -1;  
    }  
}

Testing the stencil buffer

// Step 1: Render the stencil into an offscreen texture using the FBO  
gl3.glBindFramebuffer(GL3.GL_FRAMEBUFFER, stencilFBO[0]);  
gl3.glViewport(0, 0, canvasWidth, canvasHeight);  
// Clear stencil texture  
gl3.glClearColor(0,0,0,0);  
gl3.glClear(GL3.GL_COLOR_BUFFER_BIT | GL3.GL_DEPTH_BUFFER_BIT);

// Please forgive my custom mesh shader here.
// Draw everything without lighting in flat white.
meshShader.use(gl3);  
meshShader.setMatrix4d(gl3, "viewMatrix", camera.getViewMatrix(originShift));  
meshShader.setMatrix4d(gl3, "projectionMatrix", camera.getChosenProjectionMatrix(canvasWidth, canvasHeight));  
meshShader.set1i(gl3, "useVertexColor", 0);  
meshShader.set1i(gl3, "useLighting", 0);  
meshShader.setColor(gl3,"diffuseColor",Color.WHITE);

for(var mesh : selectedMeshes) {  
    meshShader.setMatrix4d(gl3,"modelMatrix",mesh.getMatrix());
    mesh.render(gl3);
}

// resume editing the color buffer, do not change the depth mask or the stencil buffer.  
gl3.glBindFramebuffer(GL3.GL_FRAMEBUFFER, 0);  

// Step 2: Render outlines using the stencil texture  
gl3.glActiveTexture(GL3.GL_TEXTURE0);  
gl3.glBindTexture(GL3.GL_TEXTURE_2D, stencilTexture[0]);  

captureTextureData(gl3,canvasWidth,canvasHeight);

captureTextureData is used to write stencilTexture to a BufferedImage, and then I can use a breakpoint to view the stencil buffer.

Of course this only works if something there is a selected mesh!
Also… it draws upside down because the screen Y axis is reverse from a BufferedImage y axis.

Drawing the quad

Once the stencil buffer looked good it was time to disable the capture and draw the quad on screen.

//captureTextureData(gl3,canvasWidth,canvasHeight);  

outlineShader.use(gl3);  
outlineShader.set1i(gl3, "stencilTexture", 0); // Texture unit 0  
outlineShader.set2f(gl3, "textureSize", canvasWidth, canvasHeight);  
outlineShader.setColor(gl3, "outlineColor", Color.GREEN);  
outlineShader.set1f(gl3, "outlineSize", outlineThickness);  

// Render the quad with the stencil texture to the screen  
gl3.glDisable(GL3.GL_CULL_FACE);  
fullScreenQuad.render(gl3);  
gl3.glEnable(GL3.GL_CULL_FACE);

All we need is…

Testing a new outline shader

outline.vert v2

#version 330 core  

layout(location = 0) in vec3 aPosition;  
layout(location = 1) in vec2 aTexture;  

void main() {  
    gl_Position = vec4(position, 1.0);  
}

outline.frag v2

#version 330 core  

uniform vec4 outlineColor = vec4(0.0, 1.0, 0.0, 1.0);  
uniform float outlineSize = 1.0;  
uniform vec2 canvasSize;  // Size of the texture/screen  

out vec4 finalColor;  // Output fragment color  

// Sampler for the stencil texture (or depth-stencil data)  
uniform sampler2D stencilTexture;  

// the screen and the stencilTexture have dimensions from (-1,-1) to (1,1)  
// the canvasSize, outlineSize, and gl_FragCoord are in pixels from (0,0) to (width,height)  
void main() {  
    // Size of a fragment in texture coordinates  
    vec2 texelSize = 1.0 / canvasSize;  
    // Current pixel position in the stencil texture  
    vec2 textureCoord = gl_FragCoord.xy / canvasSize;  
    vec4 stencilValue = texture(stencilTexture, textureCoord);  

    // If the stencil value is not zero we're inside the stencil area so skip.  
    if (stencilValue.r > 0.0) discard;  

    int outInt = int(ceil(outlineSize));  
    float o2 = outlineSize * outlineSize;  
    // loop over all pixels within +/-outline size  
    for (int y = -outInt; y <= outInt; y++) {  
        for (int x = -outInt; x <= outInt; x++) {  
            if(x*x + y*y > o2) continue; // Skip pixels outside the circle  
            // convert pixel offset to texture coordinate offset
            vec2 offset = vec2(x, y) * texelSize;  
            // Sample the stencil texture at the offset position  
            vec4 neighbor = texture(stencilTexture, textureCoord + offset);  
            if(neighbor.r > 0.0) {  
                // We're in range, set the outline color and exit  
                finalColor = outlineColor;  
                return;  
            }  
        }  
    }  
    // If no neighboring pixels are found with a stencil value do nothing.
    //finalColor = vec4(0,0,1,1);  // make the background blue for testing
    //finalColor = vec4(0,(textureCoord.x+1)/2,(textureCoord.y+1)/2,1);
    // gradient for testing
}

Tada!

Final thoughts

The approach I took is considered “Brute force”. Ben Golus’ Jump Flood Algorithm (JFA) is the final answer for most people. It’s the hardest to do and the most efficient at run time. Presently I am not doing huge outlines that would require the run time cost savings. Also I have not seen a GLSL step-by-step… so… Perfect is the enemy of good enough.

I write 3D stuff mainly to simulate robots I want to build. The app is called Robot Overlord and it’s totally open source, so feel free to click the link and join in the latest fun. Show the world how smart you are and make a PR with working JFA.

Seriously, if you know someone learning OpenGL… make them start with 2.0. If you can choose the language for them, pick Java. It’s just so much easier and more likely to succeed. Watching my nephew do a ten chapter tutorial on Vulkan for one triangle… that’s crazy pants.