Stencil Buffer and Stencil Test

Introduction to Stencil

The concept of Stencil is closely tied to the Stencil Buffer.

We know that the screen has a depth buffer. By default, if the distance of the pixel being rendered is greater than the value in the depth buffer, the current pass is discarded.

The Stencil Buffer is also a buffer, but unlike the depth buffer, which has built-in comparison behavior, the Stencil Buffer requires us to explicitly define how it should be processed.

The official docs¹² cover this, though they can be a bit abstract. Let’s start with an example to make it easier to understand what it can actually do.

Stencil Test

Suppose we have a Cube and a square plane Quad. When the Quad is in front of the Cube, it naturally occludes part of it.

Now, let’s replace the Quad with the following Shader. This Shader does the following:

• Blend Zero One means the original color is preserved, making the object effectively transparent.
• ZWrite Off disables writing to the depth buffer.
• The Stencil block defines the stencil test. For now, all we need to know is that Comp Always means the test always passes, and Pass Replace means that after the pass, the stencil buffer value is replaced with the Ref value.

Shader "Custom/Stencil/StencilFilter"
{
    Properties
    {
        [IntRange] _StencilID ("Stencil ID", Range(0, 255)) = 1
    }

    SubShader
    {
        Tags {
            "RenderType" = "Opaque"
            "RenderPipeline" = "UniversalPipeline"
            "Queue" = "Geometry"
        }
        
        Blend Zero One
        ZWrite Off
            
        Stencil
        {
          Ref [_StencilID]
          Comp Always
          Pass Replace
        }
        
        Pass
        {
        }
    }
}

Now the Quad is transparent, but it also overwrites the Stencil Buffer values for its entire region.

Next, let’s replace the Cube’s Shader:

• Here we only care about the Stencil block. Comp Equal means the pass continues if the Ref value equals the current Stencil Buffer value; otherwise, rendering stops.

Shader "Custom/Stencil/StencilTest"
{
    Properties
    {
        [IntRange] _StencilID ("Stencil ID", Range(0, 255)) = 0
        [MainColor] _BaseColor("Base Color", Color) = (1, 1, 1, 1)
        [MainTexture] _BaseMap("Base Map", 2D) = "white"
    }

    SubShader
    {
        Tags {
            "RenderType" = "Opaque"
            "RenderPipeline" = "UniversalPipeline"
            "Queue" = "Geometry"
        }
        
        Stencil
        {
            Ref [_StencilID]
            Comp Equal
        }

        Pass
        {
            HLSLPROGRAM

            #pragma vertex vert
            #pragma fragment frag

            #include "Packages/com.unity.render-pipelines.universal/ShaderLibrary/Core.hlsl"

            struct Attributes
            {
                float4 positionOS : POSITION;
                float2 uv : TEXCOORD0;
            };

            struct Varyings
            {
                float4 positionHCS : SV_POSITION;
                float2 uv : TEXCOORD0;
            };

            TEXTURE2D(_BaseMap);
            SAMPLER(sampler_BaseMap);

            CBUFFER_START(UnityPerMaterial)
                half4 _BaseColor;
                float4 _BaseMap_ST;
            CBUFFER_END

            Varyings vert(Attributes IN)
            {
                Varyings OUT;
                OUT.positionHCS = TransformObjectToHClip(IN.positionOS.xyz);
                OUT.uv = TRANSFORM_TEX(IN.uv, _BaseMap);
                return OUT;
            }

            half4 frag(Varyings IN) : SV_Target
            {
                half4 color = SAMPLE_TEXTURE2D(_BaseMap, sampler_BaseMap, IN.uv) * _BaseColor;
                return color;
            }
            ENDHLSL
        }
    }
}

Set both the Quad and the Cube’s _StencilID to the same value, say 1.

Now when you move the Quad, you’ll see something interesting: only the regions occluded by the Quad will render the Cube.

If we have two Cubes with different _StencilID values and two Quads with matching _StencilID values, each Quad will only reveal the Cube with its corresponding ID, ignoring the other.

Here’s what’s happening:

• By default, the entire screen’s Stencil Buffer value is 0, so both Cubes won’t render outside of the Quad’s coverage.
• Once a Quad is rendered, it changes the Stencil Buffer values of its pixels. Because the Cube’s shader uses Comp Equal, it only renders in areas where the stencil values match.

Render Order

From the above, we can see a potential issue: the Quad must render before the Cube so that it updates the Stencil Buffer first.

If their render queues are the same, flickering may occur.

We can fix this by setting the Quad’s Queue to a smaller value than the Cube’s, ensuring it renders first.

Now the Cube renders stably without flicker:

Stencil Test Syntax

You can check the official docs² or this Chinese article³ for more details.

The core syntax looks like this:

Stencil
{
    Ref <ref>
    ReadMask <readMask>
    WriteMask <writeMask>
    Comp <comparisonOperation>
    Pass <passOperation>
    Fail <failOperation>
    ZFail <zFailOperation>
}

• <ref> is the reference value, and how it’s compared with the Stencil Buffer depends on Comp.
• ReadMask applies a bitmask when comparing, and WriteMask applies a mask when writing.
• Comp defines how Ref compares to the Stencil Buffer Value. The default is Always (always passes). Equal passes when equal, and there are also options like Less, LEqual, Greater, NotEqual, GEqual, etc.
• Pass defines the operation performed on the Stencil Buffer when Comp passes, such as Keep (leave as is), Replace (replace with Ref), Zero, IncrSat, DecrSat, Invert, IncrWrap, DecrWrap.
• Fail defines the operation if Comp fails.
• ZFail defines the operation when Comp passes but the depth test fails.

You can also specify front and back face operations separately (see the official docs).

A minimal example to modify the stencil buffer:

Stencil
{
     Ref 1
     Comp Always
     Pass Replace
}

A minimal example to test the stencil buffer:

Stencil
{
    Ref 1
    Comp Equal
}

Renderer Objects Feature

If every time we add a new Shader we also need to add Stencil Buffer logic and set _StencilID, that quickly becomes tedious—it means modifying a lot of materials.

Instead, you can use the Renderer Objects feature, as explained in impossible-geom-stencils⁴, to set Stencil without editing every shader.

For example, imagine a scene with three Cubes and two Quads, where one purple Cube uses a transparent material:

First, replace the two Quads with the StencilFilter shader from earlier.

Then create two new Layers: Stencil Object 1 and Stencil Object 2, and assign the three Cubes to either of them.

Next, open the Universal Renderer Data settings and disable rendering for those two layers:

Now add a Render Objects Feature. In Filters, set the Layer Mask to Stencil Object 1/2. You’ll need four such features in total (2 queues × 2 layer masks).

In the Override section, enable Stencil, so you can override existing stencil settings. For example, set Comp to Equal and Value to match the intended ID.

Also pay attention to the Event timing:

• For Opaque, use After Rendering Opaques.
• For Transparent, use After Rendering Transparents.

Here’s the result:

If we tweak the scene a bit, disable the skybox, and add color to the StencilFilter, we can create a Magic Cube: