opengl - How does mask affect stencil value according to stencil op? -

the documentation in opengl reference pdf (both opengl 3.3 , 4.5 specs) not clear happens stored stencil value when mask applied.

in example if have following mask:

glstencilmask( 0x06); 

and stored in stencil buffer there value:

0x06 

if stencil operation gl_incr_wrap

what should happens when stencilop correctly invoked on pixel?

basically have mask:

00000110 

and value

00000110 

and try increment it, wrapped?

00000010 

or zeroed? (00000110 + 1) & mask

00000000 

section 17.4.2 "fine control of buffer updates" of opengl 4.5 core profile specification states:

the commands void stencilmask( uint mask ); void stencilmaskseparate( enum face, uint mask ); control writing of particular bits stencil planes. least significant s bits of mask, s number of bits in stencil buffer, specify integer mask. 1 appears in mask, corresponding bit in stencil buffer written; 0 appears, bit not written.

the glstencilmask() parameter controls bit planes written stencil buffer. not control read, or how glstencilop operates.

section 17.3.5 "stencil test" states (my emphasis):

for purposes of increment , decrement, stencil bits considered unsigned integer. incrementing or decrementing saturation clamps stencil value @ 0 , maximum representable value. incrementing or decrementing without saturation wrap such incrementing maximum representable value results in 0, , decrementing 0 results in maximum representable value.

the stencil mask not relevant in stage of pipeline. applied when fragment written framebuffer, gl*mask() functions.

so having value 0110 in buffer , applying gl_incr_wrap leads 0111 , when written buffer, mask applied, end 0110 again (and not 0).

also note there mask parameter in glstencilfunc() defining bit mask applied before stencil test. quoting section 17.3.5 again: (my emphasis):

stencilfunc , stencilfuncseparate take 3 arguments control whether stencil test passes or fails. ref integer reference value used in unsigned stencil comparison. stencil comparison operations , queries of ref clamp value range [0; 2^s - 1], s number of bits in stencil buffer attached draw framebuffer. the s least significant bits of mask bitwise anded both reference , stored stencil value, , resulting masked values participate in comparison controlled func.

so if want wrap around @ value 2^n-1, can ignore additional bits in stencil buffer , test these bits in stencil test.