Fragment shader: highp precision issue

This topic contains 1 reply, has 1 voice, and was last updated by  bert 6 years ago.

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
  • #30689


    I’m trying to do some image processing on the Nokia N9 (SGX 530). In a simple first step I want to convert a RGB image to grayscale with the following fragment shader:

        uniform sampler2D rgbImage;

        varying highp vec2 texCoord0;

        void main()


            highp vec4 color = texture2D(rgbImage, texCoord0);

            highp float luminance = 0.23 * color.r + 0.59 * color.g + 0.11 * color.b;

            gl_FragColor = vec4(luminance, 0.0, 0.0, 0.0);


    However, probably due to lower precision some pixels of the resulting grayscale image differ from the result of the same shader running in PVR GLES2 emulation on a desktop GPU. The differences are -1 or +1. If I leave out the add operations and only inspect how e.g. the red channel is modified by
    0.23 * color.r, similar errors occur. I also tried converting and rounding the texel values to integers in range [0 255] before multiplying the constant, multiplying then, rounding to the nearest integer again, and then dividing by 255 before setting gl_FragColor. Still, those +/-1 errors kept occuring.

    Since highp floats are supposed to have 32 bit precision [1], this sort of confuses me. Am I missing something here? Is this expected behavior?

    This is what
    glGetShaderPrecisionFormat() returns for GL_FRAGMENT_SHADER on the device:

        GL_LOW_FLOAT:    range: [1 1], precision: 8
        GL_MEDIUM_FLOAT: range: [14 14], precision: 10
        GL_HIGH_FLOAT:   range: [126 126], precision: 23

    bert2011-11-16 22:24:33



    Alright, this seems to be expected behavior. The PVR GLES2 emulator kind of reports 24 bits of precision for my desktop GPU. I reproduced the problem calculating the values in question by hand, each for a 23 and for a 24 bit significand, and it really came down to rounding errors. After all, that’s obvious, but since I’m not too savvy with floating point arithmetic I couldn’t believe it first…
    bert2011-11-16 23:35:43

Viewing 2 posts - 1 through 2 (of 2 total)
You must be logged in to reply to this topic.