7

Does any body know how to find average luminosity for a texture in a fragment shader? I have access to both RGB and YUV textures the Y component in YUV is an array and I want to get an average number from this array.

3 Answers 3

7

I recently had to do this myself for input images and video frames that I had as OpenGL ES textures. I didn't go with generating mipmaps for these due to the fact that I was working with non-power-of-two textures, and you can't generate mipmaps for NPOT textures in OpenGL ES 2.0 on iOS.

Instead, I did a multistage reduction similar to mipmap generation, but with some slight tweaks. Each step down reduced the size of the image by a factor of four in both width and height, rather than the normal factor of two used for mipmaps. I did this by sampling from four texture locations that were in the middle of the four squares of four pixels each that made up a 4x4 area in the higher-level image. This takes advantage of hardware texture interpolation to average the four sets of four pixels, then I just had to average those four pixels to yield a 16X reduction in pixels in a single step.

I converted the image to luminance at the very first stage using a dot product of the RGB values with a vec3 of (0.2125, 0.7154, 0.0721). This allowed me to just read the red channel for each subsequent reduction stage, which really helps on iOS hardware. Note that you don't need this if you are starting with a Y channel luminance texture already, but I was dealing with RGB images.

Once the image had been reduced to a sufficiently small size, I read the pixels from that back onto the CPU and did a last quick iteration over the remaining few to arrive at the final luminosity value.

For a 640x480 video frame, this process yields a luminosity value in ~6 ms on an iPhone 4, and I think I can squeeze out a 1-2 ms reduction in that processing time with a little tuning. In my experience, that seems faster than the iOS devices normally generate mipmaps for power-of-two images at around that size, but I don't have solid numbers to back that up.

If you wish to see this in action, check out the code for the GPUImageLuminosity class in my open source GPUImage framework (and the GPUImageAverageColor superclass). The FilterShowcase example demonstrates this luminosity extractor in action.

9
  • Follow-up question: why did you pick a four-samples-at-a-time reduction? With eight varyings available it feels like you'd be able to sample down, say, an 8x4 area per step before hitting dependent texture reads.
    – Tommy
    Commented Aug 29, 2012 at 2:02
  • Thank you brad for your response it's complete and seems you've been in my shoe long time before, I think I had looked at GPUImage before and will look again for this solution. great job.
    – Rhm Akbari
    Commented Aug 29, 2012 at 2:12
  • @Tommy - Good question. The four sampling approach was something that the Core Image engineers had used for their color averaging filter (which they describe in this GPU Gems article: http.developer.nvidia.com/GPUGems3/gpugems3_ch26.html), so I wanted to replicate that first. You're right, though, in that if I really wanted to optimize this I should try to use all 8 of the relatively inexpensive texture reads. I might try that next. Commented Aug 29, 2012 at 2:19
  • That's interesting, especially the performance measurements. I just had the same task on the CPU (calculate the average luminosity of a UIImage). On an iPhone 4 my CPU based approach took exactly 6 ms. So there really should be room for optimization on the GPU. Commented Aug 29, 2012 at 8:36
  • @NikolaiRuhe - Thanks for the link to the sample project. With a 640x480 image (the size I used here), I see closer to 8 ms for the merge and 18 ms for the straight average on an iPhone 4. I should point out that ~2 ms of the 6 total in my benchmark is from capturing the image from the camera and displaying an average color onscreen, but I'm still surprised that the CG interpolation comes close to this. This gives me an idea for something else I could try. Commented Aug 29, 2012 at 18:10
3

You generally don't do this just with a shader.

One of the more common methods is to create a buffer texture with full mip-maps (down to 1x1, this is important). When you want to find luminosity, you copy the backbuffer to this buffer, then regenerate mips with a nearest neighbor algorithm. The bottom pixel will then have the average color of the entire surface and can be used to find average lum through something like (c.r * 0.6) + (c.g * 0.3) + (c.b * 0.1) (edit: if you have a YUV, then do similar and use the Y; the trick is just averaging the texture down to a single value, which is what mips do).

This isn't a precise technique, but is reasonably fast, especially on hardware that can generate mipmaps internally.

1
  • Thank you for your response Peachykeen your solution is great. I voted up.
    – Rhm Akbari
    Commented Aug 29, 2012 at 22:22
3

I'm presenting a solution for the RGB texture here as I'm not sure mip map generation would work with a YUV texture.

The first step is to create mipmaps for the texture, if not already present:

glGenerateMipmapOES(GL_TEXTURE_2D);

Now we can access the RGB value of the smallest mipmap level from the fragment shader by using the optional third argument of the sampler function texture2D, the "bias":

vec4 color = texture2D(sampler, vec2(0.5, 0.5), 8.0);

This will shift the mipmap level up eight levels, resulting in sampling a far smaller level.

If you have a 256x256 texture and render it with a scale of 1, a bias of 8.0 will effectively reduce the picked mipmap to the smallest 1x1 level (256 / 2^8 == 1). Of course you have to adjust the bias for your conditions to sample the smallest level.

OK, now we have the average RGB value of the whole image. The third step is to reduce RGB to a luminosity:

float lum = dot(vec3(0.30, 0.59, 0.11), color.xyz);

The dot product is just a fancy (and fast) way of calculating a weighted sum.

3
  • Thank you for your response Nikolia I am sure this very helpful for non iOS devices, but as I am on an iOS device should take other methods but this idea is very interesting.
    – Rhm Akbari
    Commented Aug 29, 2012 at 2:07
  • 1
    @robcyr - I should point out that both Nikolai and peachykeen's answers are perfectly viable on iOS. My answer is merely an alternative approach. By stretching an NPOT texture to the next power-of-two size and then generating a mipmap from it, you'd get very close to what I suggest. I need to benchmark against mipmap generation to see if my approach has any performance advantages. Commented Aug 29, 2012 at 2:39
  • 1
    and if you stretch down to the next PoT, you can get a head-start and skip a mip level. I'd be interested in seeing what the benchmarks turn up, since generating mips is usually acceptably fast.
    – ssube
    Commented Aug 29, 2012 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.