3

I'm having a problem with color picking and antialiasing in OpenGL. When AA is activated results from glReadPixels are obviously wrong on object edges and object intersections. For example:

I render a box #28 (RGBA: 28, 0, 0, 0) near a box #32 (RGBA: 32, 0, 0, 0). With AA, I can get a wrong ReadPixel value (e.g. 30) where the cube and triangle overlap, or value of 14 on boxes edge, due to the AA algorithm.

I have ~4000 thousand objects I need to be able to pick (it's a jigsaw puzzle game). It is vital to be able to select objects by shape.

I've tried to disable AA with glDisable(GL_MULTISAMPLE) but it does not works with certain AA modes (I read it depends on AA implementation - SS, MS, CS ..)

So, how do I pick an underlying object?

  1. A way do temporary disable AA?
  2. Using a different buffer or even rendering context?
  3. Any other suggestion?
5
  • 1
    Most graphic drivers have option to force anti aliasing ignoring your settings. Try using FBO. May 25, 2011 at 11:21
  • How about implementing picking independent from the rasterizer/OpenGL? There are easy to use, high performant libraries for ray-triangle intersection testing, like OPCODE codercorner.com/Opcode.htm
    – datenwolf
    May 25, 2011 at 14:37
  • @datenwolf As he works with jigsaw puzzles (which I would implement as alpha tested quads or something similar), that won't work for him, I think. May 25, 2011 at 14:39
  • @Christian: But then the test was even easier outside OpenGL (translate the picking coordinate into a coordinate of the Jigsaw texture and test for alpha value there, bypassing a slow OpenGL readback roundtrip). Honestly I was thinking about a 3D pieces jigsaw puzzle.
    – datenwolf
    May 25, 2011 at 14:42
  • @datenwolf Ok, haven't thought about that. But still his color picking approach should be a bit easier, as without some space partitioning checking 4000 pieces won't be that efficient. May 25, 2011 at 16:36

2 Answers 2

7

Why not use an FBO as your pick buffer?

2
  • I've rigged up FBO with color and depth attachements (for picking and unprojecting) and it worked out great. Thanks!
    – Kromster
    May 26, 2011 at 17:44
  • 1
    The only drawback is that some GPUs don't support FBO and yet have AA, then I need to fallback to main buffer colorpicking and problem returns
    – Kromster
    May 13, 2012 at 18:06
2

I use this hack: pick not just one pixel, but all the 3x3=9 pixels around the picking point. If they are all same, we are safe. Otherwise, it must be on edge and we can skip that.

int renderer::pick_(int x, int y)
{
    static_assert(__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__,
            "only works on little-endian architecture");
    static_assert(sizeof(int) == 4,
            "only works on architecture that has int size of 4");

    // sort of edge detection. selection only happens at non-edge
    // since the edge may cause anti-aliasing glitch
    int ids[3*3];
    glReadPixels(x-1, y-1, 3, 3, GL_RGBA, GL_UNSIGNED_BYTE, ids);
    for (auto& id: ids) id &= 0x00FFFFFF;       // mask out alpha
    if (ids[0] == 0x00FFFFFF) return -1;        // pure white for background

    // prevent anti-aliasing glitch
    bool same = true;
    for (auto id: ids) same = (same && id == ids[0]);
    if (same) return ids[0];

    return -2;                                  // edge
}
1
  • Thanks, I was trying to figure out a method to deal with antialiasing in p5.js and this technique of sampling 3x3 is a very creative solution!
    – Andor
    Apr 6, 2022 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.