7

I'm looking for the fastest way to test if a 128 NEON register contains all zeros, using NEON intrinsics. I'm currently using 3 OR operations, and 2 MOVs:

uint32x4_t vr = vorrq_u32(vcmp0, vcmp1);

uint64x2_t v0 = vreinterpretq_u64_u32(vr);
uint64x1_t v0or = vorr_u64(vget_high_u64(v0), vget_low_u64(v0));

uint32x2_t v1 = vreinterpret_u32_u64 (v0or);
uint32_t r = vget_lane_u32(v1, 0) | vget_lane_u32(v1, 1);

if (r == 0) { // do stuff }

This translates by gcc to the following assembly code:

VORR     q9, q9, q10
VORR     d16, d18, d19
VMOV.32  r3, d16[0]
VMOV.32  r2, d16[1]
VORRS    r2, r2, r3
BEQ      ...

Does anyone have an idea of a faster way?

5 Answers 5

7

While this answer may be a bit late, there is a simple way to do the test with only 3 instructions and no extra registers:

inline uint32_t is_not_zero(uint32x4_t v)
{
    uint32x2_t tmp = vorr_u32(vget_low_u32(v), vget_high_u32(v));
    return vget_lane_u32(vpmax_u32(tmp, tmp), 0);
}

The return value will be nonzero if any bit in the 128-bit NEON register was set.

2

If you're targeting AArch64 NEON, you can use the following to get a value to test with just two instructions:

inline uint64_t is_not_zero(uint32x4_t v)
{
    uint64x2_t v64 = vreinterpretq_u64_u32(v);
    uint32x2_t v32 = vqmovn_u64(v64);
    uint64x1_t result = vreinterpret_u64_u32(v32);
    return result[0];
}
1

You seem to be looking for intrinsics and this is the way:

inline bool is_zero(int32x4_t v) noexcept
{
  v = v == int32x4{};

  return !int32x2_t(
    vtbl2_s8(
      int8x8x2_t{
        int8x8_t(vget_low_s32(v)),
        int8x8_t(vget_high_s32(v))
      },
      int8x8_t{0, 4, 8, 12}
    )
  )[0];
}

Nils Pipenbrinck's answer has a flaw in that he assumes the QC, cumulative saturation flag to be clear.

1

If you have AArch64 you can do it even easier. They have a new instruction for designed for this.

inline uint32_t is_not_zero(uint32x4_t v)
{
    return vaddvq_u32(v);
}
3
  • 1
    This seems vulnerable to overflows, though.
    – wolfv
    Mar 9, 2018 at 13:59
  • If you are using the full 32 bit, yes, it can wrap around and cause a false positive, but if you know your numbers are smaller than 2³²/4 it is safe. Apr 7, 2018 at 20:20
  • 4
    Probably better though to just use vmaxvq_u32 then. A similar reduction instruction, but one returning the maximum value instead of the sum. Apr 7, 2018 at 20:24
0

I'd avoid functions returning integer values that should only be interpreted as bool. A better way would be, for instance, defining a helper function to return maximum unsigned value of 4 lanes:

inline uint32_t max_lane_value_u32(const uint32x4_t& v)
{
#if defined(_WIN32) && defined(_ARM64_)
    // Windows 64-bit
    return neon_umaxvq32(v);
#elif defined(__LP64__)
    // Linux/Android 64-bit
    return vmaxvq_u32(v);
#else
    // Windows/Linux/Android 32-bit
    uint32x2_t result = vmax_u32(vget_low_u32(v), vget_high_u32(v));
    return vget_lane_u32(vpmax_u32(result, result), 0);
#endif
}

you can then use:

if (0 == max_lane_value_u32(v))
{
    ...
}

in your code, and such function might also be useful elsewhere. Alternatively, you can use the exact same code to write a is_not_zero() function, but then it's best form to return a bool.

Note that the only reason you'd need to define a helper function is because vmaxvq_u32() is not available on 32-bit, and may not be aliased from neon_umaxvq32() in arm64_neon.h on Windows.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.