62

I read this article. It talked about why AVX-512 instruction:

Intel’s latest processors have advanced instructions (AVX-512) that may cause the core, or maybe the rest of the CPU to run slower because of how much power they use.

I think on Agner's blog also mentioned something similar (but I can't find the exact post).

I wonder what other instructions supported by Skylake have the similar effect that they will lower the power to maximize the throughput later? All the v prefixed instructions (such as vmovapd, vmulpd, vaddpd, vsubpd, vfmadd213pd)?

I am trying to compile a list of instructions to avoid when compiling my C++ application for Xeon Skylake.

16
  • 2
    Trevis Down (aka Beeonrope on OS) wrote about this in the comments in this post and continued the discussion here. He found that each ties (scalar, AVX/AVX2, AVX-512) has "cheap" (no FP, simple operations) instructions and "heavy" instruction. Cheap instructions drop the frequency to the one of the next higher tier (e.g. cheap AVX-512 inst use the AVX/AVX2 tier) even if used sparsely. Heavy inst must be used more than 1 every ... Commented Jul 2, 2019 at 13:06
  • 2
    ... two cycles and drop the frequency according to their tier (e.g. AVX-512 heavy instrs drop the frequency to the AV-512 base). Travis also shared the code he used to test here. You can find the behaviour of each instruction with a bit of patience or by his rule of thumb. Finally note that this frequency scaling is a problem iif the ratio of vector to scalar instruction is low enough so that the drop in frequency is not balanced by the bigger width at which data is processed. Check the final binary to see if you really gained anything. Commented Jul 2, 2019 at 13:10
  • 1
    @HCSF You can make three builds, one without AVX, one with AVX/AVX2 and one with AVX-512 (if applicable) and profile them. Then take the fastest one. Commented Jul 2, 2019 at 14:52
  • 1
    Peter mentioned the -mpreferred-vector-width=256 option. I don't know if it prevents gcc from ever producing AVX-512 instructions (outside of direct intrinsic use), but it is certainly possible. I am not aware of any option which distinguishes between "heavy" and "light" instructions however. Usually this isn't a problem, since if you turn off AVX-512 and don't have a bunch of FP ops, you are probably targeting L0 anyways, and AVX-512 light is still L1.
    – BeeOnRope
    Commented Jul 3, 2019 at 6:57
  • 1
    @HCSF important routines in libc are generally compiled multiple times for different ISAs and then the version appropriate for the current CPU is selected at runtime using the dynamic loader's IFUNC capability. So you'll usually get a version optimized for your CPU (unless your libc is quite old and your CPU quite new).
    – BeeOnRope
    Commented Jul 4, 2019 at 0:16

2 Answers 2

92

On Intel chips, the frequency impact and the specific frequency transition behavior depends on both the width of the operation and the specific instruction used.

As far as instruction-related frequency limits go, there are three frequency levels – so-called licenses – from fastest to slowest: L0, L1 and L2. L0 is the "nominal" speed you'll see written on the box: when the chip says "3.5 GHz turbo", they are referring to the single-core L0 turbo. L1 is a lower speed sometimes called AVX turbo or AVX2 turbo5, originally associated with AVX and AVX2 instructions1. L2 is a lower speed than L1, sometimes called "AVX-512 turbo".

The exact speeds for each license also depend on the number of active cores. For up to date tables, you can usually consult WikiChip. For example, the table for the Xeon Gold 5120 is here:

Xeon Gold 5120 Frequencies

The Normal, AVX2 and AVX512 rows correspond to the L0, L1 and L2 licenses respectively. Note that the relative slowdown for L1 and L2 licenses generally gets worse as the number of cores increase: for 1 or 2 active cores the L1 and L2 speeds are 97% and 91% of L0, but for 13 or 14 cores they are 85% and 62% respectively. This varies by chip, but the general trend is usually the same.

Those preliminaries out of the way, let's get to what I think you are asking: which instructions cause which licenses to be activated?

Here's a table, showing the implied license for instructions based on their width and their categorization as light or heavy:

   Width    Light   Heavy  
 --------- ------- ------- 
  Scalar    L0      N/A
  128-bit   L0      L0     
  256-bit   L0      L1*    
  512-bit   L1      L2*

*soft transition (see below)

So we immediately see that all scalar (non-SIMD) instructions and all 128-bit wide instructions2 always run at full speed in the L0 license.

256-bit instructions will run in L0 or L1, depending on whether they are light or heavy, and 512-bit instructions will run in L1 or L2 on the same basis.

So what is this light and heavy thing?

Light vs Heavy

It's easiest to start by explaining heavy instructions.

Heavy instructions are all SIMD instructions that need to run on the FP/FMA unit. Basically that's the majority of the FP instructions (those usually ending in ps or pd, like addpd) as well as integer multiplication instructions which largely start with vpmul or vpmad since SIMD integer multiplication actually runs on the SIMD unit, as well as vplzcnt(q|d) which apparently also runs on the FMA unit.

Given that, light instructions are everything else. In particular, integer arithmetic other than multiplication, logical instructions, shuffles/blends (including FP) and SIMD load and store are light.

Transitions

The L1 and L2 entries in the Heavy column are marked with an asterisk, like L1*. That's because these instructions cause a soft transition when they occur. The other L1 entry (for 512-bit light instructions) causes a hard transition. Here we'll discuss the two transition types.

Hard Transition

A hard transition occurs immediately as soon as any instruction with the given license executes4. The CPU stops, takes some halt cycles and enters the new mode.

Soft Transition

Unlike hard transitions, a soft transition doesn't occur immediately as soon as any instruction is executed. Rather, the instructions initially execute with a reduced throughput (as slow as 1/4 their normal rate), without changing the frequency. If the CPU decides that "enough" heavy instructions are executing per unit time, and a specific threshold is reached, a transition to the higher-numbered license occurs.

That is, the CPU understands that if only a few heavy instructions arrive, or even if many arrive but they aren't dense when considering other non-heavy instructions, it may not be worth reducing the frequency.

Guidelines

Given the above, we can establish some reasonable guidelines. You never have to be scared of 128-bit instructions, since they never cause license related3 downclocking.

Furthermore, you never have to be worried about light 256-bit wide instructions either, since they also don't cause downclocking. If you aren't using a lot of vectorized FP math, you aren't likely to be using heavy instructions, so this would apply to you. Indeed, compilers already liberally insert 256-bit instructions when you use the appropriate -march option, especially for data movement and auto-vectorized loops.

Using heavy AVX/AVX2 instructions and light AVX-512 instructions is trickier, because you will run in the L1 licenses. If only a small part of your process (say 10%) can take advantage, it probably isn't worth slowing down the rest of your application. The penalties associated with L1 are generally moderate - but check the details for your chip.

Using heavy AVX-512 instructions is even trickier, because the L2 license comes with serious frequency penalties on most chips. On the other hand, it is important to note that only FP and integer multiply instructions fall into the heavy category, so as a practical matter a lot of integer 512-bit wide use will only incur the L1 license.


1 Although, as we'll see, this a bit of a misnomer because AVX-512 instructions can set the speed to this license, and some AVX/2 instructions don't.

2 128-bit wide means using xmm registers, regardless of what instruction set they were introduced in - mainstream AVX-512 contains 128-bit variants for most/all new instructions.

3 Note the weasel clause license related - you may certainly suffer other causes of downclocking, such as thermal, power or current limits, and it is possible that 128-bit instructions could trigger this, but I think it is fairly unlikely on a desktop or server system (low power, small form factor devices are another matter).

4 Evidently, we are talking only about transitions to a higher-level license, e.g., from L0 to L1 when a hard-transition L1 instruction executes. If you are already in L1 or L2 nothing happens - there is no transition if you are already in the same level and you don't transition to lower-numbered levels based on any specific instruction but rather running for a certain time without any instructions of the higher-numbered level.

5 Out of the two AVX2 turbo is more common, which I never really understood because 256-bit instructions are as much associated with AVX as compared to AVX2, and most of the heavy instructions which actually trigger AVX turbo (L1 license) are actually FP instructions in AVX, not AVX2. The only exception is AVX2 integer multiplies.

15
  • Comments are not for extended discussion; this conversation has been moved to chat. Commented Jul 3, 2019 at 5:35
  • 1
    @Zboson - yeah it would be interesting to explore exactly when the transition takes place. I think @ Mysticial has said that the transition function isn't that smart, i.e., that it decides to make a transition to a slower speed even when the stready-state code will objectively run slower after the transition (e.g., code with a 50/50 mix of FMA and non-FMA would be better of not transitioning since you only need ~1 FMA, you can get in the faster license but instead it transitions).
    – BeeOnRope
    Commented Aug 2, 2019 at 16:10
  • 1
    @Zboson - I think it first showed up in the Haswell server chips, i.e., Haswell-EP or whatever it was called. The name AVX2 turbo speed never made much sense to me: it mostly affects FP instructions from the AVX set, not AVX2 which was mostly integer (integer mul is an exception). Intel themselves use AVX, not AVX2 in early documents. People seem to like to call it AVX2 though, maybe because it came out in Haswell where AVX2 was the new ISA?
    – BeeOnRope
    Commented Aug 2, 2019 at 16:17
  • 1
    @BeeOnRope Might be changing for Saphire Rapids. There don't appear to be any license transition events anymore.
    – Noah
    Commented Apr 24, 2022 at 0:26
  • 1
    @user997112 also FWIW, I know we plan to enable avx512 by default for SPR in GLIBC because the freq throttling isn't a concern although thats only for the "light" instructions.
    – Noah
    Commented Jul 4, 2023 at 2:58
17

It's not the instruction mnemonic that matters, it's 512-bit vector width at all that matters.

You can use the 256-bit version of AVX-512VL instructions, e.g. vpternlogd ymm0, ymm1, ymm2 without incurring the AVX-512 turbo penalty.

Related: Dynamically determining where a rogue AVX-512 instruction is executing is about a case where one AVX-512 instruction in glibc init code or something left a dirty upper ZMM that gimped max turbo for the rest of the process lifetime. (Or until a vzeroupper maybe)

Although there can be other turbo impacts from light / heavy use of 256-bit FP math instructions, and some of that is due to heat. But usually 256-bit is worth it on modern CPUs.

Anyway, this is why gcc -march=skylake-avx512 defaults to -mprefer-vector-width=256. For any given workload, it's worth trying -mprefer-vector-width=512 and maybe also 128, depending on how much or how little of the work can usefully auto-vectorize.

Tell GCC to tune for your CPU (e.g. -march=native) and it will hopefully make good choices. Although on a desktop Skylake-X, the turbo penalty is smaller than a Xeon. And if your code does actually benefit from 512-bit vectorization, it can be worth it to pay the penalty.

(Also beware the other major effect of Skylake-family CPUs going into 512-bit vector mode: the vector ALUs on port 1 shut down, so only scalar instructions like popcnt or add can use port 1. So vpand and vpaddb etc. throughput drops from 3 to 2 per clock. And if you're on an SKX with two 512-bit FMA units, the extra one on port 5 powers up, so then FMAs compete with shuffles.)

21
  • I have been using -march=generic for a long time for my binary. So I think even -march=skylake-avx512 -mpreferred-vector-width=128 would make some optimization kick in without the heavy penalty from using avx-256 (as I ask for 128). Thought?
    – HCSF
    Commented Jul 3, 2019 at 7:19
  • @HCSF: Well sure, skylake + width=128 should be strictly better than generic for running on SKX. GCC could do worse if it bloats the code-size with AVX512 EVEX-encoded instructions unnecessarily (e.g. vmovdqu64 xmm instead of vmovdqu xmm, when not using xmm16..31), and generally compare-into-mask should be good vs. the SSE/AVX way of compare-into-vector and blend. But you should definitely test with the default width=256, too, in case the turbo penalty is worth it for your code. Doing twice as much work per uop is very good, and the big penalties only kick in with 512-bit vectors. Commented Jul 3, 2019 at 7:34
  • 1
    I actually see what you just mentioned -- vmovdqu64 (%rdx),%xmm0, vmovdqu64 0x10(%rsi),%xmm6, etc when I compiled with -march=skylake-avx512 -mprefer-vector-width=128. It seems like GCC 8.2 isn't doing it right (or not what you expected)?
    – HCSF
    Commented Jul 3, 2019 at 7:39
  • 1
    @user997112: I haven't looked into any changes in the interconnect since Ice Lake, although chipsandcheese.com/2023/03/12/a-peek-at-sapphire-rapids did some cache and memory latency testing back in March. The cores and L1 cache should be identical to Alder Lake P-cores (except with a second 512-bit FMA unit in some models), so uops.info and Agner Fog's stuff should be applicable for execution ports. Intel's optimization guide might have some stuff. Commented Nov 6, 2023 at 17:47
  • 1
    @user997112: I'm skeptical of that microbenchmark result. chipsandcheese.com/2023/11/07/… reports worst-case core-to-core latency of 81ns within one socket, and it looks like 155 ns worst-cse across sockets. (Average 59 and 138 ns). Maybe they're testing differently, like with only those 2 cores active at once, vs. with all cores actively spamming inter-core traffic creating more contention in the mesh interconnect and the links between sockets? (Real workloads don't spend all their time on inter-core transfers.) Commented Nov 7, 2023 at 23:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.