161

This is a somewhat low-level question. In x86 assembly there are two SSE instructions:

MOVDQA xmmi, m128

and

MOVNTDQA xmmi, m128

The IA-32 Software Developer's Manual says that the NT in MOVNTDQA stands for Non-Temporal, and that otherwise it's the same as MOVDQA.

My question is, what does Non-Temporal mean?

4
  • 8
    Note that SSE4.1 MOVNTDQA xmmi, m128 is an NT load, while all the other NT instructions are stores, except for prefetchnta. The accepted answer here only seems to be talking about stores. This is what I've been able to turn up about NT loads. TL:DR: hopefully the CPU does something useful with the NT hint to minimize cache pollution, but they don't override the strongly-ordered semantics of "normal" WB memory, so they do have to use the cache. Commented Nov 23, 2016 at 7:25
  • 7
    Update: NT loads may not do anything useful except on UCSW memory regions on most CPUs (e.g. Intel SnB family). NT/streaming stores definitely work on normal memory, though. Commented Jul 8, 2017 at 1:18
  • 4
    @Peter: You mean USWC memory right? I've never heard of UCSW or USWC memory before. Googling the wrong acronym wasn't helpful :-) Commented Aug 15, 2017 at 11:29
  • 5
    @AndrewBainbridge: Yes, the WC memory type attribute. Uncacheable Speculative Write-Combining. I think I was capitalizing UnCacheable and remembering that it was supposed to be 4 letters long. :P Commented Feb 26, 2019 at 20:00

4 Answers 4

194

Non-Temporal SSE instructions (MOVNTI, MOVNTQ, etc.), don't follow the normal cache-coherency rules. Therefore non-temporal stores must be followed by an SFENCE instruction in order for their results to be seen by other processors in a timely fashion.

When data is produced and not (immediately) consumed again, the fact that memory store operations read a full cache line first and then modify the cached data is detrimental to performance. This operation pushes data out of the caches which might be needed again in favor of data which will not be used soon. This is especially true for large data structures, like matrices, which are filled and then used later. Before the last element of the matrix is filled the sheer size evicts the first elements, making caching of the writes ineffective.

For this and similar situations, processors provide support for non-temporal write operations. Non-temporal in this context means the data will not be reused soon, so there is no reason to cache it. These non-temporal write operations do not read a cache line and then modify it; instead, the new content is directly written to memory.

Source: http://lwn.net/Articles/255364/

6
  • 18
    Nice answer, I would just like to point out that on the kind of processor with NT instructions, even with a non-non-temporal instruction (i.e. a normal instruction), the line cache is not "read and then modified". For a normal instruction writing to a line that is not in the cache, a line is reserved in the cache and a mask indicates what parts of the line are up-to-date. This webpage calls it "no stall on store": ptlsim.org/Documentation/html/node30.html . I couldn't find more precise references, I only heard about this from guys whose job is to implement processor simulators. Commented May 4, 2010 at 20:03
  • 2
    Actually ptlsim.org is a web site about a cycle-accurate processor simulator, exactly the same kind of thing the guys who told me about "no stall on store" are doing. I'd better mention them too in case they ever see this comment: unisim.org Commented May 4, 2010 at 20:06
  • 1
    From the answers and comments here stackoverflow.com/questions/44864033/… it seems SFENCE may be not needed. At least in the same thread. Could you also look? Commented Jul 2, 2017 at 10:32
  • 2
    @SergeRogatch it depends on what scenario you are talking about, but yes there are scenarios where sfence is required for NT stores, whereas it is never required just for normal stores. NT stores aren't ordered with respect to other stores (NT or not), as seen by other threads, without an sfence. For reads from the same thread that did the stores, however, you never need sfence: a given thread will always see its own stores in program order, regardless of whether they are NT stores or not.
    – BeeOnRope
    Commented Nov 8, 2018 at 18:59
  • Therefore non-temporal stores must be followed by an SFENCE instruction in order for their results to be seen by other processors in a timely fashion. I don't know why non-temporal stores must be followed by an SFENCE. Then non-temporal stores doesn't allow memory reordering? Commented Jun 11, 2021 at 7:51
48

Espo is pretty much bang on target. Just wanted to add my two cents:

The "non temporal" phrase means lacking temporal locality. Caches exploit two kinds of locality - spatial and temporal, and by using a non-temporal instruction you're signaling to the processor that you don't expect the data item be used in the near future.

I am a little skeptical about the hand-coded assembly that uses the cache control instructions. In my experience these things lead to more evil bugs than any effective performance increases.

4
  • question about "hand-coded assembly that uses the cache control instructions." I know you explicitly said "hand-coded" what about something like a JavaVM. Is this a better use case? The JavaVM/Compiler has analyzed the static and dynamic behavior of the program and uses these non-temporal instructions.
    – Pat
    Commented Dec 1, 2015 at 18:21
  • 10
    Exploiting known locality properties (or lack thereof) of your problem domain, algorithm or application shouldn't be shunned. Avoiding cache pollution is indeed a very attractive and effective optimisation task. Also, why the aversion toward assembly? There are vast amounts of opportunities for gains available which a compiler cannot possibly capitalise on
    – awdz9nld
    Commented Dec 21, 2015 at 23:44
  • 7
    It's definitely true that a knowledgeable low-level programmer can outperform a compiler for small kernels. This is great for publishing papers and blogposts and I've done both. They're also good didactic tools, and help understand what "really" going on. In my experience though, in practice, where you have a real system with many programmers working on it and correctness and maintainability are important, the benefit of low-level coding is almost always outweighed by the risks.
    – Pramod
    Commented Dec 22, 2015 at 22:23
  • 8
    @Pramod that same argument easily generalises to optimisation in general and is not really in scope of the discussion -- clearly that trade-off has already been considered or otherwise been deemed irrelevant given the fact that we are already talking about non-temporal instructions
    – awdz9nld
    Commented Oct 31, 2016 at 13:59
12

According to the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1: Basic Architecture, "Programming with Intel Streaming SIMD Extensions (Intel SSE)" chapter:

Caching of Temporal vs. Non-Temporal Data

Data referenced by a program can be temporal (data will be used again) or non-temporal (data will be referenced once and not reused in the immediate future). For example, program code is generally temporal, whereas, multimedia data, such as the display list in a 3-D graphics application, is often non-temporal. To make efficient use of the processor’s caches, it is generally desirable to cache temporal data and not cache non-temporal data. Overloading the processor’s caches with non-temporal data is sometimes referred to as "polluting the caches". The SSE and SSE2 cacheability control instructions enable a program to write non-temporal data to memory in a manner that minimizes pollution of caches.

Description of non-temporal load and store instructions. Source: Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 2: Instruction Set Reference

LOAD (MOVNTDQA—Load Double Quadword Non-Temporal Aligned Hint)

Loads a double quadword from the source operand (second operand) to the destination operand (first operand) using a non-temporal hint if the memory source is WC (write combining) memory type [...]

[...] the processor does not read the data into the cache hierarchy, nor does it fetch the corresponding cache line from memory into the cache hierarchy.

Note that, as Peter Cordes comments, it's not useful on normal WB (write-back) memory on current processors because the NT hint is ignored (probably because there are no NT-aware HW prefetchers) and the full strongly-ordered load semantics apply. prefetchnta can be used as a pollution-reducing load from WB memory

STORE (MOVNTDQ—Store Packed Integers Using Non-Temporal Hint)

Moves the packed integers in the source operand (second operand) to the destination operand (first operand) using a non-temporal hint to prevent caching of the data during the write to memory.

[...] the processor does not write the data into the cache hierarchy, nor does it fetch the corresponding cache line from memory into the cache hierarchy.

Using the terminology defined in Cache Write Policies and Performance, they can be considered as write-around (no-write-allocate, no-fetch-on-write-miss).

Finally, it may be interesting to review John McAlpin notes about non-temporal stores.

10
  • 6
    SSE4.1 MOVNTDQA only does anything special on WC (uncacheable Write-Combining) memory regions, e.g. video RAM. It's not at all useful on normal WB (write-back) memory on current HW, the NT hint is ignored and the full strongly-ordered load semantics apply. prefetchnta can be useful, though, as a pollution-reducing load from WB memory. Do current x86 architectures support non-temporal loads (from "normal" memory)?. Commented Nov 8, 2018 at 8:11
  • 3
    That's correct, NT stores work fine on WB memory, and are weakly-ordered, and are usually a good choice for writing large regions of memory. But NT loads aren't. The x86 manual on paper allows for the NT hint to do something for loads from WB memory, but in current CPUs it does nothing. (Probably because there are no NT-aware HW prefetchers.) Commented Nov 8, 2018 at 11:02
  • I have added that relevant info to the answer. Thank you very much.
    – chus
    Commented Nov 8, 2018 at 11:26
  • 2
    @LewisKelsey: NT stores do override the memory type. That's why they can be weakly ordered on WB memory. The major effect is avoiding RFOs (apparently they send an invalidate that even clears other dirty lines when they reach mem). They can also become visible out-of-order, so they don't have to wait until after an earlier cache-miss (regular) store commits, or until an earlier cache-miss load gets data. i.e. the kind of bottleneck asked about in Is memory outside each core always conceptually flat/uniform/synchronous in a multiprocessor system?. Commented May 25, 2019 at 3:47
  • 1
    @LewisKelsey: A memory ordering machine clear could kill any loads from after a UC store that shouldn't have been done early, if necessary. Other than that, commit order doesn't come into play until after the store retires from the out-of-order back end. That can't happen until after the store-address uop has executed, at which point the memory type for the address can be checked. A store-address uop checks the TLB when it executes; that's how CPUs can detect faulting stores before they retire. It can't wait until the SB entry is ready to commit to L1d; at that point execution is past it. Commented May 13, 2020 at 14:06
0

To add context to these answers:

When you do a memory copy, the processor will typically do a read from the source (RS), then do a read from the destination (RD), and finally, when the cacheline is flushed, a write to the destination (WD).

By using the non-temporal intrinsic, you are providing a hint that the RD may not need to occur, and possibly that the cacheline should be prioritized for eviction. Intel's own documentation doesn't mention the need to do fencing [https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#ig_expand=0,6555,6556&text=movnt], so this probably isn't something that you would need to worry about.

Intel's guidance on non temporal code suggests it may impact performance in unexpected ways, for instance:

/**
 * @brief GCM-AES Encryption using 128 bit keys, Non-temporal data
 *
 * Non-temporal version of encrypt has additional restrictions:
 * - The plaintext and cyphertext buffers must be aligned on a 64 byte boundary.
 * - In-place encryption/decryption is not recommended. Performance can be slow.
 *
 * @requires SSE4.1 and AESNI
 */

[https://github.com/intel/isa-l_crypto/blob/master/include/aes_gcm.h]

As a rule of thumb, you probably shouldn't use the non-temporal version of the MOV instructions unless you need it (i.e. avoid premature optimization). An easy way to visualize if optimizations are needed are tools like Intel Performance Monitor Counter [https://www.intel.com/content/www/us/en/developer/articles/tool/performance-counter-monitor.html]. This tool can show you your actual bandwidth to/from memory and if your expected usage doesn't match your actual usage the NT instructions might be a good fix.

Lastly, if you are in a position to only target newer Intel processors, just use AVX512, which reads and writes one cacheline at at a time. This avoids any chance of an extra RD without causing a potentially incorrect hint.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.