3

I'm writing a device driver in linux for a PCIe device. This device driver performs several read and write to test the throughput. When I use the memcpy, the maximum payload for a TLP is 8 bytes ( on 64 bits architectures ). In my opinion the only way to get a payload of 16 bytes is to use the SSE instruction set. I've already seen this but the code doesn't compile ( AT&T/Intel syntax issue ).

  • There is a way to use that code inside linux ?
  • Does anyone know where I can found an implementation of a memcpy that moves 128 bits ?
  • 2
  • 4
    MMX is only 64 bits (bytes). ITYM SSE, which is 128 bits (16 bytes). – Paul R Nov 30 '15 at 16:35
  • I'm not really knowledgeable on this topic, but can't you use DMA for this ? – ElderBug Nov 30 '15 at 16:37
  • 1
    @haster8558: "doesn't work" doesn't mean anything. Anyhow, unless you are compiling just your code (and not using any header which has asm blocks) you should use .intel_syntax noprefix at the start of your asm block and .att_syntax noprefix at the end. – Matteo Italia Nov 30 '15 at 17:04
  • 3
    You should investigate whether the Linux kernel has it's own fast memcpy function you can use. It may even be tuned to the particular CPU it runs on at boot time. – Ross Ridge Nov 30 '15 at 21:12
7

First of all you probably use GCC as the compiler and it uses the asm statement for inline assembler. When using that you will have to use a string literal for the assembler code (which will be copied into the assembler code before sending to the assembler - this means that the string should contain newline characters).

Second you will probably have to use AT&T syntax for the assembler.

Third GCC uses extended asm to pass variables between assembler and C.

Fourth you should probably avoid inline assembler when possible anyway as the compiler wont have the possibility to schedule instructions past an asm statement (this was true at least). Instead you could maybe make use of GCC extensions like the vector_size attribute:

typedef float v4sf __attribute__((vector_size(16)));

void fubar( v4sf *p, v4sf* q )
{
  v4sf p0 = *p++;
  v4sf p1 = *p++;
  v4sf p2 = *p++;
  v4sf p3 = *p++;

  *q++ = p0;
  *q++ = p1;
  *q++ = p2;
  *q++ = p3;
}

has the advantage that the compiler will produce code even if you compile for a processor that doesn't have the mmx registers, but perhaps some other 128-bit registers (or doesn't have vector registers at all).

Fifth you should investigate if the provided memcpy isn't fast enough. Often the memcpy is really optimized.

Sixth you should take precaution if you're using special registers in the Linux kernel, there are registers that aren't saved during context switch. The SSE registers are a part of these.

Seventh as you using this to test throughput you should consider if the processor is a significant bottleneck in the equation. Compare the actual execution of the code with the reads from/writes to RAM (do you hit or miss the cache?) or the reads from/write to the peripheral.

Eighth when moving data you should avoid moving big chunks of data from RAM to RAM and if it's to/from a peripheral that has limited bandwidth you should definitely consider using DMA for that. Remember that if it's access time that limits the performance the CPU will still be considered busy (although it can't run at 100% speed).

  • 1
    You can't use SSE in the Linux kernel without taking extra precautions. Vector regs aren't saved/restored unless they have to be, because normal kernel code doesn't touch them. – Peter Cordes Dec 1 '15 at 3:04
  • @PeterCordes I wasn't considering that much that this were in kernel space, I've updated the answer to point that out. – skyking Dec 1 '15 at 6:21
  • Just few clarification. The problem isn't the speed of the standard memcpy, the problem is that standard memcpy doesn't allow transfer of 16 bytes on PCIe bus. The bottleneck is the CPU, I'm sure about this because I'm using a protocol analyzer. In windows, on the same hardware, I can see a payload of 16 bytes, in linux I don't. The only idea I got is SSE instruction set. – haster8558 Dec 1 '15 at 7:36
  • @haster8558 Have you ruled out the possibility that the RAM may be a bottle neck? – skyking Dec 1 '15 at 8:05
  • There isn't an evidence that the RAM could be involved in the transfer. Then the write throughput for a PCIe X1 is almost 70 MB/s on linux, and it almost double in windows and the hardware is the same. On the same hardware I want the same behaviour. The bottleneck is the CPU indeed. – haster8558 Dec 1 '15 at 8:30
4

Leaving this answer here for now, even though it's now clear the OP just wants a single 16B transfer. On Linux, his code is causing two 8B transfers over the PCIe bus.

For writing to MMIO space, it's worth trying movnti write-combining-store instructions. The source operand for movnti is a GP register, not a vector reg.

You can probably generate that with intrinsics, if you #include <immintrin.h> in your driver code. That should be fine in the kernel, as long as you're careful about what intrinsics you use. It doesn't define any globals.


So most of this section isn't very relevant.

On most CPUs (where rep movs is good), Linux's memcpy uses it. It only uses a fallback to an explicit loop for CPUs where rep movsq or rep movsb are not good choices.

When the size is a compile-time-constant, memcpy has an inline implementation using rep movsl (AT&T syntax for rep movsd), then for cleanup: non-rep movsw and movsb if needed. (Actually kinda clunky, IMO, since the size is a compile-time constant. Also doesn't take advantage of fast rep movsb on CPUs that have it.)

Intel CPUs since P6 have had at least fairly good rep movs implementations. See Andy Glew's comments on it.

But still, you're wrong about memcpy only moving in 64bit blocks, unless I'm misreading the code or you're on a platform where it decides to use the fallback loop.

Anyway, I don't think you're missing out on much perf by using the normal Linux memcpy, unless you've actually single-stepped your code and seen it doing something silly.

For large copies, you'll want to set up DMA anyway. CPU usage by your driver is important, not just the max throughput you can obtain on an otherwise-idle system. (Be careful of trusting microbenchmarks too much.)


Using SSE in the kernel means saving/restoring the vector registers. It's worth it for the RAID5/RAID6 code. That code may only run from a dedicated thread, rather than from contexts where the vector/FPU registers still have another process's data.

Linux's memcpy can be used from any context, so it avoids using anything but the usual integer registers. I did find an article about an SSE kernel memcpy patch, where Andi Kleen and Ingo Molnar both say it wouldn't be good to always use SSE for memcpy. Maybe there could be a special bulk-memcpy for big copies where it's worth saving the vector regs.

You can use SSE in the kernel, but you have to wrap it in kernel_fpu_begin() and kernel_fpu_end(). On Linux 3.7 and later, kernel_fpu_end() actually does the work of restoring FPU state, so don't use a lot of fpu_begin/fpu_end pairs in a function. Also note that kernel_fpu_begin disables pre-emption, and you must not "do anything that might fault or sleep".

In theory, saving just one vector reg, like xmm0, would be good. You'd have to make sure you used SSE, not AVX instructions, because you need to avoid zeroing the upper part of ymm0 / zmm0. You might cause an AVX+SSE stall when you return to code that was using ymm regs. Unless you want to do a full save of the vector regs, you can't run vzeroupper. And even to do that, you'd need to detect AVX support...

However, doing even this one-reg save/restore would require you to take the same precautions as kernel_fpu_begin, and disable pre-emption. Since you'd be storing to your own private save slot (prob. on the stack), rather than to task_struct.thread.fpu, I'm not sure that even disabling pre-emption is enough to guarantee that user-space FPU state won't be corrupted. Maybe it is, but maybe it isn't, and I'm not a kernel hacker. Disabling interrupts to guard against this, too, is probably worse than just using kernel_fpu_begin()/kernel_fpu_end() to trigger a full FPU state save using XSAVE/XRSTOR.

  • Thanks, I try to explain better. I've a FPGA linked to my CPU through a PCIe bus. Between CPU and FPGA there is a PCIe Protocol Analyzer. I've seen in windows that when ( in the device driver ) I call a memcpy with a block size bigger that 16 bytes, the payload of memory write is 16 bytes. In linux I can't see the same thing. In windows I can see this behaviour only in device driver built with WDF. About the fast of memcpy I don't care, because CPU takes the same time to generate the TLP ( with 8 or 16 payload ) for the roocomplex, so there isn't any differences. – haster8558 Dec 1 '15 at 7:54
  • Of course a DMA transfer is the solution for big chunc of data, but I want to be able to transfer 16 byte with only one strike. The DMA has to be programmed. It means to send at least 2 TLP to the endpoint and the wait for an interrupt. – haster8558 Dec 1 '15 at 7:57
  • @haster8558: Right, for transfering just 16B, obviously you don't want to DMA that. Your question left out way too much detail for anyone to figure out what kind of thing you were doing. Everyone assumed you were talking about bulk transfers, not a single 128b. I don't write device drivers myself, so I don't even know what a TLP is. I'm not sure what happens when you rep movsq to or from MMIO space, rather than between two normal writeback memory regions. But it sounds like you get 64bit chunks :/ – Peter Cordes Dec 1 '15 at 8:01
  • 1
    It would be interested to see some experimental evidence. I tested glibc's memcpy and gcc's built-in memcpy and found it was easy to beat them in some cases. I basically redid the tests Agner Fog did in his optimizing C++ manual. I would be surprised if Linux's memcpy is much better. I'm willing to bet for sizes much larger than the last level cache that using non-temporal stores and multiple threads will beat Linux's memcpy. – Z boson Dec 1 '15 at 8:08
  • 1
    @haster8558: I meant machine code, not high-level source... I'm wondering if Windows has a trick to do it without SSE, or if the Windows driver does just use SSE. I've never written Windows kernel / driver code, and don't know anything about it. Try disassembling the relevant function in the windows driver. – Peter Cordes Dec 1 '15 at 11:41
3

The link you mentioned is using non-temporal stores. I have discussed this several times before, for example here and here. I would suggest your read those before proceeding further.

But if you really want to produce the inline assembly code in the link you mentioned here is how you do it: use intrinsics instead.

The fact that you cannot compile that code with GCC is exactly one of the reasons intrinsics were created. Inline assembly has to be written differently for 32-bit and 64-bit code and typically has different syntax for each compiler. Intrinsics solve all these issues.

The following code should compile with GCC, Clang, ICC, and MSVC in both 32-bit and 64-bit mode.

#include "xmmintrin.h"
void X_aligned_memcpy_sse2(char* dest, const char* src, const unsigned long size)
{
    for(int i=size/128; i>0; i--) {
        __m128i xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7;
        _mm_prefetch(src + 128, _MM_HINT_NTA);
        _mm_prefetch(src + 160, _MM_HINT_NTA);
        _mm_prefetch(src + 194, _MM_HINT_NTA);
        _mm_prefetch(src + 224, _MM_HINT_NTA);

        xmm0 = _mm_load_si128((__m128i*)&src[   0]);
        xmm1 = _mm_load_si128((__m128i*)&src[  16]);
        xmm2 = _mm_load_si128((__m128i*)&src[  32]);
        xmm3 = _mm_load_si128((__m128i*)&src[  48]);
        xmm4 = _mm_load_si128((__m128i*)&src[  64]);
        xmm5 = _mm_load_si128((__m128i*)&src[  80]);
        xmm6 = _mm_load_si128((__m128i*)&src[  96]);
        xmm7 = _mm_load_si128((__m128i*)&src[ 112]);

        _mm_stream_si128((__m128i*)&dest[   0], xmm0);
        _mm_stream_si128((__m128i*)&dest[  16], xmm1);
        _mm_stream_si128((__m128i*)&dest[  32], xmm2);
        _mm_stream_si128((__m128i*)&dest[  48], xmm3);
        _mm_stream_si128((__m128i*)&dest[  64], xmm4);
        _mm_stream_si128((__m128i*)&dest[  80], xmm5);
        _mm_stream_si128((__m128i*)&dest[  96], xmm6);
        _mm_stream_si128((__m128i*)&dest[ 112], xmm7);
        src  += 128;
        dest += 128;
    }
}

Note that src and dest need to be 16 byte aligned and that size needs to be a multiple of 128.

I don't, however, advice to use this code. In the cases when non-temporal stores are useful loop unrolling is useless and explicit pre-fetching is rarely ever useful. You can simply do

void copy(char *x, char *y, int n)
{
    #pragma omp parallel for schedule(static)
    for(int i=0; i<n/16; i++) {
        _mm_stream_ps((float*)&y[16*i], _mm_load_ps((float*)&x[16*i]));
    }
}

more details as to why can be found here.


Here is the assembly from the X_aligned_memcpy_sse2 function using intrinsics with GCC -O3 -S -masm=intel. Notice that it's essentially the same as here.

    shr rdx, 7
    test    edx, edx
    mov eax, edx
    jle .L1
.L5:
    sub rsi, -128
    movdqa  xmm6, XMMWORD PTR [rsi-112]
    prefetchnta [rsi]
    prefetchnta [rsi+32]
    prefetchnta [rsi+66]
    movdqa  xmm5, XMMWORD PTR [rsi-96]
    prefetchnta [rsi+96]
    sub rdi, -128
    movdqa  xmm4, XMMWORD PTR [rsi-80]
    movdqa  xmm3, XMMWORD PTR [rsi-64]
    movdqa  xmm2, XMMWORD PTR [rsi-48]
    movdqa  xmm1, XMMWORD PTR [rsi-32]
    movdqa  xmm0, XMMWORD PTR [rsi-16]
    movdqa  xmm7, XMMWORD PTR [rsi-128]
    movntdq XMMWORD PTR [rdi-112], xmm6
    movntdq XMMWORD PTR [rdi-96], xmm5
    movntdq XMMWORD PTR [rdi-80], xmm4
    movntdq XMMWORD PTR [rdi-64], xmm3
    movntdq XMMWORD PTR [rdi-48], xmm2
    movntdq XMMWORD PTR [rdi-128], xmm7
    movntdq XMMWORD PTR [rdi-32], xmm1
    movntdq XMMWORD PTR [rdi-16], xmm0
    sub eax, 1
    jne .L5
.L1:
    rep ret
  • Thank you so much. Just a doubt, one of my colleague told me that intrinsics couldn't be use inside a kernel mode. Do I misunderstanding ? – haster8558 Dec 1 '15 at 7:44
  • 1
    @haster8558, I don't know. Why did you your colleagues say that? The only thing I could imagine is that the compiler converted the intrinsic into instructions that are not compatible with the kernel. You have to look at the assembly produced. I guess that's one disadvantage of intrinsics in that you can't guarantee you get exactly the same code with different compilers or even different version of a compiler. – Z boson Dec 1 '15 at 8:04
  • The problem is the xmmintrin.h include stdlib.h and there are several problem during compile. – haster8558 Dec 1 '15 at 8:33
  • is it possible to see the X_aligned_memcpy_sse2 disassembled in AT&T syntax ? – haster8558 Dec 1 '15 at 10:58
  • @haster8558, yes gcc -O3 -S foo.c. Note again that I don´t recommend using this code. I just wanted to illustrate how to do this with intrinsics. – Z boson Dec 1 '15 at 11:01

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.