{"id":13877,"date":"2025-08-19T00:23:54","date_gmt":"2025-08-18T22:23:54","guid":{"rendered":"https:\/\/monodes.com\/predaelli\/?p=13877"},"modified":"2025-08-19T00:23:58","modified_gmt":"2025-08-18T22:23:58","slug":"going-faster-than-memcpy","status":"publish","type":"post","link":"https:\/\/monodes.com\/predaelli\/2025\/08\/19\/going-faster-than-memcpy\/","title":{"rendered":"Going faster than memcpy"},"content":{"rendered":"\n

Going faster than memcpy<\/a><\/h2>\n\n\n\n\n\n\n\n
\n

Going faster than memcpy<\/h1>\n\n\n\n

While profiling Shadesmar<\/a> a couple of weeks ago, I noticed that for large binary unserialized messages (>512kB) most of the execution time is spent doing copying the message (using memcpy<\/code>) between process memory to shared memory and back.<\/p>\n\n\n\n

I had a few hours to kill last weekend, and I tried to implement a faster way to do memory copies.<\/p>\n\n\n\n

\n\n\n\n

<\/a>Autopsy of memcpy<\/h3>\n\n\n\n

Here\u2019s the dumb of perf<\/code><\/a> when running pub-sub for messages of sizes between 512kB and 2MB.<\/p>\n\n\n\n

 Children      Self  Shared Object      Symbol\n+  99.86%     0.00%  libc-2.27.so       [.] __libc_start_main\n+  99.86%     0.00%  [unknown]          [k] 0x4426258d4c544155\n+  99.84%     0.02%  raw_benchmark      [.] main\n+  98.13%    97.12%  libc-2.27.so       [.] __memmove_avx_unaligned_erms\n+  51.99%     0.00%  raw_benchmark      [.] shm::PublisherBin<16u>::publish\n+  51.98%     0.01%  raw_benchmark      [.] shm::Topic<16u>::write\n+  47.64%     0.01%  raw_benchmark      [.] shm::Topic<16u>::read\n<\/code><\/pre>\n\n\n\n
__memmove_avx_unaligned_erms<\/code> is an implementation of memcpy<\/code> for unaligned memory blocks that uses AVX to copy over 32 bytes at a time. Digging into the glibc<\/code> source code, I found this:<\/p>\n\n\n\n
#if IS_IN (libc)\n# define VEC_SIZE                32\n# define VEC(i)                  ymm##i\n# define VMOVNT                  vmovntdq\n# define VMOVU                   vmovdqu\n# define VMOVA                   vmovdqa\n# define SECTION(p)              p##.avx\n# define MEMMOVE_SYMBOL(p,s)     p##_avx_##s\n\n# include "memmove-vec-unaligned-erms.S"\n#endif\n<\/code><\/pre>\n\n\n\n
Breaking down this function:<\/p>\n\n\n\n
memmove<\/code>: glibc<\/code> implements memcpy<\/code> as a memmove<\/code> instead, here\u2019s the relevant source code:<\/p>\n\n\n\n
# define SYMBOL_NAME memcpy\n# include "ifunc-memmove.h"\n\nlibc_ifunc_redirected (__redirect_memcpy, __new_memcpy,\n\t\t       IFUNC_SELECTOR ());\n<\/code><\/pre>\n\n\n\n
Here\u2019s the difference between the two: With memcpy<\/code>, the destination cannot overlap the source at all. With memmove<\/code> it can. Initially, I wasn\u2019t sure why it was implemented as memmove<\/code>. The reason for this will become clearer as the post proceeds.<\/p>\n\n\n\n
erms<\/code>: E<\/em>nhanced R<\/em>ep M<\/em>ovs<\/em> is a hardware optimization for a loop that does a simple copy. In simple pseudo-code, this is what the loop implementation looks like for copying a single byte at a time (REP MOVSB<\/code>).<\/p>\n\n\n\n
void rep_movsb(void *dest, const void *src, size_t len) {\n  const uint8_t* s = (uint8_t*)src;\n  uint8_t* d = (uint8_t*)dest;\n\n  while (len--)\n    *d++ = *s++;\n\n  return dest;\n}\n<\/code><\/pre>\n\n\n\n
Since the loop copies data pointer by pointer, it can handle the case of overlapping data.<\/p>\n\n\n\n
vec<\/code>: For the above loop rather than copying around single bytes, it uses x86 vectorized instructions to copy multiple bytes in a single loop iteration (technically single instruction). vmov*<\/code> are assembly instructions for AVX which is the latest instruction set that the CPU on my laptop supports. With VEC_SIZE = 32<\/code>, it copies 32 bytes at a time.<\/p>\n\n\n\n
unaligned<\/code>: This is a generic version of memmove<\/code> that can copy between any pointer locations irrespective of their alignment. Unaligned pointers increase complexity for the copy loop when using vectorized instructions. The unaligned preceeding and trailing memory locations must be copied separately before hitting the optimized loop.<\/p>\n\n\n\n
memmove-vec-unaligned-erms.S<\/code> holds the actual implementation in assembly. A few things that the implementation does:<\/p>\n\n\n\n
    \n
  1. It uses REP MOVS<\/code> only if the data is greater than 4kB. For smaller values it uses the SSE2 optimization.<\/li>\n\n\n\n
  2. For handling unaligned<\/code> pointers, it uses the following blocks:\n