Page "Remote direct memory access" Paragraph 3

RDMA reduces network protocol overhead, leading to improvements in communication latency.

Reductions in protocol overhead can increase a network's ability to move data quickly, allowing applications to get the data they need faster, in turn leading to more scalable clusters.

However, one must be aware of the tradeoff between this reduction in network protocol overhead and additional overhead that may be incurred on each node due to the need for pinning virtual memory pages.

In particular, zero-copy RDMA protocols require that the memory pages involved in a transaction be pinned, at least for the duration of the transfer.

If this is not done, RDMA pages might be paged out to disk and replaced with other data by the operating system, causing the DMA engine ( which knows nothing of the virtual memory system maintained by the operating system ) to send the wrong data.

The net result of not pinning the pages in a zero-copy RDMA system can be corruption of the contents of memory in the distributed system.

Pinning memory takes time and additional memory to set up, reduces the quantity of memory the operating system can allocate to processes, limits the overall flexibility of the memory system to adapt over time, and could even lead to underutilization of memory if processes unnecessarily pin pages.

The net result is the introduction of latency, sometimes in linear proportion to the number of pages of data pinned in memory.

In order to mitigate these problems, several techniques for interfacing with RDMA devices were developed: