为什么我们需要 DMA 池?
Why do we need DMA pool?
我正在阅读 https://www.kernel.org/doc/Documentation/DMA-API.txt 但我不明白为什么需要 DMA 池。
为什么不让 DMA PAGE_SIZE 分配内存 dma_alloc_coherent
并使用偏移量?
此外,为什么动态 DMA 对网络设备驱动程序有用而不是重复使用相同的 DMA 内存?
<1KB 数据传输的最佳性能是什么?
警告:我不是 linux 内核方面的专家。
LDD book (which may be better reading to start) says that DMA pool works better for smaller dma regions (shorter than page) - https://static.lwn.net/images/pdf/LDD3/ch15.pdf page 447 or https://www.oreilly.com/library/view/linux-device-drivers/0596005903/ch15.html, "DMA pools" 部分:
A DMA pool is an allocation mechanism for small, coherent DMA mappings. Mappings obtained from dma_alloc_coherent
may have a minimum size of one page. If your device needs smaller DMA areas than that, you should probably be using a DMA pool. DMA pools are also useful in situations where you may be tempted to perform DMA to small areas embedded within a larger structure. Some very obscure driver bugs have been traced down to cache coherency problems with structure fields adjacent to small DMA areas. To avoid this problem, you should always allocate areas for DMA operations explicitly, away from other, non-DMA data structures. ... Allocations are handled with dma_pool_alloc
https://www.kernel.org/doc/Documentation/DMA-API-HOWTO.txt
中也有同样的说明
If your driver needs lots of smaller memory regions, you can write
custom code to subdivide pages returned by dma_alloc_coherent(),
or you can use the dma_pool API to do that. A dma_pool is like
a kmem_cache, but it uses dma_alloc_coherent(), not __get_free_pages().
Also, it understands common hardware constraints for alignment,
like queue heads needing to be aligned on N byte boundaries.
所以 DMA 池是针对较小分配的优化。您可以对每个小的 dma 内存单独使用 dma_alloc_coherent
(开销更大),或者您可以尝试构建自己的池(更多自定义代码来管理偏移量和分配)。但是DMA池已经实现了,可以使用了。
应针对您的情况分析方法的性能。
网络驱动中动态dma注册示例(用于skb片段):
https://elixir.bootlin.com/linux/v4.6/source/drivers/net/ethernet/realtek/r8169.c
static struct sk_buff *rtl8169_alloc_rx_data
mapping = dma_map_single(d, rtl8169_align(data), rx_buf_sz,
DMA_FROM_DEVICE);
static int rtl8169_xmit_frags
mapping = dma_map_single(d, addr, len, DMA_TO_DEVICE);
static netdev_tx_t rtl8169_start_xmit
mapping = dma_map_single(d, skb->data, len, DMA_TO_DEVICE);
static void rtl8169_unmap_tx_skb
dma_unmap_single(d, le64_to_cpu(desc->addr), len, DMA_TO_DEVICE);
就地为 dma 注册 skb 片段会更好(如果 sg dma is supported by NIC chip) than copying every fragment from skb into some DMA memory. Check "Understanding Linux Network Internals" book for section "dev_queue_xmit Function" and Chapter 21; and skb_linearize
DMA 池使用示例 - nvme 驱动程序(prp 为 part of Submission Queue Element、物理区域页面、64 位指针和 "PRP List contains a list of PRPs with generally no offsets."):
https://elixir.bootlin.com/linux/v4.6/source/drivers/nvme/host/pci.c#L1807
static int nvme_setup_prp_pools(struct nvme_dev *dev)
{
dev->prp_page_pool = dma_pool_create("prp list page", dev->dev,
PAGE_SIZE, PAGE_SIZE, 0);
static bool nvme_setup_prps
prp_list = dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma);
static void nvme_free_iod
dma_pool_free(dev->prp_small_pool, list[0], prp_dma);
我正在阅读 https://www.kernel.org/doc/Documentation/DMA-API.txt 但我不明白为什么需要 DMA 池。
为什么不让 DMA PAGE_SIZE 分配内存 dma_alloc_coherent
并使用偏移量?
此外,为什么动态 DMA 对网络设备驱动程序有用而不是重复使用相同的 DMA 内存?
<1KB 数据传输的最佳性能是什么?
警告:我不是 linux 内核方面的专家。
LDD book (which may be better reading to start) says that DMA pool works better for smaller dma regions (shorter than page) - https://static.lwn.net/images/pdf/LDD3/ch15.pdf page 447 or https://www.oreilly.com/library/view/linux-device-drivers/0596005903/ch15.html, "DMA pools" 部分:
A DMA pool is an allocation mechanism for small, coherent DMA mappings. Mappings obtained from
dma_alloc_coherent
may have a minimum size of one page. If your device needs smaller DMA areas than that, you should probably be using a DMA pool. DMA pools are also useful in situations where you may be tempted to perform DMA to small areas embedded within a larger structure. Some very obscure driver bugs have been traced down to cache coherency problems with structure fields adjacent to small DMA areas. To avoid this problem, you should always allocate areas for DMA operations explicitly, away from other, non-DMA data structures. ... Allocations are handled withdma_pool_alloc
https://www.kernel.org/doc/Documentation/DMA-API-HOWTO.txt
中也有同样的说明If your driver needs lots of smaller memory regions, you can write custom code to subdivide pages returned by dma_alloc_coherent(), or you can use the dma_pool API to do that. A dma_pool is like a kmem_cache, but it uses dma_alloc_coherent(), not __get_free_pages(). Also, it understands common hardware constraints for alignment, like queue heads needing to be aligned on N byte boundaries.
所以 DMA 池是针对较小分配的优化。您可以对每个小的 dma 内存单独使用 dma_alloc_coherent
(开销更大),或者您可以尝试构建自己的池(更多自定义代码来管理偏移量和分配)。但是DMA池已经实现了,可以使用了。
应针对您的情况分析方法的性能。
网络驱动中动态dma注册示例(用于skb片段): https://elixir.bootlin.com/linux/v4.6/source/drivers/net/ethernet/realtek/r8169.c
static struct sk_buff *rtl8169_alloc_rx_data
mapping = dma_map_single(d, rtl8169_align(data), rx_buf_sz,
DMA_FROM_DEVICE);
static int rtl8169_xmit_frags
mapping = dma_map_single(d, addr, len, DMA_TO_DEVICE);
static netdev_tx_t rtl8169_start_xmit
mapping = dma_map_single(d, skb->data, len, DMA_TO_DEVICE);
static void rtl8169_unmap_tx_skb
dma_unmap_single(d, le64_to_cpu(desc->addr), len, DMA_TO_DEVICE);
就地为 dma 注册 skb 片段会更好(如果 sg dma is supported by NIC chip) than copying every fragment from skb into some DMA memory. Check "Understanding Linux Network Internals" book for section "dev_queue_xmit Function" and Chapter 21; and skb_linearize
DMA 池使用示例 - nvme 驱动程序(prp 为 part of Submission Queue Element、物理区域页面、64 位指针和 "PRP List contains a list of PRPs with generally no offsets."):
https://elixir.bootlin.com/linux/v4.6/source/drivers/nvme/host/pci.c#L1807
static int nvme_setup_prp_pools(struct nvme_dev *dev)
{
dev->prp_page_pool = dma_pool_create("prp list page", dev->dev,
PAGE_SIZE, PAGE_SIZE, 0);
static bool nvme_setup_prps
prp_list = dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma);
static void nvme_free_iod
dma_pool_free(dev->prp_small_pool, list[0], prp_dma);