为 pod 设置突发带宽限制

Question

据我所知，有两种方法可以使用 k8s 限制带宽。

首先，用

配置CNI

{
   "type": "bandwidth",
   "capabilities": {"bandwidth": true},
   "ingressRate": 10000000,
   "ingressBurst": 10000000,
   "egressRate": 10000000,
   "egressBurst": 10000000
}

其次，使用以下内容注释广告连播：

annotations:
  kubernetes.io/ingress-bandwidth: 8M
  kubernetes.io/egress-bandwidth: 8M

我通读了文档，但没有找到任何方法来为 pod 配置 burst。

但是默认突发太大而无法使限制有用：

qdisc tbf 1: dev calia2333445823 root refcnt 2 rate 8Mbit burst 256Mb lat 25.0ms

而且似乎当CNI配置了带宽限制时，pod注解不会覆盖CNI的配置并生效。

那么如何为一个 pod 设置突发？

iperf 输出说明爆了：

服务器：

Accepted connection from 192.168.0.34, port 49470
[  5] local 192.168.203.129 port 1234 connected to 192.168.0.34 port 49472
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   246 MBytes  2060541 Kbits/sec                  (omitted)
[  5]   1.00-2.00   sec   935 KBytes  7657 Kbits/sec                  (omitted)
[  5]   2.00-3.00   sec   933 KBytes  7645 Kbits/sec                  (omitted)
[  5]   0.00-1.00   sec   935 KBytes  7655 Kbits/sec
[  5]   1.00-2.00   sec   935 KBytes  7659 Kbits/sec
[  5]   2.00-3.00   sec   935 KBytes  7655 Kbits/sec
[  5]   3.00-4.00   sec   933 KBytes  7645 Kbits/sec
[  5]   4.00-5.00   sec   932 KBytes  7637 Kbits/sec
[  5]   5.00-6.00   sec   935 KBytes  7657 Kbits/sec
[  5]   6.00-7.00   sec   936 KBytes  7667 Kbits/sec
[  5]   7.00-8.00   sec   932 KBytes  7632 Kbits/sec
[  5]   8.00-9.00   sec   935 KBytes  7659 Kbits/sec
[  5]   9.00-10.00  sec   933 KBytes  7647 Kbits/sec
[  5]  10.00-11.00  sec   935 KBytes  7654 Kbits/sec
[  5]  11.00-11.15  sec   141 KBytes  7582 Kbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-11.15  sec  0.00 Bytes  0.00 Kbits/sec                  sender
[  5]   0.00-11.15  sec  10.2 MBytes  7650 Kbits/sec                  receiver

客户：

[  4] local 192.168.0.34 port 49472 connected to 192.168.203.129 port 1234
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   247 MBytes  2073202 Kbits/sec    0    324 KBytes       (omitted)
[  4]   1.00-2.00   sec   700 KBytes  5732 Kbits/sec    0    366 KBytes       (omitted)
[  4]   2.00-3.00   sec  1.55 MBytes  13037 Kbits/sec    0    407 KBytes       (omitted)
[  4]   0.00-1.00   sec   891 KBytes  7291 Kbits/sec    0    448 KBytes
[  4]   1.00-2.00   sec   954 KBytes  7821 Kbits/sec    0    491 KBytes
[  4]   2.00-3.00   sec  1018 KBytes  8344 Kbits/sec    0    532 KBytes
[  4]   3.00-4.00   sec  1.06 MBytes  8858 Kbits/sec    0    573 KBytes
[  4]   4.00-5.00   sec  1.18 MBytes  9897 Kbits/sec    0    615 KBytes
[  4]   5.00-6.00   sec  1.24 MBytes  10433 Kbits/sec    0    656 KBytes
[  4]   6.00-7.00   sec  1.25 MBytes  10487 Kbits/sec    0    697 KBytes
[  4]   7.00-8.00   sec  1.25 MBytes  10488 Kbits/sec    0    766 KBytes
[  4]   8.00-9.00   sec  0.00 Bytes  0.00 Kbits/sec    0    899 KBytes
[  4]   9.00-10.00  sec  1.25 MBytes  10485 Kbits/sec    0   1.03 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.0 MBytes  8410 Kbits/sec    0             sender
[  4]   0.00-10.00  sec  10.2 MBytes  8532 Kbits/sec                  receiver

环境：

Kubernetes 1.15.7
Calico v3.11.1
带宽插件 v0.8.0
tc iproute 4.11.0-14.el7

Answer 1

不幸的是，当前的带宽控制实现不支持限制 pod 的突发。我对此进行了相同的测试。我还查看了 kubernetes github 上的 cni 代码，发现有只有 annotations 可见 ingress bandwidth 和 egress bandwidth。

    bandwidthAnnotation := make(map[string]string)
    bandwidthAnnotation["kubernetes.io/ingress-bandwidth"] = "1M"
    bandwidthAnnotation["kubernetes.io/egress-bandwidth"] = "1M"

因为 network shaping is still in alpha stage you could raise a request on github 并询问对于这个功能。

为 pod 设置突发带宽限制

Set burst for bandwidth limit for a pod

kubernetes

cni