RDS MySQL 高内存消耗
RDS MySQL High Memory consumption
我在 RDS MySQL 服务器上遇到内存利用率高的问题，
我们将 MySQL 的内存上限设置为 149 GB，但它仍然消耗大约 98% 的内存。
任何人都可以帮助我了解 RDS 内存消耗和释放行为吗？
我们如何才能知道是什么导致了内存使用率过高？
我们如何避免它？
RDS Configuration:- db.r3.8xlarge.(CPU core = 32 and Memory = 240 GB).
MySQL内存配置：-
Innodb_buffer_pool_size=144 GB
Innodb_buffer_pool_instances= 1
Innodb_page_cleaners =1
Key_buffer_size = 256 MB
query_cache_size =64 MB
tmp_table_size =4096 MB
innodb_log_buffer_size= 64 MB
Max_connections = 20484
sort_buffer_size= 2 MB
read_buffer_size = 0.128 MB
read_rnd_buffer_size = 0.256 MB
join_buffer_size = 1 MB
thread_stack =0.25 MB
binlog_cache_size = 0.031 MB
Innodb 状态的一部分
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (read thread)
I/O thread 7 state: waiting for completed aio requests (read thread)
I/O thread 8 state: waiting for completed aio requests (read thread)
I/O thread 9 state: waiting for completed aio requests (read thread)
I/O thread 10 state: waiting for completed aio requests (write thread)
I/O thread 11 state: waiting for completed aio requests (write thread)
I/O thread 12 state: waiting for completed aio requests (write thread)
I/O thread 13 state: waiting for completed aio requests (write thread)
I/O thread 14 state: waiting for completed aio requests (write thread)
I/O thread 15 state: waiting for completed aio requests (write thread)
I/O thread 16 state: waiting for completed aio requests (write thread)
I/O thread 17 state: waiting for completed aio requests (write thread)
I/O thread 18 state: waiting for completed aio requests (write thread)
I/O thread 19 state: waiting for completed aio requests (write thread)
I/O thread 20 state: waiting for completed aio requests (write thread)
I/O thread 21 state: waiting for completed aio requests (write thread)
I/O thread 22 state: waiting for completed aio requests (write thread)
I/O thread 23 state: waiting for completed aio requests (write thread)
I/O thread 24 state: waiting for completed aio requests (write thread)
I/O thread 25 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0, 0, 0, 0, 0] , aio writes: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
16300373 OS file reads, 327069064 OS file writes, 34275132 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 24.00 writes/s, 2.83 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 3690, seg size 3692, 1232846 merges
merged operations:
insert 1518763, delete mark 7967515, delete 581073
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 38278223, node heap has 53862 buffer(s)
Hash table size 38278223, node heap has 101395 buffer(s)
Hash table size 38278223, node heap has 51039 buffer(s)
Hash table size 38278223, node heap has 49662 buffer(s)
Hash table size 38278223, node heap has 31822 buffer(s)
Hash table size 38278223, node heap has 69910 buffer(s)
Hash table size 38278223, node heap has 62223 buffer(s)
Hash table size 38278223, node heap has 107050 buffer(s)
75.17 hash searches/s, 83.82 non-hash searches/s
---
LOG
---
Log sequence number 23815192515256
Log flushed up to 23815192515256
Pages flushed up to 23815192515256
Last checkpoint at 23815192515247
0 pending log flushes, 0 pending chkp writes
71186755 log i/o's done, 7.34 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 158455627776
Dictionary memory allocated 9737076
Buffer pool size 9444223
Free buffers 272250
Database pages 8645010
Old database pages 3191205
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 36040960, not young 898847533
0.00 youngs/s, 0.00 non-youngs/s
Pages read 15253773, created 43492874, written 244272465
0.00 reads/s, 0.52 creates/s, 16.07 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 8645010, unzip_LRU len: 0
I/O sum[469]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
1 queries inside InnoDB, 0 queries in queue
1 read views open inside InnoDB
Process ID=18963, Main thread ID=47543500240640, state: sleeping
Number of rows inserted 7973224537, updated 16944890, deleted 452458106, read 549621443211
6.03 inserts/s, 0.07 updates/s, 0.03 deletes/s, 229179.61 reads/s
Innodb_buffer_pool_size=144 GB 看起来像语法错误，但其他输出确认它被设置为 144G。
"Capping" at 149G -- 这是 RDS 中的内容吗？考虑到 buffer_pool_size.
似乎相当紧
tmp_table_size =4096 MB -- 由于这可以分配给每个查询，因此您冒着耗尽 RAM 的严重风险。将其降低到 1G。\
Max_connections = 20484 -- 大得离谱。减少到 1000。注意：分配了许多内存块 "per connection"。所以，如果你每个人都达到 20K，你可能会耗尽 RAM。
Innodb_buffer_pool_instances= 1 -- 更改为 16 可略微提高性能。
解决上述问题后，... buffer_pool 是减少 MySQL RAM 占用空间的简单方法。如果 98% 是个问题，请更改它。
我在 RDS MySQL 服务器上遇到内存利用率高的问题， 我们将 MySQL 的内存上限设置为 149 GB，但它仍然消耗大约 98% 的内存。
任何人都可以帮助我了解 RDS 内存消耗和释放行为吗？
我们如何才能知道是什么导致了内存使用率过高？ 我们如何避免它？
RDS Configuration:- db.r3.8xlarge.(CPU core = 32 and Memory = 240 GB).
MySQL内存配置：-
Innodb_buffer_pool_size=144 GB
Innodb_buffer_pool_instances= 1
Innodb_page_cleaners =1
Key_buffer_size = 256 MB
query_cache_size =64 MB
tmp_table_size =4096 MB
innodb_log_buffer_size= 64 MB
Max_connections = 20484
sort_buffer_size= 2 MB
read_buffer_size = 0.128 MB
read_rnd_buffer_size = 0.256 MB
join_buffer_size = 1 MB
thread_stack =0.25 MB
binlog_cache_size = 0.031 MB
Innodb 状态的一部分
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (read thread)
I/O thread 7 state: waiting for completed aio requests (read thread)
I/O thread 8 state: waiting for completed aio requests (read thread)
I/O thread 9 state: waiting for completed aio requests (read thread)
I/O thread 10 state: waiting for completed aio requests (write thread)
I/O thread 11 state: waiting for completed aio requests (write thread)
I/O thread 12 state: waiting for completed aio requests (write thread)
I/O thread 13 state: waiting for completed aio requests (write thread)
I/O thread 14 state: waiting for completed aio requests (write thread)
I/O thread 15 state: waiting for completed aio requests (write thread)
I/O thread 16 state: waiting for completed aio requests (write thread)
I/O thread 17 state: waiting for completed aio requests (write thread)
I/O thread 18 state: waiting for completed aio requests (write thread)
I/O thread 19 state: waiting for completed aio requests (write thread)
I/O thread 20 state: waiting for completed aio requests (write thread)
I/O thread 21 state: waiting for completed aio requests (write thread)
I/O thread 22 state: waiting for completed aio requests (write thread)
I/O thread 23 state: waiting for completed aio requests (write thread)
I/O thread 24 state: waiting for completed aio requests (write thread)
I/O thread 25 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0, 0, 0, 0, 0] , aio writes: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
16300373 OS file reads, 327069064 OS file writes, 34275132 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 24.00 writes/s, 2.83 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 3690, seg size 3692, 1232846 merges
merged operations:
insert 1518763, delete mark 7967515, delete 581073
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 38278223, node heap has 53862 buffer(s)
Hash table size 38278223, node heap has 101395 buffer(s)
Hash table size 38278223, node heap has 51039 buffer(s)
Hash table size 38278223, node heap has 49662 buffer(s)
Hash table size 38278223, node heap has 31822 buffer(s)
Hash table size 38278223, node heap has 69910 buffer(s)
Hash table size 38278223, node heap has 62223 buffer(s)
Hash table size 38278223, node heap has 107050 buffer(s)
75.17 hash searches/s, 83.82 non-hash searches/s
---
LOG
---
Log sequence number 23815192515256
Log flushed up to 23815192515256
Pages flushed up to 23815192515256
Last checkpoint at 23815192515247
0 pending log flushes, 0 pending chkp writes
71186755 log i/o's done, 7.34 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 158455627776
Dictionary memory allocated 9737076
Buffer pool size 9444223
Free buffers 272250
Database pages 8645010
Old database pages 3191205
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 36040960, not young 898847533
0.00 youngs/s, 0.00 non-youngs/s
Pages read 15253773, created 43492874, written 244272465
0.00 reads/s, 0.52 creates/s, 16.07 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 8645010, unzip_LRU len: 0
I/O sum[469]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
1 queries inside InnoDB, 0 queries in queue
1 read views open inside InnoDB
Process ID=18963, Main thread ID=47543500240640, state: sleeping
Number of rows inserted 7973224537, updated 16944890, deleted 452458106, read 549621443211
6.03 inserts/s, 0.07 updates/s, 0.03 deletes/s, 229179.61 reads/s
Innodb_buffer_pool_size=144 GB 看起来像语法错误，但其他输出确认它被设置为 144G。
"Capping" at 149G -- 这是 RDS 中的内容吗？考虑到 buffer_pool_size.
似乎相当紧tmp_table_size =4096 MB -- 由于这可以分配给每个查询，因此您冒着耗尽 RAM 的严重风险。将其降低到 1G。\
Max_connections = 20484 -- 大得离谱。减少到 1000。注意：分配了许多内存块 "per connection"。所以，如果你每个人都达到 20K，你可能会耗尽 RAM。
Innodb_buffer_pool_instances= 1 -- 更改为 16 可略微提高性能。
解决上述问题后，... buffer_pool 是减少 MySQL RAM 占用空间的简单方法。如果 98% 是个问题，请更改它。