MemSQL 为 10MB 数据占用 15GB 内存

MemSQL takes 15GB memory for 10MB of data

我已经使用以下资源按照以下方式安装了 memsql 5.1.2。

Google cloud server

HDD: 100GB

Machine type: n1-standard-4 (4 vCPUs, 15 GB memory)

实施:

2 MEMSQL 节点运行在同一台机器上的以下端口上

3306 Master Aggregator

3307 Leaf

资源利用率:

Memory 14.16 GB / 14.69 GB

Paging 0 B/s

Database size - 10MB

1818    memsql  1.1%    77% /var/lib/memsql/leaf-3307/memsqld --defaults-file=/var/lib/memsql/leaf-3307/memsql.cnf --pid-file=/var/lib/memsql/leaf-3307/data/memsqld.pid --user=memsql
2736    memsql  0.3%    16% /var/lib/memsql/master-3306/memsqld --defaults-file=/var/lib/memsql/master-330

注意:服务器中没有实现交换内存。

数据库大小由运行对 information_schema.TABLES 的查询获取。

所有数据都以行存储的形式存在,因为我们必须 运行 通过考虑表之间的许多关系进行查询。

一旦 memsql 启动,内存就会上升到 70% 并继续增加,2-3 小时后,当尝试连接 memsql 时,内存会出现以下错误,之后也无法连接。

OperationalError: (1836, "Leaf 'xx.xxx.x.xx':3307 failed while executing this query. Try re-running the query.")
[Mon Mar 27 09:26:31.163455 2017] [:error] [pid 1718] [remote xxx.xxx.xxx.xxx:9956]

唯一的解决办法是重启服务器,因为它已经占用了所有内存。

我能为此做什么?它的实施方式有问题吗?我应该在此处附上任何日志吗?

Show status extended; 查询给出如下结果

+-------------------------------------+------------------------------------------------------------------------+
| Variable_name                       | Value                                                                  |
+-------------------------------------+------------------------------------------------------------------------+
| Aborted_clients                     | 48                                                                     |
| Aborted_connects                    | 1                                                                      |
| Bytes_received                      | 85962135                                                               |
| Bytes_sent                          | 545322701                                                              |
| Connections                         | 1626                                                                   |
| Max_used_connections                | 69                                                                     |
| Queries                             | 364793                                                                 |
| Questions                           | 364793                                                                 |
| Threads_cached                      | 19                                                                     |
| Threads_connected                   | 50                                                                     |
| Threads_created                     | 69                                                                     |
| Threads_running                     | 1                                                                      |
| Threads_background                  | 1                                                                      |
| Threads_idle                        | 0                                                                      |
| Ready_queue                         | 0                                                                      |
| Idle_queue                          | 0                                                                      |
| Context_switches                    | 1626                                                                   |
| Context_switch_misses               | 0                                                                      |
| Uptime                              | 22270                                                                  |
| Auto_attach_remaining_seconds       | 0                                                                      |
| Data_directory                      | /var/lib/memsql/leaf-3307/data                                         |
| Plancache_directory                 | /var/lib/memsql/leaf-3307/plancache                                    |
| Transaction_logs_directory          | /var/lib/memsql/leaf-3307/data/logs                                    |
| Segments_directory                  | /var/lib/memsql/leaf-3307/data/columns                                 |
| Snapshots_directory                 | /var/lib/memsql/leaf-3307/data/snapshots                               |
| Threads_waiting_for_disk_space      | 0                                                                      |
| Seconds_until_expiration            | -1                                                                     |
| License_key                         | 11111111111111111111111111111111                                       |
| License_type                        | community                                                              |
| Query_compilations                  | 62                                                                     |
| Query_compilation_failures          | 0                                                                      |
| GCed_versions_last_sweep            | 0                                                                      |
| Average_garbage_collection_duration | 21 ms                                                                  |
| Total_server_memory                 | 9791.4 MB                                                              |
| Alloc_thread_stacks                 | 70.0 MB                                                                |
| Malloc_active_memory                | 1254.7 (+0.0) MB                                                       |
| Malloc_cumulative_memory            | 7315.5 (+0.2) MB                                                       |
| Buffer_manager_memory               | 1787.8 MB                                                              |
| Buffer_manager_cached_memory        | 77.2 (-0.1) MB                                                         |
| Buffer_manager_unrecycled_memory    | 0.0 MB                                                                 |
| Alloc_skiplist_tower                | 263.8 MB                                                               |
| Alloc_variable                      | 501.4 MB                                                               |
| Alloc_large_variable                | 2.4 MB                                                                 |
| Alloc_table_primary                 | 752.6 MB                                                               |
| Alloc_deleted_version               | 92.9 MB                                                                |
| Alloc_internal_key_node             | 72.1 MB                                                                |
| Alloc_hash_buckets                  | 459.1 MB                                                               |
| Alloc_table_metadata_cache          | 1.1 MB                                                                 |
| Alloc_unit_images                   | 34.8 MB                                                                |
| Alloc_unit_ifn_thunks               | 0.6 MB                                                                 |
| Alloc_object_code_images            | 11.6 MB                                                                |
| Alloc_compiled_unit_sections        | 17.3 MB                                                                |
| Alloc_databases_list_entry          | 17.9 MB                                                                |
| Alloc_plan_cache                    | 0.1 MB                                                                 |
| Alloc_replication_large             | 232.0 MB                                                               |
| Alloc_durability_large              | 7239.1 MB                                                              |
| Alloc_sharding_partitions           | 0.1 MB                                                                 |
| Alloc_security                      | 0.1 MB                                                                 |
| Alloc_log_replay                    | 0.9 MB                                                                 |
| Alloc_client_connection             | 3.0 MB                                                                 |
| Alloc_protocol_packet               | 6.1 (+0.1) MB                                                          |
| Alloc_large_incremental             | 0.8 MB                                                                 |
| Alloc_table_memory                  | 2144.2 MB                                                              |
| Alloc_variable_bucket_16            | allocs:10877846  alloc_MB:166.0  buffer_MB:179.0  cached_buffer_MB:1.9 |
| Alloc_variable_bucket_24            | allocs:4275659  alloc_MB:97.9  buffer_MB:106.8  cached_buffer_MB:1.9   |
| Alloc_variable_bucket_32            | allocs:2875801  alloc_MB:87.8  buffer_MB:93.4  cached_buffer_MB:1.9    |
| Alloc_variable_bucket_40            | allocs:724489  alloc_MB:27.6  buffer_MB:31.0  cached_buffer_MB:1.2     |
| Alloc_variable_bucket_48            | allocs:377060  alloc_MB:17.3  buffer_MB:19.8  cached_buffer_MB:0.9     |
| Alloc_variable_bucket_56            | allocs:228720  alloc_MB:12.2  buffer_MB:14.0  cached_buffer_MB:0.8     |
| Alloc_variable_bucket_64            | allocs:150214  alloc_MB:9.2  buffer_MB:10.1  cached_buffer_MB:0.2      |
| Alloc_variable_bucket_72            | allocs:35264  alloc_MB:2.4  buffer_MB:2.9  cached_buffer_MB:0.0        |
| Alloc_variable_bucket_80            | allocs:14920  alloc_MB:1.1  buffer_MB:1.2  cached_buffer_MB:0.0        |
| Alloc_variable_bucket_88            | allocs:5582  alloc_MB:0.5  buffer_MB:0.6  cached_buffer_MB:0.0         |
| Alloc_variable_bucket_104           | allocs:8075  alloc_MB:0.8  buffer_MB:1.0  cached_buffer_MB:0.0         |
| Alloc_variable_bucket_128           | allocs:8892  alloc_MB:1.1  buffer_MB:1.2  cached_buffer_MB:0.0         |
| Alloc_variable_bucket_160           | allocs:17614  alloc_MB:2.7  buffer_MB:3.0  cached_buffer_MB:0.0        |
| Alloc_variable_bucket_200           | allocs:30454  alloc_MB:5.8  buffer_MB:6.9  cached_buffer_MB:0.6        |
| Alloc_variable_bucket_248           | allocs:4875  alloc_MB:1.2  buffer_MB:1.5  cached_buffer_MB:0.2         |
| Alloc_variable_bucket_312           | allocs:371  alloc_MB:0.1  buffer_MB:0.2  cached_buffer_MB:0.0          |
| Alloc_variable_bucket_384           | allocs:30  alloc_MB:0.0  buffer_MB:0.1  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_480           | allocs:11  alloc_MB:0.0  buffer_MB:0.1  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_600           | allocs:57  alloc_MB:0.0  buffer_MB:0.2  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_752           | allocs:62  alloc_MB:0.0  buffer_MB:0.2  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_936           | allocs:42  alloc_MB:0.0  buffer_MB:0.2  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_1168          | allocs:106  alloc_MB:0.1  buffer_MB:0.2  cached_buffer_MB:0.0          |
| Alloc_variable_bucket_1480          | allocs:126  alloc_MB:0.2  buffer_MB:0.2  cached_buffer_MB:0.0          |
| Alloc_variable_bucket_1832          | allocs:0  alloc_MB:0.0  buffer_MB:0.2  cached_buffer_MB:0.2            |
| Alloc_variable_bucket_2288          | allocs:1  alloc_MB:0.0  buffer_MB:0.2  cached_buffer_MB:0.1            |
| Alloc_variable_bucket_2832          | allocs:33  alloc_MB:0.1  buffer_MB:1.1  cached_buffer_MB:0.2           |
| Alloc_variable_bucket_3528          | allocs:16  alloc_MB:0.1  buffer_MB:0.5  cached_buffer_MB:0.1           |
| Alloc_variable_bucket_4504          | allocs:49  alloc_MB:0.2  buffer_MB:0.8  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_5680          | allocs:66  alloc_MB:0.4  buffer_MB:1.2  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_6224          | allocs:30  alloc_MB:0.2  buffer_MB:1.0  cached_buffer_MB:0.1           |
| Alloc_variable_bucket_7264          | allocs:94  alloc_MB:0.7  buffer_MB:1.5  cached_buffer_MB:0.0           |
| Alloc_variable_bucket_9344          | allocs:70  alloc_MB:0.6  buffer_MB:2.6  cached_buffer_MB:0.2           |
| Alloc_variable_bucket_11896         | allocs:14  alloc_MB:0.2  buffer_MB:2.4  cached_buffer_MB:1.2           |
| Alloc_variable_bucket_14544         | allocs:7  alloc_MB:0.1  buffer_MB:2.4  cached_buffer_MB:1.9            |
| Alloc_variable_bucket_18696         | allocs:18  alloc_MB:0.3  buffer_MB:3.2  cached_buffer_MB:1.9           |
| Alloc_variable_bucket_21816         | allocs:4  alloc_MB:0.1  buffer_MB:0.4  cached_buffer_MB:0.0            |
| Alloc_variable_bucket_26184         | allocs:6  alloc_MB:0.1  buffer_MB:0.9  cached_buffer_MB:0.2            |
| Alloc_variable_bucket_32728         | allocs:13  alloc_MB:0.4  buffer_MB:2.4  cached_buffer_MB:1.4           |
| Alloc_variable_bucket_43648         | allocs:12  alloc_MB:0.5  buffer_MB:1.4  cached_buffer_MB:0.2           |
| Alloc_variable_bucket_65472         | allocs:7  alloc_MB:0.4  buffer_MB:2.8  cached_buffer_MB:1.9            |
| Alloc_variable_bucket_130960        | allocs:3  alloc_MB:0.4  buffer_MB:2.2  cached_buffer_MB:1.9            |
| Alloc_variable_cached_buffers       | 21.4 MB                                                                |
| Alloc_variable_allocated            | 438.7 MB                                                               |
| Successful_read_queries             | 9048                                                                   |
| Successful_write_queries            | 19096                                                                  |
| Failed_read_queries                 | 0                                                                      |
| Failed_write_queries                | 4                                                                      |
| Rows_returned_by_reads              | 75939                                                                  |
| Rows_affected_by_writes             | 245                                                                    |
| Execution_time_of_reads             | 7864 ms                                                                |
| Execution_time_of_write             | 180311 ms                                                              |
| Transaction_buffer_wait_time        | 0 ms                                                                   |
| Transaction_log_flush_wait_time     | 0 ms                                                                   |
| Row_lock_wait_time                  | 0 ms                                                                   |
| Ssl_accept_renegotiates             | 0                                                                      |
| Ssl_accepts                         | 0                                                                      |
| Ssl_callback_cache_hits             | 0                                                                      |
| Ssl_client_connects                 | 0                                                                      |
| Ssl_connect_renegotiates            | 0                                                                      |
| Ssl_ctx_verify_depth                | 18446744073709551615                                                   |
| Ssl_ctx_verify_mode                 | 0                                                                      |
| Ssl_default_timeout                 | 0                                                                      |
| Ssl_finished_accepts                | 0                                                                      |
| Ssl_finished_connects               | 0                                                                      |
| Ssl_session_cache_hits              | 0                                                                      |
| Ssl_session_cache_misses            | 0                                                                      |
| Ssl_session_cache_overflows         | 0                                                                      |
| Ssl_session_cache_size              | 20480                                                                  |
| Ssl_session_cache_timeouts          | 0                                                                      |
| Ssl_sessions_reused                 | 0                                                                      |
| Ssl_used_session_cache_entries      | 0                                                                      |
| Ssl_verify_depth                    | 0                                                                      |
| Ssl_verify_mode                     | 0                                                                      |
| Ssl_cipher                          |                                                                        |
| Ssl_cipher_list                     |                                                                        |
| Ssl_version                         |                                                                        |
| Ssl_session_cache_mode              | SERVER                                                                 |
+-------------------------------------+------------------------------------------------------------------------+

从状态输出可以看出:

  • 叶节点总内存10GB
  • 7GB Alloc_durability_large

您可以在此处查看这些变量的含义:https://help.memsql.com/hc/en-us/articles/115001091386-What-Is-Using-Memory-on-My-Leaves-

最有趣的是 Alloc_durability_large 中的金额很大,这很不寻常。您是否有大量数据库 and/or 分区? (您可以通过计算叶节点上显示数据库扩展中的行数来检查。)每个都需要固定数量的事务缓冲区内存(默认为 64 MB)。