NoValidHost:未找到有效主机。没有足够的可用主机

NoValidHost: No valid host was found. There are not enough hosts available

当我在仪表板中创建实例时,出现错误:

No valid host was found. There are not enough hosts available.

/var/log/nova/nova-conductor.log文件中,有日志:

2017-08-05 00:22:29.046 3834 WARNING nova.scheduler.utils [req-89c159c7-b40a-43eb-8f0d-9306eb73e83a 2a5fa182fb1b459980db09cd1572850e 0d5998f2f7ec4c4892a32e06bafb19df - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations
    dests = self.driver.select_destinations(ctxt, spec_obj)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 74, in select_destinations
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.

2017-08-05 00:22:29.048 3834 WARNING nova.scheduler.utils [req-89c159c7-b40a-43eb-8f0d-9306eb73e83a 2a5fa182fb1b459980db09cd1572850e 0d5998f2f7ec4c4892a32e06bafb19df - - -] [instance: 2011e343-c8fc-4ed0-8148-b0d2b5ba37c3] Setting instance to ERROR state.
2017-08-05 00:22:30.785 3834 WARNING oslo_config.cfg [req-89c159c7-b40a-43eb-8f0d-9306eb73e83a 2a5fa182fb1b459980db09cd1572850e 0d5998f2f7ec4c4892a32e06bafb19df - - -] Option "auth_plugin" from group "neutron" is deprecated. Use option "auth_type" from group "neutron".

然后我搜索了SO,找到一个相关的post:

我检查了 mysql 中的 free_ram_mb:

MariaDB [nova]> select * from compute_nodes \G;
*************************** 1. row ***************************
           created_at: 2017-08-04 12:44:26
           updated_at: 2017-08-04 13:51:35
           deleted_at: NULL
                   id: 4
           service_id: NULL
                vcpus: 8
            memory_mb: 7808
             local_gb: 19
           vcpus_used: 0
       memory_mb_used: 512
        local_gb_used: 0
      hypervisor_type: QEMU
   hypervisor_version: 1005003
             cpu_info: {"vendor": "Intel", "model": "Broadwell", "arch": "x86_64", "features": ["smap", "avx", "clflush", "sep", "rtm", "vme", "invpcid", "msr", "fsgsbase", "xsave", "pge", "erms", "hle", "cmov", "tsc", "smep", "pcid", "pat", "lm", "abm", "adx", "3dnowprefetch", "nx", "fxsr", "syscall", "sse4.1", "pae", "sse4.2", "pclmuldq", "fma", "tsc-deadline", "mmx", "osxsave", "cx8", "mce", "de", "rdtscp", "ht", "pse", "lahf_lm", "rdseed", "popcnt", "mca", "pdpe1gb", "apic", "sse", "f16c", "ds", "invtsc", "pni", "aes", "avx2", "sse2", "ss", "hypervisor", "bmi1", "bmi2", "ssse3", "fpu", "cx16", "pse36", "mtrr", "movbe", "rdrand", "x2apic"], "topology": {"cores": 2, "cells": 1, "threads": 1, "sockets": 4}}
 disk_available_least: 18
          free_ram_mb: 7296
         free_disk_gb: 19
     current_workload: 0
          running_vms: 0
  hypervisor_hostname: ha-node1
              deleted: 0
              host_ip: 192.168.8.101
  supported_instances: [["i686", "qemu", "hvm"], ["x86_64", "qemu", "hvm"]]
            pci_stats: {"nova_object.version": "1.1", "nova_object.changes": ["objects"], "nova_object.name": "PciDevicePoolList", "nova_object.data": {"objects": []}, "nova_object.namespace": "nova"}
              metrics: []
      extra_resources: NULL
                stats: {}
        numa_topology: NULL
                 host: ha-node1
 ram_allocation_ratio: 3
 cpu_allocation_ratio: 16
                 uuid: 9113940b-7ec9-462d-af06-6988dbb6b6cf
disk_allocation_ratio: 1
*************************** 2. row ***************************
           created_at: 2017-08-04 12:44:34
           updated_at: 2017-08-04 13:50:47
           deleted_at: NULL
                   id: 6
           service_id: NULL
                vcpus: 8
            memory_mb: 7808
             local_gb: 19
           vcpus_used: 0
       memory_mb_used: 512
        local_gb_used: 0
      hypervisor_type: QEMU
   hypervisor_version: 1005003
             cpu_info: {"vendor": "Intel", "model": "Broadwell", "arch": "x86_64", "features": ["smap", "avx", "clflush", "sep", "rtm", "vme", "invpcid", "msr", "fsgsbase", "xsave", "pge", "erms", "hle", "cmov", "tsc", "smep", "pcid", "pat", "lm", "abm", "adx", "3dnowprefetch", "nx", "fxsr", "syscall", "sse4.1", "pae", "sse4.2", "pclmuldq", "fma", "tsc-deadline", "mmx", "osxsave", "cx8", "mce", "de", "rdtscp", "ht", "pse", "lahf_lm", "rdseed", "popcnt", "mca", "pdpe1gb", "apic", "sse", "f16c", "ds", "invtsc", "pni", "aes", "avx2", "sse2", "ss", "hypervisor", "bmi1", "bmi2", "ssse3", "fpu", "cx16", "pse36", "mtrr", "movbe", "rdrand", "x2apic"], "topology": {"cores": 2, "cells": 1, "threads": 1, "sockets": 4}}
 disk_available_least: 18
          free_ram_mb: 7296
         free_disk_gb: 19
     current_workload: 0
          running_vms: 0
  hypervisor_hostname: ha-node2
              deleted: 0
              host_ip: 192.168.8.102
  supported_instances: [["i686", "qemu", "hvm"], ["x86_64", "qemu", "hvm"]]
            pci_stats: {"nova_object.version": "1.1", "nova_object.changes": ["objects"], "nova_object.name": "PciDevicePoolList", "nova_object.data": {"objects": []}, "nova_object.namespace": "nova"}
              metrics: []
      extra_resources: NULL
                stats: {}
        numa_topology: NULL
                 host: ha-node2
 ram_allocation_ratio: 3
 cpu_allocation_ratio: 16
                 uuid: 32b574df-52ac-43dc-87f8-353350449076
disk_allocation_ratio: 1
2 rows in set (0.00 sec)

你看到了 free_ram_mb: 7296,我只想创建一个 512mb 虚拟机,但是失败了。


EDIT-1

nova 服务全部启动:

[root@ha-node1 ~]# nova service-list
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host     | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+
| 2  | nova-consoleauth | ha-node3 | internal | enabled | up    | 2017-08-05T14:20:25.000000 | -               |
| 5  | nova-conductor   | ha-node3 | internal | enabled | up    | 2017-08-05T14:20:29.000000 | -               |
| 7  | nova-cert        | ha-node3 | internal | enabled | up    | 2017-08-05T14:20:23.000000 | -               |
| 15 | nova-scheduler   | ha-node3 | internal | enabled | up    | 2017-08-05T14:20:20.000000 | -               |
| 22 | nova-cert        | ha-node1 | internal | enabled | up    | 2017-08-05T14:20:26.000000 | -               |
| 29 | nova-conductor   | ha-node1 | internal | enabled | up    | 2017-08-05T14:20:22.000000 | -               |
| 32 | nova-consoleauth | ha-node1 | internal | enabled | up    | 2017-08-05T14:20:29.000000 | -               |
| 33 | nova-consoleauth | ha-node2 | internal | enabled | up    | 2017-08-05T14:20:29.000000 | -               |
| 36 | nova-scheduler   | ha-node1 | internal | enabled | up    | 2017-08-05T14:20:30.000000 | -               |
| 40 | nova-conductor   | ha-node2 | internal | enabled | up    | 2017-08-05T14:20:26.000000 | -               |
| 44 | nova-cert        | ha-node2 | internal | enabled | up    | 2017-08-05T14:20:27.000000 | -               |
| 46 | nova-scheduler   | ha-node2 | internal | enabled | up    | 2017-08-05T14:20:28.000000 | -               |
| 49 | nova-compute     | ha-node2 | nova     | enabled | up    | 2017-08-05T14:19:35.000000 | -               |
| 53 | nova-compute     | ha-node1 | nova     | enabled | up    | 2017-08-05T14:20:05.000000 | -               |
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+

新星列表:

[root@ha-node1 ~]# nova list 
+--------------------------------------+------+--------+------------+-------------+----------+
| ID                                   | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+----------+
| 20193e58-2c5b-44c6-a98f-a44e2001934f | vm1  | ERROR  | -          | NOSTATE     |          |

以及 nova 展示实例:

[root@ha-node1 ~]# nova show 20193e58-2c5b-44c6-a98f-a44e2001934f
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property                             | Value                                                                                                                                                                                                      |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                                                                                                                                                                       |
| OS-EXT-AZ:availability_zone          | nova                                                                                                                                                                                                       |
| OS-EXT-SRV-ATTR:host                 | -                                                                                                                                                                                                          |
| OS-EXT-SRV-ATTR:hostname             | vm1                                                                                                                                                                                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                                                                                                                                                                          |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000003                                                                                                                                                                                          |
| OS-EXT-SRV-ATTR:kernel_id            |                                                                                                                                                                                                            |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                                                                                                                                                                          |
| OS-EXT-SRV-ATTR:ramdisk_id           |                                                                                                                                                                                                            |
| OS-EXT-SRV-ATTR:reservation_id       | r-jct8kkcq                                                                                                                                                                                                 |
| OS-EXT-SRV-ATTR:root_device_name     | /dev/vda                                                                                                                                                                                                   |
| OS-EXT-SRV-ATTR:user_data            | -                                                                                                                                                                                                          |
| OS-EXT-STS:power_state               | 0                                                                                                                                                                                                          |
| OS-EXT-STS:task_state                | -                                                                                                                                                                                                          |
| OS-EXT-STS:vm_state                  | error                                                                                                                                                                                                      |
| OS-SRV-USG:launched_at               | -                                                                                                                                                                                                          |
| OS-SRV-USG:terminated_at             | -                                                                                                                                                                                                          |
| accessIPv4                           |                                                                                                                                                                                                            |
| accessIPv6                           |                                                                                                                                                                                                            |
| config_drive                         |                                                                                                                                                                                                            |
| created                              | 2017-08-05T14:17:54Z                                                                                                                                                                                       |
| description                          | vm1                                                                                                                                                                                                        |
| fault                                | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 496, in build_instances |
|                                      |     context, request_spec, filter_properties)                                                                                                                                                              |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 567, in _schedule_instances                                                                                                    |
|                                      |     hosts = self.scheduler_client.select_destinations(context, spec_obj)                                                                                                                                   |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 370, in wrapped                                                                                                                  |
|                                      |     return func(*args, **kwargs)                                                                                                                                                                           |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 51, in select_destinations                                                                                             |
|                                      |     return self.queryclient.select_destinations(context, spec_obj)                                                                                                                                         |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 37, in __run_method                                                                                                    |
|                                      |     return getattr(self.instance, __name)(*args, **kwargs)                                                                                                                                                 |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py\", line 32, in select_destinations                                                                                                |
|                                      |     return self.scheduler_rpcapi.select_destinations(context, spec_obj)                                                                                                                                    |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\", line 126, in select_destinations                                                                                                     |
|                                      |     return cctxt.call(ctxt, 'select_destinations', **msg_args)                                                                                                                                             |
|                                      |   File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", line 169, in call                                                                                                                |
|                                      |     retry=self.retry)                                                                                                                                                                                      |
|                                      |   File \"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py\", line 97, in _send                                                                                                                 |
|                                      |     timeout=timeout, retry=retry)                                                                                                                                                                          |
|                                      |   File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 464, in send                                                                                                       |
|                                      |     retry=retry)                                                                                                                                                                                           |
|                                      |   File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 455, in _send                                                                                                      |
|                                      |     raise result                                                                                                                                                                                           |
|                                      | ", "created": "2017-08-05T14:18:14Z"}                                                                                                                                                                      |
| flavor                               | m1.tiny (1)                                                                                                                                                                                                |
| hostId                               |                                                                                                                                                                                                            |
| host_status                          |                                                                                                                                                                                                            |
| id                                   | 20193e58-2c5b-44c6-a98f-a44e2001934f                                                                                                                                                                       |
| image                                | cirros-0.3.4-x86_64 (202778cd-6b32-4486-9444-c167089d9082)                                                                                                                                                 |
| key_name                             | -                                                                                                                                                                                                          |
| locked                               | False                                                                                                                                                                                                      |
| metadata                             | {}                                                                                                                                                                                                         |
| name                                 | vm1                                                                                                                                                                                                        |
| os-extended-volumes:volumes_attached | []                                                                                                                                                                                                         |
| status                               | ERROR                                                                                                                                                                                                      |
| tags                                 | []                                                                                                                                                                                                         |
| tenant_id                            | 0d5998f2f7ec4c4892a32e06bafb19df                                                                                                                                                                           |
| updated                              | 2017-08-05T14:18:16Z                                                                                                                                                                                       |
| user_id                              | 2a5fa182fb1b459980db09cd1572850e                                                                                                                                                                           |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

EDIT-2

nova-compute.log中的/var/log/nova/有用信息:

......
2017-08-05 22:17:42.669 103174 INFO nova.compute.resource_tracker [req-60a062ce-4b3d-4cb7-863e-2f9bba0bc6ec - - - - -] Compute_service record updated for ha-node1:ha-node1
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [req-7dded91f-7497-4d20-ba89-69f867a2a8fb 2a5fa182fb1b459980db09cd1572850e 0d5998f2f7ec4c4892a32e06bafb19df - - -] [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f] Instance failed to spawn
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f] Traceback (most recent call last):
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2078, in _build_resources
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]     yield resources
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1920, in _build_and_run_instance
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]     block_device_info=block_device_info)
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2584, in spawn
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]     admin_pass=admin_password)
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2959, in _create_image
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]     fileutils.ensure_tree(libvirt_utils.get_instance_path(instance))
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]   File "/usr/lib/python2.7/site-packages/oslo_utils/fileutils.py", line 40, in ensure_tree
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]     os.makedirs(path, mode)
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]   File "/usr/lib64/python2.7/os.py", line 157, in makedirs
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]     mkdir(name, mode)
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/20193e58-2c5b-44c6-a98f-a44e2001934f'
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]
2017-08-05 22:18:11.563 103174 INFO nova.compute.manager [req-7dded91f-7497-4d20-ba89-69f867a2a8fb 2a5fa182fb1b459980db09cd1572850e 0d5998f2f7ec4c4892a32e06bafb19df - - -] [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f] Terminating instance
....

首先你需要检查"nova service-list"或"openstack compute service list"的输出。它应该至少显示一个 'nova-compute' 状态为 "Up" 且状态为 'enabled'.

的服务

如果以上没有问题,则计算节点正在与调度程序正常通信。如果没有,那么您需要检查 nova-scheduler 日志。

nova-scheduler 有一系列过滤器,如内存过滤器、CPU 过滤器、聚合过滤器,它适用于根据您 selected 的风格过滤主机。也就是说,如果你 select 具有 16GB RAM 的 flavor,那么调度程序将过滤(内存过滤器)具有可用内存的计算主机。完成所有过滤后,调度程序将尝试在过滤后的主机上启动实例,如果失败,它将尝试在另一台主机上启动。默认尝试次数为 3。所有这些都可以在调度程序日志中看到。这将使您清楚地知道出了什么问题。

您还需要检查 'nova show ' 输出。如果您可以在 "OS-EXT-SRV-ATTR:hypervisor_hostname" 中看到计算主机,那么我们可以理解调度程序能够成功分配计算主机并且计算主机出现问题。在这种情况下,您需要检查该管理程序的 nova-compute 日志。

终于发现我把/var/lib/nova/挂载到nfs目录/mnt/sdb/var/lib/nova/,但是/mnt/sdb/var/lib/nova/权限是root:root,所以我改成nova:nova (与 /var/lib/nova/ 相同)。

命令:

chown -R nova:nova nova

调试

  1. 启用debug mode获取详细日志。

    在这些文件中设置debug = True

    • /etc/nova/nova.conf
    • /etc/nova/cinder.conf
    • /etc/glance/glance-registry.conf
  2. Restart 重新配置的服务

  3. 尝试重新创建实例并检查日志。

    查看 nova-scheduler.log 文件并尝试找到如下行:

    .. INFO nova.filters [req-..] Filter DiskFilter returned 0 hosts
    

    这一行上面应该是包含 Filters 详细信息的 DEBUG 日志,例如:

    .. DEBUG nova.filters [req-..] Filter RetryFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104
    .. DEBUG nova.filters [req-..] Filter AvailabilityZoneFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104
    .. DEBUG nova.filters [req-..] Filter RamFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104
    .. DEBUG nova.filters [req-..] (...) ram: 37107MB disk: 11264MB io_ops: 0 instances: 4 does not have 17408 MB usable disk, it only has 11264.0 MB usable disk. host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/disk_filter.py:70
    

过度使用

OpenStack allows you to overcommit CPU and RAM on compute nodes. This allows you to increase the number of instances running on your cloud at the cost of reducing the performance of the instances. The Compute service uses the following ratios by default:

  • CPU allocation ratio: 16:1
  • RAM allocation ratio: 1.5:1

请阅读 documentation 以获取更多信息。

您可以使用 nova.conf 更改分配比率:

  • cpu_allocation_ratio
  • ram_allocation_ratio
  • disk_allocation_ratio

在所有计算节点中编辑 /etc/nova/nova.conf 并根据您的应用程序要求进行修改。

cpu_allocation_ratio = 2.0 (双物理核心可用于总实例) ram_allocation_ratio = 2.0 (总内存的两倍可用于总实例

在所有计算节点中重启 nova 和 nova-scheduler

systemctl restart openstack-nova-*

systemctl restart openstack-nova-scheduler.service