Terraform provisioner 无法 winrm 到 Azure 上新建的 Windows VM

Terraform provisioner cannot winrm to new built Windows VM on Azure

我正在尝试使用 Terraform 在 Azure 上配置 windows VM,同时 bootstrap 它。我能想到的方法是 terraform provisioner。出于测试目的,我这样编写配置程序:

provisioner "remote-exec" {
    inline = [
      "md c:/terraform",
    ]
    connection {
      type     = "winrm"
      host   = "${azurerm_network_interface.vmstamp.private_ip_address}"
      user     = "${var.admin_username}"
      password = "${var.admin_password}"
      https    = false
      insecure = true
      timeout = "1m"
    }
}

模板已成功配置 VM,但在尝试将 VM 与 WinRM 连接时出现错误。

azurerm_virtual_machine.vmstamp: Still creating... (5m50s elapsed)
azurerm_virtual_machine.vmstamp (remote-exec): Connecting to remote host via WinRM...
azurerm_virtual_machine.vmstamp (remote-exec):   Host: 10.237.249.146
azurerm_virtual_machine.vmstamp (remote-exec):   Port: 5985
azurerm_virtual_machine.vmstamp (remote-exec):   User: azadmin
azurerm_virtual_machine.vmstamp (remote-exec):   Password: true
azurerm_virtual_machine.vmstamp (remote-exec):   HTTPS: true
azurerm_virtual_machine.vmstamp (remote-exec):   Insecure: true
azurerm_virtual_machine.vmstamp (remote-exec):   NTLM: false
azurerm_virtual_machine.vmstamp (remote-exec):   CACert: false
azurerm_virtual_machine.vmstamp: Still creating... (6m0s elapsed)

Error: Error applying plan:

1 error(s) occurred:

* azurerm_virtual_machine.vmstamp: timeout - last error: unknown error Post https://10.237.249.146:5985/wsman: dial tcp 10.237.249.146:5985: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

我尝试过的几件事:

1) 正在考虑防火墙问题。但是 packer 可以使用以下代码启动连接到同一台笔记本电脑上新建的 windows 虚拟机:

"communicator": "winrm",
"winrm_use_ssl": "true",
"winrm_insecure": "true",
"winrm_timeout": "3m",
"winrm_username": "packer",

2) 尝试了 https=truehttps=false,都失败了。

3) 尝试了 use_ntlm=trueuse_ntlm=false,都失败了。

4) 尝试了 port=5985port=5986,都失败了。 5986 实际上没有在来自市场映像的新 windows VM 中侦听。

terraform provisioner 的正确配置是什么?

针对你的问题,我做了测试。您可以通过 Packer 查看 windows 图像的结果:

可以看到winrm也有超时。另一个对 Terraform 的测试我也得到了 winrm 的超时。所以我建议您可以启用带有 vm 扩展的 winrm,确保端口 5985 已打开,然后使用配置程序执行您想要的操作。

更新

可能有什么遗漏,所以我没有像你一样成功。但是有一个过程要注意,Packer设置了证书的URL。文档 here 还展示了如何使用证书 URL 启用 winrm(如果您想使用它)。所以我认为 Packer 只是自己做,而 Terraform 不会为你做。

@Charles Xu, 注释不能放太多字符,我把输出和代码贴在这里。 加壳脚本输出:

azadmin@AZLAPSLOG1001:~/packer/winrmopen$ /usr/local/packer/packer build --var-file=variables.json Windows2016.json
azure-arm output will be in this color.

==> azure-arm: Running builder ...
    azure-arm: Creating Azure Resource Manager (ARM) client ...
    azure-arm: You have provided Object_ID which is no longer needed, azure packer builder determines this dynamically from the authentication token
==> azure-arm: Using existing resource group ...
==> azure-arm:  -> ResourceGroupName : '###Resource Group Replacement####'
==> azure-arm:  -> Location          : 'canadacentral'
==> azure-arm: Validating deployment template ...
==> azure-arm:  -> ResourceGroupName : '###Resource Group Replacement####'
==> azure-arm:  -> DeploymentName    : 'pkrdp5u8k15bqf0'
==> azure-arm: Deploying deployment template ...
==> azure-arm:  -> ResourceGroupName : '###Resource Group Replacement####'
==> azure-arm:  -> DeploymentName    : 'kvpkrdp5u8k15bqf0'
==> azure-arm: Getting the certificate's URL ...
==> azure-arm:  -> Key Vault Name        : 'pkrkv5u8k15bqf0'
==> azure-arm:  -> Key Vault Secret Name : 'packerKeyVaultSecret'
==> azure-arm:  -> Certificate URL       : 'https://pkrkv5u8k15bqf0.vault.azure.net/secrets/packerKeyVaultSecret/d1a2c8fe28c34b73b65bb53b7a9ea34f'
==> azure-arm: Setting the certificate's URL ...
==> azure-arm: Validating deployment template ...
==> azure-arm:  -> ResourceGroupName : '###Resource Group Replacement####'
==> azure-arm:  -> DeploymentName    : 'pkrdp5u8k15bqf0'
==> azure-arm: Deploying deployment template ...
==> azure-arm:  -> ResourceGroupName : '###Resource Group Replacement####'
==> azure-arm:  -> DeploymentName    : 'pkrdp5u8k15bqf0'
==> azure-arm: Getting the VM's IP address ...
==> azure-arm:  -> ResourceGroupName   : '###Resource Group Replacement####'
==> azure-arm:  -> PublicIPAddressName : 'pkrip5u8k15bqf0'
==> azure-arm:  -> NicName             : 'pkrni5u8k15bqf0'
==> azure-arm:  -> Network Connection  : 'PrivateEndpoint'
==> azure-arm:  -> IP Address          : '10.236.8.92'
==> azure-arm: Waiting for WinRM to become available...
    azure-arm: #< CLIXML
    azure-arm: WinRM connected.
    azure-arm: <Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><Obj S="progress" RefId="0"><TN RefId="0"><T>System.Management.Automation.PSCustomObject</T><T>System.Object</T></TN><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj><Obj S="progress" RefId="1"><TNRef RefId="0" /><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj></Objs>
==> azure-arm: Connected to WinRM!
==> azure-arm: Provisioning with Powershell...

可以很清楚的看到通过winrm成功连接到新建的虚拟机。 以下是我用于打包程序的脚本: 变量

{
  "tenant_id": "###Tenant ID Replacement ###",
  "subscription_id": "###Subscription ID Replacement",
  "region_name": "canadacentral",
  "virtual_network_rg": "###VNET ResourceGroupName replacement ####",
  "virtual_network_name": "###VNET Name Replacement ####",
  "virtual_network_subnet_name": "Presentation",
  "build_resource_group_name": "###Resource Group Replacement####",
  "image_resource_group_Name": "###Resource Group Replacement####",
  "chef_server": "https://chefserver01....",
  "object_id": "f3b76eb8-87e6-42d7-9d12-dafa5b124d90",
  "chef_org_url": "https://artifactory....",
  "chef_org_name": "org name replacement",
  "chef_client_binary": "https://artifactory.......",
  "chef_first_boot": "https://artifactory.......",
  "chef_client_config": "https://artifactory.......",
  "wsus_server": "http://WSUS....",
  "wsus_script": "https://artifactory......",
  "wsus_task": "https://artifactory.....",
  "chef_client_bootstrap": "https://artifactory.....",
  "chef_checkin_task": "https://artifactory......",
  "chef_checkin_xml": "https://artifactory......",
  "image_build_number": "1811261812",
  "client_id": "###Client ID replacement",
  "client_secret": "###Client Secret replacement"
}

windows.json

{
  "variables": {
    "client_secret": "{{user `client_secret`}}"
  },  
  "builders": [{
  "type": "azure-arm",
  "client_id": "{{user `client_id`}}",
  "client_secret": "{{user `client_secret`}}",
  "tenant_id": "{{user `tenant_id`}}",
  "subscription_id": "{{user `subscription_id`}}",
  "object_id": "{{user `object_id`}}",
  "build_resource_group_name": "{{user `build_resource_group_name`}}",

  "managed_image_resource_group_name": "{{user `image_resource_group_Name`}}",
  "managed_image_name": "Windows2016-dev-{{user `image_build_number`}}",

  "os_type": "Windows",
  "image_publisher": "MicrosoftWindowsServer",
  "image_offer": "WindowsServer",
  "image_sku": "2016-Datacenter",
  "image_version":"latest",
  "os_disk_size_gb":"128", 
  "vm_size": "Standard_E4S_v3",
  "virtual_network_resource_group_name":"{{user `virtual_network_rg`}}",
  "virtual_network_name":"{{user `virtual_network_name`}}",
  "virtual_network_subnet_name":"{{user `virtual_network_subnet_name`}}",



  "communicator": "winrm",
  "winrm_use_ssl": "true",
  "winrm_insecure": "true",
  "winrm_timeout": "3m",
  "winrm_username": "packer",

  "azure_tags": {
      "dept": "Platform Engineering",
      "task": "Daily Image Build",
      "CostCenter": "3357",
      "UseCase": "Standard Image Builds - GSPE",
      "ReleaseDate": "{{isotime `2006-01-02`}}"
  }
  }],
  "provisioners": [ 
    {
    "type": "powershell",
    "environment_vars": [
      "chef_org_url={{user `chef_org_url`}}",
      "chef_org_name={{user `chef_org_name`}}",
      "chef_client_binary={{user `chef_client_binary`}}",
      "chef_client_config={{user `chef_client_config`}}",
      "chef_first_boot={{user `chef_first_boot`}}",
      "chef_client_bootstrap={{user `chef_client_bootstrap`}}",
      "wsus_server={{user `wsus_server`}}",
      "wsus_script={{user `wsus_script`}}",
      "wsus_task={{user `wsus_task`}}",
      "chef_setrunonce={{user `chef_setrunonce`}}",
      "chef_server={{user `chef_server`}}",
      "chef_checkin_task={{user `chef_checkin_task`}}",
      "chef_checkin_xml={{user `chef_checkin_xml`}}"

    ],  
    "inline": [
      "$chef_org_client=$ENV:chef_org_name + '-validator'",
      "$chef_org_validator_filename=$chef_org_client + '.pem'",
      "$chef_org_validator_local='c:\chef\' + $chef_org_validator_filename",
      "$chef_org_validator_url=$ENV:chef_org_url + $chef_org_validator_filename",


      "Write-Host Setting TLS 1.2",
      "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12",

      "Write-Host Download Chef Client, Org Validator and First Boot",
      "Write-Host Full ORG Validator PEM FILE - $chef_org_validator_url",
      "Write-Host Chef Org Name - $env:chef_org_name",
      "Write-Host WSUS Server Name - $env:wsus_server",
      "Write-Host Chef Server - $env:chef_server",
      "md $env:windir\Temp\packer-chef-client -Force",
      "md $env:SystemDrive\chef\ -Force",
      "iwr -Uri $chef_org_validator_url -Outfile $chef_org_validator_local",
      "iwr -Uri $ENV:chef_client_binary -Outfile $env:windir\Temp\packer-chef-client\chef.msi",
      "iwr -Uri $ENV:chef_client_config -Outfile $env:SystemDrive\chef\client.rb",
      "iwr -Uri $ENV:chef_first_boot -Outfile $env:SystemDrive\chef\first-boot.json",
      "iwr -Uri $ENV:chef_client_bootstrap -Outfile $env:windir\Temp\packer-chef-client\chef_client_bootstrap.ps1",
      "iwr -Uri $ENV:chef_checkin_task -Outfile $env:windir\Temp\packer-chef-client\chef_checkin_task.ps1",
      "iwr -Uri $ENV:chef_checkin_xml -Outfile $env:windir\Temp\packer-chef-client\ChefRunOnce.xml",
      "iwr -Uri $ENV:wsus_script -Outfile $env:windir\Temp\packer-chef-client\WSUSUpdate.ps1",
      "iwr -Uri $ENV:wsus_task -Outfile $env:windir\Temp\packer-chef-client\DownloadStageChef-modded.ps1",

      "Write-Host Install Chef Client",
      "Start-Process 'msiexec' -ArgumentList '/qb /i C:\Windows\Temp\packer-chef-client\chef.msi' -NoNewWindow -Wait",

      "Write-Host Set Regional WSUS Server",
      "reg add 'HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate' /v WUServer /t REG_SZ /d $ENV:wsus_server /f",
      "reg add 'HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate' /v WUStatusServer /t REG_SZ /d $ENV:wsus_server /f",

      "Write-Host Run WSUS Update",
      "C:\Windows\Temp\packer-chef-client\DownloadStageChef-modded.ps1",

      "Write-Host Chef RunOnce Task",
      "C:\Windows\Temp\packer-chef-client\chef_checkin_task.ps1 -chef_org_name $env:chef_org_name -chef_server $env:chef_server"
    ]
  },
  {
    "type": "powershell",
    "inline": [
      "if( Test-Path $Env:SystemRoot\windows\system32\Sysprep\unattend.xml ){ rm $Env:SystemRoot\windows\system32\Sysprep\unattend.xml -Force}",
      "& $env:SystemRoot\System32\Sysprep\Sysprep.exe /oobe /generalize /quiet /quit",
      "while($true) { $imageState = Get-ItemProperty HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\State | Select ImageState; if($imageState.ImageState -ne 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') { Write-Output $imageState.ImageState; Start-Sleep -s 10  } else { break } }"
    ]
  }
]
}

后来发现winrm需要添加证书才能启用winrm。要添加证书,我们需要先创建一个密钥库和存储帐户。 Packer 为我们完成了这些步骤,而 Terraform 则没有。因此,我们必须在 terraform 模板中构建这些步骤以启用证书,然后启用 winrm。 该计划是 bootstrap 一个 windows 虚拟机,而无需创建额外的资源,如存储帐户或密钥存储,因为我需要在配置后清理它们,这会使模板复杂化。如果无论如何我都必须创建额外的资源,我更愿意使用 Azure VM 扩展,因为它不需要在 NSG 上启用 winrm 端口 5985,这是另一个大的治理问题。