使用 moto 与服务器通信
Using moto to communiate with server
我正在尝试使用 moto 来实施集成测试，并尝试测试一些要使用 cloudformation 实施的自定义资源。我正在使用 github page 中提供的示例。我稍微修改了示例以适合我的开发环境并接受 yaml 文件而不是 jsons 这也适合我的部署策略。我已经阅读了示例中的文档和代码，并启动了 cf.create_stack 命令所需的服务器。该示例调用方法
def wait_for_log_msg(expected_msg, log_group):
...
方法 returns True 和 set 如果 expected_msg 在收到的消息中，否则它 returns False 和a set（代码见下方）
问题：
根据示例，发送的消息是 Status code: 200，但未收到该消息，当我打印这些消息时，我得到 failed executing http.request。我假设这是由于我对 motoserver.
的一些错误方法造成的
问题：
如何获得 motoserver，特别是 logstream 以输出所需的 Status code: 200 消息？
尝试次数：
我尝试使用 docker 启动 motoserver，但出现以下错误：
{"error running docker: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))"}
然后我直接切换到 运行 motoserver 通过在终端中使用
调用它
moto_server -p5000
这会导致失败，当我打印日志消息时，我得到（经过格式化以使其更易于阅读）：
{
'\x1b[32mEND RequestId: 50fe51a1-46e2-16d2-c274-8a50604d2be9\x1b[0m',
"send(..) failed executing http.request(..): HTTPConnectionPool(host='host.docker.internal', port=5000): Max retries exceeded with url: /cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f58b9eabf40>: Failed to establish a new connection: [Errno 111] Connection refused'))",
'\x1b[32mSTART RequestId: 50fe51a1-46e2-16d2-c274-8a50604d2be9 Version: $LATEST\x1b[0m',
'',
"[WARNING]\t2022-05-01T05:07:18.969Z\t50fe51a1-46e2-16d2-c274-8a50604d2be9\tRetrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f58b9eab520>: Failed to establish a new connection: [Errno 111] Connection refused')': /cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776",
"[WARNING]\t2022-05-01T05:07:18.968Z\t50fe51a1-46e2-16d2-c274-8a50604d2be9\tRetrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f58b9eab1f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776",
'\x1b[32mREPORT RequestId: 50fe51a1-46e2-16d2-c274-8a50604d2be9\tInit Duration: 183.50 ms\tDuration: 8.25 ms\tBilled Duration: 9 ms\tMemory Size: 128 MB\tMax Memory Used: 27 MB\t\x1b[0m',
"{
'RequestType': 'Create',
'ServiceToken': 'arn:aws:lambda:us-east-1:123456789012:function:stack10d4d2-InfoFunction-M4ZP2MX8XYLL',
'ResponseURL': 'http://host.docker.internal:5000/cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776',
'StackId': 'arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776',
'RequestId': '046dc61f-d786-4672-b1a9-48eb286593e2',
'LogicalResourceId': 'CustomInfo',
'ResourceType': 'Custom::Info',
'ResourceProperties': {
'ServiceToken': 'arn:aws:lambda:us-east-1:123456789012:function:stack10d4d2-InfoFunction-M4ZP2MX8XYLL',
'Region': 'us-east-1',
'MyProperty': 'stuff'
}
}",
'http://host.docker.internal:5000/cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776',
'{
"Status": "SUCCESS",
"Reason": "See the details in CloudWatch Log Stream: 2022/05/01/[$LATEST]b39c266d1989dc1bc8ec6dfecf2154d7",
"PhysicalResourceId": "2022/05/01/[$LATEST]b39c266d1989dc1bc8ec6dfecf2154d7",
"StackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776",
"RequestId": "046dc61f-d786-4672-b1a9-48eb286593e2",
"LogicalResourceId": "CustomInfo",
"NoEcho": false,
"Data": {
"info_value": "special value"
}
}',
'Response body:',
'null',
"[WARNING]\t2022-05-01T05:07:18.970Z\t50fe51a1-46e2-16d2-c274-8a50604d2be9\tRetrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f58b9eab730>: Failed to establish a new connection: [Errno 111] Connection refused')': /cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776"
}
其中我认为最有趣的部分是：
"send(..) failed executing http.request(..): HTTPConnectionPool(host='host.docker.internal', port=5000): Max retries exceeded with url: /cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f58b9eabf40>: Failed to establish a new connection: [Errno 111] Connection refused'))",
代码：
测试代码
# Libraries
# Standard Libraries
import inspect
import json
import os
import sys
import time
from uuid import uuid4
import yaml
# Paths
PACKAGE_PARENT = ".."
SCRIPT_DIR = os.path.dirname(
os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__)))
)
sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))
# Third Party Librairies
import boto3
from moto import mock_cloudformation, mock_lambda, mock_logs, mock_s3, settings
import pytest
from pytest import assume
# User Defined Libraries
from .fixtures.custom_lambda import get_template
import src.app.App as lambda_function
from .test_awslambda.utilities import wait_for_log_msg
def get_log_group_name(cf, stack_name):
...
def get_outputs(cf, stack_name):
...
@pytest.fixture(scope="function")
def aws_credentials():
"""
Mocked AWS Credentials for moto.
"""
os.environ["AWS_ACCESS_KEY_ID"] = "testing"
os.environ["AWS_SECRET_ACCESS_KEY"] = "testing"
os.environ["AWS_SECURITY_TOKEN"] = "testing"
os.environ["AWS_SESSION_TOKEN"] = "testing"
os.environ["AWS_DEFAULT_REGION"] = "us-east-1"
@mock_cloudformation
@mock_lambda
@mock_logs
@mock_s3
def test_create_custom_lambda_resource():
#########
# Integration test using a Custom Resource
# Create a Lambda
# CF will call the Lambda
# The Lambda should call CF, to indicate success (using the cfnresponse-module)
# This HTTP request will include any outputs that are now stored against the stack
# TEST: verify that this output is persisted
##########
if not settings.TEST_SERVER_MODE:
raise pytest.skip(
"Needs a standalone MotoServer, as cfnresponse needs to connect to something"
)
# Create cloudformation stack
stack_name = f"stack{str(uuid4())[0:6]}"
code = inspect.getsource(lambda_function)
template_body = get_template(code)
cf = boto3.client(
"cloudformation",
region_name="us-east-1",
endpoint_url="http://localhost:5000",
)
cf.create_stack(
StackName=stack_name,
TemplateBody=template_body,
# TemplateBody=json.dumps(template_body),
Capabilities=["CAPABILITY_IAM"],
)
# Verify CloudWatch contains the correct logs
log_group_name = get_log_group_name(cf, stack_name)
success, logs = wait_for_log_msg(
expected_msg="Status code: 200", log_group=log_group_name
)
print(f"Logs should indicate success: \n{logs}")
# Verify the correct Output was returned
outputs = get_outputs(cf, stack_name)
with assume:
assert success is True
with assume:
assert len(outputs) == 1
所有这些几乎都是 github 页面上可用内容的复制粘贴，我在该页面上更改代码以使用 inspect 将其提供给 [=18] 从另一个文件导入函数=] 版本的 cf 模板并通过它尝试模拟自定义资源的创建
完整性wait_for_log_msg:
def wait_for_log_msg(expected_msg, log_group):
logs_conn = boto3.client("logs", region_name="us-east-1")
received_messages = []
start = time.time()
while (time.time() - start) < 30:
try:
result = logs_conn.describe_log_streams(logGroupName=log_group)
log_streams = result.get("logStreams")
except ClientError:
log_streams = None # LogGroupName does not yet exist
if not log_streams:
time.sleep(1)
continue
for log_stream in log_streams:
result = logs_conn.get_log_events(
logGroupName=log_group,
logStreamName=log_stream["logStreamName"],
)
received_messages.extend(
[event["message"] for event in result.get("events")]
)
for line in received_messages:
if expected_msg in line:
return True, set(received_messages)
time.sleep(1)
return False, set(received_messages)
这是 github 页面内容的复制粘贴。
默认情况下，外部连接无法访问 MotoServer。
用于创建 CustomResource 的 LambdaFunction 在 Docker-container 内执行，算作 'outside' 连接 - 这就是您看到 NewConnectionError.
的原因
尝试像这样启动 MotoServer，这会向外部连接打开服务器：
moto_server -h 0.0.0.0
（请注意，我没有指定端口，因为 5000 是默认端口 - 根据需要进行调整。）
据我所知，Docker-exception 与 Moto 无关 - 请参阅此 SO 问题以获取可能的解决方案：
docker.errors.DockerException: Error while fetching server API version
我正在尝试使用 moto 来实施集成测试，并尝试测试一些要使用 cloudformation 实施的自定义资源。我正在使用 github page 中提供的示例。我稍微修改了示例以适合我的开发环境并接受 yaml 文件而不是 jsons 这也适合我的部署策略。我已经阅读了示例中的文档和代码，并启动了 cf.create_stack 命令所需的服务器。该示例调用方法
def wait_for_log_msg(expected_msg, log_group):
...
方法 returns True 和 set 如果 expected_msg 在收到的消息中，否则它 returns False 和a set（代码见下方）
问题：
根据示例，发送的消息是 Status code: 200，但未收到该消息，当我打印这些消息时，我得到 failed executing http.request。我假设这是由于我对 motoserver.
问题：
如何获得 motoserver，特别是 logstream 以输出所需的 Status code: 200 消息？
尝试次数：
我尝试使用 docker 启动 motoserver，但出现以下错误：
{"error running docker: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))"}
然后我直接切换到 运行 motoserver 通过在终端中使用
moto_server -p5000
这会导致失败，当我打印日志消息时，我得到（经过格式化以使其更易于阅读）：
{
'\x1b[32mEND RequestId: 50fe51a1-46e2-16d2-c274-8a50604d2be9\x1b[0m',
"send(..) failed executing http.request(..): HTTPConnectionPool(host='host.docker.internal', port=5000): Max retries exceeded with url: /cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f58b9eabf40>: Failed to establish a new connection: [Errno 111] Connection refused'))",
'\x1b[32mSTART RequestId: 50fe51a1-46e2-16d2-c274-8a50604d2be9 Version: $LATEST\x1b[0m',
'',
"[WARNING]\t2022-05-01T05:07:18.969Z\t50fe51a1-46e2-16d2-c274-8a50604d2be9\tRetrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f58b9eab520>: Failed to establish a new connection: [Errno 111] Connection refused')': /cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776",
"[WARNING]\t2022-05-01T05:07:18.968Z\t50fe51a1-46e2-16d2-c274-8a50604d2be9\tRetrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f58b9eab1f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776",
'\x1b[32mREPORT RequestId: 50fe51a1-46e2-16d2-c274-8a50604d2be9\tInit Duration: 183.50 ms\tDuration: 8.25 ms\tBilled Duration: 9 ms\tMemory Size: 128 MB\tMax Memory Used: 27 MB\t\x1b[0m',
"{
'RequestType': 'Create',
'ServiceToken': 'arn:aws:lambda:us-east-1:123456789012:function:stack10d4d2-InfoFunction-M4ZP2MX8XYLL',
'ResponseURL': 'http://host.docker.internal:5000/cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776',
'StackId': 'arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776',
'RequestId': '046dc61f-d786-4672-b1a9-48eb286593e2',
'LogicalResourceId': 'CustomInfo',
'ResourceType': 'Custom::Info',
'ResourceProperties': {
'ServiceToken': 'arn:aws:lambda:us-east-1:123456789012:function:stack10d4d2-InfoFunction-M4ZP2MX8XYLL',
'Region': 'us-east-1',
'MyProperty': 'stuff'
}
}",
'http://host.docker.internal:5000/cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776',
'{
"Status": "SUCCESS",
"Reason": "See the details in CloudWatch Log Stream: 2022/05/01/[$LATEST]b39c266d1989dc1bc8ec6dfecf2154d7",
"PhysicalResourceId": "2022/05/01/[$LATEST]b39c266d1989dc1bc8ec6dfecf2154d7",
"StackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776",
"RequestId": "046dc61f-d786-4672-b1a9-48eb286593e2",
"LogicalResourceId": "CustomInfo",
"NoEcho": false,
"Data": {
"info_value": "special value"
}
}',
'Response body:',
'null',
"[WARNING]\t2022-05-01T05:07:18.970Z\t50fe51a1-46e2-16d2-c274-8a50604d2be9\tRetrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f58b9eab730>: Failed to establish a new connection: [Errno 111] Connection refused')': /cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776"
}
其中我认为最有趣的部分是：
"send(..) failed executing http.request(..): HTTPConnectionPool(host='host.docker.internal', port=5000): Max retries exceeded with url: /cloudformation_us-east-1/cfnresponse?stack=arn:aws:cloudformation:us-east-1:123456789012:stack/stack10d4d2/ce9420fc-d0e5-4e71-949b-893537701776 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f58b9eabf40>: Failed to establish a new connection: [Errno 111] Connection refused'))",
代码：
测试代码
# Libraries
# Standard Libraries
import inspect
import json
import os
import sys
import time
from uuid import uuid4
import yaml
# Paths
PACKAGE_PARENT = ".."
SCRIPT_DIR = os.path.dirname(
os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__)))
)
sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))
# Third Party Librairies
import boto3
from moto import mock_cloudformation, mock_lambda, mock_logs, mock_s3, settings
import pytest
from pytest import assume
# User Defined Libraries
from .fixtures.custom_lambda import get_template
import src.app.App as lambda_function
from .test_awslambda.utilities import wait_for_log_msg
def get_log_group_name(cf, stack_name):
...
def get_outputs(cf, stack_name):
...
@pytest.fixture(scope="function")
def aws_credentials():
"""
Mocked AWS Credentials for moto.
"""
os.environ["AWS_ACCESS_KEY_ID"] = "testing"
os.environ["AWS_SECRET_ACCESS_KEY"] = "testing"
os.environ["AWS_SECURITY_TOKEN"] = "testing"
os.environ["AWS_SESSION_TOKEN"] = "testing"
os.environ["AWS_DEFAULT_REGION"] = "us-east-1"
@mock_cloudformation
@mock_lambda
@mock_logs
@mock_s3
def test_create_custom_lambda_resource():
#########
# Integration test using a Custom Resource
# Create a Lambda
# CF will call the Lambda
# The Lambda should call CF, to indicate success (using the cfnresponse-module)
# This HTTP request will include any outputs that are now stored against the stack
# TEST: verify that this output is persisted
##########
if not settings.TEST_SERVER_MODE:
raise pytest.skip(
"Needs a standalone MotoServer, as cfnresponse needs to connect to something"
)
# Create cloudformation stack
stack_name = f"stack{str(uuid4())[0:6]}"
code = inspect.getsource(lambda_function)
template_body = get_template(code)
cf = boto3.client(
"cloudformation",
region_name="us-east-1",
endpoint_url="http://localhost:5000",
)
cf.create_stack(
StackName=stack_name,
TemplateBody=template_body,
# TemplateBody=json.dumps(template_body),
Capabilities=["CAPABILITY_IAM"],
)
# Verify CloudWatch contains the correct logs
log_group_name = get_log_group_name(cf, stack_name)
success, logs = wait_for_log_msg(
expected_msg="Status code: 200", log_group=log_group_name
)
print(f"Logs should indicate success: \n{logs}")
# Verify the correct Output was returned
outputs = get_outputs(cf, stack_name)
with assume:
assert success is True
with assume:
assert len(outputs) == 1
所有这些几乎都是 github 页面上可用内容的复制粘贴，我在该页面上更改代码以使用 inspect 将其提供给 [=18] 从另一个文件导入函数=] 版本的 cf 模板并通过它尝试模拟自定义资源的创建
完整性wait_for_log_msg:
def wait_for_log_msg(expected_msg, log_group):
logs_conn = boto3.client("logs", region_name="us-east-1")
received_messages = []
start = time.time()
while (time.time() - start) < 30:
try:
result = logs_conn.describe_log_streams(logGroupName=log_group)
log_streams = result.get("logStreams")
except ClientError:
log_streams = None # LogGroupName does not yet exist
if not log_streams:
time.sleep(1)
continue
for log_stream in log_streams:
result = logs_conn.get_log_events(
logGroupName=log_group,
logStreamName=log_stream["logStreamName"],
)
received_messages.extend(
[event["message"] for event in result.get("events")]
)
for line in received_messages:
if expected_msg in line:
return True, set(received_messages)
time.sleep(1)
return False, set(received_messages)
这是 github 页面内容的复制粘贴。
默认情况下，外部连接无法访问 MotoServer。
用于创建 CustomResource 的 LambdaFunction 在 Docker-container 内执行，算作 'outside' 连接 - 这就是您看到 NewConnectionError.
尝试像这样启动 MotoServer，这会向外部连接打开服务器：
moto_server -h 0.0.0.0
（请注意，我没有指定端口，因为 5000 是默认端口 - 根据需要进行调整。）
据我所知，Docker-exception 与 Moto 无关 - 请参阅此 SO 问题以获取可能的解决方案： docker.errors.DockerException: Error while fetching server API version