Pytest 在 GitHub 操作上失败但在本地成功

Question

问题与背景

我在 Python 中创建了一个基于 CLI 的应用程序，它使用 SQLite 来存储用户数据。一次为一个用户安装，与外界没有通信（应用程序更新除外），所有数据都存储在用户计算机本地。

这是使用 Pytest 测试的。在我的本地环境中，所有测试都通过了。但是，他们在 GitHub 操作上失败了。问题似乎出在 SQL 查询中的 RETURNING 关键字。

尝试修复

谷歌搜索后，我发现一个可能的修复方法是更改 OS 版本 (). A different place also suggested different SQLite versions (https://sqlite.org/forum/info/a4dde39b614ec0b2)。我试过很多方法来改变运行环境，但是没有用。

我在 GitHub 操作中尝试了不同版本的 Ubuntu。
我试过使用 container: 设置，并试过 Debian，还有 Ubuntu 21.04.
尝试了不同版本的Python。
我已经尝试过手动构建 Python，使用和不使用每个 libsqlite3-dev 和 sqlite3 apt-get 包，并且都直接使用 make 和通过 pyenv。

环境信息

使用shell命令python -c "import sqlite3; print(sqlite3.version)"，线上环境和我本地机器的版本相同（2.6.0）。

我的本地系统是 WSL 2，Ubuntu21.04。一切都更新到最新的稳定版本，Python 3.10.4 通过 pyenv。

代码和错误

旧版本的动作 YAML（失败）：

name: build and release

on: [push]

jobs:
  test:
    name: pytest
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: '3.10'
      - run: pip install -r requirements.txt pytest
      - uses: cclauss/GitHub-Action-for-pytest@0.5.0
      - run: pytest

动作 YAML 的最新版本（失败）：

name: Test, build and release

# whenever a branch or commit is pushed
on: [push]

jobs:

  # use pytest
  test:

    # used to ensure testing is done right
    env:
      DEVELOPMENT: '1'
    runs-on: ubuntu-latest

    # to avoid using old sqlite version
    container:
      image: debian:latest
      options: --user root

    steps:
      # check out repo
      - uses: actions/checkout@v2

      # prevent from asking user for input
      - run: export DEBIAN_FRONTEND=noninteractive

      # install recommended tools for building Python
      - run: apt -q update
      - run: apt -q install make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev git sqlite3 -y
      - run: apt -q upgrade -y

      # install pyenv
      - run: curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
      - run: exec $SHELL
      - run: ~/.pyenv/bin/pyenv update

      # install and set up required Python
      - run: ~/.pyenv/bin/pyenv install 3.10.2
      - run: ~/.pyenv/bin/pyenv virtualenv 3.10.2 npbc
      - run: ~/.pyenv/bin/pyenv local 3.10.2/envs/npbc

      # print version info (debugging)
      - run: ~/.pyenv/shims/python -V
      - run: ~/.pyenv/shims/python -c "import sqlite3; print(sqlite3.version)"

      # install pip packages
      - run: ~/.pyenv/shims/pip install -r requirements.txt pytest

      # run test
      - run: ~/.pyenv/shims/pytest -vv

GitHub 操作的完整错误输出。

 ~/.pyenv/shims/pytest -vv
  shell: sh -e {0}
  env:
    DEVELOPMENT: 1
============================= test session starts ==============================
platform linux -- Python 3.10.2, pytest-7.1.2, pluggy-1.0.0 -- /github/home/.pyenv/versions/3.10.2/envs/npbc/bin/python3.10
cachedir: .pytest_cache
rootdir: /__w/npbc/npbc
collecting ... collected 21 items

test_core.py::test_get_number_of_each_weekday PASSED                     [  4%]
test_core.py::test_validate_undelivered_string PASSED                    [  9%]
test_core.py::test_undelivered_string_parsing PASSED                     [ 14%]
test_core.py::test_calculating_cost_of_one_paper PASSED                  [ 19%]
test_core.py::test_validate_month_and_year PASSED                        [ 23%]
test_db.py::test_get_papers PASSED                                       [ 28%]
test_db.py::test_get_undelivered_strings PASSED                          [ 33%]
test_db.py::test_delete_paper PASSED                                     [ 38%]
test_db.py::test_add_paper FAILED                                        [ 42%]
test_db.py::test_edit_paper PASSED                                       [ 47%]
test_db.py::test_delete_string PASSED                                    [ 52%]
test_db.py::test_add_string PASSED                                       [ 57%]
test_db.py::test_save_results FAILED                                     [ 61%]
test_regex.py::test_regex_number PASSED                                  [ 66%]
test_regex.py::test_regex_range PASSED                                   [ 71%]
test_regex.py::test_regex_CSVs PASSED                                    [ 76%]
test_regex.py::test_regex_days PASSED                                    [ 80%]
test_regex.py::test_regex_n_days PASSED                                  [ 85%]
test_regex.py::test_regex_all_text PASSED                                [ 90%]
test_regex.py::test_delivery_regex PASSED                                [ 95%]
test_regex.py::test_regex_hyphen PASSED                                  [100%]

=================================== FAILURES ===================================
________________________________ test_add_paper ________________________________

    def test_add_paper():
        setup_db(DATABASE_PATH, SCHEMA_PATH, TEST_SQL)
    
        known_data = [
            (1, 'paper1', 0, 0, 0),
            (1, 'paper1', 1, 1, 6.4),
            (1, 'paper1', 2, 0, 0),
            (1, 'paper1', 3, 0, 0),
            (1, 'paper1', 4, 0, 0),
            (1, 'paper1', 5, 1, 7.9),
            (1, 'paper1', 6, 1, 4),
            (2, 'paper2', 0, 0, 0),
            (2, 'paper2', 1, 0, 0),
            (2, 'paper2', 2, 0, 0),
            (2, 'paper2', 3, 0, 0),
            (2, 'paper2', 4, 1, 3.4),
            (2, 'paper2', 5, 0, 0),
            (2, 'paper2', 6, 1, 8.4),
            (3, 'paper3', 0, 1, 2.4),
            (3, 'paper3', 1, 1, 4.6),
            (3, 'paper3', 2, 0, 0),
            (3, 'paper3', 3, 0, 0),
            (3, 'paper3', 4, 1, 3.4),
            (3, 'paper3', 5, 1, 4.6),
            (3, 'paper3', 6, 1, 6),
            (4, 'paper4', 0, 1, 4),
            (4, 'paper4', 1, 0, 0),
            (4, 'paper4', 2, 1, 2.6),
            (4, 'paper4', 3, 0, 0),
            (4, 'paper4', 4, 0, 0),
            (4, 'paper4', 5, 1, 1),
            (4, 'paper4', 6, 1, 7)
        ]
    
>       npbc_core.add_new_paper(
            'paper4',
            [True, False, True, False, False, True, True],
            [4, 0, 2.6, 0, 0, 1, 7]
        )

test_db.py:150: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'paper4', days_delivered = [True, False, True, False, False, True, ...]
days_cost = [4, 0, 2.6, 0, 0, 1, ...]

    def add_new_paper(name: str, days_delivered: list[bool], days_cost: list[float]) -> None:
        """add a new paper
        - do not allow if the paper already exists"""
    
        global DATABASE_PATH
    
        with connect(DATABASE_PATH) as connection:
    
            # check if the paper already exists
            if connection.execute(
                "SELECT EXISTS (SELECT 1 FROM papers WHERE name = ?);",
                (name,)).fetchone()[0]:
                raise npbc_exceptions.PaperAlreadyExists(f"Paper \"{name}\" already exists.")
    
            # insert the paper
>           paper_id = connection.execute(
                "INSERT INTO papers (name) VALUES (?) RETURNING paper_id;",
                (name,)
            ).fetchone()[0]
E           sqlite3.OperationalError: near "RETURNING": syntax error

npbc_core.py:410: OperationalError
______________________________ test_save_results _______________________________

    def test_save_results():
        setup_db(DATABASE_PATH, SCHEMA_PATH, TEST_SQL)
    
        known_data = [
            (1, 1, 1, 2020, '04/01/2022 01:05:42 AM', '2020-01-01', 105.0),
            (1, 1, 1, 2020, '04/01/2022 01:05:42 AM', '2020-01-02', 105.0),
            (2, 2, 1, 2020, '04/01/2022 01:05:42 AM', '2020-01-03', 51.0),
            (2, 2, 1, 2020, '04/01/2022 01:05:42 AM', '2020-01-01', 51.0),
            (2, 2, 1, 2020, '04/01/2022 01:05:42 AM', '2020-01-05', 51.0)
        ]
    
>       npbc_core.save_results(
            {1: 105, 2: 51, 3: 647},
            {
                1: set([date(month=1, day=1, year=2020), date(month=1, day=2, year=2020)]),
                2: set([date(month=1, day=1, year=2020), date(month=1, day=5, year=2020), date(month=1, day=3, year=2020)]),
                3: set()
            },
            1,
            2020,
            datetime(year=2022, month=1, day=4, hour=1, minute=5, second=42)
        )

test_db.py:291: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
npbc_core.py:339: in save_results
    log_ids = {
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

.0 = <dict_keyiterator object at 0x7f135e5b5350>

    log_ids = {
>       paper_id: connection.execute(
            """
            INSERT INTO logs (paper_id, month, year, timestamp)
            VALUES (?, ?, ?, ?)
            RETURNING log_id;
            """,
            (paper_id, month, year, timestamp)
        ).fetchone()[0]
        for paper_id in costs.keys()
    }
E   sqlite3.OperationalError: near "RETURNING": syntax error

npbc_core.py:340: OperationalError
=========================== short test summary info ============================
FAILED test_db.py::test_add_paper - sqlite3.OperationalError: near "RETURNING...
FAILED test_db.py::test_save_results - sqlite3.OperationalError: near "RETURN...
========================= 2 failed, 19 passed in 0.68s =========================
Error: Process completed with exit code 1.

GitHub 链接

完整的代码库可用。至此，我已经完成了我需要对实际代码进行的大部分更改，并将其推送到GitHub。 https://github.com/eccentricOrange/npbc/tree/5c529dacbef0f9a1f8915a49dcca47834204aa09
在这个分支中，我正在尝试修复在线 Pytest：https://github.com/eccentricOrange/npbc/tree/ci-cd-errors

更新：这已经被合并和删除。拉取请求：https://github.com/eccentricOrange/npbc/pull/24
所有这些都是升级的一部分，在本期中进行了跟踪：https://github.com/eccentricOrange/npbc/issues/20

Answer 1

解决方案是手动构建 SQLite，来自 Reddit 上的这个链：https://www.reddit.com/r/learnprogramming/comments/ulcxr0/comment/i7vryw0/?context=3

这是最终的 YAML，在“原始”运行程序而不是容器中。需要设置 LD_LIBRARY_PATH 变量以确保链接器使用正确的 SQLite。

name: Test

on: [push]

jobs:

  # test with pytest
  test:
    name: pytest
    runs-on: ubuntu-latest

    # used to ensure testing directories are used, not user directories
    env:
      DEVELOPMENT: '1'

    steps:
      - uses: actions/checkout@v2
      
      # build SQLite from source, because I need 3.35<=
      - run: |
          wget https://sqlite.org/2022/sqlite-autoconf-3380500.tar.gz
          tar -xvf sqlite-autoconf-3380500.tar.gz
      - run: |
          ./configure
          make
          sudo make install
          export PATH="/usr/local/lib:$PATH"
        working-directory: sqlite-autoconf-3380500

      # run pytest
      - uses: actions/setup-python@v2
        with:
          python-version: '3.10'
      - run: pip install -r requirements.txt pytest
      - run: pytest
        env:
          LD_LIBRARY_PATH: /usr/local/lib

Pytest 在 GitHub 操作上失败但在本地成功

Pytest is failing on GitHub Actions but succeeds locally

python

sqlite

continuous-integration

containers

github-actions

问题与背景

尝试修复

环境信息

代码和错误

GitHub 链接