无法在 Rails 测试环境中使用 TimescaleDB

Unable to use TimescaleDB in Rails test environment

我在 Rails 中无法使用 TimescaleDB - 在开发中一切正常,但在我的测试套件中我无法插入任何数据。

我试过的

A) 使用 SQL 模式转储

这导致了我看到的原始错误消息。它确实为 TimescaleDB 创建了部分模式,但不是全部。我有一个 hypertable 但它不能正常工作

B) 使用 Ruby 模式转储

这让我可以插入我的 table 但它根本不是一个 hypertable - ruby 语法丢失了与 TimescaleDB 和 hypertables 相关的所有内容。

C) 直接迁移测试数据库

我尝试使用以下内容避免 schema.structure 转储和加载:

$ rails db:drop
Dropped database 'my_app_development'
Dropped database 'my_app_test'
$ RAILS_ENV=test rails db:create
Created database 'my_app_test'
$ RAILS_ENV=test rails db:migrate
== 20200517164444 EnableTimescaledbExtension: migrating =======================
-- enable_extension("timescaledb")
WARNING:  
WELCOME TO
 _____ _                               _     ____________  
|_   _(_)                             | |    |  _  \ ___ \ 
  | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ / 
  | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \ 
  | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
  |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
               Running version 1.7.0
For more information on TimescaleDB, please visit the following links:

 1. Getting started: https://docs.timescale.com/getting-started
 2. API reference documentation: https://docs.timescale.com/api
 3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture

Note: TimescaleDB collects anonymous reports to better understand and assist our users.
For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.

   -> 0.2315s
== 20200517164444 EnableTimescaledbExtension: migrated (0.2316s) ==============

== 20200517165027 CreateAccounts: migrating ===================================
-- create_table(:accounts)
   -> 0.0095s
== 20200517165027 CreateAccounts: migrated (0.0095s) ==========================

== 20200517165103 CreateMetrics: migrating ====================================
-- create_table(:metrics)
   -> 0.0116s
== 20200517165103 CreateMetrics: migrated (0.0117s) ===========================

== 20200517170842 CreateEvents: migrating =====================================
-- create_table(:events)
   -> 0.0072s
-- remove_column(:events, :id)
   -> 0.0020s
-- execute("SELECT create_hypertable('events', 'time');\n")
   -> 0.0047s
== 20200517170842 CreateEvents: migrated (0.0142s) ============================

pg_dump: warning: there are circular foreign-key constraints on this table:
pg_dump:   hypertable
pg_dump: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints.
pg_dump: Consider using a full dump instead of a --data-only dump to avoid this problem.
pg_dump: warning: there are circular foreign-key constraints on this table:
pg_dump:   chunk
pg_dump: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints.
pg_dump: Consider using a full dump instead of a --data-only dump to avoid this problem.

但是当 运行 测试套件与尝试 A 相同时。

运行 之后的测试实际上打印了几次此消息,这让我认为 Rails 自动神奇地再次使用 structure.sql 重新创建测试数据库:


psql:/home/axel/src/my_app/db/structure.sql:16: WARNING:  
WELCOME TO
 _____ _                               _     ____________  
|_   _(_)                             | |    |  _  \ ___ \ 
  | |  _ _ __ ___   ___  ___  ___ __ _| | ___| | | | |_/ / 
  | | | |  _ ` _ \ / _ \/ __|/ __/ _` | |/ _ \ | | | ___ \ 
  | | | | | | | | |  __/\__ \ (_| (_| | |  __/ |/ /| |_/ /
  |_| |_|_| |_| |_|\___||___/\___\__,_|_|\___|___/ \____/
               Running version 1.7.0
For more information on TimescaleDB, please visit the following links:

 1. Getting started: https://docs.timescale.com/getting-started
 2. API reference documentation: https://docs.timescale.com/api
 3. How TimescaleDB is designed: https://docs.timescale.com/introduction/architecture

Note: TimescaleDB collects anonymous reports to better understand and assist our users.
For more information and how to disable, please see our docs https://docs.timescaledb.com/using-timescaledb/telemetry.

错误信息

$ rails test
Running via Spring preloader in process 107937
Run options: --seed 29840

# Running:

E

Error:
Api::EventsControllerTest#test_POST_event_data_-_new_metric:
DRb::DRbRemoteError: PG::FeatureNotSupported: ERROR:  invalid INSERT on the root table of hypertable "events"
HINT:  Make sure the TimescaleDB extension has been preloaded.
 (ActiveRecord::StatementInvalid)
    app/controllers/api/events_controller.rb:5:in `create'
    test/controllers/api/events_controller_test.rb:9:in `block in <class:EventsControllerTest>'


rails test test/controllers/api/events_controller_test.rb:8



Finished in 0.215286s, 4.6450 runs/s, 0.0000 assertions/s.
1 runs, 0 assertions, 0 failures, 1 errors, 0 skips

我感觉这与 Rails 如何使用 schema.rb(默认 config.active_record.schema_format = :ruby)或 structure.sqlconfig.active_record.schema_format = :sql. 我已经尝试了结构的 Ruby 和 SQL 设置,但都不起作用 - 开发数据库已正确迁移,但测试数据库设置不正确。

在下面的两个数据库(开发和测试)中我们可以看到唯一的区别是缺少测试数据库:Child tables: _timescaledb_internal._hyper_1_1_chunk

开发数据库

$ psql -d my_app_development
psql (12.2)
Type "help" for help.

my_app_development=# SHOW shared_preload_libraries;
 shared_preload_libraries 
--------------------------
 timescaledb
(1 row)

my_app_development=# insert into events (metric_id, time, value) VALUES (1, NOW(), 22);
INSERT 0 1
my_app_development=# \d+ events
                                              Table "public.events"
  Column   |            Type             | Collation | Nullable | Default | Storage | Stats target | Description 
-----------+-----------------------------+-----------+----------+---------+---------+--------------+-------------
 metric_id | bigint                      |           |          |         | plain   |              | 
 time      | timestamp without time zone |           | not null |         | plain   |              | 
 value     | numeric                     |           |          |         | main    |              | 
Indexes:
    "events_time_idx" btree ("time" DESC)
Triggers:
    ts_insert_blocker BEFORE INSERT ON events FOR EACH ROW EXECUTE FUNCTION _timescaledb_internal.insert_blocker()
Child tables: _timescaledb_internal._hyper_1_1_chunk
Access method: heap

测试数据库

$ psql -d my_app_test
psql (12.2)
Type "help" for help.

my_app_test=# SHOW shared_preload_libraries;
 shared_preload_libraries 
--------------------------
 timescaledb
(1 row)

my_app_test=# insert into events (metric_id, time, value) VALUES (1, NOW(), 22);
ERROR:  invalid INSERT on the root table of hypertable "events"
HINT:  Make sure the TimescaleDB extension has been preloaded.
my_app_test=# \d+ events
                                              Table "public.events"
  Column   |            Type             | Collation | Nullable | Default | Storage | Stats target | Description 
-----------+-----------------------------+-----------+----------+---------+---------+--------------+-------------
 metric_id | bigint                      |           |          |         | plain   |              | 
 time      | timestamp without time zone |           | not null |         | plain   |              | 
 value     | numeric                     |           |          |         | main    |              | 
Indexes:
    "events_time_idx" btree ("time" DESC)
Triggers:
    ts_insert_blocker BEFORE INSERT ON events FOR EACH ROW EXECUTE FUNCTION _timescaledb_internal.insert_blocker()
Access method: heap

具有 SQL 架构的 ActiveRecord

CREATE EXTENSION IF NOT EXISTS timescaledb WITH SCHEMA public;

SET default_tablespace = '';

SET default_table_access_method = heap;

CREATE TABLE public.events (
    metric_id bigint,
    "time" timestamp without time zone NOT NULL,
    value numeric
);

CREATE INDEX events_time_idx ON public.events USING btree ("time" DESC);

CREATE TRIGGER ts_insert_blocker BEFORE INSERT ON public.events FOR EACH ROW EXECUTE FUNCTION _timescaledb_internal.insert_blocker();

具有 Ruby 架构的 ActiveRecord

ActiveRecord::Schema.define(version: 2020_05_17_170842) do

  # These are extensions that must be enabled in order to support this database
  enable_extension "plpgsql"
  enable_extension "timescaledb"

  create_table "events", id: false, force: :cascade do |t|
    t.bigint "metric_id"
    t.datetime "time", null: false
    t.decimal "value"
    t.index ["time"], name: "events_time_idx", order: :desc
  end
end

注意:这会松开 ts_insert_blocker 触发器并让我插入 events table 但它不再是 hypertable:


my_app_test=# \d+ events
                                              Table "public.events"
  Column   |            Type             | Collation | Nullable | Default | Storage | Stats target | Description 
-----------+-----------------------------+-----------+----------+---------+---------+--------------+-------------
 metric_id | bigint                      |           |          |         | plain   |              | 
 time      | timestamp without time zone |           | not null |         | plain   |              | 
 value     | numeric                     |           |          |         | main    |              | 
Indexes:
    "events_time_idx" btree ("time" DESC)
Access method: heap

附加信息

相关问题:Running an RSpec test suite against a TimescaleDB database with Rails 4.2 - 这些建议对我不起作用,没有可接受的答案。

版本信息:

编辑 1

我将以下内容添加到我的 test/test_helper.rb 中,类似于 @cstabru

提到的解决方法
def execute_create_hypertable(sql)
  ActiveRecord::Base.connection.execute(sql)
rescue ActiveRecord::StatementInvalid => e
  raise e unless e.message.include? 'is already a hypertable'
end

execute_create_hypertable <<~SQL
  SELECT create_hypertable('events', 'time');
SQL

但也许我们可以在初始化程序中使用类似 SELECT create_hypertable('hypertable_name', 'time_field', if_not_exists => TRUE 的东西,而不是在数据库迁移中创建 hypertables?

我 运行 也参与其中,无论我以何种方式重新创建数据库架构(sql 或 ruby 格式),hyper table 都不会重新创建为时间刻度内部架构数据未导出。

请注意,当我使用 sql 格式恢复时,它会在 ts_insert_blocker 触发器中复制,这确实中断了 table 上的插入并出现此错误(我相信是由于触发功能不可用)

PG::FeatureNotSupported: ERROR:  invalid INSERT on the root table of hypertable "hypertable_name"
HINT:  Make sure the TimescaleDB extension has been preloaded.

要解决根本问题(sql 或 ruby 格式),我们可以通过以下方式手动重新创建 hypertable(并删除触发器)

DROP TRIGGER IF EXISTS ts_insert_blocker ON events;
DROP TRIGGER

SELECT create_hypertable('hypertable_name', 'time_field', if_not_exists => TRUE);
....
(1 row)

现在手动检查 hypertable 自 https://github.com/timescale/timescaledb/pull/862

以来是否存在
SELECT * FROM timescaledb_information.hypertable;

我已将这些 DDL 命令添加到我的 spec_helper.rb 以确保测试数据库使用实际的 hypertable。我想确保测试数据库模式反映我的生产/暂存设置。

config.before(:suite) do
  # ensure the hypertable_name hypertable is setup correctly
  ActiveRecord::Base.connection.execute(
    "DROP TRIGGER IF EXISTS ts_insert_blocker ON hypertable_name;"
  )
  ActiveRecord::Base.connection.execute(
    "SELECT create_hypertable('hypertable_name', 'time_field', if_not_exists => TRUE);"
  )
  has_hypertables_sql = "SELECT * FROM timescaledb_information.hypertable WHERE table_name = 'hypertable_name';"
  if ActiveRecord::Base.connection.execute(has_hypertables_sql).to_a.empty?
    raise "TimescaleDB missing hypertable on 'hypertable_name' table"
  end
end

如果人们觉得这很有用,我可以考虑提取到 gem 以帮助恢复 rails 环境的模式,https://github.com/timescale/timescaledb/issues/1916

以防有人在 spec_helper.rb 中使用 cstabru's 的 DDL 命令并得到错误 PG::UndefinedTable: ERROR: Relation »timescaledb_information.hypertable« does not exist

从 timescaledb 版本 2.0 开始,您必须使用复数 timescaledb_information.hypertables 并且列名称也已更改,因此现在您必须使用 hypertable_name 而不是 table_name

config.before(:suite) do
  # ensure the hypertable_name hypertable is setup correctly
  ActiveRecord::Base.connection.execute(
    "DROP TRIGGER IF EXISTS ts_insert_blocker ON hypertable_name;"
  )
  ActiveRecord::Base.connection.execute(
    "SELECT create_hypertable('hypertable_name', 'time_field', if_not_exists => TRUE);"
  )
  has_hypertables_sql = "SELECT * FROM timescaledb_information.hypertables WHERE hypertable_name = 'hypertable_name';"
  if ActiveRecord::Base.connection.execute(has_hypertables_sql).to_a.empty?
    raise "TimescaleDB missing hypertable on 'hypertable_name' table"
  end
end