Unreliable/Flakey Capybara/AngularJS 集成测试时序问题

Unreliable/Flakey Capybara/AngularJS Integration Tests With Timing Issues

如何可靠地通过这些测试?

目前这些测试是不稳定的。
有时他们会过去。有时他们会失败。
下面是演示此问题的设置、代码和输出。
解决这个问题的建议将不胜感激,我相信会帮助很多其他人,所以请发表评论!

测试代码环境

  1. Rails 3.2
  2. RSpec2.x
  3. 水豚
  4. 闹鬼
  5. PhantomJS
  6. AngularJS
  7. Google Chrome 版本 47.0.2526.106(64 位)

测试来自 Gemfile.lock

的宝石
capybara (2.1.0)
database_cleaner (0.7.1)
debug_inspector (0.0.2)
guard-bundler (0.1.3)
guard-livereload (1.2.0)
guard-rspec (2.1.2)
jasminerice (0.0.10)
pg (0.17.1)
phantomjs (2.1.1.0)
poltergeist (1.4.1)
protractor-rails (0.0.17)
pry (0.9.12)
rack (1.4.7)
rack-test (0.6.3)
rails (3.2.21)
rails-assets-angular (1.3.20)
rspec-rails (2.11.4)
simplecov (0.8.2)
sprockets (2.2.3)
zeus (0.13.3)
zeus-parallel_tests (0.2.1)

我尝试过的东西

  1. 确保我使用 Capybara 的等待 DSL 匹配器
  2. 确保我的数据库清理器设置正确
  3. 测试每个页面项目,假设它可能不在页面上并且可能仍在加载
  4. 缩小不一致的测试范围
  5. 运行 单独测试不一致
  6. 确定触发不一致测试结果的代码 Capybara DSL。

    • 即创建一条新记录并假设页面已重定向并且该记录在页面上 click_on

    • .点击不一致 'working'
  7. 将 Capybara 升级到最新版本(在单独的分支中)
  8. 已将 Poltergeist 和 RSpec 升级到最新版本(在单独的分支中,仍在进行中)

我使用的资源

[1] Capybara The DSL
[2] Capybara, PhantomJs, Poltergeist, and Rspec Tips
还有更多...

测试如何运行

rspec spec/integration/costings/show_costing_spec.rb --format documentation

测试代码

show_costing_spec.rb
require "spec_helper"

RSpec.describe "Show a new costing in the listing," do

  before :each do
    admin_sign_in
    create_costing("test1")
  end

  it "shows the costing after creation" do
    within "#costings_table" do
      expect(page).to have_css("#name", text: "test1")
    end
  end

  it "shows the details of the new costing after creation" do
    expect(page).to have_content("Costings")
    within "#costings_table" do
      expect(page).to have_content("test1")
      all("#show").last.click
    end

    expect(page).to have_content("Costing Details")
    expect(page).to have_css("#name", text: "test1")
  end
end  
spec_helper.rb
# This file is copied to spec/ when you run 'rails generate r spec:install'  
ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../../config/environment", __FILE__)
# Add library functions here so we can test them.
require File.expand_path(File.dirname(__FILE__) + "/../lib/general")
require 'rspec/rails'
require 'rspec/autorun'

# Integration Testing
require 'capybara/poltergeist'
Capybara.register_driver :poltergeist_debug do |app|
 Capybara::Poltergeist::Driver.new(app, :inspector => true)  
end
Capybara.javascript_driver = :poltergeist_debug
Capybara.default_driver = :poltergeist_debug

# Capybara Integration Test Helpers
def admin_sign_in
  visit "/login"
  #Create staff member in database
  Staff.make!(:admin)
  #Log In
  fill_in "staff_username", with: "adminstaff"
  fill_in "staff_password", with: "password"
  click_button "login"
end

def create_costing(item)
  visit "/api#/costings"
  click_on "new_btn"
  within "#form_costing" do
    find("#name", match: :first).set("#{item}")
    find("#description", match: :first).set("test description")    
    find("#from_date", match: :first).set("15/02/2016")
    find("#cost_hourly_cents", match: :first).set("1.00")
    click_on "create_btn"
  end
end

RSpec.configure do |config|
  config.before(:suite) do
    # Requires supporting ruby files with custom matchers and macros, etc,
    # in spec/support/ and its subdirectories.
    require File.expand_path(File.dirname(__FILE__) + "/support/blueprints")
    Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}
  end

  # Remove this line if you're not using ActiveRecord or ActiveRecord fixtures
  config.fixture_path = "#{::Rails.root}/spec/fixtures"

  # Allow a 'focus' tag so that we can run just a few tests which we are currently working on
  config.treat_symbols_as_metadata_keys_with_true_values = true
  config.filter_run focus: true
  config.run_all_when_everything_filtered = true
  config.filter_run_excluding :slow unless ENV["SLOW_SPECS"]

  # Defer Garbage Collection
  config.before(:all) { DeferredGarbageCollection.start }
  config.after(:all)  { DeferredGarbageCollection.reconsider }

  # If you're not using ActiveRecord, or you'd prefer not to run each of your
  # examples within a transaction, remove the following line or assign false
  # instead of true.
  config.use_transactional_fixtures = false
  # config.infer_spec_type_from_file_location!

  # Configure Database Cleaner
  config.include Capybara::DSL
  config.before(:suite) do
    DatabaseCleaner.clean_with(:truncation)
  end

  config.before(:each) do
    DatabaseCleaner.strategy = :transaction
  end

  config.before(:each, :js => true) do
    DatabaseCleaner.strategy = :truncation
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.after(:each) do
    DatabaseCleaner.clean
  end
end

测试结果

Test Run 1: Failing

Run options: include {:focus=>true} exclude {:slow=>true}

All examples were filtered out; ignoring {:focus=>true}

Show a new costing in the listing, shows the costing after creation shows the details of the new costing after creation (FAILED - 1)

Failures:

1) Show a new costing in the listing,
shows the details of the new costing after creation
Failure/Error: expect(page).to have_content("test1")
expected #has_content?("test1") to return true, got false
# ./spec/integration/costings/show_costing_spec.rb:20:in block (3 levels) in
# ./spec/integration/costings/show_costing_spec.rb:19:in block (2 levels) in

Finished in 5.46 seconds 2 examples, 1 failure

Test Run 2: Passing

Run options: include {:focus=>true} exclude {:slow=>true}

All examples were filtered out; ignoring {:focus=>true}

Show a new costing in the listing,
shows the costing after creation
shows the details of the new costing after creation

Finished in 3.57 seconds 2 examples, 0 failures

更新 1

升级测试 gems 到以下版本:
水豚 (2.6.2) 来自 (2.1.0)
database_cleaner (1.5.1) 从 (0.7.1)
debug_inspector (0.0.2)
guard-bundler (0.1.3)
guard-livereload (1.2.0)
guard-spec (2.1.2)
茉莉花 (0.0.10)
PG (0.17.1)
phantomjs (2.1.1.0)
闹鬼 (1.9.0) 来自 (1.4.1)
protractor-rails (0.0.17)
从 (0.9.12) 撬 (0.10.3)
机架 (1.4.7)
rack-test (0.6.3)
rails (3.2.21)
rails-assets-angular (1.4.9) 从 (1.3.20)
rspec-rails (3.4.2) 从 (2.11.4)
simplecov (0.8.2)
链轮 (2.2.3)
宙斯 (0.13.3)
zeus-parallel_tests (0.2.1)

Result 1

Unfortunately upgrading these gems did not seem to make a difference and my tests were still flakey.

更新 2

我采纳了 Tom Walpole 的建议。确保我的 admin_sign_in 等待 sign_in 完成。

还按照 Tom 的建议更新了我的 database_cleaner 设置。

Result 2

For my stack these changes did not seem to have an effect.

Note: If one is not using AngularJS I feel these changes would have made a difference. So thank you Tom for your suggestions.

更新 3

我需要获得有关测试 运行 期间发生的事情的更多信息。网上有建议登录,使用屏幕截图保存 gems 等,但我觉得这些不是最省时的。我想指定我希望测试暂停的位置并查看变量和表单字段的内容。在浏览器中将是理想的。

我用了什么
我正在使用 "save_and_open_page" 和 "print page.html" 进行调试。

我搬到了什么地方
因为我是 运行ning RSpec 3.4.2 我添加了一个调试辅助方法:

rails_helper.rb

def debugit
  puts current_url
  require 'pry'
  binding.pry
end

Result 3

A URL would be printed in the console and the test would be paused. At this stage I would be able to navigate to the URL, login, navigate to the test page and view what the Capybara test had done.

This enabled me to identify that the source of my problems arose when the test was using capybara's fill_in DSL. In some test runs the fields would be populated correctly and the form would be submitted. In the other scenario the form would be filled in correctly but the submit button would be hit too quickly. The result here is that a record was created but input fields of name and description were not persisted.

更新 4

我发现因为我使用的是 AngularJS 输入表单和表格,所以 AngularJS 需要一点时间来绑定到输入字段。如果这次不允许,输入的数据将不会被保存。

水豚提供"within"、"find"等等待方式。我使用了这些,但它们对 AngularJS 绑定时间问题没有帮助。 我发现 ng-if 可用于创建 if 语句以等待特定项目,这将表示 AngularJS 绑定到表单字段已完成。

所以我使用 Capybara 等待方法来等待我想要填写的字段,并且我使用 AngularJS' ng-if 在它们准备好之前不显示这些字段。

实施
index.html.erb

<div  ng-if="tableParams.data">
  <table id="costings_table ng-table="tableParams" class="table">
    <td id="field1">{{table.field1}}</td>
    <td id="field2">{{table.field2}}</td>
  </table>
</div>

Result 4

Tests finally pass! However I have all these find methods with xpath ensuring specific and difficult to target items are waited on...

更新 5

即使在我的 gem 文件中,我 运行 正在使用 gem phantomJS 版本 2.1.1,但我的命令行版本仅为 1.X。事实证明这很重要。

我将命令行 phantomJS 版本更新为 2.1.1。同时,我确保我所有的输入框、按钮、表格、标题都有唯一的 ID。然后我能够删除所有 find(:xpath) 事件而不用打破测试。

Result 5

This suite of tests now reliably pass all the time! Exactly what I wanted! Yes!

立即跳出的是您的 admin_sign_in 实际上并没有等待 sign_in 完成。这意味着您可以在未在浏览器中设置会话 cookie 的情况下调用 create_costingadmin_sign_in 方法的最后一行应该类似于

expect(page).to have_text('You are signed in') # whatever message is shown upon sign in

expect(page).to have_current_path('/') # whatever path an admin is redirected to upon signing in

这将确保登录确实已完成,因此会话 cookie 已在您的浏览器中设置。

此外,您的数据库清理器配置应该使用 append_after 块而不是之后 - 请参阅 https://github.com/DatabaseCleaner/database_cleaner#rspec-with-capybara-example

问题

当测试使用水豚的 fill_in DSL 时出现问题。在某些测试 运行s 中,字段将被正确填充并且表单将被提交。在另一种情况下,表单会正确填写,但提交按钮会被点击得太快。这里的结果是创建了一条记录,但没有保留名称和描述的输入字段。

1。填写表格时,确保AngularJS绑定完整,并使用水豚等待方法

AngularJS' 需要使用 ng-if 语句在准备好之前不显示表单字段。
这需要结合使用 Capybara 等待方法来完成,以确保 fill_in 字段仅在表单加载完成后才提交。

index.html.erb 或等效项:

<div  ng-if="tableParams.data">
  <table id="costings_table ng-table="tableParams" class="table">
    <td id="field1">{{table.field1}}</td>
    <td id="field2">{{table.field2}}</td>
   </table>
</div>

2。将 PhantomJS 的命令行版本更新到最新 (2.1.1)

这似乎使 运行 的测试无需那么多的 Capybara 等待方法即可实现可靠的测试。

更新测试代码
show_costing_spec.rb

require "rails_helper"

RSpec.describe "Show a new costing in the listing,", :type => :feature do

  before :each do
    admin_sign_in
    create_costing("test1")
  end

  it "shows the costing after creation" do
    within "#costings_table" do
      expect(page.find("#code2")).to have_content("2")
      expect(page.find("#name2")).to have_content("test1")
    end
  end

  it "shows the details of the new costing after creation" do
    within "#costings_table" do
      click_on "show2"
    end

    expect(page.find("#page_title")).to have_content("Costing Details")
    expect(page.find("#code")).to have_content("2")
    expect(page.find("#name")).to have_content("test1") 
    expect(page.find("#description")).to have_content("test description")
  end
end

rails_helper.rb

# This file is copied to spec/ when you run 'rails generate rspec:install'
ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../../config/environment", __FILE__)

# Add library functions here so we can test them.
require File.expand_path(File.dirname(__FILE__) + "/../lib/general")

require 'rspec/rails'
require 'devise'

RSpec.configure do |config|
  config.before(:suite) do
    # Requires supporting ruby files with custom matchers and macros, etc,
    # in spec/support/ and its subdirectories.
    require File.expand_path(File.dirname(__FILE__) + "/support/blueprints")
    Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}

    # Setup Devise before it is used in rails_helper
    config.include Devise::TestHelpers, :type => :controller
    Devise.stretches = 1 # Improves speed.
end

config.include Capybara::DSL, :type => :feature
  config.mock_with :rspec

# Remove this line if you're not using ActiveRecord or ActiveRecord fixtures
config.fixture_path = "#{::Rails.root}/spec/fixtures"

# Allow a 'focus' tag so that we can run just a few tests which we are currently working on
config.filter_run focus: true
config.run_all_when_everything_filtered = true
config.filter_run_excluding :slow unless ENV["SLOW_SPECS"]

# Defer Garbage Collection
config.before(:all) { DeferredGarbageCollection.start }
config.after(:all)  { DeferredGarbageCollection.reconsider }

# Integration Testing
require 'capybara/rspec'
require 'capybara/poltergeist'

Capybara.register_driver :poltergeist_debug do |app|
  Capybara::Poltergeist::Driver.new(app, {:inspector => true, js_errors: false })  
end

Capybara.javascript_driver = :poltergeist_debug
Capybara.default_driver = :poltergeist_debug

# Debugging tools
def debugit
  puts current_url
  require 'pry'
  binding.pry
end

# If you're not using ActiveRecord, or you'd prefer not to run each of your
# examples within a transaction, remove the following line or assign false
# instead of true.
config.use_transactional_fixtures = false

#Show Deprications As Errors with full backtracing
config.raise_errors_for_deprecations!

#rest of the file....
# Final part of Configure Database Cleaner

Capybara.default_max_wait_time = 5
config.use_transactional_fixtures = false

config.before(:suite) do
  if config.use_transactional_fixtures?
    raise(<<-MSG)
      Delete line `config.use_transactional_fixtures = true` from
      rails_helper.rb (or set it to false) to prevent uncommitted
      transactions being used in JavaScript-dependent specs. During 
      testing, the app-under-test that the browser driver connects to 
      uses a different database connection to the database connection 
      used by the spec. The app's database connection would not be 
      able to access uncommitted transaction data setup over the 
      spec's database connection.
     MSG
  end
  DatabaseCleaner.clean_with(:truncation)
end  

config.before(:each) do
  DatabaseCleaner.strategy = :transaction
end

config.before(:each, type: :feature) do
  # :rack_test driver's Rack app under test shares database connection
  # with the specs, so continue to use transaction strategy for speed.
  driver_shares_db_connection_with_specs = Capybara.current_driver == :rack_test

    if !driver_shares_db_connection_with_specs
      # Driver is probably for an external browser with an app
      # under test that does *not* share a database connection with the
      # specs, so use truncation strategy.
      DatabaseCleaner.strategy = :truncation
    end
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.append_after(:each) do
    DatabaseCleaner.clean
  end
end


def admin_sign_in
  visit "/login"

  #Create staff member in database
  Staff.make!(:admin)

  #Log In
  fill_in "staff_username", with: "adminstaff"
  fill_in "staff_password", with: "password"
  click_button "login"

  expect(page).to have_text('Logout')
end

def create_costing(item)
  @item = item
  visit "/api#/costings"

  expect(page).to have_selector("#new_btn")
  click_on "new_btn"

  expect(page).to have_text("New Costing")
  within "#form_costing" do
    fill_in "name", with: "#{@item}"
    fill_in "description", with: "test description"
    fill_in "from_date1", with: "15/02/2015" 
    fill_in "cost_hourly_cents1", with: "12.00"

    expect(page).to have_selector("#create_btn")
    click_on "create_btn"
  end
  expect(page.find("#page_title")).to have_content("Costings")
end