Ruby 机械化,填写动态表格/发送 JSON(Airbnb 日历)
Ruby Mechanize, fill dynamic Form / Send JSON (Airbnb calendar)
我的目标
我尝试使用 Ruby 更新我的爱彼迎日历。例如,这是日历的 URL:https://www.airbnb.com/manage-listing/ROOM_ID/calendar
问题
如果您已经在使用 Airbnb,要更新您的日历,您必须依次点击开始日期和结束日期,然后再点击表格 pop-up。
所以,当我使用 Mechanize 获取页面内容时,这个表单没有加载也没有出现(即使日历是动态加载的,也无法模拟点击),无法使用基本的 Mechanize 表单填写...
到目前为止我做了什么
我尝试使用 Chrome 中的开发人员工具来检查网络。当我使用 Chrome 更新我的日历时,在 https://www.airbnb.com/api/v2/calendars/ROOM_ID/START_DATE/END_DATE?_format=host_calendar&t=1427377357561&key=d306zoyjsyarp7ifhu67rjxn52tv0t20
有一个 JSON PUT,其中包含一些 JSON 数据,例如天数、可用性、价格...
我的第一个解决方案是尝试使用以下代码重现此 JSON 调用:
data = { "event_name" => "calendar",
"event_data" => { "page_uri" => "/manage-listing/ROOM_ID/calendar",
"controller" => "rooms",
"action" => "manage_listing",
"hosting_id" => ROOM_ID,
"start_date" => "2015-03-26",
"end_date" => "2015-03-29",
"available" => true,
"native_price" => 111,
"native_currency" => "EUR"
}
}
page = agent.post 'https://www.airbnb.com/api/v2/calendars/ROOM_ID/2015-03-26/2015-03-29?_format=host_calendar&t=1427374574309&key=d306zoyjsyarp7ifhu67rjxn52tv0t20', data.to_json, {'Content-Type' => 'application/json'}
但我收到 404 响应:
Mechanize::ResponseCodeError (404 => Net::HTTPNotFound for https://www.airbnb.com/api/v2/calendars/ROOM_ID/2015-03-26/2015-03-29?_format=host_calendar&t=1427374574309&key=d306zoyjsyarp7ifhu67rjxn52tv0t20 -- unhandled response)
您是否有任何建议发送表单,即使它不在页面内容中,或者 POST 带有 JSON 的请求?
感谢您的帮助
这是来自 Chrome 的完整 JSON 调用:
一般
Remote Address:xx.xx.xx.xx:xx
Request URL:https://www.airbnb.com/api/v2/calendars/ROOM_ID/2015-03-26/2015-03-29?_format=host_calendar&t=1427379998507&key=d306zoyjsyarp7ifhu67rjxn52tv0t20¤cy=EUR&locale=fr-CA
Request Method:PUT
Status Code:200 OK
响应Headers
cache-control:max-age=0, private, must-revalidate
connection:keep-alive
content-encoding:gzip
content-length:236
content-type:application/json; charset=utf-8
date:Thu, 26 Mar 2015 14:26:46 GMT
etag:W/"10845765865e36a6ccb1541bbda1c2a7"
server:nginx/1.7.7
status:200 OK
status:200 OK
strict-transport-security:max-age=10886400; includeSubdomains
vary:Accept-Encoding
version:HTTP/1.1
x-frame-options:SAMEORIGIN
x-hi-human:The Production Infrastructure team added this header. Come work with us! Email kevin.rice+hiring@airbnb.com
x-ua-compatible:IE=Edge,chrome=1
x-xss-protection:1; mode=block
请求Headers
:host:www.airbnb.com
:method:PUT
:path:/api/v2/calendars/ROOM_ID/2015-03-26/2015-03-29?_format=host_calendar&t=1427379998507&key=d306zoyjsyarp7ifhu67rjxn52tv0t20¤cy=EUR&locale=fr-CA
:scheme:https
:version:HTTP/1.1
accept:application/json, text/javascript, */*; q=0.01
accept-encoding:gzip, deflate, sdch
accept-language:fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
content-length:59
content-type:application/json
cookie:__ssid=4166c81a-49bd-4826-ac44-08307c5700ca; _csrf_token=V4%24.airbnb.ca%24CL1nNdfYkF0%24ulPyJJJWr1h6CvuBMf32YcXtnZssDud3_CqBQoqXOU0%3D; li=1; roles=0; _airbed_session_id=dfa72c17e6d014f9fd0b9705d097e5d8; flags=4027711488; EPISODES=s=1427377914349&r=https%3A%2F%2Ffr.airbnb.ca%2Fmanage-listing%2F5780104%2Fcalendar; _ga=GA1.2.1981489078.1427272843; fbs=not_authorized; _pt=1--WyJjZmYxZmE4N2RhOTU4NGNhYzhhN2M5YTIyNzkyMDliMDI0YTk1YWEzIl0%3D--2890e7d8df5181677516659fbdc4761e6de82a61; bev=1427272835_bw8KI59ELTQAsMt3; _user_attributes=%7B%22curr%22%3A%22EUR%22%2C%22guest_exchange%22%3A0.9134%2C%22id%22%3A29905162%2C%22hash_user_id%22%3A%22cff1fa87da9584cac8a7c9a2279209b024a95aa3%22%2C%22eid%22%3A%22FBPqvskr4MN1Rnpqf-oY-lG7-VNdCJVSYwUMUtm6YyOXzEpbRvmU9FWTxKNdf0UA%22%2C%22num_msg%22%3A0%2C%22num_h%22%3A1%2C%22name%22%3A%22St%C3%A9phane%22%2C%22is_admin%22%3Afalse%2C%22can_access_photography%22%3Afalse%7D
origin:https://www.airbnb.com
referer:https://www.airbnb.com/manage-listing/ROOM_ID/calendar
user-agent:Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36
x-csrf-token:V4$.airbnb.ca$CL1nNdfYkF0$ulPyJJJWr1h6CvuBMf32YcXtnZssDud3_CqBQoqXOU0=
x-requested-with:XMLHttpRequest
查询字符串参数
_format:host_calendar
t:1427379998507
key:d306zoyjsyarp7ifhu67rjxn52tv0t20
currency:EUR
locale:fr-CA
请求负载
{availability: "available", daily_price: "999", notes: ""}
availability: "available"
daily_price: "999"
notes: ""
我成功更新了房间的日历我使用了 JSON PUT 请求。这是我做的。
数据看起来像:
data = { "availability" => availability,
"daily_price" => price,
"notes" => note
}.to_json
检索 cookie :
cookie_csrf_token = ''
cookie_airbed_session_id = ''
agent.cookie_jar.each do |value|
if value.to_s.include? "_csrf_token"
cookie_csrf_token = value.to_s
elsif value.to_s.include? "_airbed_session_id"
cookie_airbed_session_id = value.to_s
end
end
headers :
headers = { 'X-CSRF-Token' => URI.unescape(cookie_csrf_token.scan(/=(.*)/).join(",")),
'Content-Type' => 'application/json',
'Cookie' => "#{cookie_csrf_token}; #{cookie_airbed_session_id}"
}
您唯一需要的 cookie 是彼此相关的 csrf_token 和 airbed_session_id。我的错误是使用了登录页面中的 csrf_token...您可以在 Mechanize 代理的 cookie_jar 变量中找到这些 cookie。
之后您将需要构建您的 URL。 URL 有一个特定的参数,称为 "key"。您可以在日历页面的元标记 (id='_bootstrap-layout-init'
) 中检索它。为此,我将 Nokogiri 与一些正则表达式结合使用:
param_t = Time.now.to_i
noko.xpath("//meta[@id='_bootstrap-layout-init']/@content").each do |attr|
param_key = attr.value[/key":"(.*?)"/, 1]
end
现在您可以去更新您的日历了:
url = "https://www.airbnb.com"
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
# Send the PUT request to update the calendar
res = http.start { |req|
req.send_request('PUT', "/api/v2/calendars/#{room_id}/#{start_date}/#{end_date}?_format=host_calendar&t=#{param_t}&key=#{param_key}", data, headers)
}
我的目标
我尝试使用 Ruby 更新我的爱彼迎日历。例如,这是日历的 URL:https://www.airbnb.com/manage-listing/ROOM_ID/calendar
问题
如果您已经在使用 Airbnb,要更新您的日历,您必须依次点击开始日期和结束日期,然后再点击表格 pop-up。 所以,当我使用 Mechanize 获取页面内容时,这个表单没有加载也没有出现(即使日历是动态加载的,也无法模拟点击),无法使用基本的 Mechanize 表单填写...
到目前为止我做了什么
我尝试使用 Chrome 中的开发人员工具来检查网络。当我使用 Chrome 更新我的日历时,在 https://www.airbnb.com/api/v2/calendars/ROOM_ID/START_DATE/END_DATE?_format=host_calendar&t=1427377357561&key=d306zoyjsyarp7ifhu67rjxn52tv0t20
有一个 JSON PUT,其中包含一些 JSON 数据,例如天数、可用性、价格...
我的第一个解决方案是尝试使用以下代码重现此 JSON 调用:
data = { "event_name" => "calendar",
"event_data" => { "page_uri" => "/manage-listing/ROOM_ID/calendar",
"controller" => "rooms",
"action" => "manage_listing",
"hosting_id" => ROOM_ID,
"start_date" => "2015-03-26",
"end_date" => "2015-03-29",
"available" => true,
"native_price" => 111,
"native_currency" => "EUR"
}
}
page = agent.post 'https://www.airbnb.com/api/v2/calendars/ROOM_ID/2015-03-26/2015-03-29?_format=host_calendar&t=1427374574309&key=d306zoyjsyarp7ifhu67rjxn52tv0t20', data.to_json, {'Content-Type' => 'application/json'}
但我收到 404 响应:
Mechanize::ResponseCodeError (404 => Net::HTTPNotFound for https://www.airbnb.com/api/v2/calendars/ROOM_ID/2015-03-26/2015-03-29?_format=host_calendar&t=1427374574309&key=d306zoyjsyarp7ifhu67rjxn52tv0t20 -- unhandled response)
您是否有任何建议发送表单,即使它不在页面内容中,或者 POST 带有 JSON 的请求?
感谢您的帮助
这是来自 Chrome 的完整 JSON 调用:
一般
Remote Address:xx.xx.xx.xx:xx
Request URL:https://www.airbnb.com/api/v2/calendars/ROOM_ID/2015-03-26/2015-03-29?_format=host_calendar&t=1427379998507&key=d306zoyjsyarp7ifhu67rjxn52tv0t20¤cy=EUR&locale=fr-CA
Request Method:PUT
Status Code:200 OK
响应Headers
cache-control:max-age=0, private, must-revalidate
connection:keep-alive
content-encoding:gzip
content-length:236
content-type:application/json; charset=utf-8
date:Thu, 26 Mar 2015 14:26:46 GMT
etag:W/"10845765865e36a6ccb1541bbda1c2a7"
server:nginx/1.7.7
status:200 OK
status:200 OK
strict-transport-security:max-age=10886400; includeSubdomains
vary:Accept-Encoding
version:HTTP/1.1
x-frame-options:SAMEORIGIN
x-hi-human:The Production Infrastructure team added this header. Come work with us! Email kevin.rice+hiring@airbnb.com
x-ua-compatible:IE=Edge,chrome=1
x-xss-protection:1; mode=block
请求Headers
:host:www.airbnb.com
:method:PUT
:path:/api/v2/calendars/ROOM_ID/2015-03-26/2015-03-29?_format=host_calendar&t=1427379998507&key=d306zoyjsyarp7ifhu67rjxn52tv0t20¤cy=EUR&locale=fr-CA
:scheme:https
:version:HTTP/1.1
accept:application/json, text/javascript, */*; q=0.01
accept-encoding:gzip, deflate, sdch
accept-language:fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
content-length:59
content-type:application/json
cookie:__ssid=4166c81a-49bd-4826-ac44-08307c5700ca; _csrf_token=V4%24.airbnb.ca%24CL1nNdfYkF0%24ulPyJJJWr1h6CvuBMf32YcXtnZssDud3_CqBQoqXOU0%3D; li=1; roles=0; _airbed_session_id=dfa72c17e6d014f9fd0b9705d097e5d8; flags=4027711488; EPISODES=s=1427377914349&r=https%3A%2F%2Ffr.airbnb.ca%2Fmanage-listing%2F5780104%2Fcalendar; _ga=GA1.2.1981489078.1427272843; fbs=not_authorized; _pt=1--WyJjZmYxZmE4N2RhOTU4NGNhYzhhN2M5YTIyNzkyMDliMDI0YTk1YWEzIl0%3D--2890e7d8df5181677516659fbdc4761e6de82a61; bev=1427272835_bw8KI59ELTQAsMt3; _user_attributes=%7B%22curr%22%3A%22EUR%22%2C%22guest_exchange%22%3A0.9134%2C%22id%22%3A29905162%2C%22hash_user_id%22%3A%22cff1fa87da9584cac8a7c9a2279209b024a95aa3%22%2C%22eid%22%3A%22FBPqvskr4MN1Rnpqf-oY-lG7-VNdCJVSYwUMUtm6YyOXzEpbRvmU9FWTxKNdf0UA%22%2C%22num_msg%22%3A0%2C%22num_h%22%3A1%2C%22name%22%3A%22St%C3%A9phane%22%2C%22is_admin%22%3Afalse%2C%22can_access_photography%22%3Afalse%7D
origin:https://www.airbnb.com
referer:https://www.airbnb.com/manage-listing/ROOM_ID/calendar
user-agent:Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36
x-csrf-token:V4$.airbnb.ca$CL1nNdfYkF0$ulPyJJJWr1h6CvuBMf32YcXtnZssDud3_CqBQoqXOU0=
x-requested-with:XMLHttpRequest
查询字符串参数
_format:host_calendar
t:1427379998507
key:d306zoyjsyarp7ifhu67rjxn52tv0t20
currency:EUR
locale:fr-CA
请求负载
{availability: "available", daily_price: "999", notes: ""}
availability: "available"
daily_price: "999"
notes: ""
我成功更新了房间的日历我使用了 JSON PUT 请求。这是我做的。
数据看起来像:
data = { "availability" => availability,
"daily_price" => price,
"notes" => note
}.to_json
检索 cookie :
cookie_csrf_token = ''
cookie_airbed_session_id = ''
agent.cookie_jar.each do |value|
if value.to_s.include? "_csrf_token"
cookie_csrf_token = value.to_s
elsif value.to_s.include? "_airbed_session_id"
cookie_airbed_session_id = value.to_s
end
end
headers :
headers = { 'X-CSRF-Token' => URI.unescape(cookie_csrf_token.scan(/=(.*)/).join(",")),
'Content-Type' => 'application/json',
'Cookie' => "#{cookie_csrf_token}; #{cookie_airbed_session_id}"
}
您唯一需要的 cookie 是彼此相关的 csrf_token 和 airbed_session_id。我的错误是使用了登录页面中的 csrf_token...您可以在 Mechanize 代理的 cookie_jar 变量中找到这些 cookie。
之后您将需要构建您的 URL。 URL 有一个特定的参数,称为 "key"。您可以在日历页面的元标记 (id='_bootstrap-layout-init'
) 中检索它。为此,我将 Nokogiri 与一些正则表达式结合使用:
param_t = Time.now.to_i
noko.xpath("//meta[@id='_bootstrap-layout-init']/@content").each do |attr|
param_key = attr.value[/key":"(.*?)"/, 1]
end
现在您可以去更新您的日历了:
url = "https://www.airbnb.com"
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
# Send the PUT request to update the calendar
res = http.start { |req|
req.send_request('PUT', "/api/v2/calendars/#{room_id}/#{start_date}/#{end_date}?_format=host_calendar&t=#{param_t}&key=#{param_key}", data, headers)
}