如何应对 REST API 中的机器人
How to tackle bots in REST APIs
我有一个移动应用程序，用户可以在其中投放广告，其他用户可以查看并接受。最近，我开始注意到机器人开始提供自己的广告。我有版主，但广告太多，不可能检查所有内容（另一个挑战是，讨价还价会立即实时发生）。它是经典的 REST API。我在谷歌上搜索了很多，令我惊讶的是找不到任何可以防止非法机器人活动的开源解决方案。
您如何处理此类案件？
有没有可能完全消除它，或者我只能采取一些措施让他们的生活更加困难？
阻止机器人程序是我们已经处理过的很多事情。一种常见的机制是使用验证码机制（例如 Google 的 recaptcha）。请google给它。
开源机器人检测
I googled a lot and to my surprise can't find any open source solution that protects from illegal bot activities.
如果您在 Github 中搜索 bot detection，您将进入此页面 https://github.com/topics/bot-detection?q=bot+detection&unscoped_q=bot+detection, that at this moment contains 7 results, that aren't very relevant, but if you remove the query string, and use https://github.com/topics/bot-detection，您将获得 36 个结果，其中一些可能相关，具体取决于您的后端语言。您还可以使用 browser detection、crawler detection、device detection 等字词进行搜索
有些repo依赖IP地址中的user-agent and/or来检测bot，这种方法很容易被绕过，因为user-agent头很容易被欺骗现在攻击者正在使用机器人农场来轮换 ip，因此很难阻止它们。
但是您应该使用其中一个存储库吗？为什么不呢，它是多一层防御，至少您可以阻止不 运行 来自机器人农场的不太复杂的机器人。
访问 API 服务器的 WHO 和 WHAT 的区别
Recently, I began to notice that bots started to give their own advertisements.
在我们深入探讨如何解决这个问题之前，我想首先澄清一个误解，这个误解在任何资历级别的开发人员中都很常见，Who 与 什么正在访问API服务器。
我建议您阅读文章 Why Does Your Mobile App Need An Api Key?，其中详细介绍了 Who 和 What 之间的区别访问您的 API 服务器，但为了您的方便，我将在此处提取主要内容：
The what is the thing making the request to the API server. Is it really a genuine instance of your mobile app, or is it a bot, an automated script or an attacker manually poking around your API server with a tool like Postman?
The who is the user of the mobile app that we can authenticate, authorize and identify in several ways, like using OpenID Connect or OAUTH2 flows.
所以我希望您考虑 Who 作为用户，您的 API 服务器将能够验证和授权对数据的访问，并考虑什么 作为代表用户发出该请求的软件，在您的情况下是机器人。
API 服务器防御
How do you tackle such cases? Is it possible to eliminate it at all or I can only make their life more difficult taking some measures?
您可以通过应用深度防御来增加攻击者的难度，方法是在您负担得起的情况下添加尽可能多的防御层，并且法律要求您的用例。
WAF - Web Application Firewall:
A web application firewall (or WAF) filters, monitors, and blocks HTTP traffic to and from a web application. A WAF is differentiated from a regular firewall in that a WAF is able to filter the content of specific web applications while regular firewalls serve as a safety gate between servers. By inspecting HTTP traffic, it can prevent attacks stemming from web application security flaws, such as SQL injection, cross-site scripting (XSS), file inclusion, and security misconfigurations.
针对 API 的有效性较弱，因为它是专门为不依赖 API 的网络应用程序设计的，但仍能够提供一定程度的保护.
UBA - User Behavior Analytics:
User behavior analytics (UBA) as defined by Gartner is a cybersecurity process about detection of insider threats, targeted attacks, and financial fraud. UBA solutions look at patterns of human behavior, and then apply algorithms and statistical analysis to detect meaningful anomalies from those patterns—anomalies that indicate potential threats. Instead of tracking devices or security events, UBA tracks a system's users. Big data platforms like Apache Hadoop are increasing UBA functionality by allowing them to analyze petabytes worth of data to detect insider threats and advanced persistent threats.
UBA 解决方案的一个很好的例子是 Google 的 Recaptcha，特别是 reCAPTCHA V3:
reCAPTCHA is a free service that protects your website from spam and abuse. reCAPTCHA uses an advanced risk analysis engine and adaptive challenges to keep automated software from engaging in abusive activities on your site. It does this while letting your valid users pass through with ease.
...helps you detect abusive traffic on your website without any user friction. It returns a score based on the interactions with your website and provides you more flexibility to take appropriate actions.
当您在移动应用程序中使用 recaptcha V3 时，您的 API 服务器随后可以验证该请求的分数，但请记住，这只会让绕过变得更加困难，因为如果您搜索 Google 对于 bypass recaptcha V3 你会看到很多解决方案作为攻击者的服务提供。
将 API 服务器锁定到移动应用程序
API 服务器可以高度确信请求确实来自您的移动应用程序，而不是通过使用移动应用程序证明概念来自机器人，我邀请您阅读 问题 How to secure an API REST for mobile app?，特别是 A Possible Better Solution[=98= 部分].
您想加倍努力吗？
在任何对安全问题的回复中，我总是喜欢引用 OWASP 基金会的出色工作。
对于移动应用程序
OWASP Mobile Security Project - Top 10 risks
The OWASP Mobile Security Project is a centralized resource intended to give developers and security teams the resources they need to build and maintain secure mobile applications. Through the project, our goal is to classify mobile security risks and provide developmental controls to reduce their impact or likelihood of exploitation.
OWASP - Mobile Security Testing Guide:
The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security development, testing and reverse engineering.
对于APIS
The OWASP API Security Project seeks to provide value to software developers and security assessors by underscoring the potential risks in insecure APIs, and illustrating how these risks may be mitigated. In order to facilitate this goal, the OWASP API Security Project will create and maintain a Top 10 API Security Risks document, as well as a documentation portal for best practices when creating or assessing APIs.
我有一个移动应用程序，用户可以在其中投放广告，其他用户可以查看并接受。最近，我开始注意到机器人开始提供自己的广告。我有版主，但广告太多，不可能检查所有内容（另一个挑战是，讨价还价会立即实时发生）。它是经典的 REST API。我在谷歌上搜索了很多，令我惊讶的是找不到任何可以防止非法机器人活动的开源解决方案。 您如何处理此类案件？ 有没有可能完全消除它，或者我只能采取一些措施让他们的生活更加困难？
阻止机器人程序是我们已经处理过的很多事情。一种常见的机制是使用验证码机制（例如 Google 的 recaptcha）。请google给它。
开源机器人检测
I googled a lot and to my surprise can't find any open source solution that protects from illegal bot activities.
如果您在 Github 中搜索 bot detection，您将进入此页面 https://github.com/topics/bot-detection?q=bot+detection&unscoped_q=bot+detection, that at this moment contains 7 results, that aren't very relevant, but if you remove the query string, and use https://github.com/topics/bot-detection，您将获得 36 个结果，其中一些可能相关，具体取决于您的后端语言。您还可以使用 browser detection、crawler detection、device detection 等字词进行搜索
有些repo依赖IP地址中的user-agent and/or来检测bot，这种方法很容易被绕过，因为user-agent头很容易被欺骗现在攻击者正在使用机器人农场来轮换 ip，因此很难阻止它们。
但是您应该使用其中一个存储库吗？为什么不呢，它是多一层防御，至少您可以阻止不 运行 来自机器人农场的不太复杂的机器人。
访问 API 服务器的 WHO 和 WHAT 的区别
Recently, I began to notice that bots started to give their own advertisements.
在我们深入探讨如何解决这个问题之前，我想首先澄清一个误解，这个误解在任何资历级别的开发人员中都很常见，Who 与 什么正在访问API服务器。
我建议您阅读文章 Why Does Your Mobile App Need An Api Key?，其中详细介绍了 Who 和 What 之间的区别访问您的 API 服务器，但为了您的方便，我将在此处提取主要内容：
The what is the thing making the request to the API server. Is it really a genuine instance of your mobile app, or is it a bot, an automated script or an attacker manually poking around your API server with a tool like Postman?
The who is the user of the mobile app that we can authenticate, authorize and identify in several ways, like using OpenID Connect or OAUTH2 flows.
所以我希望您考虑 Who 作为用户，您的 API 服务器将能够验证和授权对数据的访问，并考虑什么 作为代表用户发出该请求的软件，在您的情况下是机器人。
API 服务器防御
How do you tackle such cases? Is it possible to eliminate it at all or I can only make their life more difficult taking some measures?
您可以通过应用深度防御来增加攻击者的难度，方法是在您负担得起的情况下添加尽可能多的防御层，并且法律要求您的用例。
WAF - Web Application Firewall:
A web application firewall (or WAF) filters, monitors, and blocks HTTP traffic to and from a web application. A WAF is differentiated from a regular firewall in that a WAF is able to filter the content of specific web applications while regular firewalls serve as a safety gate between servers. By inspecting HTTP traffic, it can prevent attacks stemming from web application security flaws, such as SQL injection, cross-site scripting (XSS), file inclusion, and security misconfigurations.
针对 API 的有效性较弱，因为它是专门为不依赖 API 的网络应用程序设计的，但仍能够提供一定程度的保护.
UBA - User Behavior Analytics:
User behavior analytics (UBA) as defined by Gartner is a cybersecurity process about detection of insider threats, targeted attacks, and financial fraud. UBA solutions look at patterns of human behavior, and then apply algorithms and statistical analysis to detect meaningful anomalies from those patterns—anomalies that indicate potential threats. Instead of tracking devices or security events, UBA tracks a system's users. Big data platforms like Apache Hadoop are increasing UBA functionality by allowing them to analyze petabytes worth of data to detect insider threats and advanced persistent threats.
UBA 解决方案的一个很好的例子是 Google 的 Recaptcha，特别是 reCAPTCHA V3:
reCAPTCHA is a free service that protects your website from spam and abuse. reCAPTCHA uses an advanced risk analysis engine and adaptive challenges to keep automated software from engaging in abusive activities on your site. It does this while letting your valid users pass through with ease.
...helps you detect abusive traffic on your website without any user friction. It returns a score based on the interactions with your website and provides you more flexibility to take appropriate actions.
当您在移动应用程序中使用 recaptcha V3 时，您的 API 服务器随后可以验证该请求的分数，但请记住，这只会让绕过变得更加困难，因为如果您搜索 Google 对于 bypass recaptcha V3 你会看到很多解决方案作为攻击者的服务提供。
将 API 服务器锁定到移动应用程序
API 服务器可以高度确信请求确实来自您的移动应用程序，而不是通过使用移动应用程序证明概念来自机器人，我邀请您阅读
您想加倍努力吗？
在任何对安全问题的回复中，我总是喜欢引用 OWASP 基金会的出色工作。
对于移动应用程序
OWASP Mobile Security Project - Top 10 risks
The OWASP Mobile Security Project is a centralized resource intended to give developers and security teams the resources they need to build and maintain secure mobile applications. Through the project, our goal is to classify mobile security risks and provide developmental controls to reduce their impact or likelihood of exploitation.
OWASP - Mobile Security Testing Guide:
The Mobile Security Testing Guide (MSTG) is a comprehensive manual for mobile app security development, testing and reverse engineering.
对于APIS
The OWASP API Security Project seeks to provide value to software developers and security assessors by underscoring the potential risks in insecure APIs, and illustrating how these risks may be mitigated. In order to facilitate this goal, the OWASP API Security Project will create and maintain a Top 10 API Security Risks document, as well as a documentation portal for best practices when creating or assessing APIs.