如何防止 Hangfire 重复作业在连续执行 30 分钟后重新启动

Question

我正在开发 asp.net mvc-5 网络应用程序，我在使用 Hangfire 工具执行运行长运行ning 后台作业时遇到问题。 问题是如果作业执行超过 30 分钟，hangfire 会自动启动另一个作业，所以我最终会同时有两个类似的作业运行ning。

现在我有以下内容：-

Asp.netmvc-5
IIS-8
Hangfire 1.4.6
Windows 服务器 2012

现在我已经在每天 17:00 为运行定义了一个 hangfire 循环作业。后台作业主要扫描我们的网络以查找服务器和虚拟机并更新数据库，重复作业将在完成执行后发送电子邮件。当执行时间少于 30 分钟时，重复性作业过去运行良好。但今天随着我们系统的发展，重复性工作在 40 分钟后完成，而不是过去的 22-25 分钟。我收到了 2 封电子邮件而不是一封电子邮件（并且电子邮件之间的时间大约为 30 分钟）。现在我手动重新运行作业，我注意到问题如下：-

"when the recurring job reaches 30 minutes of continuous execution, a new instance of the recurring job will start, so I will have two instances instead of one running at the same time, so that why I received 2 emails."

现在，如果重复作业花费的时间少于 30 分钟（例如 29 分钟），我将不会遇到任何问题，但如果重复作业执行时间超过 30 分钟，那么出于某种原因或另一个 hangfire 将启动一个新作业。虽然当我在作业执行期间访问 hangfire 仪表板时，我发现只有一个活动作业，但当我监视我们的数据库时，我可以从 sql 分析器中看到有两个作业正在访问数据库。这发生在循环作业开始 30 分钟后（在我们的例子中是 17:30），这就是为什么我收到 2 封电子邮件，这意味着 2 个循环作业运行在后台而不是一个。

请问有人可以就此提出建议吗？如果当前重复作业执行超过 30 分钟，我如何避免 hangfire 自动启动新的重复作业？谢谢

Answer 1

您是否查看了 Hangfire docs 中的 InvisibilityTimeout 设置？

Default SQL Server job storage implementation uses a regular table as a job queue. To be sure that a job will not be lost in case of unexpected process termination, it is deleted only from a queue only upon a successful completion.

To make it invisible from other workers, the UPDATE statement with OUTPUT clause is used to fetch a queued job and update the FetchedAt value (that signals for other workers that it was fetched) in an atomic way. Other workers see the fetched timestamp and ignore a job. But to handle the process termination, they will ignore a job only during a specified amount of time (defaults to 30 minutes).

Although this mechanism ensures that every job will be processed, sometimes it may cause either long retry latency or lead to multiple job execution. Consider the following scenario:

Worker A fetched a job (runs for a hour) and started it at 12:00.

Worker B fetched the same job at 12:30, because the default invisibility timeout was expired.

Worker C (did not fetch) the same job at 13:00, because (it will be deleted after successful performance.)

If you are using cancellation tokens, it will be set for Worker A at 12:30, and at 13:00 for Worker B. This may lead to the fact that your long-running job will never be executed. If you aren’t using cancellation tokens, it will be concurrently executed by WorkerA and Worker B (since 12:30), but Worker C will not fetch it, because it will be deleted after successful performance.

So, if you have long-running jobs, it is better to configure the invisibility timeout interval:

var options = new SqlServerStorageOptions
{
    InvisibilityTimeout = TimeSpan.FromMinutes(30) // default value
};

GlobalConfiguration.Configuration.UseSqlServerStorage("<name or connection string>", options);

截至 Hangfire 1.5 this option is now Obsolete。正在处理的工作对其他工作人员是不可见的。

Say goodbye to confusing invisibility timeout with unexpected background job retries after 30 minutes (by default) when using SQL Server. New Hangfire.SqlServer implementation uses plain old transactions to fetch background jobs and hide them from other workers.

Even after ungraceful shutdown, the job will be available for other workers instantly, without any delays.

Answer 2

我很难找到有关如何为 Postgresql 数据库正确执行此操作的文档，我看到的每个示例都是使用 sqlserver，我发现隐形超时是如何在 PostgreSqlStorageOptions 对象中实现的属性，我在这里找到这个：https://github.com/frankhommers/Hangfire.PostgreSql/blob/master/src/Hangfire.PostgreSql/PostgreSqlStorageOptions.cs#L36。幸运的是，通过反复试验，我能够弄清楚 UsePostgreSqlStorage 有一个重载来接受这个对象。对于 .Net Core 2.0，当您在启动时的 ConfigureServices 方法中设置 hangfire postgresql 数据库时 class 添加这个（默认超时设置为 30 分钟）：

    services.AddHangfire(config =>
            config.UsePostgreSqlStorage(Configuration.GetConnectionString("Hangfire1ConnectionString"), new PostgreSqlStorageOptions {
                InvisibilityTimeout = TimeSpan.FromMinutes(720)

            }));

Answer 3

我在使用 Hangfire.MemoryStorage 作为存储提供程序时遇到了这个问题。使用内存存储需要在MemoryStorageOptions中设置FetchNextJobTimeout，否则默认作业会在30分钟后超时并执行新的作业。

var options = new MemoryStorageOptions
{
    FetchNextJobTimeout = TimeSpan.FromDays(1)
};
GlobalConfiguration.Configuration.UseMemoryStorage(options);

Answer 4

只是想指出，尽管如此，如下所述：

As of Hangfire 1.5 this option is now Obsolete. Jobs that are being worked on are invisible to other workers.

Say goodbye to confusing invisibility timeout with unexpected background job retries after 30 minutes (by default) when using SQL Server. New Hangfire.SqlServer implementation uses plain old transactions to fetch background jobs and hide them from other workers.

Even after ungraceful shutdown, the job will be available for other workers instantly, without any delays.

似乎对于很多使用MySQL、PostgreSQL的人来说，MongoDB、InvisibilityTimeout仍然是必经之路：https://github.com/HangfireIO/Hangfire/issues/1197

如何防止 Hangfire 重复作业在连续执行 30 分钟后重新启动

How to prevent a Hangfire recurring job from restarting after 30 minutes of continuous execution

asp.net

iis

asp.net-mvc-5

background-process

hangfire