Hangfire Dynamic Queues

Hangfire is a really nice tool that allows you to perform background processing jobs in .NET. There are a bunch of different job types and it's all backed by persistent storage technologies such as MongoDB, SQL Server and a bunch of others.

By default, Hangfire works in a distributed fashion, in other words, all background jobs are serialized and persisted to your data store, and any server that is connected to that data store will be able to pick up the scheduled job and execute it. It is super easy to set this up and it works really, really well.

Have a look at their getting started page here.

Hangfire Queues

Once you get the hang of Hangfire you'll inevitably want to schedule jobs that are specific to a certain server. The good news is that Hangfire has already made provision for that with queues. Any background job you want to exclusively run on a specific server can be decorated with a [Queue] attribute and explicit queue name. For example you can create a method called SendWelcomeEmail and decorate it with [Queue("emailonly")], and then configure your specific Hangfire server to only listen for jobs that are in the emailonly queue.

Example server setup using OWIN

GlobalConfiguration.Configuration
  .UseSqlServerStorage(ConfigurationManager.ConnectionStrings["MyDatabaseConnection"].ConnectionString);

app.UseHangfireDashboard();
app.UseHangfireServer(new BackgroundJobServerOptions
{
	Queues = new[] { "emailonly" } // This will setup the server to only process "emailonly" queues 
});

Example send email with "emailonly" queue attribute

[Queue("emailonly")]
public void SendWelcomeEmail(string emailAddress)
{
	// Send welcome email functionality...
}

This is fantastic! The bad news is that if you want to send something to a queue at run-time, attributes are not going to help really...

Example problem

Say for example you are relying on the MemoryCache to build up some lookup items for your website and you would like to cache it on startup - out of process. Something like Hangfire will work quite nicely to achieve that. Since your server and client for Hangfire is in the same solution, you don't have to create a new service to do background processing, and you'll still have access to the MemoryCache.

As with the example above, MemoryCache is definitely server specific, and if you have the same code base in a web-farm with multiple servers, there is no way to actually control which server will process the background job without queues. But how exactly will you control it?

PS: I know a distributed caching solution will work better in this case, but please bear with me...

The Hangfire server setup will be really easy. You could get the queue name from AppSettings in Startup.cs or Global.asax when you setup your Hangfire server:

GlobalConfiguration.Configuration
  .UseSqlServerStorage(ConfigurationManager.ConnectionStrings["MyDatabaseConnection"].ConnectionString);

app.UseHangfireDashboard();
app.UseHangfireServer(new BackgroundJobServerOptions
{
	Queues = new[] { ConfigurationManager.AppSettings["QueueName"] } // Specify the queue at runtime
});

But how do you deal with the QueueAttribute? You can only use constant variables with Attributes!

Cue Dynamic Queues

At first I thought I'll create a new attribute that inherits from the QueueAttribute and build up the queue name using the machine name.

Build queue name using machine name

var queueName = Environment.MachineName;
queueName = new string(queueName.Where(char.IsLetterOrDigit).ToArray()).ToLower();

(Note: Hangfire queue names need to be lower case)

But the QueueAttribute is sealed, and there is no way to inherit from a sealed class. I had a look at the source code for the QueueAttribute on Github and found that the QueueAttribute inherits from JobFilterAttribute and implements IElectStateFilter.

I dug a little bit deeper and found out that Hangfire only cares about IElectStateFilter and JobFilterAttribute. This means we can create an attribute that looks like this:

public class QueueToCurrentServerOnlyAttribute : JobFilterAttribute, IElectStateFilter
{
	public string QueueName
	{
		get
		{
			var queueName = Environment.MachineName;
			queueName = new string(queueName.Where(char.IsLetterOrDigit).ToArray()).ToLower();
			return queueName;
		}
	}

	public void OnStateElection(ElectStateContext context)
	{
		var enqueuedState = context.CandidateState as EnqueuedState;
		if (enqueuedState != null)
		{
			enqueuedState.Queue = QueueName;
		}
	}
}

And you can now change the server setup to use the machine name as a queue name:

GlobalConfiguration.Configuration
  .UseSqlServerStorage(ConfigurationManager.ConnectionStrings["MyDatabaseConnection"].ConnectionString);

var queueName = Environment.MachineName;
queueName = new string(queueName.Where(char.IsLetterOrDigit).ToArray()).ToLower();

app.UseHangfireDashboard();
app.UseHangfireServer(new BackgroundJobServerOptions
{
	Queues = new[] { queueName } // Specify the queue at runtime
});

And finally your caching method can look like this:

[QueueToCurrentServerOnly]
public void AddToCache(string key, object obj)
{
	// caching logic goes here... 
}

Now all methods decorated with the QueueToCurrentServerOnly attribute will only be executed on the Hangfire server that can process these queues.

Let me know what you think about this in the comments section below.