Autoscaling Sidekiq Workers on Convox
I’ve become a big fan of Convox over the past few years. It’s a great platform for deploying, scaling, and managing web without having to spend large amounts of time managing infrastructure. It also works with multiple clouds, so you can easily run your app on AWS, GCP, Azure, and more.
Among the features is autoscaling, which allows you to horizontally scale your application based on specific metrics. For a web application, this is usually based on the number of requests, or CPU/memory utilization. But if you’re looking to scale Sidekiq workers then you’re going to need a different metric.
Typically here you would want to look at one of these two metrics:
- Enqueued Jobs. How many jobs do you have waiting to be processed?
- Worker Utilization. How many workers do you have idling?
The idea is that once your queue depth or utilization increases, to increase throughput you would horizontally scale your worker instances to handle the demand.
Aside: Why not just increase Sidekiq thread count? #
Sidekiq lets you fine-tune how many threads are running on an instance. This roughly translates into the number of concurrent workers. By default, this is 25, but you can always increase this number.
It turns out that as you increase threads, you potentially start to run into stability issues. It’s not recommended that you exceed 50. In practice, I have observed crashing workers, or perpetually idle workers when experimenting with higher thread counts.
This blog post goes into more detail around Sidekiq concurrency and why it is better to scale processes than threads.
Sidekiq metrics #
Convox allows autoscaling based on custom CloudWatch metrics, so the first challenge is getting Sidekiq metrics into Cloudwatch.
My research turned up two Ruby gems for this:
I decided to go with the former as the latter has not been updated in 5 years at the time of writing.
Setup was simple, once added to the Gemfile, I added it to my sidekiq.rb initializer:
Sidekiq::CloudWatchMetrics.enable!
Note that this assumes that you are already defining the standard AWS Access Key, Secret, and Region environment variables. If you aren’t, you can always instantiate an Aws::CloudWatch::Client instance and pass it in.
Once done, restart your server and you should start seeing metrics make their way to Cloudwatch.
Bringing it all together #
Now that we have Sidekiq metrics in Cloudwatch, all that remains is to update the convox.yml to enable autoscaling. Within the entry for your sidekiq service, just add the following.
sidekiq:
scale:
count: 1-10
targets:
custom:
Sidekiq/EnqueuedJobs:
aggregate: max
value: 500
And voila, you should now be ready to horizontally scale!