Queue module overview

• Prerequisites
  • TL;DR
• Use cases
  • Notifications
  • Jobs
  • Events
• Horizon
• Horizon console commands
  • How it works under the hood
  • Laravel Queue Workers
  • Connections
  • Our queues
  • Balancing strategies
  • Default configuration
  • Supervisors on production
  • Number of processes
  • When to create a new queue
  • How to wipe all queues

Prerequisites

1. Read Laravel Queues documentation.
2. Read Laravel Horizon documentation.

TL;DR

Queues aim to run some code out of the request-response cycle. In order to do that, we need to have few separate PHP processes that work as daemons (“workers”) and a supervisor that will monitor these workers and re-run them if they fail or run of the memory limit.

Horizon is an all-in-one solution: workers, supervisors, queue configurators, and a dashboard for queues. There are few available drivers that allow for storing queued Jobs in different places (SQL DB, Redis, SQS or even sync that runs Jobs synchronously), but Horizon works with Redis only. Horizon can supervise workers, but still needs a system/OS supervisor to manage the master Horizon PHP process (artisan horizon) that supervises Horizon workers.

Use cases

Currently, we use Laravel queues (with a Redis driver) for processing jobs (incl. notifications) asynchronously.

Notifications

All notifications in the system, i.e. mails, SMS, Slack, and Postcard notifications are Queueable (ShouldQueue implementations) and thus being queued using a queue driver (currently in multiple queues).

Jobs

In order to be more responsive, we need to handle some jobs asynchronously. That’s where our redis queues comes in.

Events

We don’t have any async event handler, but we can add these to our infrastructure if needed.

Horizon

We use Horizon to handle queued jobs on all of our queues. When using Horizon, it’s simpler to setup queues with any number of workers, use different balancing strategies, have elastic worker processes being spawned by supervisor (it configures the supervisor instances automatically as well), and has a good GUI interface to see what’s going on (A lot of room for improvement on the usage of the dashboard).

Horizon console commands

To ensure the data on the Horizon dashboard stays up to date, it is recommended to use horizon:* commands to manage queues instead of Laravel’s queue:* commands. For example:

// BAD
php artisan queue:clear redis --queue=notifications

// GOOD
php artisan horizon:clear redis --queue=notifications

How it works under the hood

This is how queues looks like using htop (2023-09-27, production):

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 1808 root       20   0 97168  130M 30872 S  0.7  0.8  0:00.85 │  /usr/bin/python /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf
31700 forge      20   0  491M  130M 30872 S  0.7  0.8  0:09.50 │  └─ php /home/forge/www.interaction-design.org/current/artisan horizon
31756 forge      20   0  490M  128M 31104 S  0.0  0.8  0:09.18 │     ├─ /usr/bin/php8.x artisan horizon:supervisor production-web-o7Er:supervisor--notifications redis --workers-name=default --balance=auto --max-processes=6 --min-processes=2 --nice=0 --balance-cooldown=3 --balance-max-shift=1 --parent-id=31700 --auto-scaling-strategy=time --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=notifications --sleep=3 --timeout=90 --tries=3 --rest=0
 6821 forge      20   0  542M  113M 32712 S  0.0  0.7  0:02.06 │     │  ├─ /usr/bin/php8.x artisan horizon:work redis --name=default --supervisor=production-web-o7Er:supervisor--notifications --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=notifications --sleep=3 --timeout=90 --tries=3 --rest=0
 5372 forge      20   0  555M  132M 38180 S  0.0  0.8  0:02.62 │     │  └─ /usr/bin/php8.x artisan horizon:work redis --name=default --supervisor=production-web-o7Er:supervisor--notifications --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=notifications --sleep=3 --timeout=90 --tries=3 --rest=0
31754 forge      20   0  489M  127M 30768 S  0.0  0.8  0:08.53 │     ├─ /usr/bin/php8.x artisan horizon:supervisor production-web-o7Er:supervisor--long redis-long --workers-name=default --balance= --max-processes=2 --min-processes=1 --nice=0 --balance-cooldown=3 --balance-max-shift=1 --parent-id=31700 --auto-scaling-strategy=time --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=default --sleep=3 --timeout=14400 --tries=2 --rest=0
 5137 forge      20   0  458M  101M 31064 S  0.0  0.6  0:00.57 │     │  ├─ /usr/bin/php8.x artisan horizon:work redis-long --name=long-timeout --supervisor=production-web-o7Er:supervisor--long --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=long-timeout --sleep=3 --timeout=14400 --tries=2 --rest=0
 3051 forge      20   0  540M  110M 32732 S  0.0  0.7  0:00.77 │     │  └─ /usr/bin/php8.x artisan horizon:work redis-long --name=long-timeout --supervisor=production-web-o7Er:supervisor--long --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=long-timeout --sleep=3 --timeout=14400 --tries=2 --rest=0
31752 forge      20   0  490M  128M 31008 S  0.0  0.8  0:09.11 │     ├─ /usr/bin/php8.x artisan horizon:supervisor production-web-o7Er:supervisor--priority redis --workers-name=default --balance= --max-processes=6 --min-processes=2 --nice=0 --balance-cooldown=3 --balance-max-shift=1 --parent-id=31700 --auto-scaling-strategy=time --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=high,default,low --sleep=3 --timeout=90 --tries=3 --rest=0
31774 forge      20   0  559M  130M 33048 S  0.0  0.8  0:03.07 │     │  ├─ /usr/bin/php8.x artisan horizon:work redis --name=default --supervisor=production-web-o7Er:supervisor--priority --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=high,default,low --sleep=3 --timeout=90 --tries=3 --rest=0
13288 forge      20   0  539M  108M 31880 S  0.0  0.7  0:02.00 │     │  ├─ /usr/bin/php8.x artisan horizon:work redis --name=default --supervisor=production-web-o7Er:supervisor--priority --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=high,default,low --sleep=3 --timeout=90 --tries=3 --rest=0
 9097 forge      20   0  550M  122M 32912 S  0.0  0.8  0:02.23 │     │  ├─ /usr/bin/php8.x artisan horizon:work redis --name=default --supervisor=production-web-o7Er:supervisor--priority --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=high,default,low --sleep=3 --timeout=90 --tries=3 --rest=0
 7771 forge      20   0  537M  107M 31880 S  0.0  0.7  0:01.97 │     │  ├─ /usr/bin/php8.x artisan horizon:work redis --name=default --supervisor=production-web-o7Er:supervisor--priority --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=high,default,low --sleep=3 --timeout=90 --tries=3 --rest=0
 5138 forge      20   0  533M  103M 31792 S  0.0  0.6  0:00.65 │     │  ├─ /usr/bin/php8.x artisan horizon:work redis --name=default --supervisor=production-web-o7Er:supervisor--priority --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=high,default,low --sleep=3 --timeout=90 --tries=3 --rest=0
 3029 forge      20   0  458M  101M 31184 S  0.0  0.6  0:00.69 │     │  └─ /usr/bin/php8.x artisan horizon:work redis --name=default --supervisor=production-web-o7Er:supervisor--priority --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=high,default,low --sleep=3 --timeout=90 --tries=3 --rest=0
31751 forge      25   5  489M  127M 31004 S  0.0  0.8  0:07.99 │     └─ /usr/bin/php8.x artisan horizon:supervisor production-web-o7Er:supervisor--sequential redis --workers-name=default --balance= --max-processes=1 --min-processes=1 --nice=5 --balance-cooldown=3 --balance-max-shift=1 --parent-id=31700 --auto-scaling-strategy=time --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=sequential--high,sequential--low --sleep=3 --timeout=90 --tries=2 --rest=0
31851 forge      25   5  460M  104M 31176 S  0.0  0.6  0:06.57 │        └─ /usr/bin/php8.x artisan horizon:work redis --name=default --supervisor=production-web-o7Er:supervisor--sequential --backoff=0 --max-time=0 --max-jobs=0 --memory=128 --queue=sequential--high,sequential--low --sleep=3 --timeout=90 --tries=2 --rest=0

the same, simplified version

/usr/bin/python /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf
└─ horizon
   ├─ horizon:supervisor supervisor--priority redis --min-processes=2 --max-processes=6
   │  ├─ horizon:work redis --queue=high,default,low --memory=128 --timeout=90
   │  ├─ horizon:work redis --queue=high,default,low --memory=128 --timeout=90
   ├─ horizon:supervisor supervisor--notifications redis --min-processes=2 --max-processes=6
   │  ├─ horizon:work redis --queue=notifications --memory=128 --timeout=90 --tries=3
   │  └─ horizon:work redis --queue=notifications --memory=128 --timeout=90 --tries=3
   ├─ horizon:supervisor supervisor--sequential redis --min-processes=1 --max-processes=1
   │  └─ horizon:work redis --queue=sequential--high,sequential--low --memory=128 --timeout=90
   └─ horizon:supervisor supervisor--long redis-long --min-processes=1 --max-processes=2
      ├─ horizon:work redis-long --queue=long-timeout --memory=128 --timeout=14400
      └─ horizon:work redis-long --queue=long-timeout --memory=128 --timeout=14400
    

As you can see:

1. supervisord (via /usr/bin/python /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf) spawns a single artisan horizon master process. Note, /etc/supervisor/supervisord.conf includes
2. artisan horizon spawns horizon:supervisor instances as configured in horizon.php
3. each artisan horizon:supervisor instance spawns a configured number of horizon:work workers (see horizon.defaults.*.processes, where * is a horizon supervisor name)

Retrying multiple jobs at once on Horizon

Horizon has a nice GUI for managing jobs and queues which can be accessed via https://www.interaction-design.org/admin/horizon. However, one thing it doesn't yet provide is a 'retry all failed jobs' action. If you find yourself needing to retry many jobs at once (e.g., if the mail provider API stopped responding for a period of time) you can open up the 'console' tab in your browser's developer tools and copy and paste the following:

document.querySelectorAll('table tr td:nth-child(4) > a').forEach((elem) => elem.click());

That should select the 'retry' button on each row and click them all at once.

There's also another option, which retries all failed jobs at once. Open an SSH console to production, browse to the root project directory (cd ~/interaction-design.org/current), open a tinker session (php artisan tinker) and run the following:

\Illuminate\Support\Facades\DB::table('jobs--failed')
    ->where('failed_at', '>', now()->subDay()) // Add any condition
    ->pluck('uuid')
    ->mapInto(\Laravel\Horizon\Jobs\RetryFailedJob::class)
    ->each(fn ($j) => app()->call([$j, 'handle']))

Laravel Queue Workers

We also have three (3) queue workers to run our default, high, and low queues. These processes are being handled by supervisor (number of processes, the command to run, restarts, etc.).

Connections

There are two redis connections for normal and long-running queued jobs/notifications/etc.

Note: I’m not sure why we need two connections, but I’ll try to find out and document it here.

Our queues

• Each connection has a default queue (which works as the fallback for everything that doesn’t have an explicit queue specified).
• We use low queue for low-priority jobs. This queue is tolerant for time-consuming and unstable jobs. All video-related jobs should use this queue.
• We use high queue for high-priority jobs. Please do not push any long-running jobs to this queue. In order to make this queue fast, we have lower default number of tries per jobs. We use notifications queue for all notifications.
• We use the long-timeout queue on redis-long connection for any long-running job, with an extended timeout of 4 hours.
• For debugging purposes, we may create some temp queues. The goal is to monitor queue size and performance issues. After some period of monitoring, we remove such queue and move all it’s jobs to other queues.

Balancing strategies

• auto: Scales the number of horizon workers (up & down) according to the load between all the queues
• simple: Simply divides the number of available processes between queues
• false: This acts like the default Laravel queue worker (order of declared queues matters)

Read more

Default configuration

• There’s a default configuration that applies to our staging/local environments. The default configuration has lower number of processes to reduce the load.

Supervisors on production

• high, default, and low queues are all run by a single supervisor. The reason is to be able to use the false balancing strategy so that our high queue has the highest priority.
• notifications have their own separate supervisor. This will enable auto-scaling ability so that we can spawn more processes when load increases.
• default queue on redis-long has a separate supervisor. The reason is to have an extended timeout (4 hours).

Number of processes

• In order to be able to increase/decrease the min/max number of processes in production, multiple environment variables are introduced. These include: -- HORIZON_MIN_PROCESSES: minimum number of processes for supervisor--priority and supervisor--notifications Horizon supervisors -- HORIZON_MAX_PROCESSES: maximum number of processes for supervisor--priority and supervisor--notifications Horizon supervisors

When to create a new queue

1. You need special limits for memory or timeouts
2. You need special defaults for a group of Jobs (ex.: number of tries)

How to wipe all queues

In non-production environments, such as staging, it can be useful to wipe all queues and start fresh. You can use horizon:wipe command to do that. Internally it makes use of horizon:clear command and will clear all jobs from all queues across all connections.