Background processing: cron, queues, and batch

Work that is slow, periodic, or allowed to fail does not belong in a page request. Drupal gives you three tools: cron hooks for scheduled work, queues for reliable deferred work, and batches for long operations a user is waiting on. This page shows you when to use each and how to build all three.

Goal

Your module runs work in the background: on a schedule, item by item with retries, or across multiple requests while a user watches a progress bar. Pick the tool by the question it answers:

You need	Use	Runs
“Do housekeeping periodically”	hook_cron()	Every cron run, synchronously
“Do this for each item, eventually, reliably”	A #[QueueWorker] plugin	At cron or on demand, with retries
“Do something big right now, without timing out”	The Batch API	Across requests, with a progress page

Prerequisites

• A custom module; the first tutorial part gets you one.
• Services and dependency injection for the injection patterns used throughout.
• Cron itself must run on your site. On Standard-profile sites, the core Automated Cron module triggers it after page requests; in production, run drush cron from a real scheduler. ddev drush core:cron (ddev drush cron) runs it by hand.

Steps

The samples come from the examples project’s docs_snippets and module_howtos_b_demo modules. Replace their module names, queue IDs (docs_snippets_publish_reminder, docs_demo_article_sync), and state keys with your own.

Implement hook_cron()

Add a #[Hook('cron')] method to a class in your module’s src/Hook/ directory. Cron hooks may be class methods; this is worth stating because their .install-file cousins, the update hooks, must stay procedural (the Hook attribute documents exactly which hooks that rule covers). This implementation does both kinds of cron work, inline housekeeping and queueing:

src/Hook/DemoHooks.php

/**
 * Implements hook_cron().
 *
 * Cron hooks may be class methods like this one. Update hooks may
 * not: core requires hook_update_N() and friends to stay procedural.
 */
#[Hook('cron')]
public function cron(): void {
  // Fast, idempotent housekeeping can run inline on every cron run.
  $this->metrics->archiveOldDays();

  // Defer slow or fallible work (here: pushing articles to a remote
  // API) to a queue. The matching #[QueueWorker] plugin declares a
  // "cron" key, so the items are processed later in this same run.
  $last_run = $this->state->get('docs_demo.last_sync', 0);
  $nids = $this->entityTypeManager->getStorage('node')->getQuery()
    ->condition('type', 'article')
    ->condition('changed', $last_run, '>=')
    ->accessCheck(FALSE)
    ->execute();

  $queue = $this->queueFactory->get('docs_demo_article_sync');
  foreach ($nids as $nid) {
    $queue->createItem(['nid' => (int) $nid]);
  }
  $this->state->set('docs_demo.last_sync', $this->time->getRequestTime());
}

Facts about the runtime, from Cron itself:

• The whole cron run is wrapped in a lock, so two cron runs never overlap. Your hook still shares the run’s time budget with every other module, which is why anything slow goes to a queue.
• You do not control frequency. The site owner does, at Administer → Configuration → System → Cron/admin/config/system/cron or in their crontab. Gate work that has its own schedule with a timestamp in state, as the example’s last_sync value does.
• Cron runs as the anonymous user, with the resulting entity access behavior; note accessCheck(FALSE) in the example.

Create a queue worker

A queue worker is a plugin class in src/Plugin/QueueWorker/ with the QueueWorker attribute. Its id doubles as the queue name; the optional cron key asks Drupal to process the queue at cron for up to the given number of seconds.

web/modules/custom/docs_snippets/src/Plugin/QueueWorker/PublishReminderWorker.php

<?php

declare(strict_types=1);

namespace Drupal\docs_snippets\Plugin\QueueWorker;

use Drupal\Core\Entity\EntityTypeManagerInterface;
use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Drupal\Core\Queue\Attribute\QueueWorker;
use Drupal\Core\Queue\QueueWorkerBase;
use Drupal\Core\StringTranslation\TranslatableMarkup;
use Drupal\node\NodeInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;

/**
 * Publishes nodes that were queued for delayed publication.
 *
 * Add items with:
 * @code
 * \Drupal::queue('docs_snippets_publish_reminder')->createItem(['nid' => 42]);
 * @endcode
 *
 * The "cron" key asks Drupal to process this queue for up to 30 seconds
 * on every cron run. Leave it out to process the queue yourself, for
 * example with "drush queue:run".
 */
#[QueueWorker(
  id: 'docs_snippets_publish_reminder',
  title: new TranslatableMarkup('Delayed node publisher'),
  cron: ['time' => 30],
)]
final class PublishReminderWorker extends QueueWorkerBase implements ContainerFactoryPluginInterface {

  /**
   * Constructs the queue worker.
   *
   * @param array $configuration
   *   The plugin configuration.
   * @param string $plugin_id
   *   The plugin ID.
   * @param mixed $plugin_definition
   *   The plugin definition.
   * @param \Drupal\Core\Entity\EntityTypeManagerInterface $entityTypeManager
   *   The entity type manager.
   */
  public function __construct(
    array $configuration,
    string $plugin_id,
    mixed $plugin_definition,
    protected readonly EntityTypeManagerInterface $entityTypeManager,
  ) {
    parent::__construct($configuration, $plugin_id, $plugin_definition);
  }

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition): static {
    return new static(
      $configuration,
      $plugin_id,
      $plugin_definition,
      $container->get('entity_type.manager'),
    );
  }

  /**
   * {@inheritdoc}
   */
  public function processItem($data): void {
    $node = $this->entityTypeManager->getStorage('node')->load($data['nid']);
    if (!$node instanceof NodeInterface) {
      // The node was deleted in the meantime; nothing to do. Returning
      // normally removes the item from the queue. Throw an exception
      // instead to release the item for a retry on the next run.
      return;
    }
    $node->setPublished()->save();
  }

}

Items go in with createItem() on the queue, obtained from the queue factory service: \Drupal::queue('docs_snippets_publish_reminder')->createItem(['nid' => 42]), or the injected factory in a service class. The cron hook above follows the same pattern for its own docs_demo_article_sync queue. Each item should be small, serializable data (IDs, not loaded entities: the entity may change or vanish before the item is processed).

Control retries with queue exceptions

What processItem() does decides each item’s fate. The contract, straight from Cron::processQueue():

• Return normally: the item is deleted, done.
• Throw any exception: the error is logged and the item stays queued; it becomes available again when its lease expires.
• Throw RequeueException: the item is released for retry immediately.
• Throw DelayedRequeueException: the item is held back for the delay you specify (on queue backends implementing DelayableQueueInterface, which includes the default database queue).
• Throw SuspendQueueException: stop processing this whole queue for now; the item is released. Use it when a shared dependency of every item (a remote API, for example) is down.

A queue item's lifecycle

A queue item is claimed with a lease, then processItem runs. Returning normally deletes the item. Throwing any plain exception leaves the item claimed until its lease expires, when it becomes available again: a failed item is not lost. RequeueException releases it for immediate retry, DelayedRequeueException holds it back for a set delay, and SuspendQueueException releases it and stops the whole queue.

Process queues on demand

Cron is the default consumer, not the only one. Drush processes a single queue whenever you want:

ddev drush queue:list
ddev drush queue:run docs_snippets_publish_reminder

A worker without a cron key in its attribute is processed only this way, which is the setup you want when a dedicated runner (a crontab entry or a supervisor process) should own the queue instead of the general cron run.

Build a batch for long UI operations

Cron and queues run detached from any user. When an administrator clicks a button that triggers ten thousand writes, you need the work done now, without blowing the PHP time limit, and with feedback. That is the Batch API: it splits the work into operations and runs them across sequential requests behind a progress page.

Assemble the batch with BatchBuilder and hand it to batch_set(), typically from a form submit handler:

src/Batch/ArticlePublishBatch.php

/**
 * Sets up a batch that publishes the given articles.
 *
 * @param int[] $nids
 *   The IDs of the articles to publish.
 */
public static function publishAll(array $nids): void {
  $batch = (new BatchBuilder())
    ->setTitle(t('Publishing articles'))
    ->setInitMessage(t('Preparing to publish...'))
    ->setProgressMessage(t('Processed @current of @total batches.'))
    ->setErrorMessage(t('Publishing ran into an error.'))
    ->setFinishCallback([self::class, 'finished']);

  // Each operation runs in its own request, so size the chunks to
  // stay well under the PHP execution time limit.
  foreach (array_chunk($nids, 20) as $chunk) {
    $batch->addOperation([self::class, 'processChunk'], [$chunk]);
  }

  // Hand the batch to Drupal. In a form submit handler this is all
  // you do: Drupal takes over the response and shows the progress
  // page until every operation has run.
  batch_set($batch->toArray());
}

The operation and finished callbacks run in later, separate requests, so they must be statically callable and get everything through arguments, $context, or the global container:

src/Batch/ArticlePublishBatch.php

/**
 * Batch operation: publishes one chunk of articles.
 *
 * Operations run in fresh requests, after the original form request
 * has ended, so everything they need must arrive as arguments or
 * come from \Drupal::service().
 *
 * @param int[] $nids
 *   The IDs of the articles in this chunk.
 * @param array $context
 *   The batch context, persisted between operations.
 */
public static function processChunk(array $nids, array &$context): void {
  $storage = \Drupal::entityTypeManager()->getStorage('node');
  foreach ($storage->loadMultiple($nids) as $node) {
    $node->setPublished()->save();
    // $context['results'] accumulates across operations and is handed
    // to the finished callback.
    $context['results'][] = $node->id();
  }
  // Shown on the progress page while the next operation loads.
  $context['message'] = t('Published @count articles so far.', [
    '@count' => count($context['results'] ?? []),
  ]);
}

/**
 * Batch finished callback: reports the outcome to the user.
 *
 * @param bool $success
 *   TRUE if every operation completed without a fatal error.
 * @param array $results
 *   The accumulated $context['results'] values.
 * @param array $operations
 *   The operations that did not run, when $success is FALSE.
 */
public static function finished(bool $success, array $results, array $operations): void {
  $messenger = \Drupal::messenger();
  if ($success) {
    $messenger->addStatus(t('Published @count articles.', [
      '@count' => count($results),
    ]));
  }
  else {
    $messenger->addError(t('Publishing stopped early; @count operations did not run.', [
      '@count' => count($operations),
    ]));
  }
}

Update hooks reuse this machinery through their $sandbox parameter; see Custom database tables and update paths.

Verify

1. Run cron and watch your hook and queue execute:

   ddev drush core:cron
   ddev drush watchdog:show --count=20

   The log shows a completed cron run; failures in your worker appear here with the exception message.

2. Inspect queue depth before and after: ddev drush queue:list shows your queue’s item count dropping to zero once processing succeeds.

3. For a batch, trigger the form that calls batch_set(): you see the progress page, then your finished callback’s message on the destination page.

Variations

• Tune another module’s cron queue. hook_queue_info_alter() lets a site-specific module change a worker’s cron time budget without forking it.
• Queues without workers. You can create and consume queues directly through QueueInterface (claimItem(), deleteItem(), releaseItem()) with no plugin at all; you then own the processing loop. Cron::processQueues() is the reference implementation to copy.
• Swap the queue backend. The QueueFactory resolves each queue to a backend from settings: $settings['queue_service_NAME'] for one queue or $settings['queue_default'] for all of them. Contrib backends (Redis, for example) slot in here without changes to your worker.

Related

• Call external APIs applies this page’s queue pattern to outbound HTTP.
• Custom database tables and update paths covers the lock service that cron uses and batched update hooks.
• Logging is where your background work reports what happened; nobody is watching a screen.
• API topic: queue operations.