Understanding Message Queue in Magento 2: A Step-by-Step Guide

When working on large-scale e-commerce systems, sending tasks like notifications, emails, or integrations directly in the request cycle can slow down performance. Magento 2 provides a Message Queue framework to handle these tasks asynchronously.

In this blog, we will explore how Magento’s message queue works, what each configuration file does, and how everything connects together. Finally, we will see how to run queues locally.

What is a Message Queue in Magento 2?

A message queue is a buffer where messages (tasks) are stored until they are processed by a consumer.

  • Publisher: Sends messages into the queue.
  • Topic: Identifies the type of message.
  • Queue: Holds the actual messages until they are processed.
  • Consumer: Reads messages from the queue and processes them.

This decouples the request (publishing) from the execution (processing).

Required Files in Magento 2 Queue System

Magento uses XML configuration files to define publishers, topics, queues, and consumers. Let’s go through each one.

1. communication.xml

Defines the topic name and the type of message it expects.

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:Communication/etc/communication.xsd">

    <topic name="ecommet.bulk.notification.topic" request="string"/>
</config>
  • topic name: The identifier for our queue messages.
  • request="string": Messages must be passed as strings (usually JSON).
2. queue_publisher.xml

Registers which topics are available for publishing.

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/queue_publisher.xsd">

    <publisher topic="ecommet.bulk.notification.topic"/>
</config>
  • When you call $publisher->publish('ecommet.bulk.notification.topic', $data), Magento looks here to know if the topic exists.
3. queue_topology.xml

Defines how topics are bound to queues.

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/queue_topology.xsd">

    <exchange name="magento" type="topic" connection="db">
        <binding id="ecommet.bulk.notification.binding"
                 topic="ecommet.bulk.notification.topic"
                 destinationType="queue"
                 destination="ecommet.bulk.notification.queue"/>
    </exchange>
</config>
  • This tells Magento: whenever a message is published to ecommet.bulk.notification.topic, place it in the queue named ecommet.bulk.notification.queue.
  • connection="db" means we are using Magento’s database as the queue backend (no RabbitMQ required).
4. queue_consumer.xml

Registers the consumer that will process messages from a specific queue.

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/queue_consumer.xsd">

    <consumer name="ecommet.bulk.notification.consumer"
              queue="ecommet.bulk.notification.queue"
              connection="db"
              handler="Ecommet\BulkNotification\Model\Consumer\Notification::process"/>
</config>
  • name: Identifier for the consumer.
  • queue: The queue to listen to.
  • handler: The PHP class and method that will be executed when a message arrives.
5. Consumer Class

The consumer is a simple PHP class with a process method.

namespace Ecommet\BulkNotification\Model\Consumer;

use Psr\Log\LoggerInterface;

class Notification
{
    protected $logger;

    public function __construct(LoggerInterface $logger)
    {
        $this->logger = $logger;
    }

    public function process($message)
    {
        $this->logger->info("Processing message: " . $message);
        // Your logic goes here (e.g., send SMS, push notification, etc.)
    }
}
6. Publishing a Message

Anywhere in your code (e.g., a controller or service class), you can publish a message:

$this->publisher->publish(
    'ecommet.bulk.notification.topic',
    json_encode(['customer_id' => 123, 'type' => 'sms'])
);

How the Pieces Work Together

  1. Controller or service publishes a message → queue_publisher.xml validates the topic.
  2. Message goes into queue_message table in DB via queue_topology.xml binding.
  3. Consumer registered in queue_consumer.xml reads messages from that queue.
  4. Consumer class processes each message using the process() method.

Running the Queue Locally

When using the DB connection, Magento stores messages in the queue_message table.

Check registered consumers:
bin/magento queue:consumers:list
Start a consumer:
bin/magento queue:consumers:start ecommet.bulk.notification.consumer

Options:

  • --max-messages=100 → Stop after 100 messages.
  • --single-thread → Run in a single thread (simpler for debugging).
  • Add -vvv for verbose logging.

Tags

Related Posts