Fusion of Thoughts

Unlocking the Secrets of Code: Your Go-To Source for Expert Coding Insights

urfusion

Shoes-urfusion-anypage

Step-by-Step Guide to Apache Kafka Magento 2 Integration with Apache2 (2025)

by ·

Introduction To Apache Kafka Magento 2 integration

In today’s high-volume e-commerce environment, real-time event streaming is a key architectural requirement. By implementing Apache Kafka Magento 2 integration, you can handle large streams of orders, inventory updates, product catalogue changes, and customer events in a scalable, fault-tolerant way. This article walks through how to set up Apache Kafka together with Apache2 (HTTP server) and Magento 2 (e-commerce platform)—covering installation, proxy configuration, message queue topic setup, Magento 2 configuration and best practices.

By the end of this tutorial you’ll have a baseline architecture where Magento 2 publishes events (e.g., new orders, inventory changes) into Kafka topics, consumers (internal or external) pick up these events for analytics, fulfilment, shipping or other downstream systems. Let’s dive in.

 

AI-Powered Search Engines: The Future of Answer-First Discovery

 

Why integrate Apache Kafka with Magento 2?

Before jumping into commands, it’s helpful to understand why such an integration matters.

  • Apache Kafka is a distributed publish-subscribe messaging system designed for high-throughput, high-volume, fault-tolerant event streaming. (Cloudkul)
  • Magento 2 is an enterprise-grade e-commerce platform which produces many system events: order placed, payment received, shipment created, product updated, customer created, etc.
  • Using Kafka rather than simpler queue systems allows you to build real-time pipelines: Magento 2 → Kafka topic → multiple consumers (analytics, warehouse, CRM, BI, etc).
  • According to articles, the “Apache Kafka Magento 2 integration” gives better scalability for large stores compared to simpler brokers. (LinkedIn)
  • Also, using Apache2 as a front-end reverse proxy ensures secure HTTP/S access, SSL termination, and routing of REST/Webhook endpoints for custom consumers.

Hence, this integration is ideal for medium/large Magento 2 stores where you expect real-time event processing, analytics, multi-system integration, or high throughput.

Prerequisites & Environment Setup

Before you begin, ensure you have the following components and environment:

  • A server (or VM) running Ubuntu (e.g., 20.04 or later) with sudo access.
  • Apache2 installed and serving your Magento 2 site (or planned to install).
  • Magento 2 installation (preferably 2.x) with access to CLI, database, file system.
  • Java JDK installed (Kafka is Java-based). For example OpenJDK 11 or later. (Medium)
  • Basic familiarity with terminal commands, editing config files, CLI operations of Kafka & Zookeeper (or KRaft mode).
  • Enough memory/CPU for Kafka: at least 4 GB RAM recommended for dev/test. (DigitalOcean)

Step 1: Install and configure Apache Kafka

This step sets up Apache Kafka on your server. We assume a single‐node setup for dev/test; for production you’ll use a cluster.

  1. Install Java (if not already): 
    sudo apt update  
sudo apt install openjdk-11-jdk -y  
java -version
  2. Download Kafka: 
    wget https://downloads.apache.org/kafka/4.1.0/kafka_2.13-4.1.0.tgz  
tar -xzf kafka_2.13-4.1.0.tgz  
sudo mv kafka_2.13-4.1.0 /opt/kafka

    (Confirm latest version from official downloads page.) (Apache Kafka)

  3. (Optional) Create a dedicated user for Kafka: 
    sudo adduser --system --no-create-home --group kafka  
sudo chown -R kafka:kafka /opt/kafka

    (Helps with security and service automation.)

  4. Configure Kafka (for example update /opt/kafka/config/server.properties):
    • Set log.dirs to a persistent directory (e.g., /var/lib/kafka-logs).
    • Add delete.topic.enable = true if you want to allow topic deletion. (Hevo Data)
    • Configure broker.id=0, listeners=PLAINTEXT://:9092, etc.
  5. Create systemd service files for Kafka (and Zookeeper if required). Example for Ubuntu 20.04:
    /etc/systemd/system/zookeeper.service:

    [Unit]  
Description=ZooKeeper Service  
Requires=network.target  
After=network.target  

[Service]  
User=kafka  
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties  
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh  
Restart=on-failure  

[Install]  
WantedBy=multi-user.target

    /etc/systemd/system/kafka.service:

    [Unit]  
Description=Apache Kafka Server  
Requires=zookeeper.service  
After=zookeeper.service  

[Service]  
User=kafka  
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties  
ExecStop=/opt/kafka/bin/kafka-server-stop.sh  
Restart=on-failure  

[Install]  
WantedBy=multi-user.target

    Then:

    sudo systemctl daemon-reload  
sudo systemctl enable zookeeper kafka  
sudo systemctl start zookeeper kafka  
sudo systemctl status kafka

    (Cherry Servers)

  6. Test your installation:
    • Create a topic: 
      /opt/kafka/bin/kafka-topics.sh --create --topic magento_events --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
    • Start a producer: 
      /opt/kafka/bin/kafka-console-producer.sh --topic magento_events --bootstrap-server localhost:9092
> {"event":"order_created","order_id":1001}
    • Start a consumer: 
      /opt/kafka/bin/kafka-console-consumer.sh --topic magento_events --bootstrap-server localhost:9092 --from-beginning

    You should see messages you typed sent to producer appearing in consumer.

At this point, Kafka is running and ready to accept events.

Step 2: Configure Apache2 (HTTP server) for Magento 2 & optional Kafka HTTP endpoints

Since you already have or will have Apache2 serving your Magento 2 store, you may also want Apache2 to handle HTTP/S, SSL termination, reverse proxying of Kafka-provided REST endpoints (if you expose REST/Connect endpoints), and secure webhooks for events.

Here’s how to set it up:

  1. Ensure Apache2 is installed and active: 
    sudo apt update  
sudo apt install apache2 -y  
sudo systemctl enable apache2  
sudo systemctl start apache2
  2. Enable necessary Apache2 modules: 
    sudo a2enmod rewrite headers proxy proxy_http ssl  
sudo systemctl restart apache2
  3. Configure a VirtualHost in /etc/apache2/sites-available/magento.conf (adjust ServerName etc): 
    <VirtualHost *:80>
  ServerName shop.example.com
  DocumentRoot /var/www/magento2/pub
  <Directory /var/www/magento2/pub>
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>

  ErrorLog ${APACHE_LOG_DIR}/magento_error.log
  CustomLog ${APACHE_LOG_DIR}/magento_access.log combined
</VirtualHost>

    Then enable:

    sudo a2ensite magento.conf  
sudo systemctl reload apache2
  4. If you wish to expose Kafka REST APIs or HTTP endpoint for integrations, you can proxy to the Kafka port via Apache2. For example in your VirtualHost: 
    ProxyPass /kafka-rest http://localhost:8082  
ProxyPassReverse /kafka-rest http://localhost:8082

    This way your integrations can access Kafka REST or Kafka Connect via https through Apache2.

  5. Enable SSL (recommended for production): 
    sudo apt install certbot python3-certbot-apache -y  
sudo certbot --apache -d shop.example.com

    This gives you HTTPS for your Magento site and Kafka proxy endpoints.

So now Apache2 is set up both for your Magento 2 site and as a reverse proxy for Kafka-related endpoints if needed.

Step 3: Configure Magento 2 for Kafka integration

Now moving to the core of the Apache Kafka Magento 2 integration: making Magento 2 publish relevant events into Kafka topics (and optionally consume them). The integration might be custom-built or via an extension/connector.

3.1 Identify which Magento 2 events you need

Typical events you may want to publish into Kafka:

  • New orders created (sales_order_place_after)
  • Order status changed (sales_order_save_after)
  • Product created/updated (catalog_product_save_after)
  • Inventory change (cataloginventory_stock_item_save_after)
  • Customer registered/updated (customer_save_after)

3.2 Create a custom Magento 2 module for Kafka publishing

Here’s a simplified example structure:

app/code/YourCompany/KafkaPublisher/

  • registration.php
  • etc/module.xml
  • etc/events.xml
  • Observer/OrderCreated.php
  • Model/KafkaProducer.php

events.xml:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework/Event/etc/events.xsd">
  <event name="sales_order_place_after">
    <observer name="yourcompany_kafka_order_created" instance="YourCompany\KafkaPublisher\Observer\OrderCreated" />
  </event>
</config>

Observer/OrderCreated.php:

namespace YourCompany\KafkaPublisher\Observer;

use Magento\Framework\Event\ObserverInterface;
use Magento\Framework\Event\Observer;
use YourCompany\KafkaPublisher\Model\KafkaProducer;

class OrderCreated implements ObserverInterface
{
    protected $kafkaProducer;

    public function __construct(KafkaProducer $kafkaProducer)
    {
        $this->kafkaProducer = $kafkaProducer;
    }

    public function execute(Observer $observer)
    {
        $order = $observer->getEvent()->getOrder();
        $payload = [
            'event' => 'order_created',
            'order_id' => $order->getId(),
            'increment_id' => $order->getIncrementId(),
            'customer_id' => $order->getCustomerId(),
            'total' => $order->getGrandTotal(),
            'timestamp' => time(),
        ];
        $this->kafkaProducer->publish('magento_orders', json_encode($payload));
    }
}

Model/KafkaProducer.php:

namespace YourCompany\KafkaPublisher\Model;

class KafkaProducer
{
    protected $bootstrapServers;

    public function __construct(
        \Magento\Framework\App\Config\ScopeConfigInterface $scopeConfig
    ) {
        $this->bootstrapServers = $scopeConfig->getValue('kafkapublisher/general/bootstrap_servers');
    }

    public function publish($topic, $message)
    {
        // Example using PhpKafka or librdkafka
        $producer = new \RdKafka\Producer();
        $producer->addBrokers($this->bootstrapServers);

        $topicObj = $producer->newTopic($topic);
        $topicObj->produce(RD_KAFKA_PARTITION_UA, 0, $message);

        $producer->flush(10000);
    }
}

You will need to install the PHP Kafka library (e.g., rdkafka PHP extension) or use Kafka REST proxy.

3.3 Configure Magento 2 module settings

In etc/adminhtml/system.xml, provide config fields for bootstrap servers, topic names, etc. Then in app/etc/config.php set your module active. Clear caches:

php bin/magento module:enable YourCompany_KafkaPublisher  
php bin/magento setup:upgrade  
php bin/magento cache:flush

3.4 Testing the integration

  • Place a new order in Magento 2.
  • In Kafka server, run consumer: 
    /opt/kafka/bin/kafka-console-consumer.sh --topic magento_orders --bootstrap-server localhost:9092 --from-beginning
  • Verify the JSON payload appears.
  • Check for errors in var/log or Kafka logs.

Step 4: Use Apache2 + Kafka for more advanced integrations

With the basic pipeline in place, you can now build more advanced workflows leveraging Apache2 as the front-end and Kafka as the event backbone:

  1. Expose Kafka REST Proxy via Apache2
    If you have Kafka REST Proxy (from Confluent or open-source) listening on port 8082, use Apache2 reverse proxy (see above) to expose it at https://shop.example.com/kafka-rest. This allows external systems to publish or consume via HTTP/S.
  2. Multi-tenant topics
    You might create separate topics like magento_orders, magento_products, magento_inventory. Make sure your consumers subscribe appropriately.
  3. Apache2 webhooks for consumers
    Instead of long-running CLI consumers, you can set up webhooks or micro-services that subscribe to Kafka and then update external systems (ERP, CRM). These can be accessed via Apache2-secured endpoints.
  4. Securing the pipeline
    • Use SSL/TLS for Kafka endpoints or restrict via firewall.
    • Use SASL authentication for Kafka brokers in production.
    • Use Apache2 to enforce HTTP basic auth or OAuth for REST endpoints.
    • Monitor Kafka lag, topic sizes, consumer offsets.
  5. Scaling & high availability
    In production, you will run Kafka in a cluster of brokers, multiple partitions, replication factor >1, possibly KRaft mode (Zookeeperless) in newer Kafka versions. (Cherry Servers)
    Also make sure your Apache2 front-end for Magento 2 is load-balanced, and your Magento 2 message consumer services are resilient.

Step 5: Performance tuning & best practices for e-commerce

When you implement Apache Kafka Magento 2 integration, you should keep in mind the following best practices:

  • Topic partition count: More partitions = higher parallelism. Consider number of consumers when deciding partitions.
  • Replication factor: For production, set to 2 or 3 to ensure broker failure tolerance.
  • Message format: Use JSON or Avro. Define a schema for your events (event type, timestamp, payload).
  • Consumer offset management: Ensure you handle offsets properly (manual/auto). Monitor consumer lag.
  • Back-pressure & flow control: If events accumulate faster than consumers process, you’ll see lag.
  • Magento 2 bulk-publishing vs real-time: For very high volumes (e.g., flash sale orders), consider using batching or asynchronous publish to reduce latency impact on Magento storefront.
  • Apache2 caching: Ensure your web front-end (Apache2 + Magento 2) is optimized (Redis caching, Varnish, full-page cache), so event publishing doesn’t slow the user experience.
  • Monitoring & logging: Use tools like Prometheus + Grafana or Confluent Control Center to monitor Kafka cluster health, topics, consumer groups.
  • Disaster recovery & retention: Configure log retention in Kafka (log.retention.hours) to store events for a time period so replay is possible.
  • Version upgrades: Ensure you follow Kafka upgrades carefully and test your Magento 2 module compatibility.

Troubleshooting common issues

Here are some common issues you may encounter and how to address them:

  • Kafka broker not reachable: Ensure listeners and advertised.listeners in server.properties are correctly set. Ensure firewall allows port 9092.
  • Consumer sees no messages: Check that topic exists, the producer is publishing, and consumer group is correct. Use kafka-consumer-groups.sh to inspect.
  • Magento 2 events not published: Check your observer is correctly registered, module enabled, caches flushed. Check for exceptions in var/log/exception.log or system.log.
  • Apache2 proxy errors: Check logs (/var/log/apache2/error.log), and ensure ProxyPass target is correct, and proxy modules enabled.
  • High lag in consumers: Likely consumers are too slow or partitions too few. Increase partitions or optimize consumer logic.
  • Security/auth issues: If using SASL/SSL for Kafka, ensure your PHP library supports it and certificates/keys are correctly configured.

Conclusion

The Apache Kafka Magento 2 integration offers a powerful pipeline for event-driven e-commerce architecture. By combining Kafka for high-throughput streaming, Apache2 as a secure HTTP front & reverse-proxy, and Magento 2 as the core storefront and event generator, you create a scalable, future-proof system capable of handling real-time orders, inventory sync, analytics and multi-system workflows.

In this article we covered:

  • Why you’d integrate Kafka with Magento 2
  • How to install and configure Apache Kafka on Ubuntu
  • How to configure Apache2 for Magento 2 + Kafka proxying
  • How to build a Magento 2 module to publish events into Kafka topics
  • Advanced integration ideas, performance tuning and best practices
  • Troubleshooting tips

Implementing this integration gives your e-commerce platform a major architectural advantage. As your store grows in transaction volume or you add more downstream systems (CRM, ERP, analytics), this setup will scale better than traditional synchronous processing methods.

Feel free to adapt the module, topics, event payloads and consumer logic to your business needs. If you would like deeper guidance (e.g., building consumers, Kafka Streams for analytics, Sunsetting RabbitMQ in Magento 2, or multi-cluster deployments), I’m happy to help in future posts.

 

 

Tags:

You may also like...

Creating a Shopify App using Laravel How to Create Custom WordPress Plugin? How to Build a Telegram Bot using PHP How to Convert Magento 2 into PWA?