Kafka Pub/Sub Demo

A Node.js demonstration of Apache Kafka pub/sub messaging using KafkaJS with multi-partition topics and consumer groups.

Prerequisites

• Node.js v18 or higher
• Docker & Docker Compose
• Apache Kafka (running in Docker)

Setup

1. Install Dependencies

npm install

2. Start Kafka Broker

Option A: KRaft Mode (Recommended - No ZooKeeper Required)

Run the following command to start a Kafka broker with KRaft mode in Docker:

docker run -d --name kafka -p 9092:9092 `
  -e KAFKA_NODE_ID=1 `
  -e KAFKA_PROCESS_ROLES=broker,controller `
  -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 `
  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 `
  -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT `
  -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 `
  -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER `
  -e KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT `
  -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 `
  -e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 `
  -e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 `
  -e CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk `
  confluentinc/cp-kafka

Option B: Traditional Mode with ZooKeeper (Legacy)

If you prefer the traditional Kafka setup with ZooKeeper:

# Start ZooKeeper
docker run -d --name zookeeper -p 2181:2181 zookeeper

# Start Kafka with ZooKeeper
docker run -d --name kafka -p 9092:9092 `
  -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 `
  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 `
  -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 `
  confluentinc/cp-kafka

3. Create Topics

Create the rider-updates topic with admin setup:

node admin.js

Output:

Admin connecting...
Admin connected
Creating topic [rider-updates]
Topic created success [topic: rider-updates]
Admin disconnected

Usage

Producer

Send messages to the Kafka topic. Messages are partitioned based on rider location:

• North → Partition 0
• South → Partition 1

node producer.js

Input format: <riderName> <location>

Example:

> alice north
> bob south
> charlie north

Each message is sent as JSON:

{
  "name": "alice",
  "location": "north"
}

Consumer

Consume messages from the topic with consumer groups:

# Consume as consumer group 'user-1' (default)
node consumer.js

# Consume as consumer group 'user-2'
node consumer.js user-2

# Consume as any custom group
node consumer.js custom-group

Key Features:

• Each consumer group independently consumes all messages
• Multiple consumers in the same group share partitions
• fromBeginning: true ensures the consumer reads all messages from the topic start

Project Structure

• client.js - Kafka client configuration connecting to localhost:9092
• admin.js - Creates the rider-updates topic with 2 partitions
• producer.js - Interactive producer for sending location updates
• consumer.js - Consumer with support for multiple consumer groups
• package.json - Project dependencies (kafkajs)

Cleanup

Stop the Kafka container:

docker stop kafka
docker rm kafka

Dependencies

• kafkajs v2.2.4 - KafkaJS client for Node.js

License

ISC