Apache Kafka is a powerful open-source platform designed for building real-time data pipelines and streaming applications. It’s widely used by companies of all sizes to manage large volumes of data efficiently, offering high throughput, scalability, and fault tolerance. Whether you’re building a logging system, a data processing pipeline, or a stream processing application, Kafka is a reliable choice.
In this guide, we’ll walk you through the steps to install and configure Apache Kafka on Ubuntu 24.04. From setting up the required dependencies like Java, to configuring Kafka in KRaft (self-managed metadata) mode, you’ll learn how to get Kafka up and running on a fresh Ubuntu server. Whether you’re a developer testing Kafka locally or setting it up for production, this tutorial will help you get started with a solid foundation.
Commands
sudo apt install openjdk-21-jdk -y
sudo useradd -r -m -U -d /opt/kafka -s /bin/false kafka
sudo mkdir -p /opt/kafka
sudo chown -R kafka:kafka /opt/kafka
ls -ld /opt/kafka
wget https://downloads.apache.org/kafka/4.0.0/kafka_2.13-4.0.0.tgz
sudo tar -xzf kafka_2.13-4.0.0.tgz -C /opt/kafka –strip-components=1
sudo chown -R kafka:kafka /opt/kafka
sudo ls -l /opt/kafka
sudo nano /opt/kafka/config/server.properties
log.dirs=/opt/kafka/data
sudo /opt/kafka/bin/kafka-storage.sh random-uuid
sudo /opt/kafka/bin/kafka-storage.sh format -t <your-cluster-id> -c /opt/kafka/config/server.properties
sudo nano /etc/systemd/system/kafka.service
===============================================================================
[Unit]
Description=Apache Kafka Server
After=network.target
[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
=============================================================================
sudo mkdir -p /opt/kafka/data
sudo chown -R kafka:kafka /opt/kafka/data
sudo systemctl daemon-reload
sudo systemctl enable kafka
sudo systemctl restart kafka
sudo systemctl status kafka
sudo /opt/kafka/bin/kafka-topics.sh –create –topic test-topic –bootstrap-server localhost:9092 –partitions 1 –replication-factor 1
sudo /opt/kafka/bin/kafka-console-producer.sh –topic test-topic –bootstrap-server localhost:9092
sudo /opt/kafka/bin/kafka-console-consumer.sh –topic test-topic –from-beginning –bootstrap-server localhost:9092
Video:
