Apache Kafka is an open-source distributed event streaming platform developed by the Apache Software Foundation. It is written in Java and Scala and used for high-performance data pipelines and streaming analytics. Apache Kafka is a distributed message broker designed to handle large volumes of data. It is highly scalable and can run on one or more servers.
This tutorial will show you how to install Apache Kafka on Fedora.
Step 1 – Install Java OpenJDK
First, install the Java JDK using the following commands.
dnf update -y
dnf install java-11-openjdk -y
After installing Java, verify the Java installation with the following command.
java --version
Output.
openjdk 11.0.15 2022-04-19 OpenJDK Runtime Environment 18.9 (build 11.0.15+10) OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10, mixed mode, sharing)
Step 2 – Install Apache Kafka
First, visit the Kafka official download page, pick the latest Kafka version, and run the following command to download it to your server.
wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz
Next, extract the downloaded file using the following command.
tar xzf kafka_2.13-3.4.0.tgz
Next, move the extracted directory to /usr/local directory.
mv kafka_2.13-3.4.0 /usr/local/kafka
Step 3 – Create a Systemd Service File for Kafka
Next, you must create a systemd service file to manage the Kafka service via systemd.
First, create a Zookeeper service file.
nano /etc/systemd/system/zookeeper.service
Add the following configurations.
[Unit] Description=Apache Zookeeper server Documentation=http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple ExecStart=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties ExecStop=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
Save the file, then create a Kafka service file.
nano /etc/systemd/system/kafka.service
Add the following configurations.
[Unit] Description=Apache Kafka Server Documentation=http://kafka.apache.org/documentation.html Requires=zookeeper.service [Service] Type=simple Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk" ExecStart=/usr/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties ExecStop=/usr/bin/bash /usr/local/kafka/bin/kafka-server-stop.sh [Install] WantedBy=multi-user.target
Save and close the file, then reload the systemd daemon to apply the changes.
systemctl daemon-reload
Step 4 – Start Apache Kafka
Now, start the Apache Kafka and Zookeeper services and enable them to start at system reboot.
systemctl start zookeeper systemctl start kafka systemctl enable zookeeper systemctl enable kafka
You can now check the status of Kafka using the following command.
systemctl status kafka
Output.
● kafka.service - Apache Kafka Server
Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: disabled)
Active: active (running) since Sat 2023-04-22 23:21:35 EDT; 4s ago
Docs: http://kafka.apache.org/documentation.html
Main PID: 4830 (java)
Tasks: 54 (limit: 4666)
Memory: 306.7M
CPU: 6.974s
CGroup: /system.slice/kafka.service
└─4830 /usr/lib/jvm/jre-11-openjdk/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+Explicit>
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,176] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReap>
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,196] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReap>
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,200] INFO Successfully created /controller_epoch with initial epoch 0 (kafka.zk.KafkaZkClient)
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,223] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,244] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,245] INFO Feature ZK node created at path: /feature (kafka.server.FinalizedFeatureChangeListener)
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,300] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,309] INFO [MetadataCache brokerId=0] Updated cache from existing to latest FinalizedFeaturesAndEpoch(fe>
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,327] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
Apr 22 23:21:40 fedora bash[4830]: [2023-04-22 23:21:40,334] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChan>
Step 5 – Create a Topic in Kafka
At this point, Kafka is installed on your server. To test Kafka, create a topic named FedoraTopic.
cd /usr/local/kafka bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic FedoraTopic
Next, verify your created topic using the following command.
./bin/kafka-topics.sh --bootstrap-server=localhost:9092 --list
Output:
FedoraTopic
Now, execute the producer and type a few messages into the console to send to the server.
./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic FedoraTopic >Hi >How are you
Next, open another terminal and run the consumer to read data from the Kafka cluster and display messages to standard output.
./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic FedoraTopic --from-beginning Hi How are you
Conclusion
In this post, we explained how to install Apache Kafka on Fedora. We also create a sample topic and show you how producer and consume work. You can now install Kafka on dedicated server hosting from Atlantic.Net!
