Apache Kafka is an open-source stream processing and message broker software application that allows you to process data streams via a distributed streaming platform. It acts as a messaging system between the sender and the recipient. Apache Kafka is based on a distributed architecture, so it provides high fault tolerance and scalability capabilities. It was originally developed by LinkedIn, but now it is a project of Apache Software Foundation. Apache Kafka provides an interface to read and write data to Kafka clusters or to import and export data to and from third-party systems.
In this post, we will explain how to install Apache Kafka on Rocky Linux 8.
Step 1- Install Java
Apache Kafka is a Java-based application, so Java must be installed on your server. If not installed, you can install it using the following command:
dnf update -y dnf install java-11-openjdk-devel -y
Once Java is installed, verify the Java installation using the following command:
java --version
You will get the Java version in the following output:
openjdk 11.0.13 2021-10-19 LTS OpenJDK Runtime Environment 18.9 (build 11.0.13+8-LTS) OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8-LTS, mixed mode, sharing)
Step 2 – Install Apache Kafka on Rocky Linux 8
First, go to the Apache official website and download the latest version of Apache Kafka using the wget command:
wget https://dlcdn.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
Once the download is completed, extract the downloaded file using the following command:
tar -xvzf kafka_2.13-3.0.0.tgz
Once the downloaded file is extracted, move the extracted directory to /usr/local directory:
mv kafka_2.13-3.0.0/ /usr/local/kafka
Once you are finished, you can proceed to the next step.
Step 3 – Create Systemd Service File for Zookeeper and Kafka
For the production environment, it is recommended to create a systemd service file to run both Zookeeper and Kafka in the background.
First, create a systemd service file for Zookeeper with the following command:
nano /etc/systemd/system/zookeeper.service
Add the following lines:
[Unit] Description=Apache Zookeeper server Documentation=http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple ExecStart=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties ExecStop=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
Save and close the file, then create a systemd service file for Kafka using the following command:
nano /etc/systemd/system/kafka.service
Add the following lines:
[Unit] Description=Apache Kafka Server Documentation=http://kafka.apache.org/documentation.html Requires=zookeeper.service [Service] Type=simple Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk" ExecStart=/usr/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties ExecStop=/usr/bin/bash /usr/local/kafka/bin/kafka-server-stop.sh [Install] WantedBy=multi-user.target
Save and close the file, then reload the systemd daemon with the following command:
systemctl daemon-reload
Next, start both Zookeeper and Kafka services and enable them to start at system reboot:
systemctl start zookeeper systemctl start kafka systemctl enable zookeeper systemctl enable kafka
You can also check both services using the following command:
systemctl status zookeeper kafka
You will get the following output:
● zookeeper.service - Apache Zookeeper server
Loaded: loaded (/etc/systemd/system/zookeeper.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2021-10-22 07:51:56 UTC; 42s ago
Docs: http://zookeeper.apache.org
Main PID: 33802 (java)
Tasks: 28 (limit: 11411)
Memory: 91.4M
CGroup: /system.slice/zookeeper.service
└─33802 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCIn>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,676] INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeeper.>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,676] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.z>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,679] INFO Snapshot loaded in 22 ms, highest zxid is 0x0, digest is 1371985504 (o>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,679] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.z>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,683] INFO Snapshot taken in 3 ms (org.apache.zookeeper.server.ZooKeeperServer)
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,703] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookee>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,741] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apach>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,764] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=>
Oct 22 07:51:58 RockyLinux8 bash[33802]: [2021-10-22 07:51:58,765] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvid>
Oct 22 07:52:12 RockyLinux8 bash[33802]: [2021-10-22 07:52:12,102] INFO Creating new log file: log.1 (org.apache.zookeeper.server.persistence.>
● kafka.service - Apache Kafka Server
Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2021-10-22 07:52:09 UTC; 29s ago
Docs: http://kafka.apache.org/documentation.html
Main PID: 34147 (java)
Tasks: 69 (limit: 11411)
Memory: 331.6M
CGroup: /system.slice/kafka.service
└─34147 /usr/lib/jvm/jre-11-openjdk/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancy>
Oct 22 07:52:14 RockyLinux8 bash[34147]: [2021-10-22 07:52:14,881] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeC>
Oct 22 07:52:14 RockyLinux8 bash[34147]: [2021-10-22 07:52:14,899] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Starting socket server>
Oct 22 07:52:14 RockyLinux8 bash[34147]: [2021-10-22 07:52:14,981] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started data-plane acc>
Oct 22 07:52:14 RockyLinux8 bash[34147]: [2021-10-22 07:52:14,988] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started socket server >
Oct 22 07:52:15 RockyLinux8 bash[34147]: [2021-10-22 07:52:15,069] INFO Kafka version: 3.0.0 (org.apache.kafka.common.utils.AppInfoParser)
Oct 22 07:52:15 RockyLinux8 bash[34147]: [2021-10-22 07:52:15,069] INFO Kafka commitId: 8cb0a5e9d3441962 (org.apache.kafka.common.utils.AppInf>
Oct 22 07:52:15 RockyLinux8 bash[34147]: [2021-10-22 07:52:15,069] INFO Kafka startTimeMs: 1634889134988 (org.apache.kafka.
Step 4 – Create a Topic on Kafka
To test Apache Kafka, you will need to create at least one topic on the server.
Change the directory to Apache Kafka and create a test topic named topic1 with the following command:
cd /usr/local/kafka/ bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic topic1
You can now verify your created topic using the following command:
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
You will get the following output:
topic1
Kafka provides two APIs: Producer and Consumer. The Producer is responsible for creating events and the Consumer displays them on the screen:
First, run the following command to create an event named event1 using the following command:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic event1
Type some text that you want to stream and display on the Consumer.
>Hi, this is my first event
Sample output:
[2021-10-22 07:58:05,318] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {event1=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
Open another terminal and run the following command to display the generated event data in real-time:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic event1 --from-beginning
You will get the following output:
Hi, this is my first event
Conclusion
In the above guide, you learned how to install Apache Kafka on Rocky Linux 8. For more information, you can visit the Apache Kafka documentation page. Get started with Apache Kafka on VPS hosting from Atlantic.Net!
