Apache Kafka is a free, open-source, distributed event streaming platform that is used by thousands of companies for high-performance data pipelines. It is a message broker software application primarily used to build real-time streaming data pipelines and applications. Apache Kafka allows you to process data streams via a distributed streaming platform. It is based on a distributed architecture, so it provides high fault tolerance and scalability capabilities.
In this post, we will explain how to install Apache Kafka on Oracle Linux 8.
Step 1 – Install Java
Apache Kafka is based on Java, so you will need to install Java on your server. You can install it by running the following command:
dnf install java-11-openjdk-devel -y
Once Java is installed, verify the Java installation using the following command:
java --version
You will get the Java version in the following output:
openjdk 11.0.15 2022-04-19 LTS OpenJDK Runtime Environment 18.9 (build 11.0.15+10-LTS) OpenJDK 64-Bit Server VM 18.9 (build 11.0.15+10-LTS, mixed mode, sharing)
Step 2 – Install Apache Kafka on Oracle Linux 8
First, go to the Apache official website and download the latest version of Apache Kafka using the wget command:
wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz
Once the download is completed, extract the downloaded file using the following command:
tar -xvzf kafka_2.13-3.2.0.tgz
Next, move the extracted directory to /usr/local directory:
mv kafka_2.13-3.2.0/ /usr/local/kafka
Once you are finished, you can proceed to the next step.
Step 3 – Create Systemd Service File for Zookeeper and Kafka
It is a good idea to create a systemd service file to run both Zookeeper and Kafka in the background.
First, create a systemd service file for Zookeeper with the following command:
nano /etc/systemd/system/zookeeper.service
Add the following lines:
[Unit] Description=Apache Zookeeper server Documentation=http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple ExecStart=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties ExecStop=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
Save and close the file, then create a systemd service file for Kafka using the following command:
nano /etc/systemd/system/kafka.service
Add the following lines:
[Unit] Description=Apache Kafka Server Documentation=http://kafka.apache.org/documentation.html Requires=zookeeper.service [Service] Type=simple Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk" ExecStart=/usr/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties ExecStop=/usr/bin/bash /usr/local/kafka/bin/kafka-server-stop.sh [Install] WantedBy=multi-user.target
Save and close the file, then reload the systemd daemon with the following command:
systemctl daemon-reload
Next, start both the Zookeeper and Kafka services and enable them to start at system reboot:
systemctl start zookeeper systemctl start kafka systemctl enable zookeeper systemctl enable kafka
To check the Zookeeper service, run the following command:
systemctl status zookeeper
You will get the following output:
● zookeeper.service - Apache Zookeeper server
Loaded: loaded (/etc/systemd/system/zookeeper.service; disabled; vendor preset: disabled)
Active: active (running) since Sat 2022-06-11 01:06:50 EDT; 28s ago
Docs: http://zookeeper.apache.org
Main PID: 29634 (java)
Tasks: 26 (limit: 11409)
Memory: 67.9M
CGroup: /system.slice/zookeeper.service
└─29634 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCI>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,302] INFO zookeeper.commitLogCount=500 (org.apache.zookeeper.server.ZKDatabase)
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,311] INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeepe>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,312] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,321] INFO Snapshot loaded in 18 ms, highest zxid is 0x0, digest is 1371985504 >
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,321] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,324] INFO Snapshot taken in 2 ms (org.apache.zookeeper.server.ZooKeeperServer)
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,378] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zook>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,388] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apa>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,446] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalM>
Jun 11 01:06:53 oraclelinux8 bash[29634]: [2022-06-11 01:06:53,447] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProv>
lines 1-20/20 (END)
To check the Kafka service, run the following command:
systemctl status kafka
You should see the following output:
● kafka.service - Apache Kafka Server
Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: disabled)
Active: active (running) since Sat 2022-06-11 01:07:17 EDT; 55s ago
Docs: http://kafka.apache.org/documentation.html
Main PID: 29994 (java)
Tasks: 69 (limit: 11409)
Memory: 327.3M
CGroup: /system.slice/kafka.service
└─29994 /usr/lib/jvm/jre-11-openjdk/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupanc>
Jun 11 01:07:23 oraclelinux8 bash[29994]: [2022-06-11 01:07:23,975] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNod>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,001] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Starting socket serv>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,055] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started data-plane a>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,055] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started socket serve>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,157] INFO Kafka version: 3.2.0 (org.apache.kafka.common.utils.AppInfoParser)
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,157] INFO Kafka commitId: 38103ffaa962ef50 (org.apache.kafka.common.utils.AppI>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,157] INFO Kafka startTimeMs: 1654924044055 (org.apache.kafka.common.utils.AppI>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,158] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,377] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Recorde>
Jun 11 01:07:24 oraclelinux8 bash[29994]: [2022-06-11 01:07:24,378] INFO [BrokerToControllerChannelManager broker=0 name=alterPartition]: Rec>
Step 4 – Create Your First Topic on Kafka
Next, you will need to create at least one topic on the server to test Apache Kafka.
Change the directory to Apache Kafka and create a test topic named app1 with the following command:
cd /usr/local/kafka/ bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic app1
You can now verify your created topic using the following command:
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
You will get the following output:
app1
Kafka provides two APIs: producer and consumer. The producer is responsible for creating events and the consumer uses them to display it on the screen:
Now, run the producer and then type a few messages into the console to send to the server.
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic app1
Type some text that you want to stream and display on the consumer.
>Hi, this is my first kafka server
Open another terminal and run the following command to display the generated event data in real-time:
cd /usr/local/kafka/ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic app1 --from-beginning
You will get the following output:
Hi, this is my first kafka server
Conclusion
In the above guide, we explained how to install Apache Kafka on Oracle Linux 8. You can now integrate Apache Kafka with your application to collect and analyze a large amount of data. For more information, you can visit the Apache Kafka documentation page. Try it on dedicated hosting from Atlantic.Net!
