Practitioner Level

Installing a Kafka Cluster and Creating a Topic

What's a Hands-on Lab?Get Started Pricing Pricing

1 hour

7 Learning Objectives

About this Hands-on Lab

Apache Kafka is known for many things, one being the ability to process over 1 million messages per second. The ease of decoupling components makes it appealing to developers as well.

In this hands-on lab, you will build a custom Kafka cluster and create a topic. A topic consists of many partitions of messages and is the fundamental unit used in Kafka.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Download the Kafka Binaries

Use wget to download the tar file from the mirror:

wget http://mirror.cogentco.com/pub/apache/kafka/2.6.1/kafka_2.12-2.6.1.tgz

Extract the tar file and move it into the /opt directory:

tar -xvf kafka_2.12-2.6.1.tgz

sudo mv kafka_2.12-2.6.1 /opt/kafka

Change to the kafka directory and list the contents:
```
cd /opt/kafka
```
```
ls
```

Install Java and Disable RAM Swap

Use the following command to install the latest Java Developer Kit (JDK):
```
sudo apt install -y default-jdk
```
Use the following command to verify that Java has been installed:
```
java -version
```
Use the following command to disable RAM swap:
```
swapoff -a
```
Use the following command to comment out swap in the /etc/fstab file:
```
sudo sed -i '/ swap / s/^/#/' /etc/fstab
```

Create a New Directory for Kafka and Zookeeper

Use the following command to create a new directory for Kafka message logs:
```
sudo mkdir -p /data/kafka
```
Use the following command to create a snapshot directory for Zookeeper:
```
sudo mkdir -p /data/zookeeper
```

Change ownership of those directories to allow the user cloud_user control:

sudo chown -R cloud_user:cloud_user /data/kafka

sudo chown -R cloud_user:cloud_user /data/zookeeper

Specify an ID for Each Zookeeper Server

Use the following command to create a file in /data/zookeeper (on server #1) called myid with the contents "1" to specify Zookeeper server #1:
```
echo "1" > /data/zookeeper/myid
```
Use the following command to create a file in /data/zookeeper (on server #2) called myid with the contents "2" to specify Zookeeper server #2:
```
echo "2" > /data/zookeeper/myid
```
Use the following command to create a file in /data/zookeeper (on server #3) called myid with the contents "3" to specify Zookeeper server #3:
```
echo "3" > /data/zookeeper/myid
```

Modify the Kafka and Zookeeper Configuration Files

Use the following command to remove the existing server.properties file (in the config directory) and create a new server.properties file:
```
rm config/server.properties
```
```
vim config/server.properties
```

Copy and paste the following into the contents of the server.properties file and change the broker.id and the advertised.listeners:

# change this for each broker
broker.id=[broker_number]
# change this to the hostname of each broker
advertised.listeners=PLAINTEXT://[hostname]:9092
# The ability to delete topics
delete.topic.enable=true
# Where logs are stored
log.dirs=/data/kafka
# default number of partitions
num.partitions=8
# default replica count based on the number of brokers
default.replication.factor=3
# to protect yourself against broker failure
min.insync.replicas=2
# logs will be deleted after how many hours
log.retention.hours=168
# size of the log files 
log.segment.bytes=1073741824
# check to see if any data needs to be deleted
log.retention.check.interval.ms=300000
# location of all zookeeper instances and kafka directory
zookeeper.connect=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/kafka
# timeout for connecting with zookeeper
zookeeper.connection.timeout.ms=6000
# automatically create topics
auto.create.topics.enable=true

Use the following command to remove the existing zookeeper.properties file (in the config directory) and create a new zookeeper.properties file:
```
rm config/zookeeper.properties
```
```
vim config/zookeeper.properties
```

Copy and paste the following into the contents of the zookeeper.properties file (don’t change this file):

# the directory where the snapshot is stored.
dataDir=/data/zookeeper
# the port at which the clients will connect
clientPort=2181
# setting number of connections to unlimited
maxClientCnxns=0
# keeps a heartbeat of zookeeper in milliseconds
tickTime=2000
# time for initial synchronization
initLimit=10
# how many ticks can pass before timeout
syncLimit=5
# define servers ip and internal ports to zookeeper
server.1=zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888

Create the Kafka and Zookeeper Service

Create the file /etc/init.d/zookeeper on each server and paste in the following contents:

sudo vim /etc/init.d/zookeeper

#!/bin/bash
#/etc/init.d/zookeeper
DAEMON_PATH=/opt/kafka/bin
DAEMON_NAME=zookeeper
# Check that networking is up.
#[ ${NETWORKING} = "no" ] && exit 0

PATH=$PATH:$DAEMON_PATH

case "$1" in
 start)
       # Start daemon.
       pid=`ps ax | grep -i 'org.apache.zookeeper' | grep -v grep | awk '{print $1}'`
       if [ -n "$pid" ]
         then
           echo "Zookeeper is already running";
       else
         echo "Starting $DAEMON_NAME";
         $DAEMON_PATH/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties
       fi
       ;;
 stop)
       echo "Shutting down $DAEMON_NAME";
       $DAEMON_PATH/zookeeper-server-stop.sh
       ;;
 restart)
       $0 stop
       sleep 2
       $0 start
       ;;
 status)
       pid=`ps ax | grep -i 'org.apache.zookeeper' | grep -v grep | awk '{print $1}'`
       if [ -n "$pid" ]
         then
         echo "Zookeeper is Running as PID: $pid"
       else
         echo "Zookeeper is not Running"
       fi
       ;;
 *)
       echo "Usage: $0 {start|stop|restart|status}"
       exit 1
esac

exit 0

Change the file to executable, change ownership, install, and start the service:

sudo chmod +x /etc/init.d/zookeeper

sudo chown root:root /etc/init.d/zookeeper

sudo update-rc.d zookeeper defaults

sudo service zookeeper start

sudo service zookeeper status

Create the file /etc/init.d/kafka on each server and paste in the following contents:

#!/bin/bash
#/etc/init.d/kafka
DAEMON_PATH=/opt/kafka/bin
DAEMON_NAME=kafka
# Check that networking is up.
#[ ${NETWORKING} = "no" ] && exit 0

PATH=$PATH:$DAEMON_PATH

# See how we were called.
case "$1" in
 start)
       # Start daemon.
       pid=`ps ax | grep -i 'kafka.Kafka' | grep -v grep | awk '{print $1}'`
       if [ -n "$pid" ]
         then
           echo "Kafka is already running"
       else
         echo "Starting $DAEMON_NAME"
         $DAEMON_PATH/kafka-server-start.sh -daemon /opt/kafka/config/server.properties
       fi
       ;;
 stop)
       echo "Shutting down $DAEMON_NAME"
       $DAEMON_PATH/kafka-server-stop.sh
       ;;
 restart)
       $0 stop
       sleep 2
       $0 start
       ;;
 status)
       pid=`ps ax | grep -i 'kafka.Kafka' | grep -v grep | awk '{print $1}'`
       if [ -n "$pid" ]
         then
         echo "Kafka is Running as PID: $pid"
       else
         echo "Kafka is not Running"
       fi
       ;;
 *)
       echo "Usage: $0 {start|stop|restart|status}"
       exit 1
esac

exit 0

Save and close the file.

Change the file to executable, change ownership, install, and start the service:

sudo chmod +x /etc/init.d/kafka

sudo chown root:root /etc/init.d/kafka

sudo update-rc.d kafka defaults

sudo service kafka start

sudo service kafka status

Create a Topic

Use the following command to create a topic named test:

./bin/kafka-topics.sh --zookeeper zookeeper1:2181/kafka --create --topic test --replication-factor 1 --partitions 3

Use the following command to describe the topic:

./bin/kafka-topics.sh --zookeeper zookeeper1:2181/kafka --topic test --describe

Contact Sales Get Started See Pricing See Pricing

Additional Resources

The company you work for would like to start using Apache Kafka for real-time analytics of their application data. They realize the benefit to Kafka, in that it can process millions of messages in real-time. The company would like to start using Kafka as soon as possible and have tasked you with building the Kafka cluster. You will be responsible for building a Kafka cluster with three brokers, and three Zookeeper servers. Your company only gave you three servers total, and would like the cluster configured for high-availability and fault-tolerance. Your boss says that if one of the servers fail, we must ensure that the cluster still remains up and running. That means you must include at least two replicas of the data. Here are the steps you must take to build the Kafka cluster:

Download the latest Kafka binaries
Install Java
Disable RAM swap
Create a directory with appropriate permissions for Kafka logs
Create a directory with appropriate permissions for Zookeeper snapshots
Specify an ID for the Zookeeper servers to reach quorum
Modify the Kafka configuration file to reflect the appropriate settings
Modify the Zookeeper configuration file to reflect the appropriate settings
Create Zookeeper as a service, so it can run without manual intervention
Create Kafka as a service, so it can run without manual intervention
Create a topic to test that the cluster is working properly

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Plans & Pricing

Choosing is the first step

Whether you’re a small team, a global enterprise, or somewhere in between, we offer multiple plans so you can choose the one that best fits your needs.

Business Basic

^$ 33.27 / user monthly

Billed as $499 $399.20 / user yearly

Hands-on learning for smaller teams
Unlimited courses, quizzes, and practice exams
Full learn-by-doing experience with Hands-on Labs and Cloud Playground
User and skills analytics

Business Plus

^$ 46.60

Billed as $699 $559.20 / user yearly

Scalability for larger teams and entire organizations
Everything in Business Basic, plus:
Evaluate team readiness with Skills Assessment
Manage teams, single sign-on (SSO), and reporting
Fast-track learning with Study Groups and Accelerator Programs
Dedicated Customer Success Manager

Offer applies only to: new personal and business paying customers, existing personal and business customers who switch to an annual paid plan, and any additional annual plan licenses purchased by existing business customers. Offer valid for a limited time only, beginning 10:00AM CT on February 19, 2021. Offer may not be combined with other offers. Purchases are subject to applicable Terms of Use or Master Services Agreement.

Business Basic

$499 / year

Business Plus

$699 / year

Unlimited Courses

Learning without limits

Teams learn from anywhere with unlimited access to certification courses and deep dives on AWS, Azure, GCP, Linux, and so much more.

Unlimited Hands-on Labs

Learn faster by doing

Teams learn by doing with guided labs based on real-world scenarios in a secure, risk-free environment. Build new skills in as little as 15 minutes.

Learning Paths

Take your teams from novice to guru

Not sure how to proceed? We’ll guide you through the exact skills your team needs to progress from novice to guru across a variety of cloud specialties, including Architect, Security, and DevOps.

Cloud Playground - Servers

Cloud on demand

Your learners can spin up pre-configured, auto-provisioned servers in just a few clicks. Provide safe practice environments and never worry about a surprise cloud bill.

Cloud Playground - Instant Terminal

SSH on demand

Quickly and safely launch a secure, in-browser SSH terminal into any instance on any provider – even behind a firewall.

Cloud Playground - Sandboxes

Learn by doing

Teams learn faster by doing with Cloud Playground. Try out new cloud skills in live AWS, Azure, and GCP sandbox environments — without racking up a surprise bill. Cloud along with courses, test ideas, and prepare for exams.

ACG Original Series

Keep up with the cloud

Stay up-to-date on all things cloud with weekly and monthly videos about the latest developments in AWS, Azure, GCP, and Kubernetes.

Practice Exams

Put skills to the test

Learners prepare for certs with practice exams that mimic the real thing. Plus, they get personalized pointers on how to improve their scores.

Quizzes

Pop quiz, hot shot

Quickly check knowledge on a variety of topics and know where to brush up with short quizzes.

Community Forums

Cloud with friends

Learners get the help they need, when they need it, from industry experts and others in the community.

ACG Mobile Apps

Keep ACG in your pocket

Teams learn cloud from anywhere with our mobile apps for iOS and Android.

Mobile Offline Mode

Learn off the grid

No internet connection? No problem. Download courses for offline access with our app for iOS and Android.

Records of Completion

Achievement unlocked

Records of completion make it easy for your learners to mark and showcase progress.

Video Transcripts & Captions

For the record

Learners read along when sound isn’t an option and skip furiously scribbling notes. Our transcripts and captions are also fully searchable, making it easy to find exactly what they’re looking for.

Interactive Diagrams

Study aids on demand

Snag clickable, PDF versions of core concepts to revisit later. Our diagrams make great study aids, help build retention, and can act as handy cheat sheets for a quick refresher down the road.

Email Support

One dashboard to rule them all

Easily assign new users, track learning and ROI with usage and skill analytics, and manage the entire ACG for Business experience from one central location.

Unlimited Admin Seats

Manage cloud learning for teams

Scale freely and keep your ACG for Business licenses productive with unlimited free Administrators. Yes, you pay only for learners.

Skills Assessment

Know where you stand

Take stock of your organization’s cloud readiness so you can know exactly where to start.

Learning Accelerator

Get to the cloud faster

Fast-track your organization’s path to cloud fluency with sprint-based certification tracks, including weekly lessons, hands-on projects and whitepapers.

Study Groups

Learn better together

Group learners together so they can support each other and build skills faster. Discussing topics and helping others understand concepts boosts retention and builds camaraderie.

Certification Reporting

Skills at a glance

Keep tabs on skills and certifications across your organization to properly staff projects, track progress, and inform learning goals.

Team Management

Easy organization by teams

Create and assign learners to teams for easier management, tracking, and reporting. Designate team coordinators to help keep teams on track, and know who’s conquering the cloud or needs a nudge with the team leaderboard.

Single Sign-On

Invite 10 to 10,000 or more

Use your company’s ID and password to make management and scalability a breeze.

LMS Integration

Connect all the dots

Integrate results and reporting into your existing learning management systems for a 360-degree view of employee skills development.LMS Integration currently available for EdCast and Degreed.

API Access

Get your bits

Automatically transfer raw usage and skills data to your data warehouse or other repository.

Dedicated Customer Success Manager *

We’ve got your back

Get the most out of ACG for Business with a dedicated cloud learning consultant to support you with onboarding, best practices, and more.

Personalized Onboarding *

We’ll show you the ropes

Get up and running the right way. Let one of our learning gurus take you through an onboarding experience tailored to your needs.

Contact Sales

* Available with 50+ seats

Plans & Pricing

Choosing is the first step

Whether you’re an individual learner, a global enterprise, or somewhere in between, we offer several plans so you can choose the one that best fits your needs.

Free

^$ 0 / month

Limited access to basic courses and features
Rotating selection of limited courses
Stay ahead with weekly cloud roundups
Engage with a vibrant community of learners and experts

Save 20%

Personal Basic

^$ 39 ^$ 25.57 / month

Billed as $379 $303.20 / user yearly

Pay Yearly Pay Monthly

Save 35% with annual plan!

Full access to our hands-on learning library
Unlimited courses, quizzes, & practice exams
Unlimited Hands-on Labs
Practice in real multi-platform servers

Save 20%

Personal Plus

^$ 33.27 / month

Billed as $499 $399.20 / user yearly

Everything in Personal Basic, plus:
Full learn-by-doing experience in real AWS, Azure, & GCP environments!
Unlimited access to full Cloud Sandboxes
No account setup or cloud bills

Offer applies only to: new personal and business paying customers, existing personal and business customers who switch to an annual paid plan, and any additional annual plan licenses purchased by existing business customers. Offer valid for a limited time only, beginning 10:00AM CT on February 19, 2021. Offer may not be combined with other offers. Purchases are subject to applicable Terms of Use or Master Services Agreement.

Free

Personal Basic

$379 $303.20 / year

Personal Plus

$499 $399.20 / year

Unlimited Courses

limited

Learn without limits

Learn from anywhere with unlimited access to certification courses and deep dives on AWS, Azure, GCP, Linux, and so much more.

Quizzes

limited

Pop quiz, hot shot

Quickly check your knowledge on a variety of topics and know where to brush up with short quizzes.

ACG Original Series

Keep up with the cloud

Stay up-to-date on all things cloud with weekly and monthly videos about the latest developments in AWS, Azure, GCP, and Kubernetes.

Community Forums

Cloud with friends

Get the help you need, when you need it, from industry experts and other learners in the community — and help others in turn.

Learning Paths

Go from novice to guru in your chose specialty

Not sure how to proceed? We’ll guide you through the exact skills you need to gain to progress from novice to guru across a variety of cloud specialties, including Architect, Security, and DevOps.

ACG Mobile Apps

Keep ACG in your pocket

Learn cloud from anywhere with our mobile apps for iOS and Android.

Mobile Offline Mode

Learn off the grid

No internet connection? No problem. Download courses for offline access with our app for iOS and Android.

Unlimited Hands-on Labs

Get your hands cloudy

Learn by doing with guided labs based on real-world scenarios in a secure, risk-free environment. Build new skills in as little as 15 minutes.

Practice Exams

Put your skills to the test

Prepare for your certs with practice exams that mimic the real thing. Plus, get personalized pointers on how to improve your score.

Interactive Diagrams

Study aids on demand

Snag clickable, PDF versions of core concepts to revisit later. Our diagrams make great study aids, help build retention, and can act as handy cheat sheets for a quick refresher down the road.

Video Transcripts & Captions

For the record

Read along when sound isn’t an option and skip furiously scribbling notes. Our transcripts and captions are also fully searchable, making it easy to find exactly what you’re looking for.

Records of Completion

Achievement unlocked

Earn a little something when you complete courses. Records of completion make it easy to mark and showcase your progress.

Email Support

Cloud on demand

Spin up pre-configured, auto-provisioned servers in just a few clicks. Cloud along with courses, practice whenever you feel like it, and never worry about a surprise cloud bill.

Cloud Playground - Instant Terminal

SSH on demand

Quickly launch a secure, in-browser SSH terminal into any instance on any provider – even behind a firewall.

Cloud Playground - Sandboxes

Personal Plus Only

Learn by doing

Learn by doing with Cloud Playground. Try out new cloud skills in live AWS, Azure, and GCP sandbox environments — without racking up a surprise bill. Cloud along with courses, test an idea at work or prepare for exams.

Free
$0

Personal Basic
$379 / year

Personal Plus
$499 / year

Start Free

Free Trial

Browse Learning

Browse Platforms

Hands-on Labs

Installing a Kafka Cluster and Creating a Topic

About this Hands-on Lab

Learning Objectives

Additional Resources

Plans & Pricing

Choosing is the first step

Business Basic

Business Plus

Learning without limits

Learn faster by doing

Take your teams from novice to guru

Cloud on demand

SSH on demand

Learn by doing

Keep up with the cloud

Put skills to the test

Pop quiz, hot shot

Cloud with friends

Keep ACG in your pocket

Learn off the grid

Achievement unlocked

For the record

Study aids on demand

We’re here to help

One dashboard to rule them all

Manage cloud learning for teams

Know where you stand

Get to the cloud faster

Learn better together

Skills at a glance

Easy organization by teams

Invite 10 to 10,000 or more

Connect all the dots

Get your bits

We’ve got your back

We’ll show you the ropes

Plans & Pricing

Choosing is the first step

Free

Personal Basic

Personal Plus

Learn without limits

Pop quiz, hot shot

Keep up with the cloud

Cloud with friends

Go from novice to guru in your chose specialty

Keep ACG in your pocket

Learn off the grid

Get your hands cloudy

Put your skills to the test

Study aids on demand

For the record

Achievement unlocked

We’re here to help

Cloud on demand

SSH on demand

Learn by doing

How many seats do you need?

Ready to accelerate learning?

$2,495.00