Practitioner Level

Creating a Topic with Custom Configurations in Kafka

Pricing
45 minutes
  • 5 Learning Objectives

About this Hands-on Lab

There are many custom configurations that we can apply to topics in Kafka. In this hands-on lab, we’ll go through creating a topic, applying a custom configuration to that topic, and then testing the custom configuration by alerting the state of the partitions tied to that topic.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Set Up the Cluster

Use Docker Compose to build the Kafka Cluster:

cd content-kafka-deep-dive

docker-compose up -d --build

Now, let’s make sure Java is installed (if you receive a dpkg frontend lock message, wait a few minutes and then try again):

sudo apt install default-jdk

Unzip and change into the Kafka binaries directory:

tar -xvf kafka_2.12-2.2.0.tgz && mv kafka_2.12-2.2.0/ kafka
Create a Topic with Three Partitions and a Replication Factor of `3`

Create a topic named transaction:

bin/kafka-topics.sh --zookeeper localhost:2181 
--create 
--topic transaction 
--replication-factor 3 
--partitions 3
Add a Custom Configuration to the Topic

Add the custom configuration min.insync.replicas=3 to the topic transaction:

bin/kafka-configs.sh --zookeeper localhost:2181 
--alter 
--entity-type topics 
--entity-name transaction 
--add-config min.insync.replicas=3

Verify the topic configuration applied:

bin/kafka-configs.sh --zookeeper localhost:2181 
--describe 
--entity-type topics 
--entity-name transaction
Change the Replica Count for the Topic

First off, let’s create a JSON file named replicacount.json with these contents:

{"partitions":
 [{"topic": "transaction", "partition": 0,
 "replicas": [
 2
 ]
 }
 ],
 "version":1
}

Now, we can execute the replica count change but using that JSON file:

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 
--execute 
--reassignment-json-file replicacount.json

Once we get a "Successfully started…" message, let’s describe the topic to see the replica change:

bin/kafka-topics.sh --zookeeper localhost:2181 
--topic transaction 
--describe
Run a Producer to Get an Error Message

Open a producer and send some messages to your topic:

bin/kafka-console-producer.sh --broker-list localhost:9092 
--topic transaction 
--producer-property acks=all
Contact SalesSee Pricing

Additional Resources

Broker failure is a common scenario in Kafka cluster administration, and it causes data loss. If there aren't enough in-sync replicas, messages may be lost forever. To ensure that this never happens, we've been tasked with enforcing a multiple-replica policy and ensuring that creating topics without this policy will never be allowed by anyone else in the organization. Our instructions for this hands-on lab are:

  • Create a topic with a replication factor of 3 and three partitions.
  • Add a custom configuration to the topic that ensures an error will occur if the replication factor falls below 3.
  • Test that the policy is successfully applied and that it changes the replica count to 1 and opens a producer, which will send messages to the Kafka cluster.

Note: If you receive a dpkg frontend lock message when making sure the default-jdk is installed, wait a few minutes. Then try again.

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!

Don't have an account?

Get Started
Who’s going to be learning?
MyselfMy Organization