Creating a Topic with Custom Configurations in Kafka

45 minutes
  • 5 Learning Objectives

About this Hands-on Lab

There are many custom configurations that we can apply to topics in Kafka. In this hands-on lab, we’ll go through creating a topic, applying a custom configuration to that topic, and then testing the custom configuration by alerting the state of the partitions tied to that topic.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Set Up the Cluster

Use Docker Compose to build the Kafka Cluster:

cd content-kafka-deep-dive

docker-compose up -d --build

Install Java:

sudo apt install default-jdk

Unzip and change into the Kafka binaries directory:

tar -xvf kafka_2.12-2.2.0.tgz && mv kafka_2.12-2.2.0/ kafka
Create a Topic with Three Partitions and a Replication Factor of `3`

Create a topic named transaction:

bin/kafka-topics.sh --zookeeper localhost:2181 
--create 
--topic transaction 
--replication-factor 3 
--partitions 3
Add a Custom Configuration to the Topic

Add the custom configuration min.insync.replicas=3 to the topic transaction:

bin/kafka-configs.sh --zookeeper localhost:2181 
--alter 
--entity-type topics 
--entity-name transaction 
--add-config min.insync.replicas=3

Verify the topic configuration applied:

bin/kafka-configs.sh --zookeeper localhost:2181 
--describe 
--entity-type topics 
--entity-name transaction
Change the Replica Count for the Topic

First off, let’s create a json file named replicacount.json with these contents:

{"partitions":
 [{"topic": "transaction", "partition": 0,
 "replicas": [
 2
 ]
 }
 ],
 "version":1
}

Now we can execute the replica count change but using that json file:

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 
--execute 
--reassignment-json-file replicacount.json

Once we get a "Successfully started…" message, let’s describe the topic to see the replica change:

bin/kafka-topics.sh --zookeeper localhost:2181 
--topic transaction 
--describe
Run a Producer to get an error message

Open a producer and send some messages to your topic

bin/kafka-console-producer.sh --broker-list localhost:9092 
--topic transaction 
--producer-property acks=all

Additional Resources

Broker failure is a common scenario in Kafka Cluster Administration, and it causes data loss. If there aren't enough in-sync replicas, messages may be lost forever. To ensure that this never happens, we've been tasked with enforcing a multiple-replica policy, and ensuring that creating topics without this policy will never be allowed by anyone else in the organization. Our instructions for this hands-on lab are:

  • Create a Topic with a replication factor of 3, and 3 partitions.
  • Add a custom configuration to the topic that ensures an error will occur if the replication factor falls below 3.
  • Test that the policy is successfully applied, that it changes the replica count to 1, and opens a producer which will send messages to the Kafka cluster.

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Get Started
Who’s going to be learning?

How many seats do you need?

  • $499 USD per seat per year
  • Billed Annually
  • Renews in 12 months

Ready to accelerate learning?

For over 25 licenses, a member of our sales team will walk you through a custom tailored solution for your business.


$2,495.00

Checkout
Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!