Lab
A Cloud Guru

Creating a Topic with Custom Configurations in Kafka

There are many custom configurations that we can apply to topics in Kafka. In this hands-on lab, we'll go through creating a topic, applying a custom configuration to that topic, and then testing the custom configuration by alerting the state of the partitions tied to that topic.

Try for free Contact sales

Path Info

Level

Intermediate

Duration

45m

Published

Jul 17, 2019

Challenge

Set Up the Cluster
Use Docker Compose to build the Kafka Cluster:
```
cd content-kafka-deep-dive

docker-compose up -d --build
```
Now, let's make sure Java is installed (if you receive a dpkg frontend lock message, wait a few minutes and then try again):
```
sudo apt install default-jdk
```
Unzip and change into the Kafka binaries directory:
```
tar -xvf kafka_2.12-2.2.0.tgz && mv kafka_2.12-2.2.0/ kafka
```

Challenge

Create a Topic with Three Partitions and a Replication Factor of `3`

Create a topic named transaction:

bin/kafka-topics.sh --zookeeper localhost:2181 
--create 
--topic transaction 
--replication-factor 3 
--partitions 3

Challenge

Add a Custom Configuration to the Topic

Add the custom configuration min.insync.replicas=3 to the topic transaction:

bin/kafka-configs.sh --zookeeper localhost:2181 
--alter 
--entity-type topics 
--entity-name transaction 
--add-config min.insync.replicas=3

Verify the topic configuration applied:

bin/kafka-configs.sh --zookeeper localhost:2181 
--describe 
--entity-type topics 
--entity-name transaction

Challenge

Change the Replica Count for the Topic

First off, let's create a JSON file named replicacount.json with these contents:

{"partitions":
 [{"topic": "transaction", "partition": 0,
 "replicas": [
 2
 ]
 }
 ],
 "version":1
}

Now, we can execute the replica count change but using that JSON file:

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 
--execute 
--reassignment-json-file replicacount.json

Once we get a "Successfully started..." message, let's describe the topic to see the replica change:

bin/kafka-topics.sh --zookeeper localhost:2181 
--topic transaction 
--describe

Challenge

Run a Producer to Get an Error Message

Open a producer and send some messages to your topic:

bin/kafka-console-producer.sh --broker-list localhost:9092 
--topic transaction 
--producer-property acks=all

Author

A Cloud Guru

The Cloud Content team comprises subject matter experts hyper focused on services offered by the leading cloud vendors (AWS, GCP, and Azure), as well as cloud-related technologies such as Linux and DevOps. The team is thrilled to share their knowledge to help you build modern tech solutions from the ground up, secure and optimize your environments, and so much more!

What's a lab?

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.