Consuming Kafka Messages with Multiple Consumer Groups

30 minutes
  • 2 Learning Objectives

About this Hands-on Lab

In this hands-on lab, we will have the opportunity to consume data from a Kafka topic using multiple consumers groups. One of these groups will have a single consumer, and the other will have two consumers. Going through this will provide a firsthand look at how different consumers and consumer groups handle messages in Kafka. Additionally, Kafka’s persistent data structure for storing messages offers a great deal of flexibility in terms of how we want to consume messages. We can use consumer groups to perform different operations against the same set of messages while also being able to scale each consumer application horizontally.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Set Up the First Consumer Group with One Consumer
  1. Set up the first consumer as the sole consumer in its group. Consume some messages and save the output to /home/cloud_user/output/group1_consumer1.txt.

    The format should look like this:

    kafka-console-consumer --bootstrap-server localhost:9092 --topic inventory_purchases --group 1 > /home/cloud_user/output/group1_consumer1.txt
Set Up a Second Consumer Group with Two Consumers
  1. Create a consumer in a separate group, and store its output in /home/cloud_user/output/group1_consumer1.txt:

    kafka-console-consumer --bootstrap-server localhost:9092 --topic inventory_purchases --group 2 > /home/cloud_user/output/group2_consumer1.txt
  2. Create a second consumer in the same group, and store its output in /home/cloud_user/output/group2_consumer2.txt:

    kafka-console-consumer --bootstrap-server localhost:9092 --topic inventory_purchases --group 2 > /home/cloud_user/output/group2_consumer2.txt

Additional Resources

Your supermarket company is getting ready to use Kafka to process purchase data to track changes in inventory. When a customer purchases an item, data about the name of the item and quantity purchased will be published to a topic called inventory_purchases in the Kafka cluster.

The company is working on determining the best way to consume this data, and they want you to perform a proof-of-concept for the proposed consumer setup. Your task is to set up some consumers following a specified configuration, and then examine the message data processed by these consumers. Lastly, you will store some sample data that demonstrates what messages get processed by each consumer.

The system currently is set up so that a stream of sample data gets continuously produced to the topic.

Consume the data according to the following specifications:

  1. Consume the data from the inventory_purchases topic.
  2. Set up a consumer and wait for it to process some data.
  3. Store the output in /home/cloud_user/output/group1_consumer1.txt.
  4. Set up a separate consumer group with two consumers in that group.
  5. Wait for them to process some messages simultaneously.
  6. Store the output in /home/cloud_user/output/group2_consumer1.txt and /home/cloud_user/output/group2_consumer2.txt.

You have been given a three-broker Kafka cluster to complete this task but should perform all of your work on Broker 1.

If you get stuck, feel free to check out the solution video, or the detailed instructions under each objective. Good luck!

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!

Get Started
Who’s going to be learning?