Tuning a Kafka Consumer

30 minutes
  • 2 Learning Objectives

About this Hands-on Lab

You can configure your Kafka consumers in a variety of ways in order to customize their behavior. In many production scenarios, ensuring your consumers perform well for your use case is a matter of tweaking consumer configurations. In this lab, we will work with this process hands-on. We will be provided with some consumer code and a list of issues that can be addressed by changing the consumer configuration. We will then need to identify what configuration changes should be made in order to address these issues, and we will need to implement those changes in the consumer.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Clone the Starter Project from GitHub and Perform a Test Run
  1. Clone the starter project from GitHub:

    cd ~/
    git clone https://github.com/linuxacademy/content-ccdak-consumer-tuning-lab.git
  2. Perform a test run to make sure the code is able to compile:

    cd content-ccdak-consumer-tuning-lab
    ./gradlew run

    The code should compile, but the consumer will fail to run.

Implement Configuration Changes to Address the Issues
  1. Edit the Consumer class:

    vi src/main/java/com/linuxacademy/ccdak/consumer/MemberSignupsConsumer.java
  2. Increase the minimum fetch size by setting fetch.min.bytes=1024:

    props.setProperty("fetch.min.bytes", "1024");
  3. Send heartbeats more frequently by setting heartbeat.interval.ms=2000:

    props.setProperty("heartbeat.interval.ms", "2000");
  4. Fix the issue with the offset reset behavior by setting auto.offset.reset=earliest:

    props.setProperty("auto.offset.reset", "earliest");
  5. Run your code to verify it still works:

    ./gradlew run

Additional Resources

Your supermarket company has a consumer that logs data from the member_signups topic to System.out. This consumer acts as a utility to log the data for later auditing. However, there is a series of issues with this consumer that can be addressed through some minor alterations to its configuration. Examine the list of issues below and resolve them by implementing the necessary configuration changes in the consumer code.

You can find the consumer code on GitHub. Clone this project to the broker server. You can implement your configuration changes in the consumer class at src/main/java/com/linuxacademy/ccdak/consumer/MemberSignupsConsumer.java inside the project.

Make configuration changes to address the following issues:

  • This consumer does not have a high need for real-time data since it is merely a logging utility that provides data for later analysis. Increase the minimum fetch size to 1 K (1024 bytes) to allow the consumer to fetch more data in a single request.
  • Changes in consumer status (such as consumers joining or leaving the group) are not being detected quickly enough. Configure the consumer to send a heartbeat every two seconds (2000ms).
  • Last week, someone tried to run this consumer against a new cluster. The consumer failed with the following error message:

    Exception in thread "main" org.apache.kafka.clients.consumer.NoOffsetForPartitionException: Undefined offset with no reset policy for partitions: [member_signups-0]

    Ensure the consumer has an offset reset policy that will allow the consumer to read from the beginning of the log when reading from a partition for the first time.

When you have implemented your changes, you can run the code with ./gradlew run to test it against the development broker.

If you get stuck, feel free to check out the solution video, or the detailed instructions under each objective. Good luck!

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!

Get Started
Who’s going to be learning?