Tuning a Kafka Producer

30 minutes
  • 2 Learning Objectives

About this Hands-on Lab

Configuring Kafka producers is a relatively simple process. The real challenge is knowing what configuration options to change in order to tune the producer for your use case. In this lab, you will have the opportunity to work hands-on with the process of tuning a Kafka producer. You will be presented with a few issues that can be addressed through configuration tuning. You will then need to determine what configuration changes to make in order to address these issues, and you will need to implement those changes in the producer.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Clone the Starter Project from GitHub and Perform a Test Run
  1. Clone the starter project from GitHub.:

    cd ~/
    git clone https://github.com/linuxacademy/content-ccdak-producer-tuning-lab.git
  2. Perform a test run to make sure the code is able to compile and run:

    cd content-ccdak-producer-tuning-lab
    ./gradlew run

    The code should compile, and tests should succeed.

Implement the Configuration Changes in the Producer
  1. Open the file:

    vi src/main/java/com/linuxacademy/ccdak/producer/MemberSignupsProducer.java
  2. Fix the issue with data loss by setting acks=all:

    props.put("acks", "all");
  3. Fix the out-of-order record issue caused by retries by setting max.in.flight.requests.per.connection=1:

    props.put("max.in.flight.requests.per.connection", "1");
  4. Increase the batch size by setting batch.size=65536:

    props.put("batch.size", "65536");
  5. Run your code to verify that it still works:

    ./gradlew run

Additional Resources

Your supermarket company has a producer that publishes messages to a topic called member_signups. After running this producer for a while in production, three issues have arisen that require some adjustments to the producer configuration. You have been asked to make some changes to the producer configuration code to address these issues.

The source code can be found on GitHub. Clone this project to the broker. You should be able to implement all the necessary changes in the file src/main/java/com/linuxacademy/ccdak/producer/MemberSignupsProducer.java inside the project.

Make configuration changes to address the following issues:

  • Recently, a Kafka broker failed and had to be restarted. Unfortunately, that broker was the leader for a partition of the member_signups topic at the time, and a few records had been committed by the leader but had not yet been written to the replicas. A small number of records were lost when the leader failed. Change the configuration so that this does not happen again.
  • The producer is configured to retry the process of sending a record when it fails due to a transient error. However, in a few instances this has caused records to be written to the topic log in a different order than the order they were sent by the producer, because a record was retried while another record was sent ahead of it. A few downstream consumers depend on certain ordering guarantees for this data. Ensure that retries by the producer do not result in out-of-order records.
  • This producer sometimes experiences high throughput that could benefit from a greater degree of message batching. Increase the batch size to 64 KB (65536 bytes).

When you have implemented your changes, you can run the code with ./gradlew run to test it against the development broker.

If you get stuck, feel free to check out the solution video, or the detailed instructions under each objective. Good luck!

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!

Get Started
Who’s going to be learning?