This is a multipart series on Developing for Apache Kafka and Amazon MSK (Amazon Managed Streaming for Apache Kafka). Checkout these other tutorials on Apache Kafka.
How to Build a Distributed Apache Kafka Development Cluster
Developing .NET Core Applications for Apache Kafka and Amazon MSK
This is part one of a series on Developing for Apache Kafka and Amazon MSK (Amazon Managed Streaming for Apache Kafka).
Note, for this solution, Kafka was installed on an Amazon Linux 2 virtual machine. Feel free to use whichever OS you are comfortable with.
Before you start, it will help if you are familiar with Apache Kafka zookeepers and Apache Kafka brokers and their relationship. Also, we will not address securing Apache Kafka as that is out of scope for this tutorial.
Apache Kafka now comes in an installation package that is pretty much configured and ready to use for development purposes. You just need to put the pieces in the right places and start up the services. Let’s take a look.
Prerequisites
Apache Kafka has a dependency on Java. Download and install the JDK and run the following command to verify that Java was installed.
Warning: some AWS services may have fees associated with them.
$ java -version
1. Download Apache Kafka
Go to apache.org and download Apache Kafka . If you are working from the command line, considering using curl to get the file using the output option.
Example:
$ curl https://apache.claz.org/kafka/2.6.0/kafka_2.13-2.6.0.tgz ––output kafka.tgz
2. Extract Apache Kafka
Pick a suitable location on your file system to run Apache Kafka and extract it. We’ll use tar at the command line for this task.
$ tar -xzf kafka.tgz
Now that we have Apache Kafka extracted, let’s cd into the directory to get started using Kafka.
$ cd kafka_2.13-2.6.0
3. Start the Apache Kafka Zookeeper
Please note, the zookeeper and the broker are both included in the Apache Kafka platform that we just downloaded.
The first step in starting the Kafka environment is to start the Apache Kafka zookeeper.
$ bin/zookeeper-server-start.sh config/zookeeper.properties
This command starts the zookeeper, passing in the default configuration data from the file config/zookeeper.properties.
4. Start the Apache Kafka Broker
The next step is starting the broker. Open another terminal to the Apache Kafka server you are building and run the following command.
$ bin/kafka-server-start.sh config/server.properties
This command starts the broker, passing in the default configuration data from the file config/server.properties.
Complete!
That’s it! You now have an Apache Kafka development server.
How is this all working so easily? As mentioned earlier, Apache Kafka is pretty much “working” right out of the box. Let’s take a look at the configuration files to see how this is accomplished.
Taking a closer look at the config/zookeeper.properties file you will see the setting for clientPort with a value of “2181”. When you started the zookeeper in step 3, it started the zookeeper at localhost running on port 2181.
Now, lets take a look at the config/server.properties file. In this file you will see a setting for zookeeper.connect with a value of “localhost:2181”. When you started the broker in step 4, it was already configured to be assigned to the zookeeper that you started in step 3.
Test Apache Kafka Environment
So, now that I have an Apache Kafka server up and running, what can I do with it? First we will test it to make sure that everything is working. Luckily, Apache Kafka has a command line producer and consumer that let’s us easily test the installation.
1. Create an Apache Kafka Topic
Open a new terminal to the Apache Kafka server and run the following command in the directory that you installed Apache Kafka in.
$ bin/kafka-topics.sh ––create ––topic test-topic ––bootstrap-server localhost:9092
This command creates the topic, “test-topic”, using the broker that was started in step 4. 9092 is the default port that the broker runs on.
2. Produce a message to the newly created Apache Kafka topic.
In that same directory, run the following command. This will open a REPL like environment. This command produces messages to Apache Kafka through the broker that was started earlier.
$ bin/kafka-console-producer.sh ––topic test-topic ––bootstrap-server localhost:9092
To produce a message to Apache Kafka, type in the terminal and hit enter. Each line that you enter will be sent to the Apache Kafka topic (test-topic) that you just created. Use Control + C to quit the REPL like environment.
> Apache Kafka test record 1
> Apache Kafka test record 2
> Apache Kafka test record 3
3. Consume the messages that were sent to the newly created topic.
Open a new terminal to the Apache Kafka development server and run the following command in the directory that you installed Apache Kafka in. This command continually consumes messages from the Apache Kafka topic, “test-topic” via the broker that was started earlier (localhost:9092).
$ bin/kafka-console-consumer.sh ––topic test-topic ––from-beginning ––bootstrap-server localhost:9092
You should see the following output.
Apache Kafka test record 1
Apache Kafka test record 2
Apache Kafka test record 3
That’s it. You have now built and tested an Apache Kafka development server.
Want to take this to the next level?
Turn this Apache Kafka development server into a Distributed Apache Kafka Development Cluster.
Create .Net applications that produce and consume from this Apache Kafka development server.
AWS with .NET 