Using Kafka on OSX with a F# producer and consumer
This is just a quick note on installing Kafka on OSX and how to produce and consum messages with a F# client.
The easiast way to install Apache Kafka on OSX is probably with Homebrew
Open your favorite terminale and issue the following command:
brew install kafka
This will also install Zookeeper which Kafka has a strong dependency on.
Does it work
You can test the installation from the command line with the following few steps:
- start the Zookeeper server:
- start the Kafka server:
- create a topic:
kafka-topics –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic testTopic
- produce some messages:
kafka-console-producer –broker-list localhost:9092 –topic testTopic
- consume the messages:
kafka-console-consumer –zookeeper localhost:2181 –topic testTopic –from-beginning
- clean up messages:
kafka-topics –zookeeper 127.0.0.1:2181 –delete –topic testTopic
Your path must include Java for this to work. Hopefully the messages that you type into the producer command line ends up being printed on the consumer command line.
Here is a simple producer, that just pumps messages out quite fast:
The corresponding consumer looks like this. Please note the call to
Consume() is a blocking call but by turning it into an observable sequence one can work around this.
I do not consider the code snippets above production ready. There are a lot of configurations to take into account when deploying Kafka to production. Furthermore kafka-net does not seem to be actively developed for the time being. The current version was build for kafka version 0.8.0 but the parts that I have played around with also works for version 0.9.0.
Kafta is consider to be very fast and one way to obtain low latency is by design choice. It is the responsibility of the consumers to keep track of where in the message partion the consumer has handled messages for. Kafka supports the consumer by offering a way to commit offset positions but this feature is not supported by kafka-net.
The Apache Kafka website lists four DotNet clients for Kafka. In addition to kafka-net used in the examples above the other three clients are:
.NET Kafka Client. This client was last updated 2 years ago.
CSharpClient-for-Kafka by Microsoft. The project has sparse documentation and no Nuget package.
rdkafka-dotnet. This project seems to be maintained but there a hardly any test coverage of the project.
Compared to the official Java API all four project seems to be missing quite a lot of functionality. If you are using Kafka from DotNet in production I would very much like to hear about your experience. Kafka itself appears to be an attractive choice for messaging infrastructure but the support from DotNet is rather limited.
 The Log: What every software engineer should know about real-time data’s unifying abstraction.
 Kafka by example: Kafka as Unix Pipes.
 Apache Kafka for Beginners.
 Kafka in a Nutshell.
 Kafka: The Definitive Guide. Real-time data and stream processing at scale by Neha Narkhede, Gwen Shapira, Todd Palino, O’Reilly Media, 2016.
 ZooKeeper. Distributed Process Coordination by Flavio Junqueira, Benjamin Reed, O’Reilly Media, 2013.
Please send messages and comments to my twitter account Twitter.
Edit page on GitHub. Please help me to improve the blog by fixing mistakes on GitHub. This link will take you directly to this page in our GitHub repository.
There are more posts on the front page.
Content of this blog by Carsten Jørgensen is licensed under a Creative Commons Attribution 4.0 International License.