Kafka Topics UI

Effective browsing of Kafka data and more
photo of Antonios Chalkiopoulos
Antonios Chalkiopoulos
antwnis
Hey,
check out the new version for Kafka Topic UI here.

Kafka is now the de-facto platform for streaming architectures, and it’s eco-system is maturing, but is not just yet as Enterprise Ready as many people in Big | Fast Data would like it to be. Landoop is a London based start-up that wants to drive Kafka faster to the future, and thus..

We are announcing the kafka-topics-ui a User Interface that allows browsing data from Kafka Topics and a lot more

Enjoy a Live Demo

About Kafka-Topics-UI

The kafka-topics-ui is a user interface that interacts with the kafka-rest-proxy to allow browsing data from Kafka Topics. At the same time it brings visibility by providing a single entry point to explore i) Kafka data, ii) Kafka Schemas, iii) Kafka connectors and a lot more, such as partitions per topic, replication factor per topic and topic configurations.

Kafka REST Proxy

The Kafka REST Proxy - is an excellent state-less micro-service by Confluent that allows anyone to easily:

  • Create a consumer or producer via a POST request
  • Consume or Produce Avro | JSon | Binary data on a Kafka Topic
  • Delete a consumer

Utilizing it’s capabilities for building a UI was a straight-forward processes after overcoming the initial difficulties. First and foremost we had to tackle the fact you can’t tell a priory what type of data a particular topic contains. Once we figured out that, we can

  1. hard-code the type of particular config-topics
  2. automatically detect Avro type by fetching the _schemas topic
  3. in case type is still unknown iterate through Avro->Json->Binary consumers till one succeeds

Then things got so much more interesting. Another hidden gem was the flag - that allows us to efficiently stream just enough data to feed the UI and achieve the above process in just 2-3 seconds.

?max_bytes=50000

Kafka-Topics-UI features

Currently

  • Identifies and visualizes Kafka topics and config overrides
  • Automatically detects data types (Avro|Json|Binary)
  • Base64 decodes and parses of binary topics
  • Provides Table and JSon views
  • Allows downloading data from Kafka topics
  • Integrated with schema-registry-ui
  • Displays number of partitions and replication factor per topic
  • Docker container

Using it

Check https://github.com/landoop/kafka-topics-ui and follow instructions (Tip: Remember to enable CORS). The easiest way to get it running is using docker:

docker pull landoop/kafka-topics-ui

And run the container, by specifying the url of the kafka-rest service:

docker run --rm -it -p 8000:8000 \
       -e "KAFKA_REST_PROXY_URL=http://localhost:8082" \
       landoop/kafka-topics-ui

Quick preview

Conclusions

Landoop - similarly to you continuously brings to light best practises while thriving for operational excellence. Together with the open-source community we are aiming to enhance kafka-topics-ui with new features and capabilities:

  • Stream from beginning or real-time capturing
  • DC/OS universe components for your Mesos cluster
  • Real-time monitoring
comments powered by Disqus