In this mini tutorial we will explore how to create a Kafka Connect Pipeline using the Kafka Development Environment (fast-data-dev) in order to move real time telemetry data into Elasticsearch and finally visualize the positions in a Kibana Tile Map by writing zero code…!
Time-series datastores are of particular interest these days and influxDB is a popular open source distributed time-series database. In this tutorial we will integrate Kafka with InfluxDB using Kafka Connect and implement a Scala avro message producer to test the setup.
An FTP server, together with a pair of credentials is a common pattern, on how data providers expose data as a service.
In this article we are going to implement custom file transformers to efficiently load files over FTP and using Kafka Connect convert them to meaningful events in Avro format.
Landscaping Topologies Microservice architectures are all the rage and for good reason.
Small, lightweight, business focused and independently deployable services provide scalability and isolation which it hard to achieve in monolithic systems.
Depending on your Kafka set up security might involve different services such as Kerberos,
relying on multiple TLS certificates and advanced ACL setup in brokers and Zookeeper. In many cases, with encryption features enabled, performance is also taking a penalty hit.
The Internet of Things is on the rise, it was certainly a buzzword of 2016.
Gartner thinks so, they say there will be 20 billion devices online by 2020 with all of them transmitting (streaming) data. These devices are not limited to new devices,
Coyote, is an open-source tool https://github.com/Landoop/coyote we built using Go at Landoop for testing automation.
Define sequence of commands to be executed inside a YAML file and then the expected output in REST requests or log files or by executing commands to target systems.