JavaScript lib for Kafka development

Connecting to Kafka using redux-lenses-streaming
photo of Sebastian Dragomir
Sebastian Dragomir

More and more companies are embracing the data streaming revolution in order to get a better insight of their business and thus drive growth upwards. With Apache Kafka getting a lot of momentum in the space, it is normal to want to connect a Javascript UI application to it. This Javascript library build on redux brings the best practises of real time data streaming to the browser and provides a new way for building solutions to get and send data on Apache Kafka.

REST endpoints to Kafka is one way to connect to a backend, but now there are new ways to achieve low latency updates from Kafka into the browser. Lenses offers a new data streaming platform for Apache Kafka and exposes a rich set of endpoints for a Javascript application to use LSQL (Lenses SQL — Landoop SQL engine for Apache Kafka) to get messages from a Kafka topic to the browser using consumer group semantics. The communication is bi-directional; from your web app you can also push records back to Kafka.

Consider a scenario of an FX trading application; to get a list of currency exchange tickers it is now as easy as firing a request with the following:

SELECT bid, ask, _key.* as ticker
FROM tickers
WHERE _key in (‘EURUSD’, ‘GBPEUR’) and _ktype= STRING and _vtype= AVRO

In this tutorial we will show you how to setup a React/Redux application and retrieve data from Kafka, via SQL, or to connect an existing Redux application and stream data in a matter of minutes.

For this we will be using redux-lenses-streaming , a redux middleware built to make the process as fast and easy as possible.

This guide assumes some basic knowledge of React and Redux. The demo project has the minimal setup needed to connect, login, publish, subscribe and view streaming Kafka messages via the Lenses WebSocket connection.

If you already have an existing Redux project which you want to update with the library, you will need to install it alongside it’s peer dependency, RxJS. (although no knowledge is required of RxJS, we highly recommend looking into it, especially if your application deals with large streams of data / events / messages)

npm install rxjs redux-lenses-streaming --save

Setup and connection

We start off by adding the Lenses reducer to our root reducer.

import { combineReducers } from 'redux';
import { lensesReducer } from 'redux-lenses-streaming';
import { sessionReducer } from './sessionReducer';

const rootReducer = combineReducers({
  session: sessionReducer,
  lenses: lensesReducer,
});

export default rootReducer;

Next we’ll create the lenses middleware and add it in our Redux store; while doing so we can customise the connection options. ( this way the library will attempt to connect on initialisation )

import { createStore, applyMiddleware } from 'redux';
import { createLensesMiddleware } from 'redux-lenses-streaming';
import { createLogger } from 'redux-logger';
import { INITIAL_STATE } from '../reducers/sessionReducer'
import rootReducer from '../reducers/';

function configureStore() {
  // We are connecting when the middleware is setup
  // Alternatively we can dispatch the CONNECT action with the same options
  // For a full list of supported options, see documentation
  const options = {
    host: INITIAL_STATE.host,
    clientId: INITIAL_STATE.clientId,
  };

  const lensesWsMiddleware = createLensesMiddleware(options);
  const logger = createLogger({
    collapsed: true
  });
  const middleware = [logger, lensesWsMiddleware];

  const store = createStore(
    rootReducer,
    applyMiddleware(...middleware),
  );

  return store;
}

export default configureStore;

Currently the following options are supported.

var defaultOptions = {
  host: '',
  clientId: '',
  user: '',
  password: '',
  secure: false,
  timeout: 5000,
  autoCommitDelay: -1
};

In case you need to wait for data or user input, leave the options empty when setting up the middleware and dispatch a connect action with the options as payload:

import { Actions as KafkaActions } from 'redux-lenses-streaming';

class Connect extends React.Component {

...

onConnectClick() {
    const { connection } = this.props;
    const { authRequired } = this.state;

    if (connection) {
      this.props.disconnect();
    } else if (authRequired) {
      const options = {
        host: this.props.host,
        clientId: this.props.clientId,
        user: this.props.user,
        password: this.props.password,
      };
      this.props.connect(options);
    } else {
      const options = {
        host: this.props.host,
        clientId: this.props.clientId,
      };
      this.props.connect(options);
    }
  }

  onLogin() {
    const options = {
      user: this.props.user,
      password: this.props.password,
    };
    this.props.login(options);
  }
}

const mapDispatchToProps = dispatch => ({
  connect: (option) => {
    dispatch(KafkaActions.connect(option));
  },
  disconnect: () => {
    dispatch(KafkaActions.disconnect());
  },
  login: (option) => {
    dispatch(KafkaActions.login(option));
  },
});

export default connect(mapStateToProps, mapDispatchToProps)(Connect);

On successful connection, an action will be dispatched that you can listen to and respond in your reducers/middleware.

import { Type as KafkaType } from 'redux-lenses';

Once the KafkaType.CONNECT_SUCCESS has been received you will be able to access the connection via the reducer which has a connection object and subscriptions array (currently empty).

As you noticed, the options payload includes a user and password property. If they are provided during the connect stage, the library will automatically attempt to login. Another way to achieve this is by dispatching a LOGIN action at a later date using the provided action creator. (see docs for full list)

By this point we also notice KafkaType.KAFKA_HEARTBEAT messages that we can track in our reducers.

Subscribing and unsubscribing from topics

Next step is to subscribe to a topic. For this we will use the subscribe action creator that we will import in the same way we did with connect and send as the payload an object with the sqls property (String).

onSubscribe() {
  const request = {
      sqls: this.state.sqls,
  };
  this.props.subscribe(request);
}

An example sql would look like this:

On a successful subscription, the SUBSCRIBE_SUCCESS will be dispatched and the subscriptions array in the lenses reducer will be updated with the new subscription.

In order to listen to kafka messages, we’ll add an action handler in our application reducer ( named sessionReducer in the example ) for KafkaType.KAFKA_MESSAGE . The payload will have a content property with the messages array, that we can visualise in a data grid or chart.

In order to unsubscribe, all we have to do is dispatch an unsubscribe action with the following payload:

const request = {
   topics: [topic],
};

Publishing to a topic

In a similar way to subscribe, for publishing we will dispatch a publish action with the following payload.

onPublishClick() {
    const { pubKey, pubValue, selectedTopic } = this.state;

    const request = {
      topic: selectedTopic,
      key: pubKey,
      value: pubValue,
    };

    this.props.publish(request);
}

The acknowledgement will be of PUBLISH_SUCCESS type. If you have already subscribed to that topic, you should see the message streamed back to the client.

What’s next?

We are working on adding additional features to the library, such as message batching, delay and timeout customisation.

We recommend trying out the demo project and letting us know what features you would like to be implemented next.

Additional Resources

How to explore data in Kafka topics with Lenses

Find out about Lenses for Apache Kafka

Lenses documentation

Lenses SQL for Kafka

Lenses for Kafka Development The easiest way to start with Lenses for Apache Kafka is to get the Development Environment, a docker that contains Kafka, Zookeeper, Schema Registry, Kafka Connect, Lenses and a collection of 25+ Kafka Connectors and example streaming data.


Share this article

Did you like this article?

Subscribe to get new blogs, updates & news

Follow us for news, updates and releases!
@LandoopLtd
LENSES
For Apache Kafka ®
Download Now
Share this article


2 Minute Overview


Discover awesome features


Community


Join us at Landoop Community


Resources


Repos, Docs, Trainings, Tutorials


Free Download


ALL-IN-ONE free for developers!