GraphQL Subscriptions with Kafka Pub Sub

In the previous article, we have implemented a real-time messaging server using GraphQL. There we used the default pub sub system provided by Apollo. But it is not recommended for the production environments because it is an in-memory system. To avoid this issue, there are many community-created libraries have been developed by modifying the basic pub sub library. Among them, here we are going to use the Kafka pub sub library and connect it with the server we used in the previous article.

First, you need to follow that article and configure the server as instructed. After that, before moving into the next steps, I will explain the functionality of a pub sub model.

What is a Pub Sub Model?

Pub Sub model is basically a message communication system that follows the publish-subscribe messaging pattern. Instead of sending data directly to the receivers, data will be transferred through a mediator called channel(or topic). Publishers can add data into that channel and once the data is added, data will be broadcast to every subscriber of that channel. To implement this approach, every pub sub model has three basic operations.

Publish : Add data to the given channel

Subscribe : Subscribe users to a channel. It can be modified to receive a function to execute when the data is arrived.

Unsubscribe :Unsubscribe a client from a given channel

If we look into that example scenario we used in the previous article, there we used a subscription operation to receive messages whenever someone sends one. Since the pub sub model is well suited for this requirement, JavaScript has provided a pub sub library to use in GraphQL subscriptions.

Now, our goal is to replace the default pub sub library by the Kafka pub sub. For that, first you need to set up a Kafka Server in your operating system.

Setting up a Kafka Server

1. Download the Apache Kafka from here and extract it into a directory. (Recommended Version: 2.8.0)
2. Start the Kafka environment. (For that you need to have java 8 or higher installed)

First you need to start the zookeeper server. Zookeeper will monitor the Kafka clusters, topics and partitions to maintain the synchronization in the system. Zookeeper server is a must for Kafka 2.x versions. But if you are having a Kafka 3.x version, zookeeper is not necessary. To run the server you can use the following commands. I have provided commands for both Windows and Linux. Use the relevant command according to your operating system.

If you are using Linux use the following commands ,

// enter the Kafka directory
cd "your_kafka_directory_name"
// start the Zookeeper server
bin/zookeeper-server-start.sh config/zookeeper.properties

After the zookeeper server is up and running, open another terminal to start the Kafka Server

cd "your_kafka_directory_name"
// start the Kafka server
bin/kafka-server-start.sh config/server.properties

Following commands are for Windows,

// enter the Kafka directory
cd "your_kafka_directory_name"
// start the Zookeeper server
.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

After the zookeeper server is up and running, open another terminal and run this command,

cd "your_kafka_directory_name"
// start the Kafka server
.\bin\windows\kafka-server-start.bat .\config\server.properties

If they are running without any issue, you can proceed to next steps.

Install Dependencies

You need to go the project directory we used in the previous article, and install the following library.

npm install graphql-kafka-subscriptions

If you had any issues when installing the package, try to install the followings first.

npm install node-gyp 
npm install node-rdkafka

If you are using windows and the error still occurring, one thing you can do is install WSL 2 (Windows Sub System for Linux). Using WSL you can install Ubuntu(Or any preferred Linux OS) in Windows. Then you can move the project into a Ubuntu directory and install the node package there.

If you are going to continue with WSL, it is better the Kafka environment is also set up in Linux.

Configuring Kafka Pub Sub in the Project

You have to import the library you installed into the project. You can also remove the library used for the default pub sub.

const { KafkaPubSub } = require("graphql-kafka-subscriptions");

Then you need to modify the line where you initiate your simple Pub Sub object with a Kafka Pub Sub instance.

   const pubsub = new KafkaPubSub({
         topic: 'MessageService',
         host: '127.0.0.1',
         port: '9092',
         globalConfig: {} 
         // options passed directly to the consumer and producer
   })

The topic here is where you produce the messages from the server side and consume them from the client side. I use the default port of Kafka as 9092. You can change the port value from Kafka server but keep in mind don’t use the same port for both Kafka server and GraphQL HTTP endpoint. (eg: You can’t use the port 4000 here)

The resolver part in Simple Pub Sub, is the same for a Kafka Pub Sub also. Therefore no modification required in those fields.

Now, the completed file will be look like this.

Create the Topic

Since you have not create the “MessageService” topic in Kafka, you will receive an error if you run the project and send a GraphQL request. You can either automatically set the topic when a data has been publishing to a new topic or you can just create the topic manually.

If you want to set up the Kafka server as it automatically create topics, refer the instruction here. Or you can create topics manually as shown in below.

For Linux, (Run this command from the Kafka directory)

bin/kafka-topics.sh --create --topic MessageService --bootstrap-server localhost:9092

For Windows,

.\bin\windows\kafka-topics.bat --create --topic MessageService --bootstrap-server localhost:9092

Now, you can start the node project and go to a Apollo Studio IDE or any other GraphQL IDE to run the requests.

This procedure is the same as we did in the previous article. So you can follow up from there.

You can find the GitHub repository for the full code from here. Inform me if you got any issues in the code.