Introducing Real-time Pub-Sub: Building Reactive Applications with HPKV

Today we're excited to announce a powerful new feature in HPKV: real-time key monitoring through a bi-directional WebSocket pub-sub system. This feature allows your applications to receive instant notifications whenever key values change, enabling truly reactive applications with minimal latency.

Why Real-time Monitoring Matters

In today's fast-paced digital landscape, users expect applications to update in real-time. Whether it's collaborative tools, dashboards, or user interfaces, stale data leads to poor user experiences and potential inconsistencies.

Traditional approaches to this problem involve polling the server at regular intervals, which comes with several drawbacks:

• Latency: Updates are only recognized when the next poll occurs
• Resource usage: Regular polling consumes bandwidth and server resources
• Complexity: Implementing efficient polling requires careful timing and error handling

HPKV's new pub-sub system solves these problems by pushing updates to your application the moment they occur, with no polling required.

How It Works

The HPKV pub-sub system follows a straightforward workflow:

1. Generate a token with the specific keys you want to monitor
2. Connect to the WebSocket endpoint using this token
3. Receive notifications whenever any of the monitored keys change

Let's break down the architecture with a sequence diagram:

When you generate a token, the system records which keys you're interested in monitoring. The token contains encrypted information about your tenant ID and the specific keys you want to monitor. When you connect to the WebSocket endpoint with this token, HPKV automatically registers your connection to receive notifications for those keys.

Internally, a pattern registry maintains a mapping between keys and the connections that are interested in them. When a key changes (through any interface - REST or WebSocket), the system notifies all relevant connections immediately.

Implementation Example

Let's walk through a complete example of how to implement real-time monitoring in your application:

Step 1: Generate a Monitoring Token

First, generate a WebSocket token with the keys you want to monitor:

curl -X POST "https://YOUR_BASE_URL/token/websocket" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "subscribeKeys": ["user:123", "counter:456", "product:789"]
  }'

You'll receive a response containing your token:

{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
}

Step 2: Connect to the WebSocket Endpoint

Next, connect to the WebSocket endpoint using this token:

// Connect to the WebSocket using the token
const ws = new WebSocket('wss://YOUR_BASE_URL/ws?token=YOUR_TOKEN');

ws.onopen = () => {
  console.log('Connected to HPKV monitoring service');
};

// Handle incoming notifications
ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  
  if (data.type === 'notification') {
    const { key, value, timestamp } = data;
    
    if (value === null) {
      console.log(`Key ${key} was deleted at ${new Date(timestamp)}`);
      // Handle deletion (e.g., remove from UI)
      removeItemFromUI(key);
    } else {
      console.log(`Key ${key} was updated to ${value} at ${new Date(timestamp)}`);
      // Handle update (e.g., update UI with new value)
      updateItemInUI(key, value);
    }
  }
};

Step 3: Receive and Handle Notifications

When a monitored key changes, you'll receive a notification with this structure:

{
  "type": "notification",
  "key": "user:123",
  "value": "John Doe", // Will be null if key was deleted
  "timestamp": 1625884522123
}

There's no explicit "operation" field in the notification. Instead, you can deduce the operation type from the value:

• If the value is present, the key was updated (either inserted or modified)
• If the value is null, the key was deleted

Real-world Example: A Collaborative Todo Application

To demonstrate the pub-sub system in action, we've created a simple Todo application that shows real-time updates across multiple users. When one user adds, completes, or removes a todo item, all other connected users see those changes instantly.

This demo application:

1. Uses HPKV for storage via WebSockets
2. Subscribes to the keys containing todo items
3. Updates the UI in real-time when changes occur
4. Shows changes made by other users instantly

The source code is available on GitHub, and you can try the live demo.

Implementation Details

Under the hood, the pub-sub system consists of several key components:

1. Token Generation Service: Creates secure tokens containing the keys to monitor
2. Pattern Registry: Maintains the mapping between keys and subscriber connections
3. WebSocket Handler: Manages connections and delivers notifications
4. Notification Pipeline: Processes key changes and routes notifications to the appropriate connections

The system is designed to work seamlessly across multiple server instances, ensuring that notifications are delivered correctly even in a distributed environment.

Getting Started

To start using the pub-sub feature in your application:

1. Update to the latest version of HPKV
2. Check out the WebSocket API documentation for detailed usage instructions
3. Consider which keys would benefit from real-time monitoring in your application
4. Implement the WebSocket client as shown in the examples above

Conclusion

The addition of real-time pub-sub capabilities to HPKV opens up exciting new possibilities for building reactive, real-time applications. From collaborative tools to live dashboards, this feature enables a new generation of responsive user experiences with minimal implementation effort.

We can't wait to see what you build with this new feature! If you have questions or want to share what you're working on, send us a message or reach out on Twitter.

Get started with HPKV today and experience the power of real-time data with minimal latency.