Building resilient data pipelines shouldn’t require rebuilding your entire infrastructure. Imagine this: You’re building a payment processing system. Stripe sends webhooks for every transaction—succeeded, failed, refunded, canceled. You need these events in your database for analytics, compliance, and real-time dashboards. Simple enough, right? But then reality hits: What happens when your webhook handler crashes? How do you ensure messages aren’t lost during deployments? What if you need to scale horizontally to handle 10,000 events/second?

Danube Messaging is a Rust-based distributed pub/sub messaging platform inspired by Apache Pulsar. It supports reliable and non-reliable dispatch, ETCD-backed metadata, and pluggable storage. Latest improvements focus on performance, reliability, and developer experience, including async multi-threaded processing, TLS/JWT security, and enhanced CLI capabilities. Concurrent Topic Processing and Routing # Async, multi-threaded topic handling: Introduced async message processing with a TopicWorkerPool to fan topics across workers, removing single-threaded bottlenecks and enabling parallelism on hot topics.

This article may contain outdated information, for latest read Danube Documentation Danube Messaging is an open-source, distributed publish-subscribe (Pub/Sub) message broker system developed in Rust. Danube aims to be a powerful, flexible and scalable messaging solution. Allows single or multiple Producers to publish on the Topics and multiple Subscriptions to consume the messages from the Topic. This article aims to cover the latest developments on the Danube platform. Before that, here is a summary of Danube’s capabilities.

This article may contain outdated information, for latest read Danube Documentation In this article, I’ll guide you through the steps to use the Danube Go client library to interact with Danube PubSub messaging brokers running on a Kubernetes cluster hosted locally. Here’s a diagram of the setup:: The setup includes: The Producer - We’ll run the script outlined in this post, which actually creates three producers (./pubsub-go-demo producer). The Consumer - Similarly, three distinct consumers are created (.

Danube is an open-source, distributed publish-subscribe (Pub/Sub) message broker system developed in Rust. Danube aims to be a powerful, flexible and scalable messaging solution. Currently, the Danube platform exclusively supports Non-persistent messages. Meaning that the messages reside solely in memory and are promptly distributed to consumers if they are available, utilizing a dispatch mechanism based on subscription types For comprehensive information on setting up, configuring, and using Danube, please refer to the official documentation and the previous article.

This article may contain outdated information, for latest read Danube Documentation Danube is an open-source, distributed publish-subscribe (Pub/Sub) message broker system developed in Rust. Inspired by the Apache Pulsar messaging and streaming platform, Danube incorporates some similar concepts but is designed to carve its own path within the distributed messaging ecosystem. At the time of writing, the Danube platform is in the early stages of development and may have missing or incomplete functionalities.