Building Resilient Event-Driven Backend Pipelines with Go and NATS JetStream

As backend applications scale, traditional synchronous REST or gRPC microservices often become bottlenecks. When Service A makes a direct HTTP call to Service B, network blips, unexpected traffic spikes, or database locks in Service B can ripple across the entire system, causing cascading failures.

To build fault-tolerant, high-throughput backend architectures, engineering teams are increasingly turning to Event-Driven Architecture (EDA) powered by lightweight messaging systems like NATS JetStream.

In this article, we’ll explore how to design a resilient, asynchronous event stream in Go using NATS JetStream—focusing on at-least-once delivery, consumer durability, and backpressure management.

Why NATS JetStream for Modern Backends?

While Apache Kafka and RabbitMQ are long-time industry standards, NATS JetStream has gained immense traction for backend microservices due to its single-binary simplicity and ultra-low resource footprint:

In-Memory & Persistent Storage: JetStream adds built-in persistence and stream management directly on top of Core NATS.
Low Latency & High Throughput: Written in Go, it delivers sub-millisecond pub/sub latencies with minimal CPU and memory usage.
Flexible Consumption Models: Supports both push-based (real-time notifications) and pull-based (batch processing/worker pool) consumers out of the box.

System Architecture Overview

We will implement a resilient order-processing pipeline where:

An API Producer publishes ORDER_CREATED events to a NATS stream.
A Worker Pool Consumer pulls events from the stream, processes payments asynchronously, and explicitly acknowledges (ACK) successful jobs.

┌─────────────────┐       Publish Event       ┌───────────────────────────────┐
│                 │ ────────────────────────> │      NATS JetStream Stream    │
│  API Producer   │                           │     Subject: "ORDERS.created" │
└─────────────────┘                           └───────────────┬───────────────┘
                                                              │
                                                        Pull / Consume
                                                              │
                                                              ▼
                                              ┌───────────────────────────────┐
                                              │    Order Worker Consumer      │
                                              │  (Retries & Explicit ACKs)    │
                                              └───────────────────────────────┘

Implementation in Go

1. Prerequisites

First, ensure you have a running NATS server with JetStream enabled:

docker run -d --name nats-main -p 4222:4222 nats:latest -js

Install the official Go client:

go get github.com/nats-io/nats.go

2. Initializing the Stream Connection

Before publishing or consuming, we set up the NATS connection and define our stream configuration.

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"github.com/nats-io/nats.go"
	"github.com/nats-io/nats.go/jetstream"
)

func setupStream(ctx context.Context, js jetstream.JetStream) (jetstream.Stream, error) {
	// Define Stream parameters
	cfg := jetstream.StreamConfig{
		Name:     "ORDERS",
		Subjects: []string{"ORDERS.*"},
		Storage:  jetstream.FileStorage, // Persist messages to disk
		MaxAge:   24 * time.Hour,        // Retain messages for 24 hours
	}

	// Create or update the stream idempotently
	return js.CreateOrUpdateStream(ctx, cfg)
}

3. Implementing the Event Producer

The producer publishes JSON-encoded order events to the ORDERS.created subject.

type OrderEvent struct {
	OrderID   string  `json:"order_id"`
	UserID    string  `json:"user_id"`
	Amount    float64 `json:"amount"`
	Timestamp int64   `json:"timestamp"`
}

func publishOrder(ctx context.Context, js jetstream.JetStream, order OrderEvent) error {
	data, err := json.Marshal(order)
	if err != nil {
		return fmt.Errorf("failed to marshal order: %w", err)
	}

	// Synchronous publish ensuring JetStream acknowledged stream write
	ack, err := js.Publish(ctx, "ORDERS.created", data)
	if err != nil {
		return fmt.Errorf("failed to publish message: %w", err)
	}

	log.Printf("[Producer] Published Order ID %s | Stream Sequence: %d", order.OrderID, ack.Sequence)
	return nil
}

4. Implementing the Durable Consumer with Retries

To handle worker restarts or crashes gracefully, we use a Durable Pull Consumer. If a worker fails to process or crash mid-job, the message will automatically be redelivered after the acknowledgment timeout expires.

func startWorkerPool(ctx context.Context, js jetstream.JetStream) error {
	// Define durable consumer configuration
	consumerCfg := jetstream.ConsumerConfig{
		Durable:       "OrderPaymentProcessor",
		FilterSubject: "ORDERS.created",
		AckPolicy:     jetstream.AckExplicitPolicy, // Worker MUST explicitly ACK messages
		AckWait:       10 * time.Second,            // Redeliver if not ACKed within 10s
		MaxDeliver:    3,                           // Retry up to 3 times before DLQ
	}

	consumer, err := js.CreateOrUpdateConsumer(ctx, "ORDERS", consumerCfg)
	if err != nil {
		return fmt.Errorf("failed to create consumer: %w", err)
	}

	// Fetch and process messages concurrently
	for {
		select {
		case <-ctx.Done():
			return nil
		default:
			// Pull a batch of up to 10 messages from NATS
			msgs, err := consumer.Fetch(10, jetstream.FetchMaxWait(2*time.Second))
			if err != nil && err != context.DeadlineExceeded {
				log.Printf("[Worker] Error fetching batch: %v", err)
				continue
			}

			for msg := range msgs.Messages() {
				processMessage(msg)
			}
		}
	}
}

func processMessage(msg jetstream.Msg) {
	var order OrderEvent
	if err := json.Unmarshal(msg.Data(), &order); err != nil {
		// Terminate invalid message so it isn't redelivered
		log.Printf("[Worker] Invalid payload: %v", err)
		msg.Term()
		return
	}

	log.Printf("[Worker] Processing payment for Order ID: %s ($%.2f)", order.OrderID, order.Amount)

	// Simulate payment processing logic
	if err := simulatePayment(order); err != nil {
		log.Printf("[Worker] Payment failed for Order ID %s, requesting redelivery: %v", order.OrderID, err)
		// NAK signals NATS to retry delivery immediately or based on backoff
		msg.Nak()
		return
	}

	// Explicitly acknowledge successful processing
	msg.Ack()
	log.Printf("[Worker] Successfully processed & ACKed Order ID: %s", order.OrderID)
}

func simulatePayment(order OrderEvent) error {
	// Business logic goes here
	return nil
}

Production Best Practices for Event-Driven Backends

Transactional Outbox	Prevents partial states where DB updates succeed but message publishing fails.	Write events to an `outbox` database table within the same DB transaction, then use a CDC engine or background worker to publish to NATS.
Idempotent Consumers	Prevents duplicate execution when network glitches trigger redeliveries.	Store processed `order_id`s in Redis with an expiration window to guard against duplicate execution.
Dead Letter Queues (DLQ)	Prevents malformed or permanently failing messages from blocking consumers.	Set `MaxDeliver` limit on JetStream consumers and route abandoned messages to a `ORDERS.dlq` subject for inspection.

Key Rule: Always design event handlers with at-least-once delivery semantics in mind. Assume any event can and will be delivered twice in distributed environments.

Summary

By decoupling API request handlers from heavy asynchronous tasks using NATS JetStream and Go, you build backend systems that are:

Resilient: Isolated failures won't bring down your API gateways.
Scalable: Scale worker pools up or down independently based on consumer queue depth.
Maintainable: Clear event contracts eliminate tight coupling between microservices.