The Message Queue Bottleneck: Why We Are Moving to Pulsar

Let's be honest. If I see one more Zookeeper crash bring down an entire Kafka cluster because of a disk latency spike, I might just retire to a cabin in Lofoten. For the last five years, Kafka has been the default answer to "we need streaming." But for those of us managing high-throughput systems in 2021, the operational overhead is becoming difficult to justify.

I recently architected a streaming backend for a Norwegian fintech scaling out of Oslo. They needed strict data residency (thanks, Schrems II), sub-5ms latency, and the ability to scale storage independently of compute. Kafka's monolithic architecture—where storage and compute are coupled on the broker—was a non-starter.

Enter Apache Pulsar. It separates the serving layer (Brokers) from the storage layer (BookKeeper). This isn't just a theoretical advantage; it means when your disk fills up, you add storage nodes (Bookies). When your CPU spikes, you add Brokers. You don't have to rebalance terabytes of data just to handle more traffic.

The Infrastructure Requirement: It's All About IOPS

Pulsar is fast, but it eats I/O for breakfast. The architecture relies heavily on Apache BookKeeper for persistent storage. BookKeeper writes to a Journal (WAL - Write Ahead Log) and a Ledger (Long term storage).

If you put the Journal on a standard SATA SSD—or worse, network-attached block storage with noisy neighbors—your write latency will destroy your throughput. The Journal requires sequential write speed; the Ledger requires random read/write speed.

Pro Tip: Never deploy BookKeeper without physically separating the Journal and Ledger disks, or at least ensuring you are on high-end NVMe. On standard cloud providers, you often hit IOPS throttling limits before CPU limits. This is why we benchmarked CoolVDS instances; direct NVMe access without the "provisioned IOPS" tax found in AWS or Azure is critical for BookKeeper stability.

Step 1: System Tuning for 2021 Hardware

Before we touch the Java binaries, we need to prep the OS. We are using Ubuntu 20.04 LTS. The default Linux kernel settings are too conservative for high-throughput streaming.

Edit your /etc/sysctl.conf to handle the massive number of open file descriptors and TCP connections Pulsar generates:

# Optimize for high concurrency
fs.file-max = 2097152
fs.inotify.max_user_watches = 524288

# Network tuning for low latency
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 16384
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Disable swap to prevent JVM pauses
vm.swappiness = 0

Apply these with sysctl -p. If you skip this, your brokers will choke under load, regardless of how much RAM you throw at them.

Step 2: Deploying the Cluster (The Right Way)

We will use Apache Pulsar 2.8.0. This version introduced significant stability improvements for transactions. Do not use the standalone mode for production; we need a proper cluster.

Zookeeper Setup

Yes, Pulsar still needs Zookeeper (for now), but only for metadata, not for the data path. Install it on 3 small nodes.

BookKeeper Configuration

This is where the magic happens. On your storage nodes (Bookies), you must configure conf/bookkeeper.conf to utilize the NVMe storage correctly.

# Journal: Write-intensive, sequential. 
# Ensure /mnt/nvme0 is your fastest drive.
journalDirectories=/mnt/nvme0/pulsar/journal

# Ledger: Read/Write, random access.
ledgerDirectories=/mnt/nvme0/pulsar/ledgers

# Performance tuning
journalSyncData=true
journalWriteData=true
journalPreAllocSizeMB=128

# Network
bookiePort=3181

If you are running on CoolVDS, the local NVMe storage provides the necessary throughput to set journalSyncData=true without incurring a massive latency penalty. This ensures zero data loss if a node loses power.

Step 3: Broker Configuration

The Broker is stateless. It just routes messages. However, memory management is key here. In conf/pulsar_env.sh, we need to configure the JVM G1GC garbage collector carefully to avoid "stop-the-world" pauses.

# MEMORY SETTINGS
PULSAR_MEM="-Xms4g -Xmx4g -XX:MaxDirectMemorySize=4g"

# GC SETTINGS
PULSAR_GC="-XX:+UseG1GC \
  -XX:MaxGCPauseMillis=10 \
  -XX:+ParallelRefProcEnabled \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+DoEscapeAnalysis \
  -XX:ParallelGCThreads=4 \
  -XX:ConcGCThreads=2"

Note the MaxDirectMemorySize. Pulsar uses Netty for network I/O, which allocates off-heap memory. If you don't cap this, the OOM killer will terminate your process unexpectedly.

Step 4: Geo-Replication and Norwegian Data Residency

One of the strongest arguments for Pulsar over Kafka is native geo-replication. For our client, we set up a primary cluster in Oslo and a backup in a secondary data center (like Trondheim or Bergen) for disaster recovery.

With Kafka, MirrorMaker is a separate process that is notoriously difficult to manage. With Pulsar, it's built-in.

To enable replication, you simply define a tenant that spans multiple clusters:

$ bin/pulsar-admin tenants create my-tenant \ 
  --admin-roles my-admin-role \ 
  --allowed-clusters oslo-cluster,bergen-cluster

Then, create a namespace capable of replication:

$ bin/pulsar-admin namespaces create my-tenant/fin-data
$ bin/pulsar-admin namespaces set-clusters my-tenant/fin-data \ 
  --clusters oslo-cluster,bergen-cluster

Any message published to my-tenant/fin-data in Oslo is asynchronously replicated to Bergen. This architecture ensures that even if the primary data center goes dark, the data remains safely within Norwegian borders, satisfying Datatilsynet requirements.

Why Bare Metal Performance Matters

Virtualization overhead is the enemy of message streaming. When you run Pulsar inside Docker on top of a heavy hypervisor on top of shared storage, you introduce "jitter." In a high-frequency trading or real-time analytics environment, a 50ms jitter is unacceptable.

CoolVDS instances utilize KVM with strict resource isolation. More importantly, the disk I/O path is optimized. When benchmarking Pulsar 2.8.0 on a CoolVDS 8-core instance versus a comparable general-purpose instance from a US giant, the difference was stark:

The write latency difference is purely down to the storage subsystem. Pulsar waits for the Bookie to confirm the write to the Journal before acknowledging the message. Slow disk = Slow stream.

Conclusion

Apache Pulsar solves the scalability headaches that have plagued Kafka administrators for years, but it demands respect for the underlying hardware. You cannot throw it on cheap, shared storage and expect it to fly.

For Norwegian businesses facing strict data sovereignty laws and needing sub-millisecond latency, the combination of Pulsar's architecture and local, high-performance infrastructure is the only logical path forward.

Stop fighting with ZooKeeper timeouts. Deploy a proper BookKeeper cluster on CoolVDS and see what 200,000 messages per second actually looks like.