Part 13 in the “Building Event-Driven Microservices with Hazelcast” series
Over the previous twelve articles, we built an event-sourced microservices framework with Hazelcast: a Jet-powered event pipeline, materialized views, saga orchestration, circuit breakers, a transactional outbox, dead letter queues, and durable persistence with write-behind MapStore. The framework works. But “works” and “works under load” are different things.
This post covers what happened when we subjected the framework to serious performance engineering: flame graphs, micro-benchmarks, sustained load tests, A-B testing of Enterprise features, and scaling from a single Docker Compose host to a 5-node AWS EKS cluster. We found (and fixed) real bottlenecks, learned where time actually goes in an event-sourced system, and discovered that the biggest performance breakthrough came not from code optimization but from an architectural decision about clustering.
The Measurement Stack
Before optimizing anything, we needed to measure. Four tools, each covering a different level.
k6 handles HTTP load generation. The critical design choice: constant-arrival-rate executor, which sends requests at a fixed rate regardless of response time. The alternative — constant-vus — hides latency. When responses slow down, it sends fewer requests, which makes latency appear stable when it’s actually degrading. This is called coordinated omission, and it’s the most common load testing mistake. We almost made it.
async-profiler produces CPU and allocation flame graphs. We baked it into a Docker profiling image so we could capture 30-second flame graphs under load without restarting services. On macOS Docker Desktop, perf_events isn’t available, so we used itimer (wall-clock) mode.
Prometheus and Grafana handle time-series metrics. The framework registers 46 application-level Micrometer metrics across pipeline, saga, persistence, and gateway components. Six pre-built dashboards cover everything from event flow to JVM heap usage.
JMH handles micro-benchmarks. When we needed to isolate the cost of a single operation — serialization, IMap write, FlakeIdGenerator — we used JMH’s rigorous warmup-and-fork methodology to get microsecond-accurate measurements.
We also built supporting infrastructure: a Bash 3.2-compatible A-B test harness that automates clean-slate Docker Compose restarts, load generation, and comparison reports; a 30-minute sustained load orchestrator with automated Grafana dashboard screenshot capture; and a K8s TPS sweep script that tests across multiple concurrency levels.
Finding Bottlenecks with Flame Graphs
The first real surprise came from async-profiler. We ran a CPU flame graph on order-service at 25 TPS and found this:
| Subsystem | CPU % |
|---|---|
| Hazelcast (IMap ops, Jet, serialization) | 52% |
| Spring/Tomcat HTTP | 32% |
| Outbox polling | 24% |
| Compact serialization | 10% |
| Jet pipeline | 9% |
The outbox publisher — the component that reads pending outbox entries and publishes them to ITopic — was consuming nearly a quarter of all CPU. A background scheduled task. It doesn’t appear in HTTP latency metrics at all.
Why? It was calling IMap.values(predicate) on a map with no indexes, forcing a full partition scan and deserialization of every entry on every poll cycle.
The fix was three changes:
- Add a HASH index on status and a SORTED index on createdAt
- Switch from values(predicate) to a PagingPredicate with server-side sort and page limit
- Use keySet(predicate).size() instead of values(predicate).size() for count queries
After the fix:
Before: Outbox polling = 24% CPU, 28% allocations After: Outbox polling = 12% CPU, allocations -34%
CPU halved. Total allocations dropped by a third. And no new hotspot emerged — the post-optimization profile showed a healthy distribution across Hazelcast internals, Spring HTTP handling, and Jet pipeline processing.
Without the flame graph, I probably would have guessed that serialization or Jet was the bottleneck. Would have been wrong.
Measuring What Matters: JMH Micro-Benchmarks
After identifying system-level hotspots, we wanted to know the per-operation cost of framework internals. Twenty JMH benchmark methods, measuring every operation in the event processing pipeline:
| Operation | Cost (us) | % of Total |
|---|---|---|
| FlakeIdGenerator.newId() | 15.6 | 42% |
| EventStore.append (IMap.set) | 11.2 | 30% |
| ViewStore.executeOnKey | 9.3 | 25% |
| Event.toGenericRecord (14 fields) | 1.1 | 3% |
| Total per-event | ~37 |
Total per-event framework cost: 37 microseconds. Pipeline end-to-end p50: 8.7 milliseconds. Framework internals account for less than 0.5% of pipeline latency. The other 99.5% is system overhead — Jet scheduling, thread context switching, HTTP processing, cross-service saga coordination.
The surprise was FlakeIdGenerator. At 15.6 us per call, it’s the single most expensive operation — more than the IMap write it precedes. And it scales poorly under contention: 4 threads = 69 us (4.4x slower), 8 threads = 151 us (9.7x). The batch allocation mechanism (default 100 IDs per batch) amortizes this, but at high concurrency it becomes a real bottleneck. Fix: increase setPrefetchCount() for high-throughput services.
We also benchmarked the shared 3-node Hazelcast cluster. Network operations cost 270-340 us each — about 30-40x more than the equivalent embedded operation. A full saga cycle (8 cluster operations) costs ~2.4 ms, or 28% of pipeline latency. The remaining 72% is Jet + HTTP + thread coordination.
Hazelcast is a significant cost contributor at ~28%, but it’s not the bottleneck. The optimization opportunities are in system-level overhead, not in Hazelcast or framework code.
The Enterprise Edition Question
We built a reusable A-B testing harness and ran three head-to-head comparisons at 50 TPS:
| Test | Verdict |
|---|---|
| Community vs HD Memory | Community wins 4/4 p95 metrics. HD Memory adds serialization overhead crossing the JVM heap boundary. At small scale (~200 entries), the per-access cost outweighs GC savings. |
| Community vs TPC | TPC wins 3/4 p95 metrics but by only 0.5-1.1ms. 14% lower max latency. Marginal at this scale. |
| Community vs High Memory (512→1024M) | No meaningful difference. Services aren’t memory-constrained at 50 TPS for 3 minutes. |
The honest answer: at 50 TPS with a few hundred entries, Enterprise features don’t help. HD Memory’s native off-heap storage adds measurable per-access overhead that only pays off when GC pressure from millions of on-heap entries causes latency spikes. TPC’s event-loop architecture shows promise for higher concurrency but is marginal at this scale.
Enterprise Edition becomes valuable at much larger scale — millions of entries for HD Memory, hundreds of TPS per service for TPC, or when you need CP Subsystem for distributed locking (though that doesn’t apply in our dual-instance architecture). We’re not there yet with this demo workload.
Scaling to Production: Docker Compose to AWS EKS
We tested four deployment tiers across six TPS levels (10 to 500):
| Tier | Nodes | Max TPS | Saga p95 | Cost/hr |
|---|---|---|---|---|
| Local (Docker Desktop) | 1 | 50 | < 300ms | $0 |
| AWS Small (2x t3.xlarge) | 2 | 100 | < 650ms | $0.76 |
| AWS Medium (3x c7i.2xlarge) | 3 | 200* | Timeout* | $1.18 |
| AWS Large (5x c7i.4xlarge) | 5 | 200 | < 1s | $3.50 |
*Without per-service clustering.
The Medium tier told a puzzling story. HTTP request latency was excellent — sub-120ms p95 — and HPA scaled services to 3 replicas. But saga end-to-end completion hit the 10-second polling timeout at 50+ TPS. HPA was scaling pods, but the new replicas weren’t helping.
We spent a while staring at this before the root cause clicked: each embedded HazelcastInstance was running as a cluster-of-1. Kubernetes load-balanced saga poll requests across replicas, but only one pod owned the saga state IMap. About 50% of polls missed — they hit a replica that didn’t have the data.
The fix was per-service embedded clustering (ADR 013): same-service replicas form their own Hazelcast cluster via Kubernetes DNS discovery. Different service types remain isolated.
The results:
| Metric | Without Clustering | With Clustering |
|---|---|---|
| Saga E2E p95 (10 TPS) | 10,161ms (timeout) | 513ms |
| Saga E2E p95 (50 TPS) | 10,197ms (timeout) | 644ms |
| 200 TPS sub-1s saga | Impossible | Yes |
| Service CPU at 500 TPS | 9-10% (idle) | 86-98% (utilized) |
| HPA replicas (25+ TPS) | Stuck at base | 2 → 5 |
A 95% improvement in saga latency. Larger than all code optimizations combined. The clustering change didn’t make any individual operation faster — it made the entire system’s resources actually usable. The CPUs that were sitting at 9% utilization were now doing real work.
Sustained Load: What Breaks After 30 Minutes
Short load tests — 3 minutes — miss the slow killers. We ran a 30-minute sustained test at 50 TPS and found the problems that only surface over time:
Order-service hit the 512Mi memory ceiling at 12 minutes and ran under continuous GC pressure for the remaining 18 minutes. CPU spiked from ~30% to 60% at the ceiling — GC thrashing. Max latency spikes of 5.6 seconds, almost certainly Full GC pauses. Remarkably, no OOM kills. The JVM GC kept services alive. Barely.
All four services showed the same pattern: rapid memory growth, hit ceiling, GC-capped plateau. The root cause: IMap data grows monotonically (event store, views, completions) with EvictionPolicy.NONE.
We bumped container memory from 512Mi to 1Gi (JVM heap 512m → 768m), added MaxSizePolicy.PER_NODE with 50K entries as a safety net on the completions map, and — when PostgreSQL persistence is active — LRU eviction turns IMaps into bounded hot caches. After the fixes, services run comfortably at 50 TPS for 30+ minutes without approaching the memory ceiling.
What We Learned
Always profile before optimizing. The #1 CPU hotspot was a background task that doesn’t appear in HTTP latency metrics. Without flame graphs, we would have optimized the wrong thing.
Framework code is less than 1% of latency. Total per-event cost of serialization, IMap writes, and FlakeIdGenerator: ~37 us. Pipeline p50: 8.7 ms. System overhead dominates by 200x.
Architecture beats optimization. Per-service clustering improved saga latency by 95% — more than all code optimizations combined. When your code is fast but your system is slow, the answer is usually architectural.
constant-arrival-rate is non-negotiable for load testing. The difference between it and constant-vus is the difference between measuring real latency and measuring a fiction.
Enterprise features have a scale threshold. HD Memory and TPC don’t help at 50 TPS with 200 entries. They’re designed for datasets and concurrency levels 10-100x larger than our demo.
Memory limits need sustained testing. A 3-minute test at 512Mi looks fine. A 30-minute test reveals services running at 99.9% memory under GC pressure. Always test for the duration you’ll actually run.
Measure at every layer. HTTP latency, pipeline latency, saga latency, and per-operation cost tell different stories. Saga E2E can time out at 10s+ while HTTP p95 is 30ms. You need all four layers to understand what’s happening.
Tools and Resources
| Tool | Purpose | Location |
|---|---|---|
| k6 load tests | HTTP load generation at constant arrival rate | scripts/perf/ |
| async-profiler | CPU and allocation flame graphs | docker/profiling/ |
| JMH benchmarks | Micro-benchmark framework internals | framework-core/src/jmh/ |
| A-B test harness | Automated configuration comparison | scripts/perf/ab-test.sh |
| Grafana dashboards | 6 pre-built dashboards (46 metrics) | docker/grafana/dashboards/ |
| K8s perf test | Multi-tier TPS sweep | scripts/perf/k8s-perf-test.sh |
| Performance tuning guide | Full adopter guide | docs/guides/performance-tuning-guide.md |
| Production checklist | Quick-reference for deployment | docs/perf/production-checklist.md |
Every one of the findings in this post was invisible without measurement. The outbox was burning a quarter of the CPU. The clustering was the real bottleneck. The memory was quietly filling up. Enterprise features we expected to matter didn’t. That’s the point of performance engineering — your intuition about what’s slow is usually wrong. Measure first.
Previous: Event Sourcing Observability: Prometheus, Grafana, Jaeger
Nodes and Edges 