A short interstitial in the “Building Event-Driven Microservices with Hazelcast” series
Measurement is better than guessing. Who knew?
When the saga implementation was first finished and we ran through the test scenarios for the first time, there was a high incidence of saga timeouts if we ran for more than a few minutes. (5 minutes was great; 30 minutes was ugly.)
I didn’t ask Claude to investigate, or do any analysis of my own, because I had a pretty good suspicion what was going on. Everything was running on a single laptop — 16GB of memory split across a 3-node Hazelcast cluster, 4 services each running an embedded Hazelcast node, Docker itself, and I really hadn’t bothered to shut off my normal desktop workload. Web browser, email, whatever else. I figured I’d maxed the poor thing out and probably wouldn’t get a clean timeout-free run until I deployed to a multi-node cluster in the cloud.
That didn’t happen for some time. I’m cheap, and I wasn’t going to pay for cloud resources until I had a full-blown demo ready to go. When the day came — much later in the story if I was telling it chronologically — I saw the same pattern of timeouts. Turns out, it was never thread starvation or lack of resources. It was a combination of things, and none of them were what I’d assumed.
When faced with this reality, I asked Claude to troubleshoot the issue, and this is one of the times I was most impressed with how Claude approached a problem compared to how I would have.
In most debugging scenarios, I look only until I find the first reasonable suspect. Why keep looking if you’ve already found what you’re looking for? Fix, rebuild, retest, and on a good day, that’s the end of it. On a bad day, you’re still looking at the same issue, so you start hunting for suspect #2. Lather, rinse, repeat.
Claude came back with four identified problems. The main one was subtle: we generated product data up front and gave each item a reasonable starting stock. As the demo ran, orders depleted the stock, and eventually the inventory service started throwing InsufficientStockException — correct behavior, you can’t sell what you don’t have. But the circuit breaker we’d added for resilience was treating that business error the same as an infrastructure failure. Enough “failures” in the sliding window and the circuit breaker tripped open, rejecting all orders — including ones for products that still had stock. Sagas piled up with nowhere to go, the timeout detector found hundreds of them every cycle, and the system drowned in compensation events. At the peak: 64,000 timeouts from 53,000 sagas started.
The other three fixes addressed related gaps. Business failures like out-of-stock now trigger immediate saga compensation instead of waiting for the timeout detector to notice. A NonRetryableException marker interface tells the circuit breaker not to count deterministic business errors against the failure rate. And an automatic stock replenishment monitor keeps the demo in a steady state where orders can actually succeed for hours instead of wedging after the first few minutes.
I should have investigated the saga timeouts when they first appeared, rather than assuming the problem would magically go away with more hardware. And when I did get around to investigating, Claude’s approach of identifying all the contributing problems at once was considerably more effective than my usual one-suspect-at-a-time strategy.
Nodes and Edges 