Category: Microservices

  • Hazelcast Microservices Framework: Event Sourcing Demo

    How a side project connecting Event Sourcing to Hazelcast sat unfinished for years — and why I decided to bring it back with an AI collaborator.

    In my previous post, I shared some of my thinking about Event-Driven Microservices — the coupling problems, the mental shift toward thinking in events, and the patterns (Event Sourcing, CQRS, materialized views) that make it all work. That post was conceptual. This one is personal.

    I’ve been playing around with design concepts in this area for some time. While I was an employee of Hazelcast, I frequently worked with customers and prospects to show how Hazelcast Jet — an event stream processing engine built into the Hazelcast platform — could be used to build event processing solutions that would scale while continuing to provide low latency. These conversations were always framed around stream processing, though. Even when the intended use case was around microservices, we didn’t explicitly get into the Event Sourcing pattern. As someone coming from a background that was database-centric, the concept of events as the source of truth was a bit much for me.

    The Light Bulb Moment

    It was a light bulb moment when I realized that Hazelcast Jet could fit naturally into an Event Sourcing architecture — and that Hazelcast IMDG (the in-memory data grid, or caching layer) could concurrently maintain materialized views representing the current state of domain objects.

    Think about it: Event Sourcing needs an event log and a processing pipeline. Hazelcast Jet is a processing pipeline. CQRS needs a fast read-side store that’s kept in sync with the event stream. Hazelcast IMDG is a fast read-side store. Event Sourcing + CQRS maps beautifully onto Jet + IMDG (even though that acronym is officially retired — it’s all just “Hazelcast” now).

    And from there, I really wanted to demonstrate this. The original Microservices Framework project began.

    Version 1: The Proof of Concept

    The first version was focused on proving the core idea worked. Could I wire up a Hazelcast Jet pipeline to process domain events, persist them to an event store, and update materialized views — all in a way that was generic enough to work across different services?

    The answer was yes. The central pattern that emerged was straightforward: a service’s handleEvent() method writes incoming events to a PendingEvents map, which triggers a Jet pipeline that persists events to the EventStore, updates materialized views, and publishes to an event bus for other services to consume. It worked, and it was fast.

    Now, the central components of the architecture — the domain object, event class, controller, and pipeline — have survived relatively intact through multiple iterations of the implementation. The bones were good. But a lot of the specific implementation choices I made around those bones haven’t aged all that well.

    You know how it goes with side projects. Technical debt accumulates quietly, one “I’ll fix this later” at a time, until you’re looking at a codebase where you know you’d make different choices if you were starting over — but the sunk cost of time already invested keeps you from actually doing it. It’s the software equivalent of a kitchen renovation where you keep patching the old cabinets because ripping them out feels like too big a project for a weekend.

    That version of the framework is still hanging around on GitHub, although I decided not to link to it here as I may take it down at any time. (Upcoming posts will link to the improved version, so embedding links to the original will inevitably lead to someone grabbing the wrong one.)

    I got it to a working state, but there was a long list of things I wanted to add. Saga patterns for coordinating multi-service transactions. Observability dashboards. Comprehensive tests. Documentation that went beyond “read the code.” Each of these was a meaningful chunk of work, and progress slowed to a crawl.

    The Stall

    Let’s be honest about what happened: the project stalled. Not dramatically — it wasn’t ever really abandoned. It just… stopped moving. Every few months I’d open the codebase, when I had some extra time, and make a few minor, inconsequential changes while thinking of the more ambitious refactorings or added features that I’d get to when time permitted.

    If you’ve ever maintained a passion project alongside a day job, you know this feeling. The ideas don’t go away — they sit in the back of your mind, periodically surfacing with a pang of “I should really get back to that.” But the activation energy to restart is high, especially when the next step isn’t a fun new feature but the grind of scaffolding, configuration, and test coverage. So you close the laptop and tell yourself next month will be different. (It won’t be.)

    Enter AI-Assisted Development

    In early 2025, I started using Claude for various coding tasks and was genuinely surprised by the results. This wasn’t autocomplete on steroids — I could describe an architectural pattern and get back code that understood the why, not just the what. I could say “this needs to work like an event journal with replay capability” and get something that actually accounted for ordering guarantees and idempotency.

    That’s when the thought crystallized: what if I could use this to break through the stall?

    Here’s the thing — the stuff that had been blocking me wasn’t the hard design work. I knew what the architecture should look like. The bottleneck was the sheer volume of implementation grind: scaffolding new services, writing comprehensive tests, wiring up Docker configurations, producing documentation. Exactly the kind of work where you need focused hours, and a side project never has enough of those.

    Now, I want to be clear about what I mean here, because “AI wrote my code” carries a lot of baggage. This wasn’t about handing off the project and checking back in when it was done. It was about having a collaborator who could take high-level design direction and turn it into working code at a pace that made the project viable again. I’d provide the domain expertise, the architectural decisions, and the quality bar. The AI would provide the throughput.

    Making the Decision

    I decided to move forward with a clean reimplementation rather than trying to evolve the existing codebase. The core patterns from the original work — the Jet pipeline architecture, the event store design, the materialized view update strategy — were proven and would carry forward. But the project structure, package naming, dependency versions, and framework abstractions would start fresh. Sometimes the best way to fix a kitchen is to actually rip out the cabinets.

    The plan was to use Claude’s desktop interface for iterative design discussions (requirements, architecture, implementation planning) and then hand off to Claude Code for the actual coding. Design first, then build — with comprehensive documentation at every step so the AI would have rich context to work from.

    What happened next — the design phase, the handoff to Claude Code, and the surprises along the way — is the subject of the next post.

    Code: github.com/myawnhc/hazelcast-microservices-framework — clone it, docker-compose up, and the framework boots locally with sample data.

  • Event-Driven Microservices: Avoiding Distributed Monoliths

    You’ve heard the pitch for microservices. Small, independent services. Teams that can ship without waiting on six other teams to finish their sprint. No more three-month release cycles because somebody touched a shared library. It sounds great — and honestly, the core idea is great. But here’s the thing: a lot of teams adopt microservices and end up with something worse than the monolith they started with.

    I’ve spent the most recent part of my career working with distributed systems, and I’ve seen some of the ways monolith-to-microservice transitions can go awry. A team takes their monolith, draws some boxes around the major modules, splits them into separate services, deploys them independently, and declares victory. Six months later they’re debugging cascading failures at 2 AM and wondering why everything is harder than it used to be.

    What went wrong? They broke the monolith apart without actually decoupling it. And a distributed monolith — where you have all the operational complexity of microservices with none of the benefits — is arguably the worst of both worlds.

    The Coupling Problem

    Let’s be specific about what tight coupling looks like in a microservices architecture, because it’s not always obvious.

    Synchronous request-response everywhere. Service A calls Service B, which calls Service C, which calls Service D. If any one of those services is slow or down, the whole chain stalls. You haven’t built a resilient distributed system — you’ve built a monolith with network hops. And network hops are the worst kind of function calls, because now you get to deal with latency, partial failure, and timeout tuning on top of everything else.

    Shared databases. Multiple services reading from and writing to the same tables. This is the one that sneaks up on people, because the database feels like shared infrastructure rather than a coupling point. But the moment you need to change a schema, you’re coordinating across every service that touches those tables. You’re right back to “deploy everything together or deploy nothing” — which is exactly what microservices were supposed to fix.

    Data format dependencies. Service A produces a message with a certain structure. Services B, C, and D all parse that structure. Now Service A needs to add a field or change a type. Congratulations, you need buy-in from three other teams before you can ship. That’s not independent deployment — that’s a distributed approval process.

    Temporal coupling. Services that have to be running simultaneously to function. If the downstream service isn’t up right now, the upstream service can’t do its job. Your services aren’t really independent if they can only work when everyone else is awake. (Kind of like a group project where one person has to be physically present for anyone else to make progress. We’ve all been in that group project.)

    If any of this sounds familiar, you’re not alone. And the good news is that these problems are well-understood, and there are well-established patterns for solving them.

    Thinking in Events

    Here’s the mental shift that makes the difference: stop thinking about services calling each other, and start thinking about services reacting to things that happen.

    This is event-driven architecture, and at its core it’s about making your software reflect how the real world actually works. The real world doesn’t operate on synchronous request-response. Things happen — a customer places an order, a sensor reads a temperature, a payment clears — and other parts of the system respond to those events on their own terms, at their own pace.

    When you build systems this way, something interesting happens to those coupling problems:

    Synchronous chains disappear. Service A publishes an event. It doesn’t know or care who’s listening. Services B, C, and D each pick up the event and do their thing independently. If Service C is having a bad day, Services A, B, and D don’t notice — they keep right on working.

    Data ownership becomes clear. Each service owns its data, publishes events about what changed, and subscribes to the events it cares about from others. No shared databases, no schema coordination nightmares.

    Temporal coupling goes away. If a service is down when an event is published, that event waits in the stream until the service recovers and processes it. The system degrades gracefully instead of falling over.

    Now, this isn’t magic — you’ve traded one set of challenges for a different set. Event-driven systems have their own complexities: eventual consistency, event ordering, debugging asynchronous flows. We’ll get into all of that. But at least these are the right problems to have — problems that come from genuinely decoupled services rather than from a distributed monolith pretending to be something it’s not.

    Patterns That Make It Work

    If you start exploring event-driven microservices, you’ll quickly run into a set of well-known patterns that have emerged to address the practical challenges. Chris Richardson’s microservices.io is an excellent catalog of these — I’d recommend bookmarking it.

    Two patterns in particular are going to be central to what we explore in this blog, and I’ll admit it took me a while to appreciate how well they fit together:

    Event Sourcing — instead of storing the current state of your data and updating it in place, you store the sequence of events that led to the current state. Every state change is captured as an immutable event in an append-only log. This gives you a complete, auditable history of everything that happened in your system — not just “the account balance is $500” but “here’s every deposit, withdrawal, and transfer that got it there.”

    If you come from a database background (guilty), this feels deeply wrong at first. You mean I don’t just UPDATE the row? I keep every change? Forever? But once you get past the initial discomfort, the power of it becomes obvious. You can reconstruct any past state. You can answer questions you didn’t think to ask when the data was created. You have a complete audit trail for free.

    The catch is also obvious — if you need the current state, do you really have to replay every event from the beginning of time? For a system that’s been running for years, that’s not just slow, it’s unworkable.

    CQRS (Command Query Responsibility Segregation) — and this is where it gets interesting. You separate the write path (commands that produce events) from the read path (queries that serve up current state). The write side stores events. The read side maintains materialized views — pre-computed projections of whatever the read side needs, kept up to date by consuming the event stream.

    See what happens when you put these two together? Event Sourcing gives you the complete, immutable history. CQRS and materialized views give you fast reads without replaying the entire event log every time someone wants to check a balance. Each pattern solves the other’s biggest problem. It’s one of those combinations where the whole is genuinely greater than the sum of the parts — and as we’ll see in later posts, it maps onto certain technology stacks almost embarrassingly well.

    What’s Ahead

    This blog is going to be a hands-on exploration of these ideas — patterns first, then concrete implementations. I’m genuinely excited about this, because I think there’s a gap between the theoretical literature on event-driven architecture (which is excellent) and the practical “here’s how you actually build one” content (which is thinner than you’d expect). In the posts to come, we’ll dig into:

    • Resilience through decoupling — how event-driven systems degrade gracefully instead of cascading failures
    • Auditability and replay — the power of an event log as a source of truth, not just for debugging but for compliance, analytics, and the ability to answer questions you didn’t think to ask yet
    • Independent scalability — scaling the services under load without scaling everything, because your order processing pipeline doesn’t need to drag your user profile service along for the ride
    • Evolvability — adding new consumers of existing events without touching the producers, so your analytics team can tap into a data stream without filing a ticket with the team that owns it

    We’ll look at the patterns in general terms — what problem each one solves, what trade-offs it introduces, how to think about whether it’s the right fit — and then we’ll get into specific, working implementations that you can pull apart, run, and adapt to your own projects.

    If you’re a developer who’s building or maintaining a microservices architecture and found it harder than expected — or if you’re designing a new system and want to avoid the common pitfalls — this series is for you. The patterns are universal; the implementations will be specific. Let’s see where it takes us.

    Code: github.com/myawnhc/hazelcast-microservices-framework — clone it, docker-compose up, and the framework boots locally with sample data.