Tag: CQRS

  • Baseball Invented Event Sourcing 150 Years Ago

    I’ve spent the last few months building event-sourced microservices on the Hazelcast platform — immutable event logs, materialized views, CQRS, the whole stack. If you’ve been following along, you’ve seen the why and the how. Event Sourcing is elegant. It’s powerful. And it was invented in the 1870s.

    Not by a software engineer. By a baseball scorekeeper.

    The Scorecard Is an Event Log

    Here’s the thing about keeping score at a baseball game. You don’t write down “the score is 3-2.” You write down what happened: Acuna singled to left. Olson doubled to right-center, Acuna to third. Riley grounded to short, Acuna scored, Olson to third. Play by play, pitch by pitch, every event appended in order to the record.

    Sound familiar? It should. That’s an event log. Immutable, append-only, temporally ordered. The scorekeeper’s pencil is the world’s oldest event producer.

    And here’s the part that made me sit up straight the first time I really thought about it: the scorekeeper never records the score directly. The score — along with batting averages, on-base percentages, pitcher stat lines, and everything else you see on the broadcast — is derived by replaying those events. You don’t store state. You compute it.

    If you read the Hazelcast framework piece, you’ll recognize this immediately. We spent considerable effort setting up event stores and materialized views as separate concerns — events flow in, projections flow out. The scorekeeper has been doing this with a pencil and a printed card since before the lightbulb was invented.

    Materialized Views Are Everywhere at the Ballpark

    Once you see this pattern, you can’t unsee it. Every summary artifact at a baseball game is a materialized view — a read-optimized projection of the same underlying event stream.

    The box score is a materialized view. It takes the full play-by-play event log and projects it into a compact batting line per player: at-bats, runs, hits, RBI. A completely different shape than the source data, optimized for a specific read pattern.

    The line score is a materialized view. Same events, different projection — runs per inning instead of runs per player.

    The pitcher’s stat line is a materialized view. IP, hits, walks, strikeouts, earned runs — all derived from the same at-bat events, filtered by which pitcher was on the mound.

    The standings page is a materialized view built from an even larger event stream — every game’s final result across the entire season.

    This is CQRS in its purest form. The write side is the scorekeeper recording events. The read side is every statistical summary, broadcast graphic, and newspaper box score derived from those events. Multiple consumers, each projecting the same stream into the shape they need.

    Where It Gets Really Interesting

    The conceptual alignment goes deeper than “it’s a log and you read from it.” The hard problems are the same too.

    Temporal queries. “What was the score after five innings?” In an event-sourced system, you replay events up to a point in time. In baseball, you read the line score up to the fifth column. Same operation. A scorekeeper can reconstruct the exact game state — who was on base, how many outs, what the count was — at any point in the game by replaying events to that moment.

    Corrections and compensating events. The official scorer initially rules a play an error, then changes it to a hit. The original event isn’t erased — it’s annotated, and the downstream materialized views (batting averages, fielding percentages) recompute. If you’ve ever dealt with event versioning or compensating events in a production system, this should feel very familiar. The scorer is solving the same problem you are, just with a pencil eraser instead of an event migration framework.

    MLB’s replay challenge system makes this even more explicit. The umpire calls ball three — that’s an event. The catcher challenges — that’s a compensating event. The replay review overturns the call to strike three — that’s the resolution event. You don’t go back and pretend the original call never happened. You record the whole sequence: the call, the challenge, the outcome. The materialized views — the count, the at-bat result, the pitcher’s stat line — all reflect the final state, but the event log preserves the full history of how you got there. That’s not an analogy for event sourcing. That is event sourcing.

    Multiple independent consumers. The home team’s scorer, the visiting team’s scorer, the press box statistician, and the broadcast team are all consuming the same stream of game events and maintaining their own projections. They might format them differently, update at slightly different times, even disagree on a ruling until the official scorer weighs in. Independent materialized views with eventual consistency.

    Eventual consistency itself. The press box stats might lag a play behind the action. The official scorer’s ruling on a borderline hit-vs-error might not come until minutes after the play — sometimes not until after the game. During that window, different consumers have different projections of the truth. The system is eventually consistent. Always has been. Nobody panics about it because the mental model is intuitive: the ruling will come, the stats will update, and the final record will be correct.

    Where the Analogy Breaks Down

    Now, I’m not going to pretend this maps perfectly. The places where it doesn’t map are actually instructive — they highlight what makes distributed event sourcing genuinely hard.

    Ordering is free in baseball. One batter at a time. One pitch at a time. Events are naturally, globally ordered with no effort. In a distributed system, you fight for ordering. You need Lamport clocks, vector clocks, consensus protocols, or a centralized sequencer. Baseball gets this for free because the game is inherently single-threaded. Your microservices are not.

    Single source of truth. Baseball has an official scorer — one authoritative human who makes the final call. Distributed systems don’t get a benevolent dictator. They have to negotiate consensus across nodes that might disagree, might be partitioned, might be lying. The official scorer never has a network partition. (Though I’ve seen some calls that made me wonder.)

    Bounded, finite streams. A baseball game ends. The event stream is finite, relatively small, and complete. Production event stores grow without bound, need compaction strategies, deal with schema evolution over years. A scorekeeper’s biggest scaling challenge is a 19-inning game. Yours is a few billion events with a 5-year retention policy.

    These gaps are worth noting because they’re exactly the problems that make event sourcing hard to implement in software. Baseball had the luxury of solving the easy version first — sequential, single-authority, bounded. We got the distributed, multi-authority, unbounded version. Lucky us.

    So I Built an App

    This pattern recognition wasn’t just an intellectual exercise. I’ve been building BaseballScorer, an iOS app for scoring baseball games by hand — the way scorekeepers have always done it, but on a tablet instead of paper.

    The app’s data model is, naturally, event-sourced. Pitches, plays, and substitutions are the events. The score, the scorecard, the line score, the inning summaries — all materialized views, derived by replaying the event stream. It’s the same architecture I’ve been writing about in the Hazelcast series, just running on a single iPad instead of a distributed cluster.

    It’s available on the App Store now. If you’re the kind of person who keeps score at games — or the kind of person who’s curious why anyone would — I think you’ll find it interesting.

    And if you want to see what event sourcing looks like when you don’t have the luxury of a single-threaded, globally-ordered event stream, check out the Hazelcast microservices framework. Same patterns, much harder problem.

    This is part of an ongoing series on event-driven microservices at Nodes and Edges. Previous posts: Why Event-Driven Microservices? and The Hazelcast Microservices Framework.

  • Launching a Claude Code Project: Design First, Then Build

    Launching a Claude Code Project: Design First, Then Build

    I used Claude’s desktop interface for iterative design, then handed off to Claude Code for implementation.

    After deciding to revive my Hazelcast Microservices Framework (MSF) project, and to do so using Claude AI to do much of the heavy lifting, it came down to figuring out how to actually do this. I had no playbook for it. Nobody does, really — we’re all making this up as we go.

    I wanted to be transparent about my use of Claude, and at the same time I think the development process is interesting enough to be worthy of discussion. (Heck, maybe it’s more interesting than the framework blog posts I set out to write.) So I expect to end up with a dual series of blog posts: the framework posts — started by Claude, co-edited together, and given a final polish by me — interleaved with my observations on how the collaboration effort worked.

    This first “behind the scenes” post covers the design phase: going from a vague idea to a set of design documents and an implementation plan, all before writing a single line of code.

    Starting the Conversation

    Here was my original prompt to Claude:

    I want to use Claude Code to help me finish a demonstration project I started some time ago to show how to implement microservices using Hazelcast. (The main value of Hazelcast is to create materialized views of domain objects to maintain in-memory current state.) If it’s more effective, we can restart with a blank sheet rather than modify the existing project.

    I’d really like to iterate over the design several times before any coding starts — is that best done in Claude Code, or using this desktop interface? Ideally, creating various specifications or design documents before any coding starts would be perfect, if Claude can use these various documents as a guide to the coding process.

    How do we start?

    Claude immediately suggested splitting the work across two interfaces: use the desktop/web interface for design discussions and document creation, then move to Claude Code for implementation. Made sense to me — the conversational interface is better for back-and-forth design iteration, while Claude Code excels at multi-file code generation with direct access to the project directory.

    This turned out to be excellent advice. The design phase involved a lot of “what about this?” and “actually, let’s reorganize that” — the kind of exploratory conversation that works much better in a chat interface than in a code-focused tool. I tried doing some design work in Claude Code early on and it was noticeably worse — like trying to brainstorm on a whiteboard that keeps trying to compile your diagrams.

    The Design Phase: A Roadmap in Nine Documents

    What followed was an extended design conversation that produced nine documents over the course of a single session. I’m not going to walk through every one in detail — you can follow the links if you’re curious — but a few of them are worth talking about because of what they reveal about the collaboration process.

    Getting Started: Template and Domain

    Claude’s first move was to produce a comprehensive design document template covering everything from executive summary to demonstration scenarios. We never actually completed it — the conversation quickly moved in a more specific direction — but it served its purpose as a structural starting point. The architectural equivalent of a napkin sketch: useful for getting the conversation going, not meant to survive contact with reality.

    Before we could fill in any template, though, we needed to pick a domain for the demonstration. Claude laid out a comparison between eCommerce and Financial Services, and we settled on a hybrid approach: start with eCommerce (universally understood, clear event flows, and I had existing code to reference) but design the framework to be domain-agnostic so other domains could be plugged in later. We also simplified from four services down to three: Account, Inventory, and Order. (A fourth service, Payment, showed up later when we built out the saga patterns. Scope creep, but the useful kind.)

    That decision led to the eCommerce design document — a detailed Phase 1 design covering all three services, their APIs, events, and materialized views. Three view patterns came out of it: denormalized views (joining customer, product, and order data), aggregation views (pre-computing order statistics), and real-time status views (current inventory levels). If you’ve read the previous posts in this series, you’ll recognize these as exactly the kind of thing that makes Event Sourcing + CQRS worth the effort.

    Where I Pushed Back

    The conversation then turned to longer-term goals. I had ideas for observability dashboards, microbenchmarking, pluggable implementations, saga patterns, and more — far beyond what could fit in a Phase 1. Claude organized all of this into a phased requirements document spanning five phases.

    We iterated over this several times, adding and reorganizing. The most significant change I made was moving Event Sourcing from Phase 2 to Phase 1. Claude had initially positioned it as an advanced feature, but I saw it as the fundamental organizing principle of the entire framework — events are the source of truth, not database rows. Once I explained my existing Hazelcast Jet pipeline architecture (where handleEvent() writes to a PendingEvents map, which triggers a Jet pipeline that persists to the EventStore, updates materialized views, and publishes to the event bus), Claude immediately agreed and restructured the phases accordingly.

    This was one of the more interesting moments in the collaboration. Claude had made a reasonable default assumption about complexity ordering, but I had domain-specific knowledge about how the architecture should actually work. The back-and-forth was natural — I explained my reasoning, Claude incorporated it, and the result was better for it. If I’d just accepted the initial phasing without pushing back, the entire project would have been organized around a less coherent architecture. And honestly, I almost did just accept it. It looked reasonable. Sometimes the most important contribution you make is going “wait, actually…” when the first answer seems fine.

    Other additions during this iteration:

    • Vector Store integration (Phase 3, optional) for product similarity search
    • An MCP Server (Phase 3) to let AI assistants query and operate the system
    • Open source mandate — everything in Phases 1-2 must run on Hazelcast Community Edition
    • Blog post series structure — features developed in blog-post-sized chunks

    Architecture, Code Review, and the Rewrite Decision

    The next few documents came quickly. The Event Sourcing discussion led to a dedicated architecture document detailing the Jet pipeline design — based heavily on my existing implementation, but now formally documented with all six pipeline stages, the EventStore design, and how event replay would work.

    Then I uploaded several key source files from the original project for Claude to review: the EventSourcingController, DomainObject, SourcedEvent (later renamed to DomainEvent), EventStore, and EventSourcingPipeline. Claude produced a thorough code review comparing the existing code against the design documents. The verdict was encouraging — the core implementation was solid and matched the Phase 1 design almost perfectly. Claude recommended incremental enhancement: add correlation IDs, framework abstractions, observability, and tests on top of what was already there.

    I went the other way. After thinking about the package naming, dependency versions, and scope of changes needed, I decided on a clean reimplementation using the existing code as a blueprint. This let us start with the right project structure, package names (com.theyawns.framework.*), and dependency versions (Spring Boot 3.2.x, Hazelcast 5.6.0) from the beginning rather than refactoring them in later. Sometimes — as I’d noted in the previous post — the right move is to stop patching the old cabinets and start fresh.

    I won’t pretend this was a purely rational decision. Part of it was just wanting that clean-slate feeling — new project, new structure, no legacy cruft staring at me from the imports. Developers love a greenfield. We can’t help it.

    The Implementation Plan

    Once the architecture was validated and we’d agreed on the approach, Claude created a detailed Phase 1 implementation plan — a three-week, day-by-day schedule with code templates, success criteria, and task checklists:

    • Week 1: Framework core — Maven multi-module setup, core abstractions, event sourcing controller, Jet pipeline
    • Week 2: Three eCommerce services — Account, Inventory, Order with REST APIs and materialized views
    • Week 3: Integration, Docker Compose, documentation, demo scenarios

    We made a few tweaks (updating Hazelcast from 5.4.0 to 5.6.0, for instance), and then it was time to move to code.

    The Handoff to Claude Code

    Claude provided specific instructions for transitioning to Claude Code, including a context block to paste when starting the session:

    I'm building a Hazelcast-based event sourcing microservices framework.

Project location: hazelcast-microservices-framework/
Current state: Design documents complete, ready for implementation

Key decisions:
- Clean reimplementation (no existing code to port)
- Spring Boot 3.2.x + Hazelcast 5.6.0 Community Edition
- Package: com.theyawns.framework.*
- Three services: Account, Inventory, Order (eCommerce domain)
- Event sourcing with Hazelcast Jet pipeline
- REST APIs only

Implementation plan: docs/implementation/phase1-implementation-plan.md

Starting with Day 1: Maven project setup + core abstractions

Please read the implementation plan and let's begin.

    The whole point of the “design first” approach: you’re not asking the AI to guess at your architecture. You’re handing it a blueprint. The more detailed the blueprint, the less time you spend arguing about load-bearing walls later.

    Documents 7-9: Claude Code Configuration

    Before making the jump, I asked Claude about setup suggestions for Claude Code. This produced three more documents:

    CLAUDE.md (originally called .clinerules — I’m still not sure where that name came from) is the main configuration file that Claude Code reads automatically. It defines code standards, patterns, pitfalls to avoid, and documentation requirements. This file evolved a lot over the course of the project; looking at the commit history gives a good sense of how the “rules” grew and adapted as we ran into new situations. (More on that in a future post — it turned out to be one of the more interesting aspects of the whole process.)

    claude-code-agents.md defined eight specialized agent personas — Framework Developer, Service Developer, Test Writer, Documentation Writer, Pipeline Specialist, and others — each with specific rules, code patterns, and checklists. The idea was to switch between personas depending on the task at hand (e.g., “Switch to Test Writer agent. Write comprehensive tests for EventSourcingController.”). Whether this actually helped or was just a placebo is something I’m still not sure about, honestly.

    A docs organization guide rounded out the set, providing a recommended directory structure for keeping all the documentation organized as the project grew.

    What Came Next

    The resulting project grew well beyond the original three-week Phase 1 plan. At 150 commits, it now includes four microservices (Payment was added for saga demonstrations), an API Gateway, an MCP server for AI integration, choreographed and orchestrated saga patterns, PostgreSQL persistence, Grafana dashboards, and more. The three-week plan took considerably longer than three weeks. So it goes.

    But all of that implementation work — and the interesting stories about how human-AI collaboration played out during coding — is material for future posts.

    What I’d Do Differently (And What I’d Do Again)

    If you’re thinking about using AI for a non-trivial coding project, here’s what I took away from the design phase.

    Use the right tool for each phase. The conversational interface is great for the messy, exploratory work of figuring out what you’re actually building. Claude Code is great for building it.

    Iterate on design before you write code. We went through multiple rounds of revision on the requirements and architecture documents. Each round caught issues or surfaced priorities (like Event Sourcing belonging in Phase 1) that would have been much more expensive to discover during implementation. Measure twice, cut once. The carpenter’s rule exists for a reason.

    Bring your domain knowledge — and don’t be shy about pushing back. Claude made strong default recommendations, but the most valuable moments came when I disagreed based on my understanding of Hazelcast and the architecture I wanted. The AI is a powerful collaborator, but it doesn’t know what you know. If something feels wrong, say so. That’s where the real value of the collaboration happens.

    And document everything. I mean it. The design documents weren’t just planning artifacts — they became living reference material that Claude Code used throughout implementation. The CLAUDE.md file in particular became a continuously evolving guide that shaped code quality across the entire project. Every hour spent on documentation saved multiples in “no, that’s not what I meant” corrections later. I’ve never been great about documentation discipline, so having an AI that actually reads and follows the docs was a surprisingly effective motivator to keep them current.

    The Hazelcast Microservices Framework is open source under the Apache 2.0 license. You can find it at github.com/myawnhc/hazelcast-microservices-framework.

    Next up: what happened when we actually started coding. Spoiler: the plan did not survive intact.

  • The Hazelcast Microservices Framework

    How a side project connecting Event Sourcing to Hazelcast sat unfinished for years — and why I decided to bring it back with an AI collaborator.

    In my previous post, I shared some of my thinking about Event-Driven Microservices — the coupling problems, the mental shift toward thinking in events, and the patterns (Event Sourcing, CQRS, materialized views) that make it all work. That post was conceptual. This one is personal.

    I’ve been playing around with design concepts in this area for some time. While I was an employee of Hazelcast, I frequently worked with customers and prospects to show how Hazelcast Jet — an event stream processing engine built into the Hazelcast platform — could be used to build event processing solutions that would scale while continuing to provide low latency. These conversations were always framed around stream processing, though. Even when the intended use case was around microservices, we didn’t explicitly get into the Event Sourcing pattern. As someone coming from a background that was database-centric, the concept of events as the source of truth was a bit much for me.

    The Light Bulb Moment

    It was a light bulb moment when I realized that Hazelcast Jet could fit naturally into an Event Sourcing architecture — and that Hazelcast IMDG (the in-memory data grid, or caching layer) could concurrently maintain materialized views representing the current state of domain objects.

    Think about it: Event Sourcing needs an event log and a processing pipeline. Hazelcast Jet is a processing pipeline. CQRS needs a fast read-side store that’s kept in sync with the event stream. Hazelcast IMDG is a fast read-side store. Event Sourcing + CQRS maps beautifully onto Jet + IMDG (even though that acronym is officially retired — it’s all just “Hazelcast” now).

    And from there, I really wanted to demonstrate this. The original Microservices Framework project began.

    Version 1: The Proof of Concept

    The first version was focused on proving the core idea worked. Could I wire up a Hazelcast Jet pipeline to process domain events, persist them to an event store, and update materialized views — all in a way that was generic enough to work across different services?

    The answer was yes. The central pattern that emerged was straightforward: a service’s handleEvent() method writes incoming events to a PendingEvents map, which triggers a Jet pipeline that persists events to the EventStore, updates materialized views, and publishes to an event bus for other services to consume. It worked, and it was fast.

    Now, the central components of the architecture — the domain object, event class, controller, and pipeline — have survived relatively intact through multiple iterations of the implementation. The bones were good. But a lot of the specific implementation choices I made around those bones haven’t aged all that well.

    You know how it goes with side projects. Technical debt accumulates quietly, one “I’ll fix this later” at a time, until you’re looking at a codebase where you know you’d make different choices if you were starting over — but the sunk cost of time already invested keeps you from actually doing it. It’s the software equivalent of a kitchen renovation where you keep patching the old cabinets because ripping them out feels like too big a project for a weekend.

    That version of the framework is still hanging around on GitHub, although I decided not to link to it here as I may take it down at any time. (Upcoming posts will link to the improved version, so embedding links to the original will inevitably lead to someone grabbing the wrong one.)

    I got it to a working state, but there was a long list of things I wanted to add. Saga patterns for coordinating multi-service transactions. Observability dashboards. Comprehensive tests. Documentation that went beyond “read the code.” Each of these was a meaningful chunk of work, and progress slowed to a crawl.

    The Stall

    Let’s be honest about what happened: the project stalled. Not dramatically — it wasn’t ever really abandoned. It just… stopped moving. Every few months I’d open the codebase, when I had some extra time, and make a few minor, inconsequential changes while thinking of the more ambitious refactorings or added features that I’d get to when time permitted.

    If you’ve ever maintained a passion project alongside a day job, you know this feeling. The ideas don’t go away — they sit in the back of your mind, periodically surfacing with a pang of “I should really get back to that.” But the activation energy to restart is high, especially when the next step isn’t a fun new feature but the grind of scaffolding, configuration, and test coverage. So you close the laptop and tell yourself next month will be different. (It won’t be.)

    Enter AI-Assisted Development

    In early 2025, I started using Claude for various coding tasks and was genuinely surprised by the results. This wasn’t autocomplete on steroids — I could describe an architectural pattern and get back code that understood the why, not just the what. I could say “this needs to work like an event journal with replay capability” and get something that actually accounted for ordering guarantees and idempotency.

    That’s when the thought crystallized: what if I could use this to break through the stall?

    Here’s the thing — the stuff that had been blocking me wasn’t the hard design work. I knew what the architecture should look like. The bottleneck was the sheer volume of implementation grind: scaffolding new services, writing comprehensive tests, wiring up Docker configurations, producing documentation. Exactly the kind of work where you need focused hours, and a side project never has enough of those.

    Now, I want to be clear about what I mean here, because “AI wrote my code” carries a lot of baggage. This wasn’t about handing off the project and checking back in when it was done. It was about having a collaborator who could take high-level design direction and turn it into working code at a pace that made the project viable again. I’d provide the domain expertise, the architectural decisions, and the quality bar. The AI would provide the throughput.

    Making the Decision

    I decided to move forward with a clean reimplementation rather than trying to evolve the existing codebase. The core patterns from the original work — the Jet pipeline architecture, the event store design, the materialized view update strategy — were proven and would carry forward. But the project structure, package naming, dependency versions, and framework abstractions would start fresh. Sometimes the best way to fix a kitchen is to actually rip out the cabinets.

    The plan was to use Claude’s desktop interface for iterative design discussions (requirements, architecture, implementation planning) and then hand off to Claude Code for the actual coding. Design first, then build — with comprehensive documentation at every step so the AI would have rich context to work from.

    What happened next — the design phase, the handoff to Claude Code, and the surprises along the way — is the subject of the next post.