Distributed Systems: Orchestration, Invocation, Events

A very typical concern in distributed systems is the need to coordinate logic in different domains / bounded contexts. Let’s say we have two domains, A and B. When A performs a bit of logic ‘a’, B needs to perform corresponding logic ‘b’. There are generally three ways to accomplish this:

1. Create an orchestrator:

Pros:

• A and B remain simple and agnostic of each other.

Cons:

• Orchestrator absorbs the complexity of knowing about both A and B, interfacing with APIs exposed by A and B, implementing rollback and/or retry logic to mitigate failures. 
• If either A or B change, the Orchestrator has to accommodate those changes. 
• If we want to add another version of B (let’s call it B′) or add another participant C, we have to modify the Orchestrator to deal with all such changes. 
• Orchestrator is in danger of becoming a god service — one that has to know everything about everything and that has to stay in sync with changes throughout the system. 
• This is a path to a distributed monolith.

2. Direct invocation:

Pros:

• Simplicity.

Cons:

• Tight coupling between A and B: A now has to understand the logic of B and interface with its APIs. If B changes, A might have to change. 
• This effectively forces A to be the orchestrator, violating the single responsibility principle. 
• A becomes stateful, holding state until the invocation of B successfully completes or handling failures. 
• While simple with just A and B, in the context of dozens or hundreds of services, point to point invocations quickly become difficult to manage, highly dependent on discovery and topology, etc. 

3.1. Event-Driven Architecture (Uncompensated): 

Pros: 

• A and B completely agnostic of each other 
• Loose coupling, allowing adding new publishers and subscribers without having to modify existing actors 
• With event sourcing based on appropriate message transport (e.g. Kafka), ability to replay/re-process events 

Cons: 

• Assumes reliability of message transport or ability by subscribers to compensate 

3.2. Event-Driven Architecture (with Observers): 

Pros: 

• A and B completely agnostic of each other 
• Loose coupling, allowing adding new publishers and subscribers without having to modify existing actors 
• With event sourcing based on appropriate message transport (e.g. Kafka), ability to replay/re-process events 
• Observer processes can be simple, only looking to match “done ‘a’” with “done ‘b’” without knowledge of details of ‘a’ and ‘b’, and flagging missing correlates 

Cons: 

• Requires a bit of extra sophistication to implement 

Conclusion:

Option 3.2 is clearly the best pattern, even if it requires familiarity with implementation of this pattern. I’d argue that if a team or org are embarking on building distributed systems, it should invest in building the requisite expertise to do it right. 

Luckily, implementations of this core pattern are increasingly available in the open-source world, without the need to build such plumbing from scratch.

On Well-Designed Event Models

I propose that events composing a well-designed event model should be of a form:

Where:

• E_Ai^x→y is the event communicating transition of aggregate/entity A_i  from business state S^x to business state S^y

• A_i is the i-th instance of aggregate or entity A consisting of attributes {α, β... ω} 

• State transition (S^x → S^y) is accompanied by changes in a subset of attributes of A_i {α, β... μ} to new values {α', β'... μ'}, these new values (if required, with previous values {α, β... μ}) are included in the data payload of the event E_Ai^x→y{α', β'... μ'}

This approach is in contrast with a trivial event model where events are simply attribute-level changes, e.g. instead of emitting E_Ai^x→y{α', β'... μ'}, emit the following series:

Ergo, developing a business state-centric event model requires starting with business state transition diagram for entity/aggregate A:

The benefit of business state-centric event model is to make explicit the state transitions (which typically involve changes to more than one attribute), instead of those having to be inferred/reconstructed by the consumer of attribute level events E^α_Ai {α'} = A_i {α'}.

CQRS – The Sweet Spot

Experience teaches all of us that in technology there’s no panacea. Every tool has its uses and, conversely, can be misused. To really understand a piece of technology one needs to understand its design intent, what problem its creators were trying to solve, and what trade-offs they selected.

It is a truism that every engineering decision involves trade-offs – selecting certain strengths at the price of accepting certain limitations (not just in software/systems engineering, but in all engineering disciplines) at a given level of investment. Thus, every instrument has a sweet spot for its application.

CQRS is an interesting example to consider. It builds on the strengths of a more vanilla version of EDA (event-driven architecture) – uncoupling domains from each other – and adds asymmetric read/write models and event sourcing.

By enabling updates at highly granular, attribute level, coupled with arbitrarily denormalized aggregate reads, CQRS creates opportunities for independently scaling the read and write/update stores – much more so than adding read replicas to traditional, symmetric-model domain implementations.

These strengths come at a certain price of additional complexity – managing highly granular even models, ensuring interoperability with non-CQRS-based domains, dealing with compacting of logs in event sourcing, etc.

It would seem that domains that would benefit most from this approach are those where multiple actors are attempting simultaneous updates to varying attributes of the same aggregate – i.e. domains demanding high-concurrency writes.

Startup pivots - technology angle

This is what I've seen throughout my career as a typical [successful] startup lifecycle:

1. Build a product really fast - focus on experience, hopefully decent app code, basic persistence abstraction, monolithic database to keep things fast & simple.

2. If the product takes off, rapidly add features, variations for markets/segments, etc. Code grows more complex & tangled, resulting in greater brittleness and more bugs, database backend shows signs of strain.

3. Increase in number of users leads to business placing greater value on  availability and scalability, as well as performance, graceful degradation and similar matters.

Depending on the length of the the ramp up between stages 2 & 3, and the quality of tech design decisions at stage 1 despite pressure to get things out the door as fast as possible, techies at this point will be looking at a pile of technical debt sized somewhere between a molehill and a mountain.

What to do? Business still requires new features and capabilities to be added at the rapid clip. The more you build on the old without investing into refactoring or rewriting, the more the technical debt compounds (just like financial debt), increasing the tension between improving the internals vs the user-facing features.

Hopefully, as some money is now being made, reinvestment in improving technology platform is possible: break up the monolithic database (hopefully not too much logic is in the DB layer - stored procedures, etc.), try to break up increasingly tangled codebase, attempt to separate operations demanding synchronous, strong consistency guarantees from those that can be intermediated by async events. Words like SOA and EDA start to bounce around. If the business demands availability levels that require multiple datacenters, CAP theorem comes into play, eventual consistency strategies become critical, etc., etc.

In the meantime, some of the self-made components & modules may now have equivalent (or better) counterparts available as open-source or commercial software.

And it's at this point that investment in architecture, with focus on decomposition and encapsulation, as well as build/buy analysis unbiased by who wrote original in-house modules, becomes critical.

One often hears the term 'pivot' in startup parlance these days, usually referring to change in business model or product vision. I think the above illustrates that technology-wise, startups also have to execute one or more pivots to avoid being bound by the initial technical decisions that, just like business and product decisions, can rarely support the business forever.

Should we perhaps place greater emphasis on technology pivots? I think so.