Agile & Bridge Building

Sometimes there’s persistent confusion on the relationship between architecture and agile delivery.

If you’re building anything more complex than a website or a mobile app with a cloud backend for the front end (BFF), then you need to identify and decide upfront things that are going to be hard to change later. But this doesn’t mean you have to figure out every detail in advance. Architecture in agile is very much about figuring out and deciding the key things that are hard to change later – and these vary case by case.

A good analogy would be in building bridges: you have to decide what type of a bridge it will be, e.g. a suspension bridge:

Manhattan Bridge Construction, Wikimedia Commons»

Or a movable bascule bridge:

© Alex Fedorov, Wikimedia Commons

The type of the bridge you’re building has to be figured out in advance since you can’t easily change one into the other mid-construction. It is a function of core constraints and requirements – the span, the height of the ships you want to be able to pass underneath it, etc.

However, what colour to paint the bridge or how to decorate it is a very flexible choice – no need to debate it upfront as you can change it at any time later (well, ideally before you bought a million dollars’ worth of paint).

Cargo Cult Agile

It is instructive to consider human tendency towards magical thinking that finds a wonderfully didactic expression in Cargo Cults.

Magical thinking fundamentally expresses faith that performance of certain rituals alters the status of the actor and can confer benefits that are not correlated with the substance of the ritual itself.
It is not difficult to find many examples where application of Agile is an exercise in ritualistically following the form rather than focusing on substance.

The original intent of Agile was to redress the balance of power between managers/stakeholders and engineers, affording more power to the latter, and achieving better results & productivity through their empowerment.

But it is not uncommon today to find instances where Agile became co-opted as just another mechanism of control, without altering the power equation, and rests on what amounts to a magical belief that by performing formal ceremonies and rituals of Agile, the benefits will accrue even if such performance is entirely divorced from its underlying rationale.

The very same powers that demanded of engineers compliance with rigors of waterfall-style development, simply switched the rulebook and now demand compliance with the rigors of Agile. Such hollowed-out Agile practice remarkably resembles the cargo cults I referred to at the start.

Factoring and pipelining complex projects

We're all familiar with the concept of minimum viable product.

In fintech, those minimum requirements include a lot of regulatory compliance, legal and other complex concerns that can have significant ramifications on how every product feature is supposed to work. Throw in requirements from key partners and customers, and one can easily end up with an MVP with a large scope, and a lot of uncertainty and interdependencies among requirements/stories.

It's very easy then to start spending a lot of time reconciling it all together in attempt to arrive at a coherent picture all teams can align on in detail.

Finally, large investments are typically in the opposite of a skunk works/'fly under the radar until you've got something good to show' situation. Large investments imply commitments - in time as well as revenue. As in - in which quarter can it be delivered/ramped/booked. So for very good business reasons one ends up with a defined budget & timeline on one hand and a quivering mass of stories on the other. What to do?

The common mistake is to attempt to mitigate uncertainty by figuring it all out upfront in sufficient detail. Then everyone is spending most of their effort drawing and redrawing the complete picture and overall progress is very slow.

And, of course, every requirement/story has a product owner behind it, and no PO worth their salt will say 'uh, you know - the thing I'm responsible for - don't worry about it for now'.


Factoring:

In my experience, key has been to factor the entire whole into subgroups of mutually influencing requirements but with fewer fine-grained dependencies between these subgroups. Architecture can serve in leading or analytical capacity for this factoring.

Then decide by fiat which subgroup goes first and start cranking away at it. Yes, without other subgroups the 'MVP' won't see the light of day, and attacking other subgroups later doesn't mean they're less important (as their POs will be worrying about). But one has to start somewhere.

Pipelining:

And then, guess what - while subgroup 1 is being delivered, [C]POs, architects, dev leads and others with responsibilities beyond a single agile team or sprint can free up time to focus on subgroup 2, 3, n, n+1... creating a pipeline.

Seemingly simple, but requires persuading passionate people that being in n-th place in delivery pipeline doesn't imply being in the same place in terms of importance.

Agile vs. waterfall [software development] - the grand bargain

In waterfall-style development, the implied bargain is: product owner provides complete requirements and developer provides cost estimates and execution timeline. Of course, completeness/finality of requirements is pure fiction. So the bane of waterfall model is that every significant change in requirements requires re-estimation and re-planning, wasting huge amounts of time for all involved.

In agile, the core bargain is different - everyone saves time by not producing or parsing huge requirements docs (which often really are comprised of vast stretches of boilerplate with useful nuggets of info hidden here and there). Product owner gets flexibility for requirements and scope changes along the way, to a degree. Developer gets a committment to fund the implementation effort through initial launch and subsequent tweaks/experimentation.

Agile is not an excuse for lack of technical design - the architect or engineering lead still need to isolate the salient aspects of the product to ensure what they're building will at its core be ble to stand the test of time.