Startup pivots - technology angle

This is what I've seen throughout my career as a typical [successful] startup lifecycle:

1. Build a product really fast - focus on experience, hopefully decent app code, basic persistence abstraction, monolithic database to keep things fast & simple.

2. If the product takes off, rapidly add features, variations for markets/segments, etc. Code grows more complex & tangled, resulting in greater brittleness and more bugs, database backend shows signs of strain.

3. Increase in number of users leads to business placing greater value on  availability and scalability, as well as performance, graceful degradation and similar matters.

Depending on the length of the the ramp up between stages 2 & 3, and the quality of tech design decisions at stage 1 despite pressure to get things out the door as fast as possible, techies at this point will be looking at a pile of technical debt sized somewhere between a molehill and a mountain.

What to do? Business still requires new features and capabilities to be added at the rapid clip. The more you build on the old without investing into refactoring or rewriting, the more the technical debt compounds (just like financial debt), increasing the tension between improving the internals vs the user-facing features.

Hopefully, as some money is now being made, reinvestment in improving technology platform is possible: break up the monolithic database (hopefully not too much logic is in the DB layer - stored procedures, etc.), try to break up increasingly tangled codebase, attempt to separate operations demanding synchronous, strong consistency guarantees from those that can be intermediated by async events. Words like SOA and EDA start to bounce around. If the business demands availability levels that require multiple datacenters, CAP theorem comes into play, eventual consistency strategies become critical, etc., etc.

In the meantime, some of the self-made components & modules may now have equivalent (or better) counterparts available as open-source or commercial software.

And it's at this point that investment in architecture, with focus on decomposition and encapsulation, as well as build/buy analysis unbiased by who wrote original in-house modules, becomes critical.

One often hears the term 'pivot' in startup parlance these days, usually referring to change in business model or product vision. I think the above illustrates that technology-wise, startups also have to execute one or more pivots to avoid being bound by the initial technical decisions that, just like business and product decisions, can rarely support the business forever.

Should we perhaps place greater emphasis on technology pivots? I think so.

iPhone as a platform

Following Apple's announcement of iPhones 5C and especially 5S, there's been much discussion about the merits of their various features and components.

Some have regretted that Apple didn't come out with any new 'whiz bang' features. Others questioned M7 motion co-processor in 5S or outright called it a gimmick.

I think such arguments reveal a superficial understanding of Apple's strategy. A few years back, everyone in the Internet industry couldn't stop talking about 'platforms'. Unsurprisingly, despite the hype, few companies actually figured out how to make money on their 'platform'.

Now, I think Apple has figured out how to both create a hardware+software platform AND make money on it - this is the entire iOS ecosystem.

So when Apple introduces new capability to its platform - whether it's 64-bit A7, or low-power-consumption M7 - it's not about what those things do by themselves, it's what they enable millions of developers to do. That's what happened with adding accelerometer, compass and other capabilities in the past.

Finally, from my own experience with various apps that attempt to track your activity (like the number of steps you've made) throughout the day, they consume way too much power to run all the time, reducing battery life unacceptably. M7 should be an awesome soltuion for that. Also, most avid joggers I know use their iPhones as fitness trackers and media players in one and are thrilled about the prospect of not having to recharge their iPhones more often.

Code maintenance & oil changes

Efficient organizations are typically KPI/metric-driven to a significant degree. Technology investments have to be translatable into business benefits, typically within 1 year horizon.

However, all code rots, with cruft and technical debt gradually accumulating within the codebase, necessitating refactoring and other maintenance that is bound to consume time and scarce resources. How can one demonstrate the necessity of such investment against competing priorities?

An analogy that comes to mind is engine oil changes. We spend time and money on them even though there's no direct, near-term proof that they are necessary. We essentially believe in their necessity because the experts tell us so. If we wait until the car starts showing signs of trouble, then it is probably too late and the repair will be far more expensive than an oil change.

So while managing technical debt is important, it is difficult to attach metrics to it that reliably demonstrate near-term benefits to business. Allocation of sufficient resources for it is then essentially a matter of trust between business and technology organizations.

Web apps - back to client-server?

Are we moving back to client server architecture for web applications?

The web started as a medium for sharing and linking content, with very limited interactivity. In time, ubiquity, elimination of software deployment to and maintenance on the clients, ability to update software on the server side transparently, SaaS and other factors made the web an application delivery channel as much as a content medium.

The biggest limitation that soon emerged is the quality of experience when using server-side UI state management - due to communication latency, responsiveness of UI leaves much to be desired, even as we moved from full page refreshes to partial updates, AJAX, parallelization of requests, etc.

Now that JavaScript has become a powerful, fully featured language, are we effectively moving back to client server model, in with rich JavaScript clients are running in browser containers and maintaining UI state on the client, with server side basically providing data and business logic processing?

The answer could well be yes, but to really match native apps in terms of quality of UI/experience, we need better resource management and control of the 'active' vs 'inactive' apps in the browser, when there are many windows & tabs open, refreshing, etc.

I think web as the medium for application delivery and browser as container is a trend that will continue and in time will force improvements in browsers, comm protocols and markup to enable better UI/experience.

Hiring Solution Architects!

Like many platform companies, PayPal has many strategic partners. Important partners come with their own requirements, often expressed as desired tweaks to core logic in multiple domains.

Needless to say, directly, programmatically implementing such tweaks across multiple domains would result in brittle, poorly maintainable code. The challenge is to ensure that there's enough flexibility in the key domains to support plurality of partner requirements without creating cruft/adding technical debt to the overall system.

This is best accomplished by technical solution architects on one hand working with partners to help shape their requirements against the fundamental rules and assumptions of the platform, and on the other hand, working with core domains comprising this platform to ensure enough flexibility is built in to support these requirements.

I'm currently looking for highly qualified individuals to fill the role of solution architects. Since all comments on this blog are moderated, you can apply by commenting on this post (I will not make submissions public).

People, Process & Org structures

Perhaps I'm biased by my start-up experience, but I strongly believe that having the right talent in the critical roles is the most important enabler of consistent, successful execution.

To put it another way, having the very best process and well-thought-out org structure in place will help, but will not guarantee success if you don't have the right people in critical roles. Conversely, if you do have the right people in those roles, they will find a way to work around the deficiencies in or change the process and the org structures.

In a start-up, you live and die by the quality of your team. In my experience, even large organizations still depend on a contingent of critical talent. So the most dangerous thing a technology company can do is to start treating its technical talent as a fungible resource - a pure cost to be managed down on a per-unit basis.

Paul Graham's analysis of Yahoo is a great illustration of this.

Errors & exceptions - client vs service

Let's say you're designing a service that exposes a getByQuery() interface. The client is responsible for composing a query and interpreting the result set returned.

Interesting question to consider: if a query results in an empty record set, is it because the query is valid but no matching records are found, or that the query is invalid in that particular context?

The designer of the service can choose to be helpful and indicate error conditions to distinguish the two cases, but at a cost of having to be more aware of the calling application's context.

Alternatively, the service could be context-agnostic and just return an empty record set. It then would be up to the client to have a complete enough suite of unit tests to have an assertion fail and indicate  an error in such a case.

I personally lean towards the services being more agnostic of client context and not getting encumbered by context-specific helper methods too early in their lifecycle. YMMV.

Why is PayPal interesting technology-wise?

I work at PayPal and find that the set of problems we have to solve here is both quite interesting and challenging. How come?

From the outside, PayPal seems simple - a multi-channel, global payment platform.

Consider this, however - every government in the world forcefully imposes and continuously tweaks the rules governing money movements.

Typically, financial institutions have separate operating units in every market, with independent software/systems implementing those rules as well as market-specific product features. Sometimes, you even find major financial institutions with entirely separate systems serving different regions within US - so much so that in at least one instance an account registered on the East Coast could not be accessible under the same set of online credentials as accounts established on the West Coast.

Not so for PayPal: As a true Internet company, PayPal relies on a single code base and (distributed) infrastructure that serves all customers around the world. The benefit to customers is that you can instantly pay anyone globally. The flip side is a highly complex code base that has to support constant and rapid evolution and significant requirements flux.

This presents an interesting problem set in the areas of software engineering & architecture, otherwise rarely found on such a scale. And because of its phenomenal growth, PayPal has the wherewithal to tackle these problems and aggressively pursue innovation.

Pivoting towards rules-based application logic

Many applications are initially simple and straightforward, but gradually become more and more complex once specialization based on geographical market, user segment, etc. is required. The same problems arise when SaaS apps transition to true multi-tenant implementation from a set of bespoke, tweaked solutions for anchor clients.

At some point, managing programmatically-implemented use cases through complex, branching conditionals becomes too onerous. At this point, it's common to see the architeture pivot towards declarative, rules-based implementations.

The key to making this transition successful is in the interface between the app and the rules-based framework. Since these rules subsume portions of the app logic, one often has to pass a fairly rich application context as argument at the interface between them.

Constructing the context that is rich enough to maximize the diversity of effectively executable rules yet still well-structured is a good exercise in abstracting key inputs to app logic from mere artifacts of app's internal state.

What rules-based framework returns back to the app is effectively a set of decisions expressed as a set of parameters that define the behavior of the remaining app logic.

One final thought: mature enterprise apps often have multiple sub-domains implementing rules-based logic. In such cases, providing a façade that hides orchestration over all such sub-domains simplifies the app even more. However, this requires an even more careful modeling of both the context as well as result set.

Technical architecture - what's on your mind?

Questions I typically ask myself when designing a system:

1. Isolation/decomposition - what are the separable concerns?
    
2. Encapsulation - how to optimally package those concerns?
    
3. Intermediation & integration - how to compose and orchestrate over multiple components? Synchronous/blocking? Asynchronous/even-driven? Asynchronous/batch-driven?
    
4. Consistency/Availability/Partition-tolerance trade-offs? Eventual consistency? Does 'OK' mean 'fully committed transaction' or 'i've recorded enough information to achieve globally consistent commit at some point in the future'?
    
5. Business logic - programmatic, data-driven & rules-based?
    
6. Entity models - extensibility/flexibility vs. performance optimization?

These are rather generic, off the top of my head. 

Loose coupling & integration - the pendulum swings

We've all learned that separation of concerns is a fundamental good in software/systems design. Often, what is seen as a logical next step is to package these concerns into separately deployable artifacts - as services requiring remote invocation, for example.

Many a startup goes through these phases - start simply, with monolithic codebase and relational DB as a universal persistence mechanism. If the startup is successful, its growth inevitably requires more - more features, more specialization based on market/locale and user segments, more experimentation, integration of new and acquired capabilities, external systems, partners, etc.

At some point, the monolithic code base becomes too complex and the engineering organization too large for every engineer to understand most of the code. So the order of the day is... isolation! Draw reasonable boundaries within the code and attempt to create good interfaces between isolatable portions (hopefully hiding implementation details of each from the others).

And how can one isolate various code domains from each other in a way that makes enforcement of dependency management simple? Well - SOA (service-oriented architecture) of course!

So soon enough, instead of a monolithic code base, there's a rich panoply of services. But not without a cost - creating a lot of services and forcing remote invocation in the name of isolation is far more expensive than running execution within the memory space of a single process. Intermediation between a multitude of services becomes the next challenge - with far from trivial issues of orchestration, latency, topology & discovery, geo-distributed failover, etc. If processing of the same request now requires a cascade of a dozen or more service calls, how does one still make it performant? How to handle versioning required across common interfaces - supporting both the new and the legacy clients?

Attacking these issues by sheer brute force is usually an expensive proposition - with costs rapidly rising. The only answer I found is ensuring an adequate investment in quality technical design/architecture.

Processing batch/bulk requests

It's fairly common to get requests for creating a batch version of a previously transactional API. What are the main questions to ask when considering batch implementation:

1. Atomicity - what operations should be considered atomic, all-or-nothing (i.e. not admitting partial failures)?
    
2. Since sizable batches cannot be processed atomically, how to handle partial failures? This quickly leads to:
    
3. Idempotency - how to prevent erroneously submitted duplicate requests from creating snowballing failures and data corruption throughout the system?
    
4. Downstream effects - how to ensure that downstream systems that depend on asynchronous processes, such as ETL, work well with different load patterns created by upstream batch requests?

So introducing batch requests into the system without compromising consistency and while maintaining load & performance SLAs is not always a trivial task, which makes it interesting!

Agile vs. waterfall [software development] - the grand bargain

In waterfall-style development, the implied bargain is: product owner provides complete requirements and developer provides cost estimates and execution timeline. Of course, completeness/finality of requirements is pure fiction. So the bane of waterfall model is that every significant change in requirements requires re-estimation and re-planning, wasting huge amounts of time for all involved.

In agile, the core bargain is different - everyone saves time by not producing or parsing huge requirements docs (which often really are comprised of vast stretches of boilerplate with useful nuggets of info hidden here and there). Product owner gets flexibility for requirements and scope changes along the way, to a degree. Developer gets a committment to fund the implementation effort through initial launch and subsequent tweaks/experimentation.

Agile is not an excuse for lack of technical design - the architect or engineering lead still need to isolate the salient aspects of the product to ensure what they're building will at its core be ble to stand the test of time.

Against ivory towers

Does taking an architecture role relegate one to an ivory tower of high-level abstractions, processes and governance? Absolutely not!

Architects are responsible for ensuring there's no gap between high-level designs they create and what is ultimately implemented by developers. If an architect articulates those designs in high-level pattern language that is not readily consumable by the engineers, it's not good enough.

Architects have to provide clear translation of what those patterns mean in terms of CODE - be it draft implementation of core abstractions and key interfaces, pseudo-code explaining the patter, etc. But 'naked' high-level design documents are usually insufficient.

The art of abstraction

Technical architecture is in many ways the art of finding the right abstractions. This is why, beyond very rough guidance such as 4+1 view, I don't see architecture easily amenable to templates. Depending of what you're designing, the right set of abstractions will be different.

[Software] engineering & flying cars

Engineering in general is the art of selecting the right trade-offs: we choose to optimize one set of design characteristics at the expense of others. For a design to be demonstrably good, it's important to make those trade-off decisions explicit.

Otherwise, it's very easy to end up with a compromise design. Like flying cars - there are some prototypes in existence today, but very clearly they're neither very good cars, nor very good planes. The only thing they're good at is combining (some) driving ability with (some) flying ability.

Even software engineering is ultimately limited by physics and so, since we can't have it all, we should choose wisely. Some design decisions are hard to change later on.

Organizational map-reduce & the role of technical architects

What if you have a huge code base and a large organization with its multiple subdivisions responsible for owning/maintaining various parts of it?

Typically, any significant, core-affecting product development requires involvement of many of these sub-units. How does the organization manage such involvement?

The bad way: Meetings are set up, inviting everyone remotely related to the new thing so they can supply estimates for 'impact' to their domains. The most knowledgeable, and thus busy, invitees decide this is a waste of time and don't show up. Those who do show up throw some estimates into the ring just in case. Each participant may have their own idea/interpretation of what the target solution is.

The good way: The product owners work with their technical architects from the start, and by the time the grand map-reduce is required, it can be done in a context of a reasonable technical solution, which can be vetted by and singed up to by all the key affected parties.

Which would you choose?

Technical architects - what are they?

I'm often asked to explain what a 'software architect' or 'technical architect' does. Is this just a title for highly-tenured engineers? A vanity title? A distinct role?

I believe software/systems architecture is a distinct role, and an important one, especially in the context of software that requires a more complex stack than simply PHP or RoR or (put your favorite tech here) backed up by a database:

1. Architects have to be comfortable working with higher-level abstractions - thinking in terms of separation and packaging of concerns, integration/intermediation and other patterns, problem and solution spaces before they zero in on specific point solutions.
    
2. Architects, compared with similar level engineers, have to trade some of the depth for breadth. This does not mean being superficial or glossing over important technical details. It means having enough depth - understanding the salient aspects but of more domains. A seasoned engineer is expected to have a more detailed knowledge, but of fewer domains.
    
3. Architects have to be comfortable with weaker ownership ties to many code artifacts - they may own an interface definition, but not its implementation, for example. Since most of us come to architecture after being seasoned developers, it's not always an easy transition.

Not every organization has the title 'architect', but most of those of sufficient scale do have people who play that role.