Week of 2022-08-29
Where I bravely explore the top of the S-curve. First, with yet another take on what might lead us there, and then a possible means to manage it. The latter involves setting our sights on the stars.
The Plumber Age
This framing jumped out of a conversation with the fellow FLUX-ers and I just gotta write it down. It appears that every software project, upon hitting its stride of success, eventually arrives at what I call the “plumber age”. Here in this story, I use “plumbing” to refer to the work of improving the quality of the code – as opposed to “feature work” that adds new functionality.
The Plumber Age is the period of time when making further progress depends primarily on the quality/velocity of refactoring and cleaning up the code base.
I like to think of it as a season in the lifecycle of a software engineering adventure. This season is characterized by the rapid rise in importance of refactoring and rewriting existing code. At some point, a software project encounters this interesting moment, where the number of new features and capabilities slows down to a trickle, because adding them takes longer and longer. Everyone on the team wants to keep going, and there’s a lot of frustration and tension, but the code seemingly begins to actively resist being changed. Every new bit of code breaks some existing behavior, blows up performance metrics, or other shenanigans.
Most useful software is born out of a bunch of hacks. Sometimes they are really solid hacks, and sometimes they are nearly literally scotch tape and popsicle sticks. And it makes sense. We want to go fast, get something in the users’ hands, and iterate like crazy, listening to what our users tell us. In the process, we accumulate hacks – also colloquially known as “accruing technical debt”. Sometimes, the hacks aren’t born as hacks. We might conceive of a rather elegant design at the start, but the contact with reality unveils our modernist naivete, and what seemed like a paragon of software engineering turns rapidly into an annoying vestige – now covered in hacks.
At some point, the hacks congeal into a solid mass that becomes nearly impossible to reason about. Veteran engineers will stare carefully at the code and move with artful precision, only for the code to laugh back at them. Often, crossing into the Plumber Age happens rather suddenly, almost like being locked out of the house. It feels like just yesterday we were able to land new bits and then – bam! – a Cthulhu is staring back at us through our code editors. The pile of hacks filigreed into a tangled network and now, this network transitioned into its next phase.
Enter the Plumber Age. When embraced, this season is marked by the emergence of the demand for, well, plumbers. People who enjoy (or at least tolerate) moving the code around become more and more common – and sometimes prominent – within the organization. The nature of the project changes. That crazy idea of tests and test infrastructure, previously mostly ignored, is considered a critical investment. Talk of build and development metrics crop ups everywhere. Words like “services” and “layering” start showing up in everyday work conversations. Plumbing becomes a full-time job, and there may even pop up a team whose job it is to maintain and modernize some stable “core” of the software. They allow other teams to continue to move forth with feature and capability development.
Most interesting to the leaders of an organization that needs to embrace the Plumber Age is the change in how the teams are structured. The loose and free-spirited flat setup that works so well to get our enterprise going becomes a hindrance. In the Plumber Age, code ownership and clear delineation of roles and responsibilities becomes a necessity. Hierarchies and RACI matrices start popping up all over the place. This may give us a queasy feeling – are we becoming a rigid bureaucracy?! Not necessarily. If we are to continue on this path and keep increasing the user value of our software, a more structured organization is needed. How carefully and thoughtfully we structure it is up to us. My intuition is that it is the fear of becoming bureaucratic that actually turns engineering organizations into bureaucracies. Adding more structure intentionally is not bad. However, pretending to not be bureaucratic will guarantee the feared outcome.
Some organizations try to reject the Plumber Age, and attempt to skirt around it in various ways. Few build second systems, and are rarely heard from again. It’s hard to see all the miracles that led our horrifying pile of hacks to become a successful first system. Others push brashly forward, chewing through talent and becoming more and more challenging to move – until they keel over and die. There are probably other paths and combinations, each leading to similar outcomes.
My guess is that the Plumber Age is inevitable. It is something that every team will encounter in due course. And just like the seasons, it is best experienced when prepared. This does not mean that each new team must start with a thoroughly designed infrastructure or hire a cadre of plumbers. That’s a whole other trap that I will save for later discussion. However, as the team starts to feel the lift of successful engagement with its users, it needs to anticipate the transition. Start having regular conversations about architecture and layering. Draw sketches of where the core of the system begins and feature work ends. Consider how features and capabilities can be added without affecting the core. Think about how the team might be structured. Are there tech leads emerging who could lead the sub-teams? Oh yeah – almost forgot: teach your code to be testable. All of these will come handy, kind of like those coats and hats you’re storing for winter.
🔗 https://glazkov.com/2022/09/01/the-plumber-age/
The Starship of Theseus
This framing of the Starship of Theseus has been on my mind for a little while, and I finally decided to write it down. Basically, in a marketplace with any sort of selection pressure, organizations vie to both retain their identity (what makes them the organization) and adapt to changing conditions. A possible path to resolving the tension of these two opposing forces – stay the same, yet evolve – is aptly described by the “Starship of Theseus” analogy.
In FLUX issue 51, we had this really nice riff on the Ship of Theseus experiment. If every part of a ship is eventually replaced, is it still the same ship? Or something entirely different?
The “Starship of Theseus” kicks the experiment up a notch. What if, in addition to replacement of parts as part of the routine maintenance, the process must incorporate improvements?
In this metaphor, the ship is seen as a collection of capabilities. Some of these capabilities are transient and some essential. Transient capabilities can be substituted for other, different capabilities without affecting the distinguishing quality – the “shipness” – of the ship. Conversely, essential capabilities are those that constitute this quality. Exchanging them for another capability means destroying the Ship of Theseus.
For instance, we can confidently state that the hull, crew quarters, and navigation are essential, while the material of which the hull is built or its particular shape are transient. Yet some capabilities that we thought of as essential are revealed to be transient in due time. We might believe that the proud sails of the Ship of Theseus were its essential capability for many generations. When the vessel is finally fit with the powerful steam engine, we recognize that the sails were transient.
To keep the ship afloat and continuing to fulfill its purpose in the changing environment, we must continuously replace its obsolete transient capabilities with more technologically advanced alternatives, while preserving and maintaining the full efficiency of the essentials. If we are successful, the steam engine is replaced by a diesel one and later, a nuclear reactor, while the hull graduates to steel, with aluminum not far behind.
Technological progress guides our understanding of what the essential capabilities are. And conversely, this understanding is guiding the evolution of the ship. A repeatedly, doggedly asked question: “What are we really about?” spurs the transformation. We resolve the tension of “stay the same while evolving” through a dynamic inquiry: the answer to this question changes over time.
If we initially thought our particular “shipness” was about the number of masts and a distinctive figurehead, we revise our answer as we adapt. And a couple of millenia later, it is the spirit of frontier exploration – the unchanged essence of our ship – that compels us to take the vessel to the stars.
Here’s my guess. Intentionally or not, any large organization that perseveres through time follows the pattern described by the Starship of Theseus story. And I wonder: would such an organization benefit from leaning into this pattern more consciously?
First, this dynamic nature of the inquiry tells us that we do not have to know the ultimate purpose of our ship. Put it even more forcefully, we must not try to have the essential capabilities captured and solidified, for an organization that knows with certainty what it is about has already sealed its fate. It will never get to orbit, let alone see other galaxies.
Second, we might be better off viewing a portfolio of our capabilities as something that will evolve over time. At any given time, we also might want to have a good sense at which capabilities are essential and which ones are transient. This sense would need to be grounded in an understanding that the essential/transient breakdown will change over time.
Third, it would have to be our collective responsibility to think in multi-year timelines. We must dream of starfaring even when we’re still powered by sails. Unlike a static vision, the vision formed as the question – “How will we go to the stars?” – can adapt to the changing environment in productive and inspiring ways.