Week of 2022-04-04
Cheesecake and Baklava
I have been reading Alex Komoroske’s Compendium cards on platforms, and there’s just so much generative thinking in there. There’s one concept that I kept having difficulty articulating for a while, and here’s my Nth attempt at defining it. It’s about the thinness and thickness of layers.
Given that layers have vectors of intentions, we could imagine that the extent to which these intentions are fulfilled is described by the length of the vector. Some layers will have super-short vectors, while others’ are quite protracted. To make this an easier visual metaphor, we can imagine that layers with longer intention vectors are thicker than layers with shorter vectors of intention.
For example, check out the 2D Canvas API. A long time ago, back when I was part of the WebKit project, I was amazed to discover that Canvas’ Context API was basically a wrapper around the underlying platform API, the GCContext. Since then, both APIs have moved apart, but even now you can still see the resemblance. If we look at these two adjacent layers, the intention of this particular Web platform API was perfectly aligned with the underlying platform’s API intention and the length of the vector was diminutively tiny — being literally a pass-through. If you wanted to make graphics on the Web, this was as close to the metal you could get, illustrating our point of thinner layers yielding shorter intention vectors.
To compare and contrast, let’s look at Cascading Style Sheets. It’s fairly easy to see the intention behind CSS. It strives toward this really neat concept of separating content of a Web document from its presentation. When I first learned about CSS, I was wholly enamored—and honestly, still am—with such a groundbreaking idea. I even wrote an article or two or three (Hello, 2005! I miss your simplicity) about best practices for content/presentation separation.
We can also see that this vector of intention extends much farther than that of the 2D Canvas API. Especially from the vantage point of a WebKit engineer, if seems like CSS took something as simple (and powerful) as GCContext and then charged daringly forward, inventing its own declarative syntax, a sophisticated model for turning these declarations into rendering, including deeply nuanced bits like formatting and laying out text. CSS is a much thicker layer. It’s the whole nine yards.
The question that inevitably arises for the aspiring platform designers is “which kind is better?” Why would one decide to design their layer thin or thick? It’s a good question. Now that we’ve learned about the pace layers, we can seek insights toward answering this question through this thought experiment. Let’s pretend to be designing a platform in two alternate realities of extremes. In the first, we boldly decide to go for the cheesecake approach: a single layer that has one vector of intention, fulfilled as completely as possible. In the second, we, just as boldly, had chosen the baklava approach: our layers are countless and are as thin as possible, just like in filo dough. Anybody hungry yet?
Applying the pace layer dynamic to both alternatives, we can see that the baklava platform is better equipped to withstand it: the multiple layers can choose their pace and change independently of each other. Poor cheesecake will have a harder time. The single layer will inevitably experience a sort of shearing force, pushing its upper part to change faster, with the bottom part staying relatively still. If I were a poetic kind, I would say something like: “this is how layers are born – in the struggle with the shearing force of innovation.” But even if I weren’t, I can describe it as a pretty significant source of suffering for developers. Especially in monolith repositories, where everyone can depend on everyone and dependencies are rarely curated for proper layering (visualize that dense mass of sugar-laden dairy), developers will keep finding themselves battered by these forces, sometimes even without realizing that they are sensing the rage of pace layers struggling to emerge. Using CSS as the reference point, I remember having conversations with a few Javascript framework developers who were so fed up with the inability to reach into the CSS monolith, they contemplated — and some partially succeeded! — rolling their own styling machinery using inline styles. There’s no good ending to that particular anecdote.
Surprisingly, baklava will experience a different kind of force, caused by the same pace layer dynamic. If I design a platform that consists of a multitude of thin layers, I am now responsible for coordinating the moving of the ladder across them. As you can imagine, the costs of such an enterprise will accrue very quickly. I was once part of a project that intended to design a developer platform from scratch. As part of the design, several neatly delineated baklava-like layers were erected, with some infrastructure to coordinate them as they were developed. Despite having no external developers and still being at an early-days stage, the project rapidly developed a fierce bullwhip effect akin to the one in the infamous beer game, threatening the sanity of engineers working at the higher layers. Intuitively, it felt right that we would design a well-layered developer surface from the ground up and keep iterating within those boundaries. It just turned out that there are many different ways in which this layering could happen, and picking one way early on can quickly lead to unhealthy outcomes. Layers accrete. Imagining that they can be drawn from whole cloth is like planning to win a lottery.
Well, great. Our baklava keeps wanting to turn into cheesecake, and cheesecake keeps wishing to be baklava. Is there a golden middle, the right size for your layer? Probably. However, I have not found a way to determine it ahead of time. Platform engineering is not a static process. It is a messy dynamic resolution of layer boundaries based on developers interacting with your code — and dang it, the “right answer” to layering also changes over time. API layering is inherently a people problem. Despite our wishes to get it “right this time”, my experience is that until there is a decent number of developers trying to build things with my APIs, it is unknowable for me what “right” looks like.
When gearing up for shipping developer surfaces, bring lots of patience. The boundaries will want to move around, evolve, and reshape across the pace layer continuum. Layers will want to become thinner and then change their minds and grow a little cheesecake girth. Getting frustrated about that or trying to keep them still is a recipe for pain. Platform engineering is not about getting the API design right. It’s about being able, willing, and ready to dance with it.
🔗 https://glazkov.com/2022/04/05/cheesecake-and-baklava/
Minimum viable legibility
Having seen folks struggle through another Google perf season and supporting them through it, I figured I’d write down some of the new things I learned. Specifically, I wanted to focus on the quadrant of performance management blind spot I talked about a while back.
If you do find yourself in the performance management blind spot, congratulations and my condolences. Congrats, because you are probably doing something so interesting and unusual enough that it is not (yet!) captured by the organization’s rubric. Condolences, because you need to do some extra legwork to show that you do indeed provide value to the organization.
Think of it as the minimum viable legibility: the additional effort of making your work recognizable as valuable when viewed through the lens of the performance management rubric, but not so much that it changes its nature. Minimum viable legibility is kind of like a tax, and there are probably lots of different ways to pay it. Based on my experience, it’s a combination of these three ingredients: sponsorship, network of support, and the “big idea”. Depending on the nature of the rubric, each of these may offer varying degrees of effectiveness.
Sponsorship is the situation in which an executive vouches for your work. They expend their own political capital to state that the work is important. If you can find a sponsor like that, you’re usually set. Things to look out for are the reserves of said political capital over time, the commitment to follow through, and the amount of cover given. There also may be additional logistics. For example, you might need to bug them regularly or connect them to the right places in the perf process if they are not in your reporting chain.
Less effective than sponsorship, the network of support may still work reasonably well in environments with peer-driven performance management systems. Kind of like a more distributed variant of sponsorship, the trick here is to grow a robust network of peers (the broader the better) who understand your work and are willing to provide peer feedback on it. It helps significantly if they can articulate well why they see your work as important, so you may have to invest some time into helping them do that – in addition to maintaining the network. When your manager’s sponsorship cover is limited, the network of support can really come through in a pinch.
Finally, sometimes effective, but also fraught with peril, the “big idea” refers to connecting your work to some important — and usually new — initiative. For example, if your organization suddenly sees the need to build a special kind of wooden spoon after focusing on sporks for the last decade, tying your work to the company’s Wooden Spoon OKR might be a tactic to try. The tactic only works when the connection is clear and not dubious, and in combination with other ingredients. Otherwise, it might backfire and actually do harm to your attempts at establishing legibility, becoming a MacGuffin-like distraction.
The unfortunate news is that this is not a reliable recipe. I’ve talked with folks whose extensive and enthusiastic support networks ended up amounting to very little. I know peeps who were trapped several layers of management deep in sponsorship deficit. In such situations, there’s very little that can be done to establish the minimal amount of legibility necessary. Blind spots are tough. However, if you truly believe that you’re doing good work that’s invisible, please give boosting minimum viable legibility a try.