Discover more from Chaos Engineering
Shipping Fast 🚀
Speed as a Competitive Advantage
Why should you care?
Anyone working for a digital company should care1 about shipping fast because as competition increases differentiation only happens from distribution, product, and speed.
Distribution for internet companies tends to converge to social media, search, content marketing, streaming, and other tactics that I’m surely forgetting, but of course some companies are able to charm their customers in novel ways.
Product, the value you exchange to your customers, tends to get copied surprisingly quickly, yet even so some legendary companies have been created through meaningful product differentiation (Apple, obviously).
Speed, the pace at which you innovate your product, is a significant competitive advantage and it’s my personal favorite because maintaining product and engineering velocity in a mature organization (nay, codebase) is extremely hard.
Thanks for reading Chaos Engineering! Subscribe for free to receive new posts and support my work.
On the implosion of Fast
Well obviously I can’t ignore that whole thing.
Fast imploded2 (my former employer3 and once hot startup), it’s been written about enough and aside from some well deserved Twitter roasts, not much attention was given to some of the good things that we accomplished.
Of course we failed at the most important thing (surviving) but that’s obvious and I thought there was value in writing about what we did well: ship Fast™.
Why is speed important?
Strategy is a critical foundation to any business but where I’ve mostly seen larger organizations struggle is in the speed of execution.
This isn’t to say that goals aren’t accomplished, eventually they are—typically long after the goals are relevant, which is entirely the problem.
Innovation happens through fast execution.
In the modern era, that is how progress happens and it is primarily how a small technology startup can eat a big company’s lunch4.
Why is shipping fast hard?
Some obvious things:
Bad or archaic software
Some not so obvious things:
Time wasted on consensus
Too much writing, not enough coding
If your team is struggling to ship products (i.e., code) at the velocity that you want then I recommend you mentally run through the first checklist.
If you don’t have any obvious problems, then I recommend reflecting on the not so obvious things, as these aren’t as painfully visible and require more reflection.
Let’s discuss each one in more depth.
You should always expect requirements to change.
It’s the nature of problem solving, you start projects with a goal and a lot of uncertainty and your job as a team is to reduce the uncertainty and hit the target but if you continue to change what you plan on building then basically you’ve kept the uncertainty the same. If the nature of the problem has a lot of moving pieces (i.e., chaos5), just tell people that the timelines are going to change and why, it’s much more reasonable and people will mostly understand, plus this is realistically the only path to an actual solution so you don't end up failing.
Increasing scope is typically subtle but any request requires more time, sometimes small, sometimes huge. Know this when asking for something.
People often say “Can you make this work for a group of people”, “Do you think you can add X,Y, and Z? It should be easy, right?”—maybe but that’s a leading question 😉 and leading questions tend to frustrate people for obvious reasons. I’ll add that engineers often do this during code reviews and it’s important to understand when you’re requesting a change to some code that’s really increasing the scope of the problem and not a part of the goal of the work being done.
If you create goals that are too ambitious and outright unrealistic, not only will you likely fail, you will probably annoy and disappoint your team in the process.
People want to accomplish things but “setting them up for failure” is a great morale killer. Don’t do it.
Your documentation is probably outdated and wrong. This will tax your team’s productivity6.
Make sure people are incentivized to update documentation, not just create it. This is important for both good citizenship and a sustainable business. If a system (and thereby a team) isn’t operationally effective without a key member, you have a key-person risk and that can be de-risked through documentation.
Bad or archaic software
How your team does the thing (i.e., writes the code) matters a lot. If people aren’t writing code very often, that’s probably not a good sign7.
In some cases, identifying bad or archaic software is somewhat challenging as it can be hard to distinguish but, in general, there are a few tips:
After an hour of review by a new senior software engineer of a single service, they can’t understand it end-to-end
A lot of code is needed in order to accomplish something truly simple
Too few or too many tests (that’s not terribly helpful 🥲)
Unfortunately these things do require some real technical expertise but hiring even one talented engineer will have high yield. They may not be able to solve your problems right away but they’ll be able to tell you what the problems are and develop a plan of attack.
Time wasted on consensus
Consensus is valuable but time is finite and decisions should be made in reasonable time frames.
Generating consensus is great! But oftentimes getting 100% alignment is really not optimal. I am a huge believer in “Disagree and Commit”, as members of a team our goal is not to always agree but to always work together to accomplish a goal, even if you disagree with the approach.
Too much writing, not enough coding
Docs and specs aren’t code.
Of course, they are valuable, but eventually it becomes people deliberating over things that don’t really matter. Everyone is bounded by time with only 24 hours in a day so in a very literal sense time spent deliberating is time spent not executing on the work. Preparation and knowing what you are solving for is critical but there is diminishing marginal returns to this and it’s important to know when you’ve gotten enough information to start the work, otherwise you’re just wasting time.
As a brief aside, decision making and execution under uncertainty is rather challenging both as a mathematical problem and as a business one but there is always a tradeoff here.
Most decisions should probably be made with somewhere around 70% of the information you wish you had. If you wait for 90%, in most cases, you're probably being slow. Plus, either way, you need to be good at quickly recognizing and correcting bad decisions. If you're good at course correcting, being wrong may be less costly than you think, whereas being slow is going to be expensive for sure.
Before commenting on something ask yourself if your feedback is going to be constructive and make a material difference.
It’s a subtle filter but one I find that helps me listen more and withhold useless criticism. This also helps prevent you being perceived as a frustrating person to work with, which kind of makes sense, right? Imagine if I went to you after you spent a lot of time on something and gave you harsh criticism that was not only not constructive but utterly useless, you’d be frustrated too. I have friends and former colleagues that do this and, unfortunately, it makes lots of people frustrated and they talk about it. Don’t be this person, be constructive and a good team player 😊—your outcomes will be better anyways.
One team, one dream.
If the team is dysfunctional, you’ll see symptoms. Some of them are: complaining, blaming others, or not compromising about things. If you see any of these symptoms, you probably have a culture problem. Teams should be empowered to make decisions and be autonomous but they should be hyper collaborative and nice. The former will make you effective at your job, the latter will make you happy in life.
Lastly, when it comes to a large code base or lots of microservices, you often have to partner with lots of other teams to understand if you’re making breaking changes, what dependencies you have to manage, and if there are other teams that may want to collaborate on a similar use case. It’s incredibly important to be a good citizen and doing things that are dysfunctional is not only interfering with your goals, it’s, frankly, a little immature.
What are the tradeoffs from building fast and breaking things?
While speed is an extraordinary competitive advantage it certainly isn’t everything. For Fast it was no panacea for lacking enough revenue, but beyond that there are certainly other tradeoffs.
Stability doesn’t have to be compromised if you build your systems right. Basically this means investing in good Continuous Integration/Continuous Deployment (CICD) pipelines. Doing this well at scale is hard but doing this poorly will make you a sloth.
Control has to be given more to smaller teams. Said another way, you have to trust your teams to be autonomous and execute, which necessarily means you have to take your hands off of the steering wheel. That means that things won’t be done exactly the way you want them to because other people obviously have a different brain than you. This is a good thing. Remember you work with adults, do not babysit them, and if you feel that is not a reasonable option, you may need new adults.
Failure will happen. Maybe more, maybe less; it’s mostly unclear and you won’t have the counterfactual to know anyways…But I invite you to reflect on the quote above from Jeff Bezos, one of the greatest entrepreneurs of our lifetime, and I will repeat it for emphasis “If you're good at course correcting, being wrong may be less costly than you think, whereas being slow is going to be expensive for sure.”
It is better to fail fast than to die slow. 😉
During my year at Fast I interviewed over 200 people, many of whom were staff or senior engineers from the best technology companies. I asked every candidate, “Why Fast?” and I found it rather interesting that they all had very similar responses: they wanted to (1) have autonomy, (2) make an impact, and (3) move fast.
That is the single greatest benefit of shipping fast: creating a culture where extremely talented people want to come build incredible things with you.
Happy shipping. 🚢 🤠
Did you like this post? Do you have any feedback? Do you have some topics you’d like me to write about? Do you have any ideas how I could make this better? I’d love your feedback!
Feel free to respond to this email or reach out to me on Twitter!
I’ll try to do this for most of my posts so that the intended audience is clear and so you won’t have to read it if it doesn’t fall under your set of interests.
A very fortunate outcome was that I get to continue my fast shipping endeavors with the majority of Fast’s engineers at Affirm, which is an incredible place to call home. 🤗
These are good market behaviors where competition is naturally incentivizing faster innovation and rewarding innovators (though not always).
I do this for my own entertainment.
Wrong documentation is far worse than no documentation, as you can waste a lot of time doing things that are wrong!
I do not recommend using any crude (or maybe even sophisticated) metric for performance calibration for software engineers. It is trivially gamified and rendered either useless or counter-productive. Instead, I recommend monitoring the metric to assess the quality. Lots of code written isn’t necessarily a good sign but no code written is definitely a terrible one.