Through the Stack 2-09 (Week 14)

This is "Through the Stack," a weekly focus on a topic and links relevant to Lead developers (actual or aspiring) working with an internet-related product.

Many lead developers, tech lead, and staff engineers have their hands in many projects and influence many layers in their organizations. As such, they work Through the Stack: from Strategy, through Operational to Tactical work. This newsletter reflects those dimensions.
If you have comments or content to suggest, please reach out to us by email.

This week ...

... we focus on the concept of "Fast iterations". This topic can be helpful to anyone within the engineering team: junior, mid-level, senior engineers, and above. This is an excellent example of concepts we started this newsletter for: we want to showcase things that can help the whole team on their learning journey.

Before jumping in, though, a few articles have popped up in our feeds, so we will, as every week, start with those.

As every episode is now also posted on our Substack we are testing out their Chat feature. You can join the conversation there.

Links of the week

• Process Experiments, J. Gothelf: a good read; you might see parallels too with the rest of this episode
• The big lie of Strategic Planning, HBR: a long read but worth it, especially interesting to senior and C-level people
• 3 steps to Telling a Compelling Story, J. Gothelf: a good read for anyone looking to write or speak about a topic
• @arvidkahl thread on how he got a new background for his landing page (AI-related)

Iterating slow

Following last week's focus on the Pareto principle, let's discuss fast iterations.

As our 'Failure Factory' article discussed, a team falling or already within a Feature Factory pattern focuses on shiny objects and deliverables. Due to those, the team will group and delivery changes in one go, with a lot of time passing between releases.

This pattern tends to cause several issues:

• Large code changes to review: with several 100 lines changed (or more) the code reviews are challenging to do and often skipped
• Too many changes to track: with many things done at once, release notes and commit messages will likely gather the main points leaving many details out
• Too many changes to trust the release: one reason why teams don't like to release on a Friday afternoon: each release is so voluminous that they don't trust it will go fine
• Version management conflicts: with a long-running branch, the delta with the main branch will keep on growing, adding more complexity always to keep it in sync

Besides the Feature Factory pattern, this approach is also used by many teams using, as an excuse, for the fact that they can't split their work into smaller chunks. Yet, the same issues will appear.

Iterating fast

One solution is to iterate much faster: a set of changes should only take a few hours or less than a day from being started to deploying in production.

Such an approach will force the team to change a few parts of their work culture. It will fail if that does not happen, and they keep shoving their old ways into that pattern.

The main idea is not to jump right away into writing code. Each iteration must be purposeful and prepared: the developer, pair of developers, or group of developers must have a clear purpose for the iteration's work. This work must be done quickly and self-sufficient enough to allow for one or more subsequent steps afterward.

To be clear: the work that will be done might not be enough to fulfill the complete feature requested. It will rarely be. Sometimes the code in question won't be "in use" by the rest of the code until the next iteration (some might oppose this idea). Yet, this won't be the case for long, as the next iteration might only be hours away.

Here we are considering the following points:

• whatever we are doing needs to be self-sufficient and tested
• whatever gets merged in the main branch must always be deployable
• it's more important to limit the difference between each deployment than the always have something to show

This allows us to take care of several of the issues listed before:

• with a limited amount of code to review, code reviews can happen
• with little differences between each deployment, there are fewer risks to deploy
• with more minor changes in each commit and a more apparent purpose, there are better-quality commit messages

Helping iterations

There are ways to help such an approach by adopting tools such as feature switches and canary deployments.

Feature switches

By now, all programming languages have a way to handle feature switches: a programmatic way to turn on or off a feature. The switch can happen based on an environment variable, a database value, or the current user's profile.

This is handy for hiding away a feature while the initial work is done. This can also allow having both versions of a feature co-existing in the code base, only to have one replace the other at a specific date or under certain conditions.

While it can add complexity to the code base, it can help the team iterate faster. With modern feature switches, the result of each iteration can be displayed and used by only a specific set of users, thus allowing for a gradual rollout. This will avoid leaving code unused for too long.

Canary deployments

A more advanced approach is to have a limited release of the deployment onto a subset of servers or only apply to a subset of users. The subset, called a canary group, will be monitored for the duration of the experiment to ensure all is well.

This is particularly handy for testing the whole application in a different setting rather than just a minor change. Depending on how it's done, this can be risky as it might entice the team to move back to large releases.

Remember the previous episode

Last week we covered the Pareto Principle. Iterating fast goes hand in hand with this principle: you want to keep moving forward, always aiming to use the fruits of the 20% of investment rather than lose time on hypothetical gains.

Knowing if an idea is good is more important than fulfilling your vision. Iterating fast will allow you to expose a lot quicker your idea to stakeholders and, ultimately, users.

This approach will require the whole engineering team and company to reimagine their work. Yet, as the engineering team starts to deliver continuously, everybody will become more confident that any change that can be imagined is only ever a few iterations away. Even if those changes begin in a limited manner, adjustments will be less costly and more prompt.

Yet, as discussed in a few posts, this approach works best in environments where:

• cooperation within the team and company is high
• messengers of issues and ideas are supported and trained
• bridging between teams is encouraged (no silos!)
• failures are studied and used to improve
• novelty is not feared and is implemented

Those traits are related to generative cultures (R. Westrum). If you are considering such an approach, be aware that you will need to question several cultural points within your evolving organization.

Key takeaways

Before parting ways, review what you should take away from this post.

1. Long iterations are a source of risks due to the growing size of code changes, the low quality of code reviews they imply, high uncertainty they cause
2. Short iterations help reduce the risks by ensuring changes happen in smaller portions, thus removing two causes of risk: large size of the change, skimmed code reviews
3. Short iterations promote a purposeful approach to any code change, thus helping engineers keep their work atomic, clean, reviewable
4. Short iterations go hand in hand with Feature Switches, which allow for a soft rollout of what they introduce

Who are we, by the way?

This content is written and published by Imfiny/Pier22, a consulting company based in France. We help CTOs, and Tech Leads grow their engineering team and stack.