Effective software engineering
How to ship and iterate faster
Software development is hard. Computers are dumb1 and programming is complicated. Even if our ideas about which feature or product to build and how to build it are brilliant, consumer is hard and people don’t always behave the way we want. Over time, companies realize iteration appears to be the only successful strategy instead of simply outsourcing.
Problems arise when the business views the software engineering team as a traditional cost center rather than a value driver. Sometimes you see the business having one foot in the cost center model and the other in the value driver. This dichotomy is probably not the only one out there, but the juxtaposition of differences invites us to understand why and how one can be more than merely problematic than the other.
Cost Center
Top-down solutions
Engineering builds what they are told
Value is created through other people ideating
Engineering is measured by cost to build and velocity against project roadmap
In the cost center view, the engineering team is merely implementing solutions that the planning people have already devised. In some cases, there is no representation from the engineering side in a planning meeting to come up with solutions. This increases coordination costs that exacerbate time spent back and forth communicating, discussing, justifying, and prioritizing the work to be done.
Value Driver
Engineering is tasked with problems that ladder up to company strategy
Engineering has access to relevant data to evaluate itself and make adjustments
Engineering is measured by the impact against those problems over time
On the other hand, the value driver ways of working prioritizes effectively iterating over problem space in place of upfront designing of solutions. The people doing the work need to be the same people who do the planning and evaluation. Otherwise you pay overhead cost. The people doing the work need to understand the problem they are trying to solve, not the solution other people think is best. The people doing the work need to be able to evaluate their progress. Otherwise, how do they know if they are doing the right thing?
These people are not only product and engineering but should include a dedicated data person or at least be given access to data to be able to independently rate their progress and improve the core business metrics over time.
Below are the steps you could take to transition engineering from cost center to value driver ways of working.
Step 1
Team leads including leadership agree on top-level company metrics and the company strategy to get there. This is simple. Almost all companies already do this to certain extent.
Step 2
The product team takes the company strategy and produces a product strategy that reflects what we should focus the engineering teams on to achieve our company strategy. This is not a product roadmap. For example, for the company strategy of selling product through third parties, we need to build capability to take orders from third parties and fulfill them and invest in experiences to encourage users that buy product from third parties to engage and purchase with us directly.
As part of the product strategy, the product team will work with engineering to understand what can be split into parallel teams and what metrics those teams will be responsible for. Those metrics will not be the top-level company metrics or high-level business levers such as LTV and new user acquisition but some lower-level proxies that the teams can impact directly.
Team leads will review and approve the product strategy and figure out which teams they need representation on and assign people. A team focused on new user growth should probably have someone from growth marketing. A team focused on personalizing the user experience should have someone from data science. It needs to feel like a single team, else we will waste overhead in periodic syncs and information gaps2
Step 3
Teams will iterate on the problem space and periodically share their learnings to the outside stakeholders e.g. team leads. Teams need agency to make decisions on their own without needing approval from outside stakeholders.
This continues until the list of company problems needs to be reprioritized, and ideally, reprioritization is on a predictable cadence like annual planning.
Benefits
Teams doing the work are directly connected to the impact of their work via the metrics we all agree on.
Different parts of the company can plug in at the execution level instead of the planning level. If different groups need to be involved to do effective planning, then we also need them involved for effective execution, otherwise the team is boxed in to whatever roadmap we decide without room or permission to maneuver.
Giving the teams control to change their roadmap allows them to reprioritize based on whether their metrics are heading in the right direction. Doing this at the local level versus a broader sync meeting eliminates overhead and allows the teams to be self-responsive rather than waiting for direction from above. It is critical that teams feel they have permission to make decisions.
With greater context and lesser overhead, all this leads to faster shipping and iteration which brings us closer to the vision of where the company can be.
Caveats
Of course, every way of working is not without drawbacks. In order for this to work well, we need to assume you have great people who are curious and driven to understand how their work is connected to broader business goals. Team leads must also ensure they are actively setting the right context and promoting parallel teams to have greater ownership and accountability for their roadmap. It’s all too easy to fall back on the old ways of working when in difficulty.
There will inevitably be occasional redundancy and ambiguity around which project falls under which parallel team. We could potentially mitigate this with consistent, creative communication tactics for team leads to actively resolve it and keep the entire team informed. Sometimes the size of parallel teams can get so large to the point that it can make people difficult to speak up in planning meetings.
While these are valid concerns, completely rectifying them could come at too great a cost to speed of delivery and iteration. If you want to build great feature or product that users will love, then we need to resist the tempting clarity of top-down mandates and move engineering closer to the value it brings to the business.
Ending Note
Thanks to Rachel Moon on Data Science Doodles for taking the time to review this post. Improvements have been made on the diagram thanks to her suggestions.
Will computers always remain dumb with the advent of AI? See: https://www.businessinsider.com/googles-artificial-intelligence-chief-john-giannandrea-says-computers-are-dumb-2015-10
The number of relationships (r) grows as a function number of people (n) per this equation: r = (n^2-n) / 2. Each relationship produces some amount of overhead costs. See: Hackman, J. Richard. Leading teams: setting the stage for great performances. Boston, Mass.: Harvard Business School Press, 2002. Print.