Contagion is exactly why interfaces are one of the most important pieces of design and should be given significant thought. A beautiful interface with a suboptimal implementation can be easily cleaned up when time is allotted. The reverse is rarely true.
I gotta say, it's pretty amazing to me that this was written by an engineering manager. None of the EMs I've worked with would be capable of discussing our codebase at this level of technical detail. Even the ones that used to be engineers.
Although to be fair, we don't have any EMs who were promoted from within. We have a bad habit of hiring managers from outside, as nobody internally really wants to stop doing engineering (myself included).
Great article, from a technical perspective! I would say it’s more a “nomenclature” than a “taxonomy” because it’s neither exhaustive nor discrete (by design), but I might be mistaken there. I loved the physical examples for each especially, really thought provoking.
As always, I have a philosophical nit to pick: the “three axes” introduced at the top are just “Return” and “Investment” from good ol’ RoI, with a subcategory added for a particular type of forward-looking/conditional Return. I’m guessing this decision has worked in practice and I don’t expect video game development practices to be absolutely scientifically sound, but some extra philosophical certainty never hurts!
One important aspect is when you knowingly take on tech debt in return of some short-term benefit. Then this benefit becomes an other axis to weigh against.
My experience at big corporate is that (edit: unmanageable) tech debt is caused by undisciplined and unorganized scrum team.
When you have a proper backlog of tickets, including tech debt tickets, the team will eventually fix the tech debt when there are not enough feature tickets to exhaust capacity.
Agile is perfectly optimised for creating tech debt. Corporate software is almost always impossible to change once released so it’s obvious that frequent iterative deliverables that you can only code around or on top of propagate technical debt
> I have yet to visit this misterious universe you describe.
The trick is to have 1 backlog. Tech debt and features live on the same list and it is up to the PM to prioritize. Engineering’s job is to argue cost.
Good PMs will prioritize relevant tech debt or pull it in with feature work in the same area. They understand the tradeoff of go slow to go fast. They also understand when tech debt will never become relevant (because the feature is getting nixed, or hasn’t shown desired impact yet, or because the cost of interest is waaaay lower than the cost of paying it off in many cases).
This only works when engineers have the discipline to look stinky awful code in the eye and say “not today” and stay within agreed timeboxes. You blow this estimate once or twice, get the PM in hot water with leadership, and you’ve lost the trust.
All of the teams I've been on have used one list. I've never seen a PM prioritize the technical work. I still think it's a good idea for it to be one list, but it's not sufficient.
For teams that don't have a good PM, you also need a tech champion. Failing that, engineers need to inflate estimates and do tech work under other stories. Then everything becomes less predictable and teams never develop trust.
All PMs I have seen so far were just passing on management’s desire for more features quickly. The only approach I have seen work is if engineering adds refactoring as part of the normal work that needs to be done without asking for permission.
That's the practical advice to engineers who are stuck in a dysfunctional organization where they can't really effect change, which is probably 90%+ of all organizations.
> For teams that don't have a good PM, you also need a tech champion.
That's part of the role of a Technical Program Manager. The Eng Manager, Product Manager, and TPM should form a holy trinity of mutual support, filling in for each other's gaps. When that happens, you get much better odd of having a high performing team.
Source: I've been both an engineering manager and a TPM. Never the PM, though.
> For teams that don't have a good PM, you also need a tech champion
Yes. And to add some nuance, you need a [trusted] engineer who can say “This will take 3 weeks because of tech debt items A, B, C. We can fix those in 1 week and then take 1 week to implement this. How would you like to proceed?”
Any decent PM will take the 2 week option that also cleans up the codebase.
But if fixing the tech debt would take 3 weeks and then another 2 weeks to build the feature, then any decent PM will take the option that doesn’t fix tech debt unless there’s a bunch more stuff coming in this area in which case taking 3 weeks to fix stuff is totally worth it.
Their job is to make those tradeoffs. Our job is to highlight the tradeoffs they’re making so they can make informed decisions.
> Our job is to highlight the tradeoffs they’re making so they can make informed decisions.
This is an oft-stated thing that I oft-disagree with. It states that engineers ought to be subordinate to PMs, which shouldn't always be the case.
If you have shit engineers and great PMs, the best outcome is likely to shift decision making to PMs. If you have great engineers and shit PMs, decision making should shift towards engineers.
If they are both equivalently shit or great, it should be a balance. I believe this is the most likely scenario. I believe that balance is thrown out the window if engineers "highlight the tradeoffs" while the actual decision making is lies with the PMs.
How to actually achieve balance is extremely idiomatic to the team and organization. It's hard to get people to have adult, non-confrontational discussions about this sort of thing, however. Too many people will treat it as a negotiation.
> This is an oft-stated thing that I oft-disagree with. It states that engineers ought to be subordinate to PMs, which shouldn't always be the case.
I think of it more as a partnership.
If I’m in charge of getting groceries and you’re in charge of budgets, we need to have an informed discussion on what exactly is our budget and what food we need so we don’t starve. Sure I could blow the whole budget on steak and I might even love eating nothing but steak for 3 days, but eventually some carbs would be nice. Likewise neither of us will be happy if I go max stingy and buy nothing but bags of rice for the week.
The reason I think PMs should make the final call is not that engineers are subordinate, it’s that PMs are accountable. (RACI – responsible, accountable, consulted, informed). The person whose ass is on the line makes the call.
Usually when I ask engineers if they want to be accountable for making the call (and its outcome), things get real quiet real fast :)
“Not enough feature tickets to exhaust capacity” - I don’t think I’ve ever seen this happen :) PMs and sales always manage to book all available capacity.
> My experience at big corporate is that (edit: unmanageable) tech debt is caused by undisciplined and unorganized scrum team.
Yeah, this is 100% correct. I comically left Riot after ~6 months for this exact reason. Obviously it's a large company with many different flavors of teams, and it sounds like this team maybe has gotten it together, but by in large most haven't.
While I was there I was working on some of their core games tooling and felt uneasy about my day-to-day. My teams tech debt was quite literally owning them. Constantly missing sprint scopes, spending countless hours arguing and debating about trivial stuff, it was all a mess. They ended up laying off a number of people from that team in a pretty shifty manner so maybe things have gotten better since then.
Contagion is exactly why interfaces are one of the most important pieces of design and should be given significant thought. A beautiful interface with a suboptimal implementation can be easily cleaned up when time is allotted. The reverse is rarely true.
I gotta say, it's pretty amazing to me that this was written by an engineering manager. None of the EMs I've worked with would be capable of discussing our codebase at this level of technical detail. Even the ones that used to be engineers.
Although to be fair, we don't have any EMs who were promoted from within. We have a bad habit of hiring managers from outside, as nobody internally really wants to stop doing engineering (myself included).
Discussed at the time:
A Taxonomy of Technical Debt - https://news.ycombinator.com/item?id=16810092 - April 2018 (113 comments)
also this bit:
A Taxonomy of Tech Debt (2018) - https://news.ycombinator.com/item?id=39782923 - March 2024 (1 comment)
Great article, from a technical perspective! I would say it’s more a “nomenclature” than a “taxonomy” because it’s neither exhaustive nor discrete (by design), but I might be mistaken there. I loved the physical examples for each especially, really thought provoking.
As always, I have a philosophical nit to pick: the “three axes” introduced at the top are just “Return” and “Investment” from good ol’ RoI, with a subcategory added for a particular type of forward-looking/conditional Return. I’m guessing this decision has worked in practice and I don’t expect video game development practices to be absolutely scientifically sound, but some extra philosophical certainty never hurts!
Great article. The "contagion" factor is a useful concept that I hadn't seen before. Needs a [2018] tag.
One important aspect is when you knowingly take on tech debt in return of some short-term benefit. Then this benefit becomes an other axis to weigh against.
My experience at big corporate is that (edit: unmanageable) tech debt is caused by undisciplined and unorganized scrum team.
When you have a proper backlog of tickets, including tech debt tickets, the team will eventually fix the tech debt when there are not enough feature tickets to exhaust capacity.
Agile is perfectly optimised for creating tech debt. Corporate software is almost always impossible to change once released so it’s obvious that frequent iterative deliverables that you can only code around or on top of propagate technical debt
> the team will eventually fix the tech debt when there are not enough feature tickets to exhaust capacity
I have yet to visit this misterious universe you describe.
> I have yet to visit this misterious universe you describe.
The trick is to have 1 backlog. Tech debt and features live on the same list and it is up to the PM to prioritize. Engineering’s job is to argue cost.
Good PMs will prioritize relevant tech debt or pull it in with feature work in the same area. They understand the tradeoff of go slow to go fast. They also understand when tech debt will never become relevant (because the feature is getting nixed, or hasn’t shown desired impact yet, or because the cost of interest is waaaay lower than the cost of paying it off in many cases).
This only works when engineers have the discipline to look stinky awful code in the eye and say “not today” and stay within agreed timeboxes. You blow this estimate once or twice, get the PM in hot water with leadership, and you’ve lost the trust.
All of the teams I've been on have used one list. I've never seen a PM prioritize the technical work. I still think it's a good idea for it to be one list, but it's not sufficient.
For teams that don't have a good PM, you also need a tech champion. Failing that, engineers need to inflate estimates and do tech work under other stories. Then everything becomes less predictable and teams never develop trust.
All PMs I have seen so far were just passing on management’s desire for more features quickly. The only approach I have seen work is if engineering adds refactoring as part of the normal work that needs to be done without asking for permission.
That's the practical advice to engineers who are stuck in a dysfunctional organization where they can't really effect change, which is probably 90%+ of all organizations.
> For teams that don't have a good PM, you also need a tech champion.
That's part of the role of a Technical Program Manager. The Eng Manager, Product Manager, and TPM should form a holy trinity of mutual support, filling in for each other's gaps. When that happens, you get much better odd of having a high performing team.
Source: I've been both an engineering manager and a TPM. Never the PM, though.
Perhaps that can work, but I'm skeptical whenever the solution is "another manager".
> For teams that don't have a good PM, you also need a tech champion
Yes. And to add some nuance, you need a [trusted] engineer who can say “This will take 3 weeks because of tech debt items A, B, C. We can fix those in 1 week and then take 1 week to implement this. How would you like to proceed?”
Any decent PM will take the 2 week option that also cleans up the codebase.
But if fixing the tech debt would take 3 weeks and then another 2 weeks to build the feature, then any decent PM will take the option that doesn’t fix tech debt unless there’s a bunch more stuff coming in this area in which case taking 3 weeks to fix stuff is totally worth it.
Their job is to make those tradeoffs. Our job is to highlight the tradeoffs they’re making so they can make informed decisions.
I agree with you completely that you need trust.
> Our job is to highlight the tradeoffs they’re making so they can make informed decisions.
This is an oft-stated thing that I oft-disagree with. It states that engineers ought to be subordinate to PMs, which shouldn't always be the case.
If you have shit engineers and great PMs, the best outcome is likely to shift decision making to PMs. If you have great engineers and shit PMs, decision making should shift towards engineers.
If they are both equivalently shit or great, it should be a balance. I believe this is the most likely scenario. I believe that balance is thrown out the window if engineers "highlight the tradeoffs" while the actual decision making is lies with the PMs.
How to actually achieve balance is extremely idiomatic to the team and organization. It's hard to get people to have adult, non-confrontational discussions about this sort of thing, however. Too many people will treat it as a negotiation.
> This is an oft-stated thing that I oft-disagree with. It states that engineers ought to be subordinate to PMs, which shouldn't always be the case.
I think of it more as a partnership.
If I’m in charge of getting groceries and you’re in charge of budgets, we need to have an informed discussion on what exactly is our budget and what food we need so we don’t starve. Sure I could blow the whole budget on steak and I might even love eating nothing but steak for 3 days, but eventually some carbs would be nice. Likewise neither of us will be happy if I go max stingy and buy nothing but bags of rice for the week.
The reason I think PMs should make the final call is not that engineers are subordinate, it’s that PMs are accountable. (RACI – responsible, accountable, consulted, informed). The person whose ass is on the line makes the call.
Usually when I ask engineers if they want to be accountable for making the call (and its outcome), things get real quiet real fast :)
> it is up to the PM to prioritize. Engineering’s job is to argue cost.
That’s a lot of words to say “more features lol” which is basically what every PM I’ve worked with has only wanted.
“Not enough feature tickets to exhaust capacity” - I don’t think I’ve ever seen this happen :) PMs and sales always manage to book all available capacity.
if tech debt would depend on some kind of methodology it would not pop up with XP/Kanban/waterfall.
techdebt can even pop up in unorganized slowmo opensource software.
> My experience at big corporate is that (edit: unmanageable) tech debt is caused by undisciplined and unorganized scrum team.
Yeah, this is 100% correct. I comically left Riot after ~6 months for this exact reason. Obviously it's a large company with many different flavors of teams, and it sounds like this team maybe has gotten it together, but by in large most haven't.
While I was there I was working on some of their core games tooling and felt uneasy about my day-to-day. My teams tech debt was quite literally owning them. Constantly missing sprint scopes, spending countless hours arguing and debating about trivial stuff, it was all a mess. They ended up laying off a number of people from that team in a pretty shifty manner so maybe things have gotten better since then.