Systems-Thinking for Software Projects
Software projects keep failing, not because we don’t have the right team or tools but because our software development system is broken. Out of the Software Crisis is a guide to fixing your software projects with systems-thinking making them more resilient to change and less likely to fail.
Most software projects fail…
- If they don’t go over time or over budget, they don’t fulfil their requirements.
- If they fulfil the requirements, then the requirements were wrong.
- If we did the user research, got the requirements right, and delivered on time and within budget, then we still fail because bugs spawn out of nowhere as soon as it gets in front of an actual user.
- The frameworks and dependencies that were supposed to save time keep changing, forcing rewrites.
- All this causes subsequent sprints and projects to go over time and over budget.
Even when a project goes well, that never lasts because software isn’t just for one project. Software is forever. Or, it feels like it is. It always seems to go wrong eventually.
We try to fix it with better tools and processes. Things improve for a while. Then everything starts to fall apart again, just in slower motion.
Coders and non-coders, designers and non-designers, managers and direct reports, we all try our best, and stick to our field’s best practices—we’re doing the right thing—but success never materialises.
None of our projects work nearly as well as we’d hoped. Our careers are a succession of half-baked software.
We keep writing code, but we never seem to get anywhere. We tick all the boxes and follow all the processes, but the schedule keeps slipping. Bugs keep multiplying. We’ve tried every tool and framework, but nothing seems to help…
We’ve got the design team running smoothly. The team has been churning out great designs that the managers love, but all we got after the release were complaints. The users hate it…
We’re keeping pace, every sprint is technically a success, but it keeps getting harder and harder. I don’t know how much longer we can keep this up. Our core dev has just started flaking on us…
We’re doing everything right—we’re agile and follow all of the best practices—but this still isn’t working.
You think you’re ready to make software
You’re a manager and you’ve got your processes running like clockwork. The deadlines are realistic. The requirements are clear. You’re using a hot new framework. It got you up and running in record time. The designer is cranking out designs like a machine. Your team is fired up. You’re fired up.
Or, you’re the tech lead or senior developer and you’ve been working with your manager on platform choices, testing strategy, language choice, and framework. The project’s technical decisions are sound—based on current best practices. It’s exciting. This time you got to make the right technical choices. You didn’t have to compromise because of managerial whim. It’s going to be great.
Then it turns to shit.
Nothing goes as planned. Everybody underdelivers and underperforms. You keep working on the process. You try everything. You cut back on your technologial ambitions. You try keep the code free of defects but time is running out. You follow all the best practices but aren’t getting the results you expected.
It should have been great, but it’s garbage.
You aren’t alone
You don’t suck.
Your team isn’t uniquely awful.
You aren’t doing any worse than average.
60-90% of all software projects fail. Big projects fail more than small ones. It all comes down to one inescapable fact: most software projects fail.
There must be a process out there that will get us on the right track? A method, tactic, or strategy? Maybe something technological? A programming language, or framework?
None of those are actual solutions. You don’t need a new framework. You don’t need a new programming language. You don’t need a new process. You can’t fix this with better tools because the tools aren’t broken.
The tools didn’t work for you because what was missing was a functioning system for the tools to work in.
Software development tools and processes, old and new, won’t work for you unless you understand software development as a holistic, integrated system.
Software development is an economic system whose primary purpose is to achieve business goals.
Fix your software development process with systems-thinking
Out of the Software Crisis is a framework-agnostic book for coders and non-coders. Some technical knowledge is helpful for reading but not necessary. If you know what a web server is but couldn’t program one, this book is still for you. If you are a coder, the book’s concepts will help you in your work and collaboration. This book is intended to be read by the entire, cross-functional team.
This book contains everything I wished everybody on every team I’ve worked on knew and understood about software development condensed into 155 pages. It’s for anybody who has authority over processes and technical decisions, even if it’s only partial, even if your ‘team’ is just yourself.
It will explain…
-
The foundation concepts that software development is built on, why processes work for some but not others, which activities are actually responsible for generating value and which aren’t, and what it is exactly that frameworks are doing for you.
-
How all of these disparate elements of the software development process come together to form a system and how you can improve your project outcomes by adjusting the whole—how they integrate—without throwing everything out and starting from scratch.
-
The basics of how to see software development through the lens of systems-thinking and it goes into detail on how the elements of software development differ from most other products or projects.
-
The business value of unit testing and types—why skipping them will have a direct effect on economic outcomes.
-
How design creates value for the business through experimentation.
-
How to understand frameworks as systems, what they are, what they do for you, how they help, and why they can sometimes be harmful.
Praise for Out of the Software Crisis
The only issue so far with @baldur’s book is how distracted I am by the typography. It’s a bit too good 😅
I’ve bought your book based solely on the extract about the business value of unit testing on the website. I read that and thought “YES! At last! Someone’s said it.” and wondered what else you have to say. I’m looking forward to reading it. :)
I read @baldur’s book ‘Out of the Software Crisis’ on the flight over yesterday and I’d strongly recommend reading it.
The book tries to frame a lot of software failures by moving up an abstraction from tactics like “hey did you assign the right story points” to a more nuanced look at how companies manage software as a whole
Hoping to beat my roughly 50/50 ratio of big software projects succeeding in my career with (Thing I’m working on) and this book gives me new primitives for thinking about it.
Reading Out Of The Software Crisis by @baldur and this hits hard:
“Most software companies overshoot their software development system complexity by a substantial margin because the margins on software businesses are so good. When times are good, everything you do seems to lead to growth. You begin to associate your random actions with random fluctuations in the market. Managers build mental models for how the market works that have no relation with reality.”
This is basically a book full of hot takes and I’m here for it.
“Out of the Software Crisis” by @baldur is a great book and well worth the read. Highly quotable, and really challenged my view of software development in a few places.
Too many great quotes, but here’s one that hit me in the gut:
“Truly catastrophic software failures rarely get discovered in a sprint review or shipping retrospectives. The serious bugs that induce panic attacks only appear once the software is distributed widely enough to begin to experience edge cases. Most of the time, that only happens long after the software development team has moved on and marked the project as successful or challenged.”
Love this quote about design from “Out of the software crisis” by @fakebaldur: “Decoration [is] the least important thing design does for software…[aesthetics] is part of what design does. It isn’t what you hire designers to do.”
Really great as an introduction to systems thinking as applied to software design. It isn’t used as much as a guide or resource, but as a way to get a lay of the land.
This book “out of the software crisis” by @baldurbjarnason is really good 👍
I cannot recommend it highly enough
Just finished @baldur’s new book “Out of the Software Crisis”. It’s a whistle-stop tour of the history and current state of #software development, and an intro to systems thinking. Very relatable if you’re a #webdev, and has inspired me to read more…
“Out of the Software Crisis” by @baldur is so good that I shared it with our executive management team. It’s short, non-technical, well-researched and identifies the key reasons software projects fail and how to improve the chance of success. Hoping this authoritative little book will help our team look beyond the waterfall/agile/safe/whatever dogmas to the importance of the wider organization in determining success.
Buy the bundle
Get the four-book bundle of my ebooks, Yellow: principles (or useless aphorisms) for software dev (ebook edition), The Intelligence Illusion and Out of the Software Crisis, and Bad Writing, in PDF and EPUB, for $49 USD together, almost a 45% discount off the combined price.
Buy the ebook bundle for $49 (plus tax and shipping)
Get the ebook
The book is 180 pages long, with the bonus essays, and comes in PDF and EPUB formats.
Every edition will get all future corrections to the ebook for free. If I decide to add bonus materials, every edition will get them, for free, in the next update.
Get the ebook for $35 (plus VAT, if applicable)
Excerpts from the book
Introduction
That is because the hardships have been lost in a haze of nostalgia. Software has always been bad. These apps may have been better designed, but they also didn’t have to account for as much variety in screen sizes or inputs. Most of them didn’t support proper multitasking, and stability was next to nonexistent. They crashed, a lot. Accessibility just wasn’t a thing. It’s like admiring the architecture of a classic building while forgetting that it’s a condemned deathtrap and every other building like it burned down, killing every inhabitant.
Try not to kill your software development system
The concept is that the world operates as multiple layers of interconnected systems and that the systems, not the elements of the system, are what defines overall behaviour and output.
A company’s performance isn’t down to the CEO or the employees but the system they operate in. A hospital’s overall effectiveness at healing people isn’t down to the skill of its doctors but how it’s organised and structured.
Much of the field is focused on theories on how you can change or modify systems—the levers you can pull to effect lasting change—but my focus here is simple: let’s try not to murder the system. We quite often begin with functioning development systems in software, and we don’t seem to have problems getting started. But we do have a knack for just absolutely murdering the hell out of the system afterwards.
Processes aren’t about managing the team or the project. They are feedback loops that provide elements of the system—in software that would be the team—with the information they need to adjust Flows and thereby affect Stock. The process’s effectiveness depends on the circumstance and how it connects to the rest of the system. There is no such thing as a prefabricated process. You always need to implement and adjust it to your needs. In software development, all processes work up to a point and in specific contexts. When in doubt, start simple and only add complexity if you absolutely have to. You may be told that you need to implement scrum or SAFe, but how you do it and hook it up to the rest of the system matters almost as much as the process itself. Details matter.
Variability is poison to software development
While defects are very important to any software project, the business benefit of unit tests is to preserve the future economic value of the code. This isn’t quite the same thing. Unit tests make the code resistant to being devalued by change. In software, the biggest threat to the economic value of code is that everything is always in flux. The business purpose of unit tests and types isn’t to prevent defects but to prevent variations in quality. Unexpected improvements in quality are just as undesirable as discovering a bug. Other modules may well be depending on the previous unimproved state. Unit tests serve as an early warning check that tells you something changed when it shouldn’t have.
Research & taste
I don’t care if you’re a backend developer, front-end developer, product manager, project manager, designer, or Swift/Java engineer, you need to collect good software, try it, experience it, and get to know what well-made feels like. You think a boom operator on a film set isn’t a film enthusiast? Half of them, easy, can recite the best lines from Coppola’s The Conversaton from memory.
Programming as theory-building
Software is the insights of the development team made manifest. Software has no life on its own but exists as a kind of cyborg simultaneously in the programmers and the code. To reuse Donna Haraway’s words, software is simultaneously fiercely material and irreducibly imaginary (p. xii, Gray 1995). Software can be real and unreal at the same time. The written code is how software interacts with end users and other software systems. The insights and knowledge that exist in the minds of the developers are how software lives, changes, and grows. Without the code, it has no way to interact with the real world. Without the knowledge in the minds of the developers, it has no way to adapt and survive.
Maturity, Complexity, & Layers
This is the flip side of the maturity question. Many mature frameworks have evolved alongside their exemplars—the apps and software that exemplify how the framework should be used. When you attempt to build something similar using that same framework, you will often fail miserably because you are trying to extend a complex system with complex interactions. The exemplary apps won’t have the same problem because the system was simple when their projects began and grew with it. This is a frequent issue with single-vendor frameworks. Whenever you have a framework whose primary purpose is to support a single product or service, you often end up with interwoven complexity that makes using it for a new service much harder.
Table of Contents
INTRODUCTION
PART ONE: SOFTWARE DEVELOPMENT SYSTEMS
- It was great until it wasn’t
- Try not to kill your software development system
- How do you make good software?
- Variability is poison to software development
- Design practice requires experimentation
- Intrinsic versus extrinsic innovation
- Programming as product design
- Research & taste
- Programming as theory-building
PART TWO: HOW WE FAIL
- The first Software Crisis
- The agile reaction
- The many kinds of software failures
- Silver bullets
PART THREE: FRAMEWORKS
- WTF do you mean when you say ‘framework’?
- You need to study exemplary software
- Check your needs
- Understand the paradigms
- Maturity, Complexity, & Layers
- In the end, what else is there but work?
WHAT NEXT?
- Further reading
- References
Refund Policy
If you aren’t happy with the purchase, I’ll give you a refund, no questions asked, even beyond the EU’s 14-day statutory requirement, up to a year from purchase.
The Author
My name is Baldur Bjarnason.
I’ve been a web developer for over twenty-five years. I’ve worked as a lead developer in the past. Lately, I’ve been working as a consultant. My clients in the past have included companies small and large, startups and enterprises, and both not-for-profits and for-profits.