This is a continuation of the blog post 12 Principles of Good Software Design.
Conclusions.
Unlike most professions, software developers face an interesting and rather unique conundrum. Our job fundamentally boils down to realizing a vision of a product–often defined by someone else (such as a project manager or by business needs), entirely out of thin air, creating out of whole cloth something sophisticated and advanced with few constraints except for the desired results.
And because we are creating out of whole cloth with few constraints–at most we may be told our software has to run on an iPhone or be deployed to an existing server cluster, or run on an embedded processor with far more memory and computing power than an advanced workstation from 20 years ago–we are completely free to choose how we express ourselves in code. So long as the results work, and can be maintained by a future generation of software developers, we can do anything we want.
It’s heady stuff.
The problem, however, is that despite our freedom of choice, despite a lack of constraints on how we do our job, despite a lack of prescriptions on how to design or build our projects that other professions face, the failure rates in the software development industry are amazingly high. Some studies suggest as many as half of all software projects will fail–either by far exceeding the original budgets or being canceled outright. Even in the more limited field of Enterprise Resource Planning, where software packages and best practices are well defined and the project is simply adopting existing tools to an existing business, 51% of companies viewed their ERP implementation as unsuccessful. Worse, 31% of projects will be canceled before they are ever completed, while only 16% of projects (from the Chaos Report) are completed on-time and on-budget.
These are older reports and undoubtedly success rates have increased as we see a shift to more well established software practices. But not by much; my own personal experience is that the larger the project (as measured by programmers working on a project), the more likely it is that project will fail and code will never see the light of day.
“Failing is OK. Failing can even be desirable.” But we are not learning from our failures, and even practices put into place in order to “learn” from our failures (such as project postmortems and team introspection) offer us little insight, simply because the people making the mistakes are seldom the right people to ask about their mistakes. (It’s why so many postmortems of failed projects I’ve been on either turn into a lot of hand-wringing over “unforeseeable” errors, or turn into massive blame sessions and finger-pointing. Because human psychology being what it is, the most frightening thing any of us can face is the prospect of being wrong.)
I propose that the real problem is the fact that software developers face few constraints in how we build our code. So long as it works, we are theoretically free to build code any way we wish, testing our own pet theories or writing code we are comfortable with–or just banging out something that meets the requirements and makes our bosses happy (and gets us that promotion) without considering what it is we’re writing.
And I propose that our industry is slowly coming around to this view, and attempting to fix the lack of constraints with recommendations that attempt to impose constraints in the name of “clean code” or “best practices.”
But because as software developers we don’t like to face the prospect that our failures are being caused by too much freedom, people who propose best software practices face a delicate balancing act. Which is why so many discussions about software development describe “patterns” and treat them as if we are discovering fundamental truths buried in the aether, rather than building constraints which keep programmers from going off into the weeds.
Unfortunately because of this–because most people describing constraints to software development practices as if they are uncovering truths rather than laying out constraints–most of us feel free to “discover” (read: “invent”) our own “truths”, and strike out on our own, testing our own pet theories and pet methodologies on software projects. It’s as if we are the first to discover the Higgs Boson and are now using it to build our own anti-matter machines, not realizing that it’s a figment of our imagination and to outsiders we sound like a crazed hermit banging out screeds against modernity on our mechanical typewriters.
In other words, because we haven’t addressed the real problem–too much freedom–we’ve created an industry of lecturers who go around beating their own drum about best practices, which often either exacerbate the problem, or at the very least do nothing to address the failures.
And our projects continue to fail; they continue to be harder to maintain, continue to run over budget, continue to be canceled outright after wasting billions in our economy.
Research shows, by the way, that people who feel their choices are constrained are in fact happier than those whose freedoms are not similarly constrained. We do know that religious conservatives in the United States are in fact happier (despite being seen as stick in the muds incapable of happiness), in large part because their moral framework reduces the viable choices they see themselves facing in life.
I personally suspect the reason why people with fewer choices are happier is for the same reason that Conservatives are happier: not because we don’t swim in an endless sea of choices, but because a guiding principle reduces those choices from an overwhelming array to a manageable framework. By that same logic one would expect other groups employing the same strategy, constraining their choices to an acceptable array by using some core consistent philosophy, would see greater happiness regardless of political orientation. And it would suggest that moral relativists–people who do not believe morality is an absolute but is a choice relative to circumstances–would have greater levels of unhappiness because they have no framework to winnow down choices to a manageable set.
Certainly we also see this constraining of choices being used in other areas, such as by certain restaurants. It’s why fast food restaurants have resorted to allowing people to order meals by the numbers: reducing the size of the menu makes ordering from the menu less stressful. Higher-end restaurants (especially those which feature “farm to table” or “locally grown” choices) also effectively reduce choice to its patrons, but in the process increases satisfaction. We also see this constraining of choice at certain high-end gourmet food stores. Limiting choice to “organic” or “non-GMO” or “locally sourced” reduces the bewildering array of decisions shoppers face to an acceptable subset–even as those stores introduce unique products or flavors not seen elsewhere.
It’s not to suggest that outside forces should restrain choice for us, by the way. When our choices are constrained for us, we become increasingly unhappy as the choices we would like to make for ourselves are taken away, and our own internal agency is denied. Certainly restaurants who reduce the size of their menus don’t constrain our overall choice–simply because we could eat elsewhere, or go home and make food our ourselves and our friends.
But by having a framework for choice–a framework whereby we winnow down the external pressures shoving a million offerings down our throat to a handful of decisions that we find acceptable–we make our selves happy by reducing the stress that unlimited choice threatens to impose.
And I propose the same can be said about the unlimited choices faced by software developers on how we create our own applications.
For me, the goals of this series of blog posts was to propose a set of principles for code “idioms”–ways we write code–which make it easier on us, by constraining our choices to things that I personally found have worked for me.
Those “idioms” are based on the idea of constraining our choices according to certain ideas that I suspect most would find true–that software should be kept simple, understandable, and written in such a way so that teammates and future maintenance developers can understand and maintain the code.
Of course, we can’t always do that. Time pressures make us sloppy, and sometimes algorithms we need to implement are rather specialized or requires specialized understanding. But we certainly can strive to make our code less sloppy and to help future developers understand what is supposed to be hard (because of algorithmic complexity) and what should have been simple (but was over-engineered or improperly architected).
Certainly we can realize that these rules are not “discoveries” which express an underlying truth, such as we see in mathematics which seems like a blank slate but in fact describes the physical world. And because they’re not discoveries, perhaps we can realize that when we try to find our own “truths” and seek out our own “discoveries”–new design patterns, for example, or new ways to organize our code–we haven’t discovered anything.
We’ve simply gone off the rails.
Thus my “12 principles.” Because they’re less about design patterns and more about the goals we should have when we write code.
Keep it simple. Keep it concise. Keep it clear. Keep it easy to read. Don’t be “clever.” Don’t go off the rails.
Ultimately, it’s about recognizing our real problem: too much choice. And suggesting by keeping to a set of simple principles we can reduce choice–and reduce job stress, reduce uncertainty and reduce decision complexity in a way which allows us to increase our chances of success.
If suddenly your job becomes too simple because you stick to a set of simple, concise and clear expressions that you develop an itch to explore the frontiers of your knowledge–there are plenty of open source projects out there looking for volunteers.
Development Chaos Theory 