Complexity kills. Complexity sucks the life out of users, developers and IT. Complexity makes products difficult to plan, build, test and use… And as time goes on, and as software products mature—even with the best of intent—complexity is inescapable.~ Ray Ozzie, former chief software architect at Microsoft.
There are subjects…that can be appreciated only if you make an effort proportional to their innate complexity. To hearken back to something like the language of courtly love in medieval France, there are certain objects of our affection that reveal their beauty and charm only when we make a chivalrous but determined assault on their defenses; they remain impregnable if we don’t lay siege to the fortress of their inherent complexity.~ Christian Queinnec, in Lisp In Small Pieces (Cambridge University Press, 2003), p. xvi
The first of the two quotes atop this essay (from Ray Ozzie) clearly reflects the angst that accompanies the predicament of having to deal with accidental complexity—everything that does not directly relate to the solution of a given problem, but that we have to deal with nonetheless—while the second quote (from Christian Queinnec’s acrostically-titled Lisp book) reflects the gist of essential complexity, which is the core problem that we have to solve, usually consisting of the legitimately difficult problems to unravel (in software development as well as in other fields).
Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it.
~ Alan Perlis (Epigram # 58), in Epigrams In Programming
Enthusiasm is excitement with inspiration, motivation, and a pinch of creativity. Enthusiasm is not the same as just being excited. One gets excited about going on a roller coaster. One becomes enthusiastic about creating and building a roller coaster. ~ Bo Bennett
I think that in complex and ambiguous settings, people should actively speculate, instead of passively absorbing data… Experts distinguish themselves by their ability to anticipate what might happen next… By forming sharp expectancies, experts can notice surprises more readily.
We often have to make sense of highly uncertain situations, but uncertainty is isn’t always reduced by gathering more and more information. Sense-making is not just a matter of connecting the dots; sense-making determines what counts as a dot!
Turing discussed the dilemma between having machines programmed and having them learn… Why exactly are the results of human learning so difficult to replicate otherwise? … A programmer has to go much further in adapting the system being programmed than a teacher has to the learner…. Learning can be accomplished without needing anyone to explicitly understand either the state of the learner or the complexities of the world. A programmer would need to understand both to be successful
~ Leslie Valiant, in Probably Approximately Correct: Nature’s Algorithms for Learning and Prospering in a Complex World (Basic Books, 2013), pp. 164-65
Neal Ford, an application architect at ThoughtWorks, has a keenly discerning section in his excellent book The Productive Programmer (O’Reilly, 2008) that’s dedicated to bringing out the distinction between exactly the two types of complexities that we’re talking about here. He notes
No one sets out to spend more time dealing with accidental complexity than essential complexity. But lots of organizations end up doing little more than servicing the accidental complexity layers that accrue over time. Much of the current fad for SOA (service-oriented architecture) derives from companies trying to mitigate the amount of accidental complexity that has built up over time. SOA is an architectural style that binds disparate applications together so that they can communicate. Very rarely would you ever define that problem as the driving force of your business. It is, however, exactly what you’d do if you had lots of uncommunicative applications around that needed to share information. Sounds like accidental complexity to me. The vendorization of the SOA architectural style is the Enterprise Service Bus (ESB), whose chief selling point seems to be that the solution to the problem of middleware is more middleware. Does adding complexity reduce complexity? Hardly…
Identification is the first step in ridding yourself of accidental complexity. Think about the processes, policies, and technical mazes with which you currently deal. Noticing what is essential may lead you toward abandoning something whose contribution to the real problem outweighs the amount of complexity it introduces to the overall problem. For example, you may think you need a data warehouse, but the amount of complexity it adds to the overall problem isn’t worth the benefits it might provide. You can never kill all accidental complexity with software, but you can continually try to minimize it.
Java and C# in particular were designed to make C++ easier and less error prone, so the developers built-in some fairly serious restrictions in the interest of keeping average developers out of trouble…
Because these languages are general-purpose languages, you can get pretty much anything done…with enough effort. Java kept bumping into stuff that would be nice to do but was way too much work, so frameworks were built. And built. And built. Aspects were added. More frameworks were built…
~ Neal Ford, The Productive Programmer (O’Reilly, 2008)
Gradually an industry consensus emerged that the original EJB specification’s primary virtue—enabling transactional integrity over distributed applications—was of limited use to most enterprise applications, and the functionality delivered by simpler frameworks like Spring and Hibernate was more useful. Accordingly, the EJB 3.0 specification (JSR 220) was a radical departure from its predecessors, following this new paradigm…
Accordingly, in practical terms, EJB 3.0 is much more lightweight and nearly a completely new API, bearing little resemblance to the previous EJB specifications.
Since the previous two examples relate to accidental complexity, let’s look at one final example, this one rooted in essential complexity: Consider a comprehensive account of the semantics and the implementation of the whole Lisp family of languages, namely Lisp, Scheme and related dialects. All of that in the guise of detailed descriptions of 11 interpreters and two compilers, including very recent techniques of interpretation and compilation. Roll that up into a book and you have Christian Queinnec’s Lisp In Small Pieces (Cambridge University Press, 2003), which is mentioned atop this essay, in the second of the two quotes. I’ve read only portions of that book; if it isn’t essential complexity that Queinnec is dissecting in exquisite details, then, frankly, I don’t know what essential complexity is. Period. But I have a hunch—given the extremely high Signal-to-Noise Ratio (SNR) of ideas-to-pages, to the point where the defined features are reduced to a simple lambda-calculus, along with the introduction of denotational semantics—that Queinnec is indeed talking about the good type of complexity…
I will leave you with two interesting books to look up. It would be unfair to draw comparisons between those two books, though I’ll say this much that the second of the two is quite a bit more readable, while the first is more theoretical. Note, too, that the publication of these two books is spaced apart by a decade, so perhaps the intervening period of time led to a refinement in thinking about how better to tackle complexity in the heart of software…
Domain-Driven Design: Tackling Complexity in the Heart of Software~ Eric Evans (Addison-Wesley Professional, 2003)
Implementing Domain-Driven Design~ Vaughn Vernon (Addison-Wesley Professional, 2013)
It is hard enough to remember my opinions, without also remembering my reasons for them!~ Friedrich Nietzsche, as quoted in Cristopher Moore’s and Stephan Mertens’ The Nature of Computation (Oxford University Press, 2011)