sftwr2020

Taming Complexity in Software, and Elsewhere

Complexity kills. Complexity sucks the life out of users, developers and IT. Complexity makes products difficult to plan, build, test and use… And as time goes on, and as software products mature—even with the best of intent—complexity is inescapable.
 ~ Ray Ozzie, former chief software architect at Microsoft.
There are subjects…that can be appreciated only if you make an effort proportional to their innate complexity.  To hearken back to something like the language of courtly love in medieval France, there are certain objects of our affection that reveal their beauty and charm only when we make a chivalrous but determined assault on their defenses; they remain impregnable if we don’t lay siege to the fortress of their inherent complexity.
~ Christian Queinnec, in Lisp In Small Pieces (Cambridge University Press, 2003), p. xvi

The first of the two quotes atop this essay (from Ray Ozzie) clearly reflects the angst that accompanies the predicament of having to deal with accidental complexity—everything that does not directly relate to the solution of a given problem, but that we have to deal with nonetheless—while the second quote (from Christian Queinnec’s acrostically-titled Lisp book) reflects the gist of essential complexity, which is the core problem that we have to solve, usually consisting of the legitimately difficult problems to unravel (in software development as well as in other fields).

I will present some ideas from thoughts leaders in diverse fields on taming complexity in the fields of software, and beyond; OK, I confess to being a die-hard programmer, but nobody has accused me of being parochial, at least not yet. So think of this essay as a survey of sorts, rather than a precise study or, heaven forbid, a thesis thereof 😉
Fools ignore complexity. Pragmatists suffer it. Some can avoid it. Geniuses remove it. 
 ~ Alan Perlis (Epigram # 58), in Epigrams In Programming
As Russ Olsen observes in his fine book Eloquent Ruby (Addison-Wesley Professional, 2011)—incidentally, along with David Flanagan’s The Ruby Programming Language (O’Reilly, 2008), it is, in my opinion, the finest book ever written on the Ruby programming language—it takes a lot of self-discipline to deal with either accidental complexity or to engage in people management (a role that Olsen notes that he took on for a period of time) and which is the prime motivation for his pithy observation that follows. It is, in fact, a preamble to his thoughts on the subject, and where I elide to leave the rest to your imagination. Olsen notes: Instead of rolling into work every morning, eager to do battle with the essential complexity of the universe…
I find this notion—different strokes for different folks, or the seemingly endless variation in individuals as to how we get a kick out of our particularly chosen field(s) of pursuit—an endless source of ever-changing, ever-refining, kaleidoscope-like fascination. Psychologist and philosopher William James’ The Varieties of Religious Experience anyone? Speaking for myself, I love working as a software developer, but I also digress from the subject at hand, so I’ll refrain from tangentially veering off any further, save the following lovely quote that I recently came across 😉
Enthusiasm is excitement with inspiration, motivation, and a pinch of creativity. Enthusiasm is not the same as just being excited. One gets excited about going on a roller coaster. One becomes enthusiastic about creating and building a roller coaster. ~ Bo Bennett
While writing about the subject of anticipatory thinking, Gary Klein observes in his remarkable book Streetlights and Shadows: Searching for the Keys to Adaptive Decision Making (A Bradford Book, The MIT Press, 2011) that
I think that in complex and ambiguous settings, people should actively speculate, instead of passively absorbing data… Experts distinguish themselves by their ability to anticipate what might happen next… By forming sharp expectancies, experts can notice surprises more readily. 
We often have to make sense of highly uncertain situations, but uncertainty is isn’t always reduced by gathering more and more information. Sense-making is not just a matter of connecting the dots; sense-making determines what counts as a dot!
Leslie Valiant—the T. Jefferson Coolidge Professor of Computer Science and Applied Mathematics at Harvard University—shares some delightful insights that vivify and underscore the nature of essential complexity in the computational enterprise when he notes in the chapter entitled Machines as Ecorithms: Why is artificial intelligence difficult to achieve? in his fine book Probably Approximately Correct that

Turing discussed the dilemma between having machines programmed and having them learn… Why exactly are the results of human learning so difficult to replicate otherwise? … A programmer has to go much further in adapting the system being programmed than a teacher has to the learner…. Learning can be accomplished without needing anyone to explicitly understand either the state of the learner or the complexities of the world. A programmer would need to understand both to be successful
~ Leslie Valiant, in Probably Approximately Correct: Nature’s Algorithms for Learning and Prospering in a Complex World (Basic Books, 2013), pp. 164-65

Neal Ford, an application architect at ThoughtWorks, has a keenly discerning section in his excellent book The Productive Programmer (O’Reilly, 2008) that’s dedicated to bringing out the distinction between exactly the two types of complexities that we’re talking about here. He notes

No one sets out to spend more time dealing with accidental complexity than essential complexity. But lots of organizations end up doing little more than servicing the accidental complexity layers that accrue over time. Much of the current fad for SOA (service-oriented architecture) derives from companies trying to mitigate the amount of accidental complexity that has built up over time. SOA is an architectural style that binds disparate applications together so that they can communicate. Very rarely would you ever define that problem as the driving force of your business. It is, however, exactly what you’d do if you had lots of uncommunicative applications around that needed to share information. Sounds like accidental complexity to me. The vendorization of the SOA architectural style is the Enterprise Service Bus (ESB), whose chief selling point seems to be that the solution to the problem of middleware is more middleware. Does adding complexity reduce complexity? Hardly…
Identification is the first step in ridding yourself of accidental complexity. Think about the processes, policies, and technical mazes with which you currently deal. Noticing what is essential may lead you toward abandoning something whose contribution to the real problem outweighs the amount of complexity it introduces to the overall problem. For example, you may think you need a data warehouse, but the amount of complexity it adds to the overall problem isn’t worth the benefits it might provide. You can never kill all accidental complexity with software, but you can continually try to minimize it.
To my mind—and as it would apply to effectively maintaining existing (software legacy) code bases—Ford’s preceding remark about continually try[ing] to minimize it (viz. complexity) sounds like a call to continually refactor software!  Martin Fowler has literally written the landmark, canonical book on refactoring—the process of restructuring existing computer code without changing its external behavior—and the details are well worth looking up by starting here…
In rounding out the discussion (as it applies to the two types of complexities that we’re talking about here), Ford notes that
Java and C# in particular were designed to make C++ easier and less error prone, so the developers built-in some fairly serious restrictions in the interest of keeping average developers out of trouble… 
Because these languages are general-purpose languages, you can get pretty much anything done…with enough effort. Java kept bumping into stuff that would be nice to do but was way too much work, so frameworks were built. And built. And built. Aspects were added. More frameworks were built…
~ Neal Ford, The Productive Programmer (O’Reilly, 2008) 
At this point, I’m going to throw this out there: Anyone even want to remember what it was like dealing with Enterprise JavaBeans (EJB), which is a managed, server-side component architecture for modular construction of enterprise applications? Dare I say that they can admirably serve as the poster child of accidental complexity? In all fairness, the EJB specification has of late undergone an extreme makeover
Gradually an industry consensus emerged that the original EJB specification’s primary virtue—enabling transactional integrity over distributed applications—was of limited use to most enterprise applications, and the functionality delivered by simpler frameworks like Spring and Hibernate was more useful. Accordingly, the EJB 3.0 specification (JSR 220) was a radical departure from its predecessors, following this new paradigm… 
Accordingly, in practical terms, EJB 3.0 is much more lightweight and nearly a completely new API, bearing little resemblance to the previous EJB specifications.
Funny, isn’t it, that we keep coming back to the bare simplicity of the basics, that we almost sense the comfort of having arrived when we find ourselves dealing with the good type of complexity? Yes, the good type of complexity—essential complexity, which consists of the legitimately difficult problems to solve, as well as hard material to master, in software development as well as in other fields—is always there to be mined, but to reach out to it, we have to rise above the currents swirling around in the sea of accidental complexity so we can focus on the essential complexity, and thereby make headway and progress toward our goals.

Since the previous two examples relate to accidental complexity, let’s look at one final example, this one rooted in essential complexity: Consider a comprehensive account of the semantics and the implementation of the whole Lisp family of languages, namely Lisp, Scheme and related dialects. All of that in the guise of detailed descriptions of 11 interpreters and two compilers, including very recent techniques of interpretation and compilation. Roll that up into a book and you have Christian Queinnec’s Lisp In Small Pieces (Cambridge University Press, 2003), which is mentioned atop this essay, in the second of the two quotes. I’ve read only portions of that book; if it isn’t essential complexity that Queinnec is dissecting in exquisite details, then, frankly, I don’t know what essential complexity is. Period. But I have a hunch—given the extremely high Signal-to-Noise Ratio (SNR) of ideas-to-pages, to the point where the defined features are reduced to a simple lambda-calculus, along with the introduction of denotational semantics—that Queinnec is indeed talking about the good type of complexity…

I will leave you with two interesting books to look up. It would be unfair to draw comparisons between those two books, though I’ll say this much that the second of the two is quite a bit more readable, while the first is more theoretical. Note, too, that the publication of these two books is spaced apart by a decade, so perhaps the intervening period of time led to a refinement in thinking about how better to tackle complexity in the heart of software…

  • Domain-Driven Design: Tackling Complexity in the Heart of Software
     ~ Eric Evans (Addison-Wesley Professional, 2003)
  • Implementing Domain-Driven Design
     ~ Vaughn Vernon (Addison-Wesley Professional, 2013)
…or at least that’s what I think at this point in time, because a decade is an eternity for any field—the software development field conspicuously included—that lives at the intersection of technology and creativity 😉
It is hard enough to remember my opinions, without also remembering my reasons for them!
 ~ Friedrich Nietzsche, as quoted in Cristopher Moore’s and Stephan Mertens’ The Nature of Computation (Oxford University Press, 2011)

One comment

Your Comment Here!

This site uses Akismet to reduce spam. Learn how your comment data is processed.