0. Intro 🎈

It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of light, it was the season of darkness, it was the spring of hope, it was the winter of despair.
― Charles Dickens, A Tale of Two Cities

Check those avid aficionados above on the craggy ocean shore—this just might be the place where Kool and the Gang used to hang out—and see how they go about the time-honored pastime of spotting microservices, complete with a tripod-mounted camera, positively trained in a promising direction. Oh my, some of our aficionados can be seen excitedly pointing toward a fine (microservice) specimen that they’ve just spotted, amirite?

Hold up a sec!

Lest we get ahead of ourselves, let’s backtrack and pick things right up from where we left them (in Part 1 of this two-part series) when, among other points—including (1) the wisdom of “smart endpoints and dumb pipes“, and (2) the “advisability of instrumenting your microservices before putting them in production,” and so on—we had made a play by bidding farewell to your once-glorious monolithic applications, even as we cruised through the startlingly newfangled, (microservices) observability landscape.

Hence the Dickens quote above, which, in turn, warrants a deeper look into the ins and outs of  the “Goodbye, Monolith, and Hello Whodunit” notion, if you recall the hilarious—though altogether true—observation that “boring” got replaced by the “mysterious,” because

We replaced our monolith with micro services so that every outage could be more like a murder mystery.
@honest_update (on Twitter — Oct 7, 2015)

Do we have too much of a good thing on our hands here?

I say we sally forth and find exactly that out.

The alert reader—that’s you, of course—will have noted Charles Dickens make another marquee appearance on our blog, twice in a row most recently, in the previous essay with his much-maligned “Fog everywhere…” quote (from his direly-named novel Bleak House) and in this essay with his much-beloved “It was the best of times…” quote (from his noble novel named A Tale of Two Cities.)

1. We Emerge From The Mists Of Fog 🐳

A person with a new idea is a crank until the idea succeeds.
– Mark Twain

If the novelty of a tech idea be an indicator of—in the sense of being inversely proportional to—the adoption of that idea, I say we consider the following proposition: Ideas lose their novelty; some sooner, others after a while. And the notion of observability (as applied to microservices) falls squarely in the second camp. Put another way, while microservices themselves are mainstream, observing them isn’t. At least, not yet.

But to emerge from the potentially protracted fog of less-than-optimally managing microservices—clearly, nobody wants to be in that spot, with or without the headlights on—we clearly need to think outside the box instead of remaining reflexively hemmed-in, amirite?

Oh, and as we concluded last time (in Part 1), tend to their instrumentation  (i.e. add spans, tag, and all, to your marvelous microservices), and you’ll have a decent chance of keeping tabs on what they’re up to at any given time during their whizzbang happenings.

So yes, ditching that monolith made all the sense, but remember, too, to capitalize on making your software system observable. Do so and you’ll reap the rewards in spades.

With that, let’s haul in the catch, mate?

2. Did We Cast Our Net Too Wide? 🏃

Success is a science; if you have the conditions, you get the result.
– Oscar Wilde

Oh my, perhaps we should’ve kept our feet planted on the ground and not sidestepped extravagance. See that humongous net above? Yeah, the better part of reason—had it prevailed, that is—would surely have had us instead proceed with more focus and a smaller scope.

All is not lost, though. We learn from our mistakes, we all do; some of us, motivated by the impulse to memorialize, go one step further and commit our foibles, in the spirit of full disclosure. But I digress.

Refocusing, let’s just say that our current dilemma is no exception: Go for the low-hanging fruits first, and only then widen your net to catch bigger fish.

Lest this sound too abstract, allow me to bring it down to earth, because a fishy story this is not: Instrument away (the code that powers your microservices) at the highest level—think public-facing APIs—and you’ll get the biggest bang for the buck.

And even though the inimitable Wilde was likely not thinking microservices when he had opined as above—”Success is a science; if you have the conditions, you get the result”—the same principle is hard at work here.

And with that, we continue here in Part 2 what we had begun last time, in Part 1: You can expect to see, overlaid on the accompanying pictures, a layer of one or the other, the frolicking rain drops or those tiny, endearing bricks. (Oh, the trouble yours truly goes to in order to keep you inspired, this time with the metaphor of rain drops and bricks, to better symbolize the fine-grained nature of the critters—microservices—that we got on our hands.)

3. Goodbye Spaghetti Code, Hello Whodunit 📦

Memory is deceptive because it is colored by today’s events.
– Albert Einstein

Anyone remember a harangue or two about “spaghetti code”? I do, at least from some readings I did of admittedly lesser recency.

But hey, Einstein sure got this one, too, exactly right (“Memory is deceptive because it is colored by today’s events“) yet again in that it wasn’t too long ago that microservices—albeit in a more primitive form—were all the rage of our industry under the banner of the SOA (Service-Oriented Architecture) buzzword. (Buzz-phrase, I reckon: These buzzwords, and others, become eventually consistent; that’s my theory, anyway.)

What I’m getting at is this: Powers of memory notwithstanding, shifts in industry signify—or at least they should—shifts in our operating stance. Put another way, if “spaghetti code” was at least easily observable (though definitely not pretty to work with, or even to look at), distributed code is pretty to work with (but hard to observe!)

So if we don’t want to have an unending series of whodunits on our hands, it behooves us to—you guessed it!—make our distributed code ever more observable. We’ll all sleep better at night.

Trust me on this one.

Onward.

4. Tech Gridlocks (And How To Extricate) 🚧

When our vices leave us, we like to imagine it is we who are leaving them.
– Francois de La Rochefoucauld

I don’t know about you, but I’d rather not be in the driving seat—or the passenger seat, for that matter—of the bright-red car above, mirthlessly gridlocked as it is in a flock of spunky sheep who have clearly declared strong intentions of meandering in totally random directions and as they see fit; nothing sheepish there about their designs on driving cars.

Me, I prefer the linear route: Start at departure point “A”, arrive at point “B.” Sheep, not so much. Result: Gridlock.

“Akram, can you, like, stop speaking in parables?“, I hear someone as they sound a polite note or two of civilized demurral.

Sigh. (I pursue—and love—tech stuff all day long, for crying out loud. Can’t I so much as slip in a metaphor or two, and that too edgewise, on the rare occasion? Heh.)

So I was going to say: Pay special attention to structuring your application with observability in mind—remember, we’re in observability central—and while the (unique) twists and turns of your own journey will invariably unfold and run their course, at least you’ll be far likelier to get to your destination “linearly.”

Next, you might ask: What do you mean, “linearly?“

I could say, let’s start with “linearizable.”

Then there would be trouble; and that’s stuff I don’t want. So yeah, we’re not going there; at least not this time.

Onward.

5. Running Like Clockwork, Eh ⛩

Time destroys the speculation of men, but it confirms nature.
– Cicero

I love trains. And don’t anyone get me started on the fabulous Island of Sodor, an island where trains ruled…

But that image of slick trains above caught my attention. In turn, I wished to share it to better portray the clockwork rhythms of machines—mechanical or digital—running smoothly.

Talk of NTP notwithstanding—those time-synchronization undertakings sure can at times turn into the stuff of nightmares—this stuff (i.e. making microservices run like clockwork) can be cajoled into tractability. They can be mastered into submission; not a walk in the park by any means, but doable.

We humans are visual creatures, which begs the question: “Why aren’t we making the management of microservices (far) more visual?”

Put another way, let’s assume that you’ve done the legwork of putting in those crucial “spans” in your code—aka you’ve instrumented the glorious code that powers your marvelous microservices. Once you got that, it’s off to the observability races, amirite?

You’re off the hook, yay?!

Not so fast, pal. There’s still that soulful matter of some additional automation. But hey, Rome wasn’t built in a day, was it?

Heh, let’s just say that Cicero nailed it on the head in observing that “Time destroys the speculation of men, but it confirms nature.” So yeah, which shall it be: Our speculations on the one hand, or the juggernaut-like heft of nature (think here to the laws of Physics here) on the other?

6. Peering Up The Complexity Tower 🌋

Joy in looking and comprehending is nature’s most beautiful gift.
– Albert Einstein

It’s getting closer to parting time, and we haven’t even brought in a tower or two or three. (No, not that, you computer science aficionado, you: It’s not the Towers of Hanoi—the toy of choice for a countably infinite generations of budding computer scientists grappling with the concept of recursion for the first time—that we have in mind here.)

It’s the good old—or bad old, depending on your perspective—tower of complexity that we’re after. And why a tower? Why not a high rise, or, for that matter, why not a pyramid? Darn. (I sense a punitive tinge in thine tone.)

Before anyone makes me feel sorry for having quoted Einstein above—”Joy in looking and comprehending is nature’s most beautiful gift”—allow me to set matters straight: Complexity will smolder and rage, eventually blazing out of control, if not smothered while it’s merely kindling. Or, we gotta nip it in the bud, at its foundation, should you prefer the metaphor of structures.

Lest I tire you with this revel in metaphors—me, I never metaphor I didn’t like—think of it this way: A software application (or a system) is useful only insofar as it remains tractable, one that we can reason through with confidence. The moment your application starts to slip on the cloak of opaqueness, operationally speaking—bye bye, observability of application behavior—chances are that trouble is brewing.

Slay complexity—and it comes in many guises—and you emerge victorious, having, but of course, gloriously parted the fog of confusion while you were at it, amirite?

7. But Do We Have Telemetry? 🏰

In the case of news, we should always wait for the sacrament of confirmation.
– Voltaire

Check that shiny observatory above, that of course after wiping away those water-drops, to witness it the better, and in all its glory. (Surely you remember our running metaphor of rain drops and bricks to symbolize the fine-grained nature of the critters—microservices—that we’re finessing?)

With that, I urge you to mark the words of Voltaire, who happened to be a really smart dude from yesteryear: “In the case of news, we should always wait for the sacrament of confirmation.”

Telemetry is what happens, IMHO, when you got a channel—unidirectional or bidirectional or whatever—over which data is being streamed. Once you get the data on your end, this being Voltaire’s vaunted “sacrament of confirmation,” we make hay while the sun shines, so to say.

And exactly how do we know we’ve been made privy to that oh-so-gravitas-smacking “sacrament of confirmation“?

Enter, one more time, the powers of visualizing your splendid software system in action—that, too, in real-time—fellow observability aficionados.

We got this.

8. Keep Those Container-laden Ships Chugging Along 📺

We must cultivate our own garden. When man was put in the garden of Eden he was put there so that he should work, which proves that man was not born to rest.
– Voltaire

Goodness, this Voltaire guy is oh-so quotable; for one thing, he sure has the right metaphors when it comes to making microservices observable! Bro, that’s for sure: “Man“—and woman, I’m sure he meant to say, heh—”was not born to rest.”

I mean, ask the poor sysadmin who’s up at 2:00 AM, feverishly deploying a fistful of rapidly-assembled bash scripts to prevent a cascading meltdown of the software system, all because some service downstream was being recalcitrant, being stubbornly unresponsiveness. Sigh. You get the picture, right?

(And just between you and me: That hapless sysadmin will have a thing or two to tell you about toil. Chaos engineering, anyone? But I digress. Wow, I haven’t done that in a while, have I?)

In the end, do your bit to keep those “containerized”—given the prevalence of a certain containerization tech, “Dockerized“—microservices well-orchestrated and observable. Kubernetes will help with the former; and OpenTracing with the former. 

All right then, this is where we part ways, for now anyway, our journey drawing to a close.

So come visit us again. Soon.

Meanwhile, please do your part and stay safe; you and I, we’ll all be glad we did the right thing during the global pandemic.

Meanwhile, farewell, and be well.