0. Intro 🎈

Fog everywhere. Fog up the river where it flows among green airs and meadows; fog down the river, where it rolls defiled among the tiers of shipping, and the waterside pollutions of a great (and dirty) city… Chance people on the bridges peeping over the parapets into a nether sky of fog, with fog all round them, as if they were up in a balloon and hanging in the misty clouds.
― Charles Dickens (in Bleak House)

Oh, the irony. The macro and the micro about to commingle, at least judging by the title of this essay—Macro Fog: Or, Where Do My Micro Services Keep Going?—signifying that forces tiny and forces gargantuan have been set on a collision course. (And what to make of that fog metaphor? Gah.)

Relax.

All will become clear soon enough. And that’s an ironclad guarantee, or your money refunded…

Relax, too, secure in the knowledge that we’ll steer clear of micro-economics—and macro-economics, to be sure. Lovely subject both, but they’re for another day. What we’re after today is insight into the fine art of observability, especially as applied to the wonderful world of microservices, an art that has remained shrouded far too long in the mists of fog.

With that curious intro, what better than to dive right into parting the macro-fog. Inspired, though, by the dictum of all things proactive—”Think global, act local“—we will start with parting the micro-fog, and then work our way to the grand scheme of things, the macro-fog.

Ready?

1. Where Do Those Microservices Keep Going? 🐳

All that is gold does not glitter,
Not all those who wander are lost;
The old that is strong does not wither,
Deep roots are not reached by the frost.
― J.R.R. Tolkien (in The Fellowship of the Ring)

It starts innocently enough: You add a service—or two, or three—to your glorious application and soon enough the beatific menagerie of services appears headed in the direction of a bestial mess, one veering toward the vortex of chaos. (And no, we’re not even broaching the admittedly crucial topic of chaos engineering.)

What I’ve got for you here is vanilla observability (as applied to microservices, of course.)

Think of it this way: A software application (or a system) is useful only insofar as it remains tractable, one that we can reason through with confidence. The moment your application starts to slip on the cloak of opaqueness, operationally speaking—bye bye, observability of application behavior—chances are that trouble is brewing.

Yes, ditching that monolith made all the sense, but what have we got ourselves into where every outage (of your microservices-based application) has devolved into a murder mystery on a stormy night?

And be sure to think containerized services, such as the ones that may well be decking the galleys of the big ship above!

2. It’s Relay Races All The Way Down! 🏃

Successful innovation is not a single breakthrough. It is not a sprint. It is not an event for the solo runner. Successful innovation is a team sport, it’s a relay race.
– Nguyen Quyen

If all that talk above of (potential) doom-and-gloom got you down, I hasten to add: All is not lost. Oh no, not by a long shot. We software types are an optimistic breed. We got this. There’s nothing to fear but fear itself, amirite?

So let’s take a whiff of inspiration from advice I had doled out last time—Slow Down To Speed Up—and take a few deep breaths to soothe our senses.

Feeling a bit relaxed, we come to the juncture of metaphors that you’ll find suffused throughout this essay: Rain drops (inasmuch as they converge into a flood), and bricks (inasmuch as they “coalesce” into a magnificent edifice.)

Add to that dynamo duo of metaphors the ingredient of overlaying the accompanying pictures with a layer of one or the other, alternating between rain drops and bricks—oh, the trouble yours truly goes to in order to keep you inspired—and we have an incendiary combination of creativity on our hands.

Relay race-style, passing the baton on to the next runner, microservices are in the same groove, interoperating, coordinating one with another to get stuff done.

3. Boxed-Up, Sure, But Where Do We Ship ‘Em? 📦

It is no use to wait for your ship to come in unless you have sent one out.
– Belgian Proverb

“You scratch my back, and I’ll scratch yours,” so goes the adage. So yeah, the Belgians got this one right: “It is no use to wait for your ship to come in unless you have sent one out.”

Here’s the deal: Your daintily-boxed microservices—yep, the ones you’ve lavished your attention and care on, tending and growing them unto fruition—are now ready for prime-time. The engines are revving. But oh, did we instrument them (adding spans, tag, and all, to those microservices) so we can have a prayer of a chance to keep tabs on them?

You see, nothing beats a visualization of your system’s behavior, one that is dynamic and can be drilled-down into (to your heart’s content.) And that, exactly, is what becomes possible when you lean on a distributed systems tracing infrastructure. Gotta love open source!

So keep shipping those microservices; just be sure to instrument them before putting them into their proverbial boxes. “Once the horse has left the stable, a saying goes…

Onward.

4. Pipes & Filters, Good Old Pipes & Filters 🚧

Smart endpoints and dumb pipes
– Martin Fowler (with the lowdown on microservices)

Reams of paper have already been unrolled in the service of explaining the stateless nature of the Internet protocol that is HTTP. Instead of adding even more spools to that exercise, allow me to share a story, one that’s possibly apocryphal.

One evening, so the story goes, the late, legendary Norbert Wiener—yep, the originator of cybernetics, that being the science of communication as it relates to living things and machines—was heading for his home. This, according to that story, is what happened, reinforcing, if anything, Wiener’s eccentric and forgetful nature:

Norbert Wiener was renowned for his absent-mindedness. When he and his family moved from Cambridge to Newton his wife, knowing that he would be of absolutely no help, packed him off to MIT while she directed the move. Since she was certain that he would forget that they had moved and where they had moved to, she wrote down the new address on a piece of paper, and gave it to him. Naturally, in the course of the day, some insight occurred to him. He reached in his pocket, found a piece of paper on which he furiously scribbled some notes, thought it over, decided there was a fallacy in his idea, and threw the piece of paper away in disgust.

At the end of the day he went home – to the old address in Cambridge, of course. When he got there he realized that they had moved, that he had no idea where they had moved to, and that the piece of paper with the address was long gone. Fortunately inspiration struck. There was a young girl on the street and he conceived the idea of asking her where he had moved to, saying, “Excuse me, perhaps you know me. I’m Norbert Wiener and we’ve just moved. Would you know where we’ve moved to?” To which the young girl replied, “Yes Daddy, Mommy thought you would forget.”

Oh my! Evidently, Wiener was doing stateless before statelessness was anywhere near being enshrined in the HTTP protocol, amirite? The Internet, as we know it today—TCP/IP, HTTP, and friends—rests on the foundation laid by by the statelessness meme. (Oh, those memes, those naughty memes)

Which brings me to that “pipes and filters” reference you saw quoted up atop this section: Listen to the smart dude Martin Fowler when he opined on the wisdom of smart endpoints and dumb pipes. In a nutshell, aim for your applications to be decoupled and yet as cohesive as possible. Get there by leaning on the principles and protocols that power the world-wide web. (Also, while you’re at it, think to the abundantly wise UNIX philosophy in such matters.)

5. Tell Me, Is That Span Enough? ⛩

The longest bridge is the one where you don’t know which side to turn your face for the exit!
― Mehmet Murat ildan

Getting close to the end as we are, I say it’s time for some reinforcement, this having nothing to do—all you AI/ML aficionados, in case your ears pricked up—with RL (Reinforcement Learning.) Instead, it’s a brief throwback to the goodness of using spans, tag, and all while preparing your microservices for deployment to production.

So yeah, do look into instrumenting your microservices (adding spans, tag, and all, to those microservices.) You will sleep better. Much better. That’s the bus to get on board. “The wheels on the bus,” as the timeless rhyme goes…

If you’re on board, the question then becomes one of making sure that we are “spanning” enough, instrumenting enough, and doing so with gumption.

Don’t let this bridge be the one that’s far away… It’s all within reach, fellow observability aficionados. Let’s do our bit so that this fine art—it’s beautiful art in my book—emerges in all its glory. Bye bye, shrouding (and confounding) fog.

It’s nearly time to part. As will the fog (in good time.)

6. Goodbye, Monolith(s) 🌋

We replaced our monolith with micro services so that every outage could be more like a murder mystery.
@honest_update (on Twitter — Oct 7, 2015)

Undertaking the break-up of your majestic monolith (of an application) into loosely-couple microservices is an admirable (and smart!) idea in many situations. But, be prepared for some new challenges.

(You have that winning willpower. You got this.)

Since we’re talking about making life easier for us—remember, we’re in observability central—it behooves us practitioners to be mindful of the new twists and turns this journey will (surely) bring.

So do not take this journey lightly: There are no silver bullets.

Done right, microservices are doable. I’ve done them. And so can you.

7. Let’s Run With The Span, My Man! 🏰

Art is a dialogue. I’m throwing rocks across a chasm and hoping people catch them on the other side.
– Raphael Bob-Waksberg

Woohoo! The sprinter above—check the relay race baton-wielding leitmotif pops up again—has decided to run with it, collared shirt and all. Good for him.

I say that the Scottish Highlands are a perfect setting to inspire your foray into all things observability. The high altitude of the highlands, for one thing, provides the perfect testbed for checking out the parting of the fog we look forward to witnessing.

First, though, you got to put those crucial “spans” in your code—aka instrument the glorious code that powers your marvelous microservices. Once you got that, it’s off to the observability races, amirite?

Let’s make a dash, shall we?

(Hey, speaking of the highlands, is it time for another ode?)

8. Pieces Of The Puzzle Fall Into Place… 📺

I wanna make a jigsaw puzzle that’s 40,000 pieces. And when you finish it, it says “Go outside.”
~ Demetri Martin

Even as the pieces begin falling into place, we have this nagging feeling that there’s more. And right you are. You see, I could fit only so much into the space (and time) allotted to me; NP completeness notwithstanding, that is.

Don’t be sad, though. We’ll chat again. Soon enough, too. In fact, the rest of the story will be told in the second (and final) part of this two-part essay series, coming up next time.

So come visit us again. Soon.

Meanwhile, please do your part and stay safe; you and I, we’ll all be glad we did the right thing during the global pandemic.

Adios then, and be well.