Whoa, Help Us Get Our Bearings!

It was a dark and stormy night… The pitch black sky remained sunk under mounds of darkness, only occasionally lighting up from the sporadic dazzling displays of lightning strikes. The stark storm bolts reminding me of the awesome power of nature; for all practical purposes, it was only during those fleeting moments of stormy lights that I was able to maintain my bearings on the desolate hillside where I found myself that night in the rustic countryside  ⛈

As I stood there, gingerly balanced on the parapet of the castle on the hill overlooking a deep gorge—peering intently into the deep gloom fore and aft which had swallowed up the night—an  especially ominous storm bolt seared the nighttime sky… 🏰

A Storm Bolt Kicks Off The Drama…

Did someone say something about… a Storm bolt, amirite? 💫

How could it be, far removed from civilization as I was at that time, that my geek psyche—upon the mere mention of a storm bolt—had reflexively jump-started my mind into high gear with the recent adventure with, what else, but a Storm bolt, of course.

All Alone In The Supercomputing Lab

It was a dark and stormy night… There I was, all alone in the supercomputing lab, with the power of practically limitless computing heft at my disposal: My fingers were flying across the keyboard as I wrestled with, what else, but a Storm bolt of course. Scheherazade-style, I immediately set about crafting the telling of the story in one of the languages I know best: the language of computers! 💻

Recursion And Automagic, Oh My!

Recursion writ large… Anyhow, here was the problem I was trying to solve (should any of my software savvy sisters and brothers wish to give me a helping hand, mostly for a software postmortem at this time) at the witching hour, all alone; supercomputers haven’t quite achieved sentience, yet, so I still feel confident that my earlier statement about my lonesomeness holds water 👻

Ah, yes, I was going to tell you about the vexing problem with the Storm bolt: If you all recall the beauty—so the adage that beauty is in the eye of the beholder is doubly true when it comes to us troublesome and troubled techie types—of using the Java class named BaseBasicBolt as our base class is that it “automagically” give us programmers two lovely gifts: (1) anchoring, and (2) acking. Storm does them both—anchoring and acking—equally well, swimmingly well! 🏊

Brave New World Lies Yonder

(Fair warning: Not that it’s a bad thing at all—in all fairness though to my non-technical readers—but the next two paragraphs contain technical content… I straddle the two worlds, the technical and the non-technical, caring deeply for each in equal parts, because that’s my thing: crafting essays at the intersection of culture, software, technology, and engineering. In fact, it was a scant few minutes ago that I heard from one of my dear, non-technical readers—she was providing feedback on an earlier draft of this essay—where she had the following poignant observation to share with me, far too valuable to miss sharing with you: “If George Orwell or Aldous Huxley were living today, what would they have to say about today’s computers and the way they speak in tongues? It’s the Tower of Babel all over again, as the uninitiated grope around blindly in a brave new world.” Ah, for one thing, it sure rings true with me, and for another, now if only I could write like she does; maybe one day, though I wouldn’t hold my breath; you’re pretty much stuck with me for now!) 👺

Eureka! I was onto my moment of enlightenment, woohoo! The lightbulb went off in my head; everything was clear, even as I pleaded the heavens to make it just so… Oh, to be unshackled from the chains of human bondage, all the while profusely apologizing to the hovering spirit of Somerset Maugham out there, somewhere, perhaps in the vicinity of where the caged bird sings… I know it now: the caged bird soared heavenward. I saw it with my own eyes.

All Your Dreams Are Strange, Love Comes Walking In…

Woohoo, problem solved, so it must be time for yours truly to go home, amirite? Well, not so fast, buster…

It slowly dawned on me that everything that I’ve finished narrating at this point—my adventure of deriving a unique solution all alone in the supercomputing lab—was merely a dream. Rats! A gloomy night like the one in which I found myself enshrouded, right out of something from the dark imagination of Edgar Allen Poe, sure can do strange things to one’s consciousness… 🎥

As I braced myself to absorb the full import of this realization, I took an ill-advised step on the castle’s parapet where, as you’ll recall, I was gingerly balanced all along, and then… I slipped!

Hurtling Down Into Empty Spaces

Next thing I knew, I was hurtling through space (barely half a foot away from the stony wall of the castle) but luckily, on making impact with the steep embankment at the foot of the wall, I fell—miraculously enough—such that my back rammed flush against the thick grassy slope, and whose cushion proved to be a lifesaver and a half!

Nonetheless, states change, shift happens, and I kept sliding down at breakneck speed down the grassy slope and headed straight for a Kafkaesque cave that threatened to swallow me whole 🙈

Remember how Neo—in the first movie of The Matrix trilogy—had perilously slid down the chute, thanks in large part to the rescue mission headed by Trinity, and commandeered, of course, by Morpheus? (Or perhaps like Michael Douglas and Kathleen Turner both riding the mudslide somewhere in Central America, in the fun movie Romancing the Stone!) I found myself in an eerily similar predicament… 🙉

Oh. My. God.

And Into The Kafkaesque Cave!

For a few moments—for all I know it might have been at eternity—I think I lost consciousness. When I came to, I found myself not in a Kafkaesque abyss, but rather what looked like the maw of a gloomy cave… Stalactites and stalagmites all over the place, no matter in which direction I don’t my head. It sure felt like being in a spooky cave… A handful of sonar detection-savvy bats screeched past my ears and confirmed as much! Oh my, out of the fire, and into the frying pan, oh my! 🐌

Mercifully enough, though, I had somehow managed to careen down the perilous slope largely unscathed other than some ragtag bruises along my back and legs. How that Kafkaesque cave came to be at the foot of the castle, or how that castle came to be at the top of the hillside above the cave, I will perhaps never find out… At that point In time, frankly, I couldn’t have cared less for such epistemological concerns!

Did someone say something about… Kafka, amirite? 💧

It was a dark and stormy night… There I was, all alone in the basement of a Menlo Park startup in California, with the power of practically unlimited computing heft—thanks to the marvels of cloud computing—at my disposal: My fingers were flying across the keyboard as I grappled with, what else, but a Kafka spout of course (I don’t think I was dreaming this time, or was I?) 👓

A Smashing Time With Cloud Computing

Needless to say, this particular Kafka spout—a programming conceptual construct of sorts—was the conduit out of the Storm topology and into my Apache Kafka topic! And there I was: I sipped cool lemonade from a tall tumbler standing next to my keyboard as I listened to a tune by Mark Knopfler (the song Golden Heart, if you really must know!) Life was good.

(Another fair warning: Not that it’s a bad thing—in all fairness though to my non-technical readers—but the next two paragraphs contain a fair amount of technical material… No rocket science for you techie types; just saying…) 🚀

So here was the problem—not half as vexing as the one involving the Storm topology which I had dreamily wrestled with earlier on, in the supercomputing lab as you’ll recall, but an interesting problem nonetheless—that I had set about solving with the Kafka topics at my beck and call: Remember good old compaction? Why yes, you know the routine where Kafka will normally store incoming messages for fixed periods of time, afterwards purging messages older than whatever the retention period happens to be…

Will You Change, Or Will You Stay The Same?

More interestingly, and closer to the topic at hand (pun intended even though we are generally chatting about Kafka topics), a software application can blithely use Kafka to store its current state. Anytime—make that every time—the state of the application changes, the application simply writes out the new state into a Kafka topic. And this is where the resilience guarantee of Kafka comes in: all that an application needs to recover (its latest state) is to reads those messages from Kafka to recover from a crash, and you’re done! 🚑

Alas, Reality Intrudes…

But here is the rub, the fly in the ointment, reality intruding into our serene tub: Remember what we read above about good old—make that bad old—compaction? Indeed. So there I was, having a blast with handling the power to change the retention policy on a topic from “delete” (recall how it deletes events that are older than retention time to “compact”, which, in turn, only stores the most recent value, per-each-key, in the topic)?

Ooh la la,  here, then, was the dastardly fly in the ointment—fly-swatter anyone?—cheekily staring me in the eyes! Darn, my Kafka topic contains null keys, and you know what happens in such situations, right? Yep, compaction will fail, miserably so. Yoohoo, Tony Hoare, now did you really have to make that billion dollar mistake, did you, really? Arghh… (Someone hand me Maalox, and that, too, by the bowls so I may ingest the antacid into my bowels!) 😭

The Joys Of Eventual Consistency

Hmm…  I wonder, I thought to myself, if perhaps Apache Kafka will allow me to change the retention policy on a topic from “delete” (which, as you might know, will delete events older than the retention time) to “compact” (which, as you might also know, will store only the recent-most value for every topic key available). I wonder…

Indeed, such are the joys of eventual consistency 😉

Just as I was getting warmer—not in the physical sense since it was a dark and stormy night, for crying out loud, dear well-wishers of mine—by the minute, realizing that I was on to a fabulous solution, I was awakened by another batch of stealthy bats screeched away, whooshing barely inches from my ears… 😳

I Make It To The Lip Of The Cave

Groaning and positively nonplussed that my magnificent Kafka solution would have to wait for another day, I creakily stood up, my back still hurting from the long skid along the hillside by the castle, and made it to the lip of the cave 😵

Darn, I Feel Dwarfed

Feeling small and lowly—like the boy Jack in the palm of the outstretched hand of his nemesis The Giant—there I was, all tattered and torn, broken and bereft. Thus it was that I looked up into the sky and saw… a spark! 🌟

Surely this couldn’t be happening, I thought to myself, rubbing my eyes in disbelief. My head reflexively jerked forward—the full impact of having been hit by the proverbial wall of bricks was only now beginning to make itself known to my body—and then, just as suddenly, my head jerked backward and I looked up again…

It was still there, that spark, hovering barely five feet above my head, suspended like a hummingbird in its selfsame incandescence!

Rescuing Machine Learning

Did someone say something about… Spark, amirite? 🔥

It was a dark and stormy night… There I was, all alone in the Miami penthouse, with the raw power of an extensive server farm of brawny computer at my beck and calling: My fingers were flying across the keyboard as I put together a Machine Learning (ML) solution with, what else, but a Spark topology of course 👕

Only For You

Due to its confidential nature—it’s for your eyes only and on a strictly need-to-know-basis—I can’t talk much about the monumental significance (both in scale and sophistication) of the ML solution I was carefully crafting in the luxurious Miami penthouse overlooking the soft and sandy moonlit beach which was frequented at this time by only the occasional crab or crustacean going out for their midnight snack… 🐙

Of Infinite Loops And Recursion

Surely I’m not dreaming this time, or am I? 😂

Immersed in my suspended state of recursion-writ-large (you know the routine about sometimes-non-terminating babushka doll inside babushka doll inside…), just as I was contemplating the lack of macros in the otherwise awesome language that is Scala—my ML solution sure could have used a Lisp style macro-driven boost right at that time—a gust of wind brazenly blew straight into my face and woke me up! Darn it!! 😴

Yet There I Was…

I let out an audible groan this time—darn those oh-so-elusive tantalizing dreams—doubly reverberating my  earlier contemplation of how I had, and not too long ago either, creakily stood up, my back still hurting from the long skid along the hillside by the castle, and made it to the lip of the cave. But there I was, standing above the cave’s lip 🚧

I looked up and momentarily saw—thanks wholly to the light from yet another lightning streak in the  nighttime skies above—a winding staircase carved into the rocky hillside merely a hundred feet away from where I stood. Yet again, as had happened just a bit earlier in the Kafkaesque cave, my head reflexively jerked down—the physical agony of having been hit by the wall of bricks was increasingly making itself known to my body—and then I looked up again…

Hope Springs Eternal In The Human Breast!

Woohoo! This would be my ticket, my escape from the sordid mess in which I found myself—you know the Kafkaesque abyss and the equally Kafkaesque cave where we had been spelunking. Truth be told, I wanted to give that staircase a bear hug right then and there (Hmm… On second thoughts, that might prove a tad too painful, you know, hugging the staircase and all!) 🐻

The Word In Spanish, I Mean, In Italian?

All the same, there it was, the loveliest staircase on which any human had cast their eyes. So lovely, in fact, that I got to thinking about the Italian word for a staircase: Scala!

Wait, wait, cut! (Part IV: Scala) 🎬 🎬 🎬 🎬

It Still Was A Dark And Stormy Night All Right

Did someone say something about… Scala, amirite? 🏃

It was a dark and stormy night… There I was, all alone in the Austrian Alps, listening to the sound of music softly playing all around me, surrounded even as I already was in fields of bright flowers that stretched in every direction until I just couldn’t make out the horizon 🌻 🌻 🌻 🌻

Somewhere In The Austrian Alps Now…

By the way, the transatlantic flights of late and cheap restaurant food have not done wonders for my health; in particular, my throat was hoarse as hell. So a word to the wise—especially for those of you into home remedies—that as I sat there squatting on a tuffet somewhere in the Austrian Alps (and hey, I wasn’t eating my curds and whey!) with my trusty MacBook balanced on my lap, I was gently sucking on some Ricola throat lozenges… (Good stuff, I’m telling you, that Ricola).

Dude, like, what was I doing all alone in the Alps? Frankly, I couldn’t care less how I had got there; all I knew—and frankly cared for—was that I found myself in a paradise-on-earth in the middle of springtime! On top of all that, I was hacking Scala code for crying out loud (If this be a foretaste of things to come, I’m toast!) Woohoo, life was good… 🏄

Let’s Get The Loveliness Of Macros Into Scala!

So here was the problem that I was trying to solve: implement and add the capability into the Scala programming language for users to create their own macros (Lisp-style). Oh yes!! (Out of thin air, the solution suggested itself to me; it came to me unbidden, so I really can’t even take credit for it). And did I mention that I make it a point to carry with me during my travels—transatlantic and otherwise—a vellum-bound notebook, Leonardo da Vinci-style, in which I write especially significant results and observations? For sure, my MacBook remains trusty as ever; it’s just that using a vellum-bound notebook on special occasions makes them, those occasions that is, well, special occasions! 🎨

Why, Hello Kitty

All excited, I sat down to scribble the magnificent solution—we programmers tend to think that something on the order of implementing and adding the capability to the Scala programming language to empower users with creating their own macros (Clojure-style) would qualify as a special occasion—in my vellum-bound notebook. But wait, did I hear a cat’s soft meow, almost a purr? Surely this wasn’t true; I mean, what would a true-blue member of the feline species be doing in the Austrian Alps, for crying out loud?! 🐱

In Honor Of Fermat’s Last Theorem

But hey—being the cat-lover that I am—I needed to get to the bottom of where the tantalizing feline sound was coming from… But before getting up, I scribbled something in the margin of my notebook only this much, even as the siren sounds of a cat somewhere in time beckoned wildly to me. I wrote: “Today I have discovered a truly remarkable software design (and implementation) for macros in the Scala programming language which, alas, this margin is too small to contain. Sigh.“

Then I went a-hunting for cats and, as they say, the rest is history… 📓

Right at that moment, a particularly stiff breeze blew into my face and woke me up 😴

Darn, waking up at the most momentously wrong moment—times number four, for crying out loud—is no fun! Dang!!

I let out yet another mournful groan—darn those oh-so-elusively tantalizing dreams—on realizing that Scala programmers hankering for macros (myself prominently included among them) would have to wait for another day. Oh well ⛵

Back At The Ranch

Meanwhile, back in the real word, things were looking up: the storm had mysteriously subsided into nothingness, into nothingness in the blink of an eye, and I found myself looking up into a clear nighttime sky. Much more importantly—for someone like me looking to lift himself out of the Kafkaesque furnace—the staircase ahead, a mere hundred feet away, was going to be my ticket to freedom 👟

So I took a step in that direction… Everything went well—and this is primarily for those dear readers who will settle for nothing less than the full gory details of our story in its winding down (denouement aka unraveling) phase—as I climbed up the staircase back to be reunited with civilization, I met an individual sure looked like he had been a sailor at one time or another in his life.

Popeye The Sailor Man, Toot Toot!

Shiver Me Timbers! There he was, standing upright, busily puffing away as pipe, evidently thinking deep thoughts, and only occasionally would he look up. That lonesome soul had clearly not seen a fellow human in ages because his eyes lit up on spotting a fellow human: me, having just climbed up the final step of the staircase.

He uttered to me (almost as if the desolate soul thought that I was perhaps a fellow seafaring chap) the following fateful words as best as I can recall: “It was a dark and stormy night and the captain said to the first mate, ‘mate tell us a story,’ so the mate began ‘it was a dark and stormy night.’”

No sooner had the crusty sailor uttered those fateful words that the sky suddenly darkened—just as suddenly as I had witnessed it clear up—and it was a dark and stormy night… all over again!

It’s been said that, All’s well that ends well. Would that that were true for yours truly at that juncture, hey, even half-true, amirite?