Let us prepare to grapple with the ineffable itself… 🏈
~ Douglas Adams (in Dirk Gently’s Holistic Detective Agency)

 

Preamble ⛱

No doubt your mind is already reeling as you take in the admittedly enigmatic title of this essay—Krazy About Kafka—as you desperately try to get your bearings. And remember Dr. David “Dave” Bowman (the protagonist and commander of the crew aboard the forsaken spaceship faring through outer space, alongside HAL, the artificially-intelligent, sentient, and synthetic life-form) from the movie 2001: A Space Odyssey? Yeah, so maybe you’re even echoing his astonished ejaculation (“My God, it’s full of stars!“) as you frantically try to connect the dots. You’re, like… Arghh! 🚀

Okay, let’s all of us take a deep breath now… Let’s size things up with a cool mind. For warm-up only, here’s a starter-list of grievances that I’ve just been handed in a sealed envelope:

• This blogger guy clearly doesn’t know how to spell (I mean, spelling the word “crazy” with a… “K” as in “Krazy” — Sheesh!) 🙊
• For crying out loud, like, we don’t have even the foggiest idea whether this essay is going to be about Apache Kafka (the distributed streaming platform) or about Franz Kafka (the writer renowned for his visionary and profoundly enigmatic stories)? 🙈
• Did the blogger practically write this up on a sugar high at his local Krispy Kreme doughnuts shop or something? (It sure has been tasting pallid and sugary so far… All fluff, no stuff!) 🙉

And the questions are piling on… You get the idea, right? Oh, you don’t! Well… 😹

Let’s do this: We will proceed with patching up the already-rocky start to our relationship by stopping at our nearest Krispy Kreme shoppe. Better still, let’s you and I head on over to the off-the-wall, monstrous doughnut joint below, which sure looks far more personalized and much less of a cookie-cutter doughnut shop than… 🍩

I mean, what better venue in which to cement our relationship, just you and me: Reader and Blogger, face-to-face!

(We’ll get this Kafka kraziness all sorted out lickety-split over some krispy doughnuts and maybe even sneak in a krafty slice or two of, what else, Kraft cheese! And hey, where’s our Kool-Aid to go along with the munchies? Oh, and we will be sure to capture photographic evidence with, what else, a Kodak!) 📷

Enter The Antechamber 🍩

 

Having dunked a handful of chocolate doughnuts—yes, I do see the pile of glazed jalapeño doughnuts so heaven help me with whatever sordid mess I’ve gotten myself into—we are talking freely like brothers in arms. So in full candor, then, here is what’s going to happen: We are going to have our cake and eat it too. Woohoo! 🍰

Now how about that? And we’re going to have our cake and eat it while remaining comfortably seated in the ever-nutritious environs of our local doughnuts shop, even though I’m already beginning to miss the less seedy grounds of those Krispy Kreme shops! Anyhow, live and learn…

 — “Okayyyyy… How, exactly, do you intend to pull this off?”, I hear thou, dear Reader, asking of me, thine servant the Blogger, with more than a quizzical touch in your voice 👒
— “Are you trying to tell us—your long-suffering woebegone readers—that we can have our cake and eat it too by your somehow managing to create an admixture of two disparate elements (the two Kafkas) that our poles apart?” 🎭
— “Never a problem, always happy to solve the unsolvable”, I reply with perfect equanimity 🐙
— “Hah, this one we’ve got to see to believe, Mr. Smarty-pants Blogger!” 🐌
— “I mean, how can you possibly unify two elements that simply don’t mix; they are like oil and water. Or—if you are into pets—ever tried to get cats and dogs along with one another? Hah, so there, Mr. Blogger-pants! (And don’t forget to check out the selfsame, fractured-and-scattered-but-stitched-back-together Humpty Dumpty diaspora of sorts in the picture below showing the less-than-stellar results of attempts to get a cat and a dog to see each other in the eye. See their frosty-eyed glares? Yes, exactly those kinds of, um, results!)” 🐱 🐶
— Without so much as breaking a sweat, I smile back and say “Be careful with your pronouncements now. Wait till you see what I’ve got for you next…” 😉

The Philosophy Behind Mixing (The Two) Kafkas 🍭

 

And taking a cue here from the quote atop this essay—with Douglas Adams babbling, Babel fish-style of course, about getting ready “…to grapple with the ineffable itself”—we reveal here in full candor the oh-so-subtle and easy-to-miss nexus between the Kafkaesque philosophy ala Franz Kafka (i.e. individuals burdened with guilt, isolation, and anxiety making futile attempts at personal salvation) and the distributed systems world as symbolized by the Apache Kafka mindset (i.e. building real-time streaming applications that transform or react to the streams of data) 🎭

Entering The Chamber Itself! 🍩 🍩 🍩 … 🍩

 

Which brings me to a lovely quotation— it appeared in an intriguing article called Love Is Not Algorithmic—from the marvelous magazine we all love and know as The Atlantic:

Now how about that? Ready to get yourself a subscription to The Atlantic like me? (In full disclosure, I have zero vested interests, monetarily, in the venerable magazine; I am, however, fully vested—or should I say fully invested—in soaking up the amazing writing which their writing staff churns out on a regular basis!) 🎯

The Elements Of This Essay 👒

 

Without further ado, here then are the pit-stops in our Kafkaesque ride that’s about to begin:

1. Start With What Is Right 🙋
2. Transforming (Gasp! Into An Enormous Insect…) 🐛
3. Decisive Moments In Human Evolution Are Perpetual 🎡
4. Back The World 🐘
5. Place Permanent Trust In Something Indestructible 🏰 

Seated in the salubrious environs of the cantankerous—and cavernous—boutique doughnut shop, the Blogger’s eyes are searching for the Reader’s, eager for the highly anticipated platonic rendezvous to begin… 👦

Then somebody puts a fistful of quarters in the jukebox and the shop is filled with INXS music:

Wow, what a start—one surely mind-to-mind—to the part where the Reader and Blogger meet!

(And what do we get? The collage, of course, which  is coming right up!) 🎁

1. Start With What Is Right 🙋

As we dunk our very first doughnut, I look at the Kafka quotation above one more time—don’t worry, I won’t spell quotation with a “K”, at least not for now—and couldn’t help but wonder whether Kafka perhaps did some software design on the side, moonlighting or something… 🌘

Then again, maybe not; his wild imagination (conjuring up human-to-giant-cockroach transformations at the drop of a hat and stuff like that), after all, is way too pedestrian for us technology types. Hey hey, way too prosaic. We are, after all, known for taking on far heftier problems, and for grappling with mind-bending issues that mere mortals dread! 🐜

Nonetheless, in advising us to “Start with what is right”, Kafka sure had the right sort of idea, an idea that lies at the core of the design of the eponymous software library: Apache Kafka. While there is a lot—and I mean a whole lot more—that I can say about this topic of going about making the right choices (in software design), specifically in the context of Kafka, suffice it to see that the following passage does a good job of capturing an especially important slice of design decisions:

In addition to adding consumers in order to scale a single application, it is very common to have multiple applications that need to read data from the same topic. In fact, one of the main design goals in Kafka was to make the data produced to Kafka topics available for many use cases throughout the organization… Unlike many traditional messaging systems, Kafka scales to a large number of consumers and consumer groups without reducing performance 🚉
~ Neha Narkhede, Gwen Shapira, and Todd Palino (in Kafka: The Definitive Guide — O’Reilly)

Meanwhile, we’ve got a decision of our own to make: Which doughnut will we have the pleasure of munching next as we get deeper into plumbing the depths of Kafka… So let’s move right on to the next element in our collage!

2. Transforming (Gasp! Into An Enormous Insect…) 🐛

As Gregor Samsa awoke one morning from uneasy dreams he found himself transformed in his bed into an enormous insect 🐞

~ Franz Kafka (in The Metamorphosis)

As the INXS song died down, somebody else walked up to the jukebox as the two of us—Reader and Blogger—sat face-to-face with all our energies focused on an intellectually charged and eminently platonic discussion of all things Kafka. And then we heard the next song playing in the background…

Exactly the right sort of theme to mirror the transformational aspects embodied by all things Kafka; messiahs and reformers are of course the ones who help bring about transformations and reformations!

So we dunk our next doughnut, and talk some about extreme processing; how anticlimactic… 🍩

But then again, it’s data transformations that we are talking about—data pipelines too, for crying out loud! Visualize how, from nearly nothing (imagine the chrysalis in the picture above) we witness the unfolding of transformational beauty in the wings of a fully-formed Monarch butterfly. How beautiful is that? 🏄

Indeed, while Apache Kafka is agnostic to whatever transformations we choose to apply to the data sitting in its wondrous topics, the beauty lies in the results obtained—think of all the delighted customers of our business—which enable end-users to do more with their lives!

It’s all about data pipelines and, of course, connecting them—or wiring them if you will—to get out applications and our architectural designs to delight our customers. How we choose to aggregate data and, even more importantly, how we go about making sure that raw data—which I like to think of as the system of record—remains preserved in its pristine condition (right from the time we received it from our applications business transactions) 🚃

Oh, by the way, we need to do all the above in a safe, efficient, and agile way! 🙀

3. Decisive Moments In Human Evolution Are Perpetual 🎡

I wonder what your reaction will be to the Kafka statue above…

Meanwhile, perfect timing—as the song I Would Die 4 U was winding down—yet another doughnut aficionado walks up to the jukebox, feeds the insatiable dragon, and we hear…

Oh. My. God.

Coming on the heels of a princely song—I mean we had just finished listening to Prince And The Revolution—we are slightly unnerved as we try to absorb the uncanny coincidence of now being serenaded to queenly music (Queen’s good old Freddie Mercury about to sing what is probably their deepest song ever).

But we—Reader and Blogger—try our best to remain unfazed by the digital (anyone see analog or, dare I say, analogous?) gusts of probabilistic uncertainties buffeting us. We are brave souls and proceed intrepidly with dunking our third doughnut each. Yeah!! (Gotta’ keep all these fabulous doughnut shops in business, after all. In full disclosure, we technology types are, at times, driven by the noblest of causes!)

Look, the animus of human evolution, as ever, remains in perpetual flight 🐉

Taking a cue from the world of academia— specifically the stark reminder to “publish or perish”—two of us get to talking about how Apache Kafka as evolved (with its latest stage being the cool work taking place at Confluent) to enable us technology types to take advantage of the ever increasing capabilities of Kafka. It’s a great story, and it’s awesome to be a part of it! That much we agree on; how we get from point a to point B is a different story…

Remember how the mesmerizing movie The Highlander would seamlessly segue between past and present, foreground and background, reality and fantasy? Well, we are feeling bold enough to try some transmogrifying of our own! Our hope in doing so is to find a nexus between the decisive moment in human evolution being perpetual on the one hand and Kafka having evolved (from a messaging queue) into a full-fledged streaming platform on the other 🚀

And it’s really not as complicated as it might seem at first blush: all we are really talking about is how messages are merely opaque byte arrays to Kafka itself even as we suffer developers remain tantalizingly beguiled by the likes of JSON (JavaScript Object Notation) and good—or bad old, depending on your opinion—XML (Extensible Markup Language) and stuff like that.

Did anyone see schema? And did I really hear some noise in the background about strongly typed data typing and schema evolution, not to mention the lovely concerns of compatibility—both backward and forward to be sure! (Apache Avro anyone? Yoohoo, anyone?) 🐘

(Meanwhile, the two of us can’t help but wonder what fate has in store for us next… We wonder which free song—Reader and Blogger are both confirmed cheapskates who would prefer keeping our hard-earned quarters in our wallet and purse—will one of our fellow doughnut patrons play for us next on the jukebox? More importantly, which kind of doughnut—cinnamon-sugar or Bavarian creme or apple-crumb or just plain old glazed—shall we treat ourselves to next? For anyone snickering at our priorities, allow me to solemnly remind you that our flights of intellectual virtuosity need to be fueled by that most refined of things: refined sugar, what else?) 🍩

4. Back The World 🐘

And then the music stopped. It did. It just did 🎶

It was followed by… eerie silence. We politely cleared our throat wondering—in a fitful acuteness which seemed to grow by the second—which brave doughnut patron would step up to face the “dragon” (excuse me, I got a bit carried away there as all I had in mind was the “jukebox”) and replace the steely quietness with the soft ambience of tunes? Anyone? 👹

Our prayers were answered as we heard quarters making rattling sounds on their way to being consumed by the fire-breathing dragon—darn those Freudian slips! You know the story by now, don’t you? Yep, exactly. As the quarters sank into the bottomless maw of the jukebox, we heard…

There was no talk this time around. Instead, being the true techies that we are, the two of us—Reader and Blogger—sitting face-to-face (with a new doughnut each in our hands of course, what else did you think?), our mental energies remained, as ever, laser-focused on discussing ever-widening facets of all things Kafka 👻

This time around, with the Bob Seger song playing in the background, on parts naturally turn to an especially important and rock solid aspects of Kafka: scaling out our application and enabling us to handle failures gracefully!

(The discussion was so heady that all I can recall is… Nothing! Clearly, more refined sugar is warranted: onto the next doughnut then?) 🍩

In the travails of fading retentiveness, the few wisps of memory that survive point in the direction, ironically enough, of surviving failures. Aha! That’s got to be the stuff of—no, not magic, sorry, nothing as fantastical as that going on here—scaling our software applications even as we ensure their ability to gracefully handle failures. What to do, Scooby Doo? 🐶

Not to fear, my dear, the solution’s rather linear—Kafka to the rescue: It’s highly available; in the face of application failure, a Kafka-based application can simply look up its last position in the stream (from Kafka) and continue processing from the last committed offset.

Easy as pie! 🍮

5. Place Permanent Trust In Something Indestructible 🏰

Something magical happened. Making a clean break with years and years of cheapskate habits, I stood up and walked to the jukebox (Remember now that I’m a regular at this—and pretty much all other—doughnut shops where I’m on a first-name basis with everyone!)

Everyone gasped… 😮

Witnessing a sight—your Blogger actually plunking down his own money to get the jukebox to play a song—that no one, myself included, had witnessed before, I reached for quarters in my pocket and… I came up empty. Panic started to grip me 😧

Detecting telltale signs of defeat in my eyes, my loyal Reader stood up and, without skipping a beat, graciously came to my rescue. Woohoo!

Gotta’ love my readers 💝

Reaching into her purse, she deftly fished out the exact number of quarters (clearly, the alien act of using one’s own money for a jukebox wasn’t alien at all to my comrade, the Noble Reader) that would make the dragon (correction, the jukebox!) cough up another song to fill the solemn hush that had descended on the doughnut parlor right at the moment when we were feeling emboldened enough—having of course just listened to Bob Seger’s rock solid song—to pursue one final element of Kafka coolness before bidding our farewells, the Reader and I… 👋

As the quarters sank in—not from the Blogger’s pocket to jukebox but from the Reader’s purse to its maw though I wouldn’t quite pronounce chivalry dead yet—we heard an oldie goldie… 🏆

Oh. My. God (Doing a double take, times two!) 👀

A flood of thoughts cascades in our minds, and we are left wondering if we are perhaps owners of suspicious minds… I mean, that oh-so-crucial quality of placing our permanent trust—the absence of which of course is the motif we were hearing the King cry about plaintively in his lovely song: suspicion.

On the one hand, listening to Kafka, we feel pulled toward his pronouncement that humans “…cannot live without a permanent trust in something indestructible in himself, though both the indestructible element and the trust may remain permanently hidden from him.” On the other hand, knowing full well, as technologists, the criticality of immutable data records, we feel pulled in a different direction. The result: cognitive dissonance <insert one primal scream here!> 😱

What happened next? How did we—who else but us, Reader and Blogger, still sitting face-to-face with another delicious doughnut each in our hands—manage to reconcile the conundrum in which we found ourselves?

Sigh… While all is good, and even though the outcome was great, the ensuing discussion took up  another full two hours. And yes, as you may have guessed by now, there lies the rub: our discussion was extensive enough, and then some, to the point. It’s simply impossible to even begin summarizing the high points. Basically, trying to catch the deluge in a paper cup has been tried before—with scant meager success—so I’m not going there… 🌊

This point was doubly hammered home for us revelers when—completely oblivious to our surroundings as we had become by that time—the neighborly owners of the doughnut shop stopped by our table and told us, in the gentlest of words (knowing full well, what’s more, that they wanted us to remain their loyal customers forevermore) that the doughnut shop, um, needed to close for the night… 🚧 🍩 🚧

The rest, as they say, is history. Speaking of which, I remind us all of the marvelous message in the words—and here I’m paraphrasing from memory—of American philosopher George Santayana that

Those who cannot remember the past are condemned to repeat it 🗿

At this point, we knew that if we wanted to come back the doughnut shop—and we certainly did—we decided not to run the risk of getting kicked out. So we politely stood up with an all-knowing look in our eyes that told everyone that we, of course, knew what we were doing all along, we departed our own many ways. Of course we knew that the shop was about to close. Of course we were about to head out anyway. Of course… 😎

Before we head into the sunset, let’s have a listen to these eloquent words:

A tad anticlimactic—we were expecting the origins of  the name chosen for what became Apache Kafka to be a bit more dramatic—but pretty cool on the whole eh? 🎠

Dedication 🌹

 

I fondly dedicate this essay to my muse. I revere her; my very DNA undeniably reveres her 🐱

(No muse, no essay; all I would otherwise do is sachet, and you ain’t gonna get no essay that way!)