Krazy About Kafka!

Let us prepare to grapple with the ineffable itself… 🏈
~ Douglas Adams (in Dirk Gently’s Holistic Detective Agency)


Preamble ⛱

No doubt your mind is already reeling as you take in the admittedly enigmatic title of this essay—Krazy About Kafka—as you desperately try to get your bearings. And remember Dr. David “Dave” Bowman (the protagonist and commander of the crew aboard the forsaken spaceship faring through outer space, alongside HAL, the artificially-intelligent, sentient, and synthetic life-form) from the movie 2001: A Space Odyssey? Yeah, so maybe you’re even echoing his astonished ejaculation (“My God, it’s full of stars!“) as you frantically try to connect the dots. You’re, like… Arghh! 🚀

Okay, let’s all of us take a deep breath now… Let’s size things up with a cool mind. For warm-up only, here’s a starter-list of grievances that I’ve just been handed in a sealed envelope:

  • This blogger guy clearly doesn’t know how to spell (I mean, spelling the word “crazy” with a… “K” as in “Krazy” — Sheesh!) 🙊
  • For crying out loud, like, we don’t have even the foggiest idea whether this essay is going to be about Apache Kafka (the distributed streaming platform) or about Franz Kafka (the writer renowned for his visionary and profoundly enigmatic stories)? 🙈
  • Did the blogger practically write this up on a sugar high at his local Krispy Kreme doughnuts shop or something? (It sure has been tasting pallid and sugary so far… All fluff, no stuff!) 🙉

And the questions are piling on… You get the idea, right? Oh, you don’t! Well… 😹

Let’s do this: We will proceed with patching up the already-rocky start to our relationship by stopping at our nearest Krispy Kreme shoppe. Better still, let’s you and I head on over to the off-the-wall, monstrous doughnut joint below, which sure looks far more personalized and much less of a cookie-cutter doughnut shop than… 🍩

I mean, what better venue in which to cement our relationship, just you and me: Reader and Blogger, face-to-face!

(We’ll get this Kafka kraziness all sorted out lickety-split over some krispy doughnuts and maybe even sneak in a krafty slice or two of, what else, Kraft cheese! And hey, where’s our Kool-Aid to go along with the munchies? Oh, and we will be sure to capture photographic evidence with, what else, a Kodak!) 📷
2 s

Enter The Antechamber 🍩


Having dunked a handful of chocolate doughnuts—yes, I do see the pile of glazed jalapeño doughnuts so heaven help me with whatever sordid mess I’ve gotten myself into—we are talking freely like brothers in arms. So in full candor, then, here is what’s going to happen: We are going to have our cake and eat it too. Woohoo! 🍰

Now how about that? And we’re going to have our cake and eat it while remaining comfortably seated in the ever-nutritious environs of our local doughnuts shop, even though I’m already beginning to miss the less seedy grounds of those Krispy Kreme shops! Anyhow, live and learn…

 — “Okayyyyy… How, exactly, do you intend to pull this off?”, I hear thou, dear Reader, asking of me, thine servant the Blogger, with more than a quizzical touch in your voice 👒
— “Are you trying to tell us—your long-suffering woebegone readers—that we can have our cake and eat it too by your somehow managing to create an admixture of two disparate elements (the two Kafkas) that our poles apart?” 🎭
— “Never a problem, always happy to solve the unsolvable”, I reply with perfect equanimity 🐙
— “Hah, this one we’ve got to see to believe, Mr. Smarty-pants Blogger!” 🐌
— “I mean, how can you possibly unify two elements that simply don’t mix; they are like oil and water. Or—if you are into pets—ever tried to get cats and dogs along with one another? Hah, so there, Mr. Blogger-pants! (And don’t forget to check out the selfsame, fractured-and-scattered-but-stitched-back-together Humpty Dumpty diaspora of sorts in the picture below showing the less-than-stellar results of attempts to get a cat and a dog to see each other in the eye. See their frosty-eyed glares? Yes, exactly those kinds of, um, results!)” 🐱 🐶
— Without so much as breaking a sweat, I smile back and say “Be careful with your pronouncements now. Wait till you see what I’ve got for you next…” 😉

17 s

The Philosophy Behind Mixing (The Two) Kafkas 🍭


And taking a cue here from the quote atop this essay—with Douglas Adams babbling, Babel fish-style of course, about getting ready “…to grapple with the ineffable itself”—we reveal here in full candor the oh-so-subtle and easy-to-miss nexus between the Kafkaesque philosophy ala Franz Kafka (i.e. individuals burdened with guilt, isolation, and anxiety making futile attempts at personal salvation) and the distributed systems world as symbolized by the Apache Kafka mindset (i.e. building real-time streaming applications that transform or react to the streams of data) 🎭
12 s

Entering The Chamber Itself! 🍩 🍩 🍩 … 🍩


Which brings me to a lovely quotation— it appeared in an intriguing article called Love Is Not Algorithmic—from the marvelous magazine we all love and know as The Atlantic:

There is no higher praise these days than being data-driven. A person who is data-driven is free of bias, and cuts through arguments with a sword of truth. No longer do we need to fumble through life. The answers will come. We will know how to respond, just what to do. We will let the data tell us!

And so it goes with Christian Rudder’s new book Dataclysm: Who We Are (When We Think No One’s Looking), a synopsis of insights he gleaned from analytics while working at the company he co-founded, OKCupid. His company, he tells us, could easily sport the tagline “Making the Ineffable Totally Effable”  🙌

Now how about that? Ready to get yourself a subscription to The Atlantic like me? (In full disclosure, I have zero vested interests, monetarily, in the venerable magazine; I am, however, fully vested—or should I say fully invested—in soaking up the amazing writing which their writing staff churns out on a regular basis!) 🎯
3 s

The Elements Of This Essay 👒


Without further ado, here then are the pit-stops in our Kafkaesque ride that’s about to begin:

  1. Start With What Is Right 🙋
  2. Transforming (Gasp! Into An Enormous Insect…) 🐛
  3. Decisive Moments In Human Evolution Are Perpetual 🎡
  4. Back The World 🐘
  5. Place Permanent Trust In Something Indestructible 🏰 

Seated in the salubrious environs of the cantankerous—and cavernous—boutique doughnut shop, the Blogger’s eyes are searching for the Reader’s, eager for the highly anticipated platonic rendezvous to begin… 👦

Then somebody puts a fistful of quarters in the jukebox and the shop is filled with INXS music:

I was standing
You were there
Two worlds collided
And they could never tear us apart

I told you
That we could fly
‘Cause we all have wings
But some of us don’t know why

~ INXS (Lyrics from Never Tear Us Apart)

Wow, what a start—one surely mind-to-mind—to the part where the Reader and Blogger meet!

(And what do we get? The collage, of course, which  is coming right up!) 🎁

23 s

1. Start With What Is Right 🙋

Start with what is right rather than what is acceptable 👣

~ Franz Kafka

Any time scientists disagree, it’s because we have insufficient data. Then we can agree on what kind of data to get; we get the data; and the data solves the problem. Either I’m right, or you’re right, or we’re both wrong. And we move on 👺
~ Neil deGrasse Tyson

As we dunk our very first doughnut, I look at the Kafka quotation above one more time—don’t worry, I won’t spell quotation with a “K”, at least not for now—and couldn’t help but wonder whether Kafka perhaps did some software design on the side, moonlighting or something… 🌘

Then again, maybe not; his wild imagination (conjuring up human-to-giant-cockroach transformations at the drop of a hat and stuff like that), after all, is way too pedestrian for us technology types. Hey hey, way too prosaic. We are, after all, known for taking on far heftier problems, and for grappling with mind-bending issues that mere mortals dread! 🐜

Nonetheless, in advising us to “Start with what is right”, Kafka sure had the right sort of idea, an idea that lies at the core of the design of the eponymous software library: Apache Kafka. While there is a lot—and I mean a whole lot more—that I can say about this topic of going about making the right choices (in software design), specifically in the context of Kafka, suffice it to see that the following passage does a good job of capturing an especially important slice of design decisions:

In addition to adding consumers in order to scale a single application, it is very common to have multiple applications that need to read data from the same topic. In fact, one of the main design goals in Kafka was to make the data produced to Kafka topics available for many use cases throughout the organization… Unlike many traditional messaging systems, Kafka scales to a large number of consumers and consumer groups without reducing performance 🚉
~ Neha Narkhede, Gwen Shapira, and Todd Palino (in Kafka: The Definitive Guide — O’Reilly)

Meanwhile, we’ve got a decision of our own to make: Which doughnut will we have the pleasure of munching next as we get deeper into plumbing the depths of Kafka… So let’s move right on to the next element in our collage!

2. Transforming (Gasp! Into An Enormous Insect…) 🐛

As Gregor Samsa awoke one morning from uneasy dreams he found himself transformed in his bed into an enormous insect 🐞
~ Franz Kafka (in The Metamorphosis)

[Extreme processing] …some processing of data is inherent to data pipelines. After all, we are moving data between different systems where different data formats make sense and different use cases are supported. However, too much processing ties all the downstream systems to decisions made when building the pipelines. Decisions about which fields to preserve, how to aggregate data, etc. 🚏
~ Neha Narkhede, Gwen Shapira, and Todd Palino (in Kafka: The Definitive Guide — O’Reilly)

As the INXS song died down, somebody else walked up to the jukebox as the two of us—Reader and Blogger—sat face-to-face with all our energies focused on an intellectually charged and eminently platonic discussion of all things Kafka. And then we heard the next song playing in the background…

I’m your messiah and you’re the reason why
‘Cuz U – I would die 4 u, yeah
Darling if u want me 2
U – I would die 4 u

~ Prince And The Revolution (Lyrics from I Would Die 4 U)

Exactly the right sort of theme to mirror the transformational aspects embodied by all things Kafka; messiahs and reformers are of course the ones who help bring about transformations and reformations!

So we dunk our next doughnut, and talk some about extreme processing; how anticlimactic… 🍩

But then again, it’s data transformations that we are talking about—data pipelines too, for crying out loud! Visualize how, from nearly nothing (imagine the chrysalis in the picture above) we witness the unfolding of transformational beauty in the wings of a fully-formed Monarch butterfly. How beautiful is that? 🏄

Indeed, while Apache Kafka is agnostic to whatever transformations we choose to apply to the data sitting in its wondrous topics, the beauty lies in the results obtained—think of all the delighted customers of our business—which enable end-users to do more with their lives!

It’s all about data pipelines and, of course, connecting them—or wiring them if you will—to get out applications and our architectural designs to delight our customers. How we choose to aggregate data and, even more importantly, how we go about making sure that raw data—which I like to think of as the system of record—remains preserved in its pristine condition (right from the time we received it from our applications business transactions) 🚃

Oh, by the way, we need to do all the above in a safe, efficient, and agile way! 🙀

3. Decisive Moments In Human Evolution Are Perpetual 🎡

The decisive moment in human evolution is perpetual. That is why the revolutionary spiritual movements that declare all former things worthless are in the right, for nothing has yet happened 🙆
~ Franz Kafka

Kafka has quickly evolved from messaging queue to a full-fledged streaming platform 🏂
~ Confluent (in What is Apache Kafka?)

I wonder what your reaction will be to the Kafka statue above

Meanwhile, perfect timing—as the song I Would Die 4 U was winding down—yet another doughnut aficionado walks up to the jukebox, feeds the insatiable dragon, and we hear…

Is this the real life?
Is this just fantasy?
Caught in a landslide,
No escape from reality

~ Queen ( Lyrics from Bohemian Rhapsody) 😵

Oh. My. God.

Coming on the heels of a princely song—I mean we had just finished listening to Prince And The Revolution—we are slightly unnerved as we try to absorb the uncanny coincidence of now being serenaded to queenly music (Queen’s good old Freddie Mercury about to sing what is probably their deepest song ever).

But we—Reader and Blogger—try our best to remain unfazed by the digital (anyone see analog or, dare I say, analogous?) gusts of probabilistic uncertainties buffeting us. We are brave souls and proceed intrepidly with dunking our third doughnut each. Yeah!! (Gotta’ keep all these fabulous doughnut shops in business, after all. In full disclosure, we technology types are, at times, driven by the noblest of causes!)

Look, the animus of human evolution, as ever, remains in perpetual flight 🐉

Taking a cue from the world of academia— specifically the stark reminder to “publish or perish”—two of us get to talking about how Apache Kafka as evolved (with its latest stage being the cool work taking place at Confluent) to enable us technology types to take advantage of the ever increasing capabilities of Kafka. It’s a great story, and it’s awesome to be a part of it! That much we agree on; how we get from point a to point B is a different story…

Remember how the mesmerizing movie The Highlander would seamlessly segue between past and present, foreground and background, reality and fantasy? Well, we are feeling bold enough to try some transmogrifying of our own! Our hope in doing so is to find a nexus between the decisive moment in human evolution being perpetual on the one hand and Kafka having evolved (from a messaging queue) into a full-fledged streaming platform on the other 🚀

And it’s really not as complicated as it might seem at first blush: all we are really talking about is how messages are merely opaque byte arrays to Kafka itself even as we suffer developers remain tantalizingly beguiled by the likes of JSON (JavaScript Object Notation) and good—or bad old, depending on your opinion—XML (Extensible Markup Language) and stuff like that.

Did anyone see schema? And did I really hear some noise in the background about strongly typed data typing and schema evolution, not to mention the lovely concerns of compatibility—both backward and forward to be sure! (Apache Avro anyone? Yoohoo, anyone?) 🐘

(Meanwhile, the two of us can’t help but wonder what fate has in store for us next… We wonder which free song—Reader and Blogger are both confirmed cheapskates who would prefer keeping our hard-earned quarters in our wallet and purse—will one of our fellow doughnut patrons play for us next on the jukebox? More importantly, which kind of doughnut—cinnamon-sugar or Bavarian creme or apple-crumb or just plain old glazed—shall we treat ourselves to next? For anyone snickering at our priorities, allow me to solemnly remind you that our flights of intellectual virtuosity need to be fueled by that most refined of things: refined sugar, what else?) 🍩

4. Back The World 🐘

In the struggle between yourself and the world, back the world 🚚
~ Franz Kafka

Data written to Kafka is written to disk and replicated for fault-tolerance. Kafka allows producers to wait on acknowledgement so that a write isn’t considered complete until it is fully replicated and guaranteed to persist even if the server written to fails 🏧
~ Kafka (Online documentation)

And then the music stopped. It did. It just did 🎶

It was followed by… eerie silence. We politely cleared our throat wondering—in a fitful acuteness which seemed to grow by the second—which brave doughnut patron would step up to face the “dragon” (excuse me, I got a bit carried away there as all I had in mind was the “jukebox”) and replace the steely quietness with the soft ambience of tunes? Anyone? 👹

Our prayers were answered as we heard quarters making rattling sounds on their way to being consumed by the fire-breathing dragon—darn those Freudian slips! You know the story by now, don’t you? Yep, exactly. As the quarters sank into the bottomless maw of the jukebox, we heard…

Stood there boldly
Sweatin’ in the sun
Felt like a million
Felt like number one
The height of summer
I’d never felt that strong
Like a rock

Like a rock, I was strong as I could be
Like a rock, nothin’ ever got to me
Like a rock, I was something to see
Like a rock

~ Bob Seger (Lyrics from Like A Rock)

There was no talk this time around. Instead, being the true techies that we are, the two of us—Reader and Blogger—sitting face-to-face (with a new doughnut each in our hands of course, what else did you think?), our mental energies remained, as ever, laser-focused on discussing ever-widening facets of all things Kafka 👻

This time around, with the Bob Seger song playing in the background, on parts naturally turn to an especially important and rock solid aspects of Kafka: scaling out our application and enabling us to handle failures gracefully!

(The discussion was so heady that all I can recall is… Nothing! Clearly, more refined sugar is warranted: onto the next doughnut then?) 🍩

In the travails of fading retentiveness, the few wisps of memory that survive point in the direction, ironically enough, of surviving failures. Aha! That’s got to be the stuff of—no, not magic, sorry, nothing as fantastical as that going on here—scaling our software applications even as we ensure their ability to gracefully handle failures. What to do, Scooby Doo? 🐶

Not to fear, my dear, the solution’s rather linear—Kafka to the rescue: It’s highly available; in the face of application failure, a Kafka-based application can simply look up its last position in the stream (from Kafka) and continue processing from the last committed offset.

Easy as pie! 🍮

5. Place Permanent Trust In Something Indestructible 🏰

Man cannot live without a permanent trust in something indestructible in himself, though both the indestructible element and the trust may remain permanently hidden from him💋
~ Franz Kafka

[Immutable data records]… Events, once occurred, can never be modified. A financial transaction that is cancelled does not disappear. Instead, an additional event is written to the stream, recording a cancellation of previous transaction… If you are familiar with binlogs, WALs, or redo logs in databases you can see that if we insert a record into a table and later delete it, the table will no longer contain the record, but the redo log will contain two transactions—the insert and the delete 📕
~ Neha Narkhede, Gwen Shapira, and Todd Palino (in Kafka: The Definitive Guide — O’Reilly)

Something magical happened. Making a clean break with years and years of cheapskate habits, I stood up and walked to the jukebox (Remember now that I’m a regular at this—and pretty much all other—doughnut shops where I’m on a first-name basis with everyone!)

Everyone gasped… 😮

Witnessing a sight—your Blogger actually plunking down his own money to get the jukebox to play a song—that no one, myself included, had witnessed before, I reached for quarters in my pocket and… I came up empty. Panic started to grip me 😧

Detecting telltale signs of defeat in my eyes, my loyal Reader stood up and, without skipping a beat, graciously came to my rescue. Woohoo!

Gotta’ love my readers 💝

Reaching into her purse, she deftly fished out the exact number of quarters (clearly, the alien act of using one’s own money for a jukebox wasn’t alien at all to my comrade, the Noble Reader) that would make the dragon (correction, the jukebox!) cough up another song to fill the solemn hush that had descended on the doughnut parlor right at the moment when we were feeling emboldened enough—having of course just listened to Bob Seger’s rock solid song—to pursue one final element of Kafka coolness before bidding our farewells, the Reader and I… 👋

As the quarters sank in—not from the Blogger’s pocket to jukebox but from the Reader’s purse to its maw though I wouldn’t quite pronounce chivalry dead yet—we heard an oldie goldie… 🏆

Oh. My. God (Doing a double take, times two!) 👀

So, if an old friend I know
Drops by to say hello
Would I still see suspicion in your eyes?

Here we go again
Asking where I’ve been
You can’t see these tears are real
I’m crying

We can’t go on together
With suspicious minds
And we can’t build our dreams
On suspicious minds 😼

~ Elvis Presley (Lyrics from Suspicious Minds)

A flood of thoughts cascades in our minds, and we are left wondering if we are perhaps owners of suspicious minds… I mean, that oh-so-crucial quality of placing our permanent trust—the absence of which of course is the motif we were hearing the King cry about plaintively in his lovely song: suspicion.

On the one hand, listening to Kafka, we feel pulled toward his pronouncement that humans “…cannot live without a permanent trust in something indestructible in himself, though both the indestructible element and the trust may remain permanently hidden from him.” On the other hand, knowing full well, as technologists, the criticality of immutable data records, we feel pulled in a different direction. The result: cognitive dissonance <insert one primal scream here!> 😱

What happened next? How did we—who else but us, Reader and Blogger, still sitting face-to-face with another delicious doughnut each in our hands—manage to reconcile the conundrum in which we found ourselves?

Sigh… While all is good, and even though the outcome was great, the ensuing discussion took up  another full two hours. And yes, as you may have guessed by now, there lies the rub: our discussion was extensive enough, and then some, to the point. It’s simply impossible to even begin summarizing the high points. Basically, trying to catch the deluge in a paper cup has been tried before—with scant meager success—so I’m not going there… 🌊

This point was doubly hammered home for us revelers when—completely oblivious to our surroundings as we had become by that time—the neighborly owners of the doughnut shop stopped by our table and told us, in the gentlest of words (knowing full well, what’s more, that they wanted us to remain their loyal customers forevermore) that the doughnut shop, um, needed to close for the night… 🚧 🍩 🚧

The rest, as they say, is history. Speaking of which, I remind us all of the marvelous message in the words—and here I’m paraphrasing from memory—of American philosopher George Santayana that

Those who cannot remember the past are condemned to repeat it 🗿

At this point, we knew that if we wanted to come back the doughnut shop—and we certainly did—we decided not to run the risk of getting kicked out. So we politely stood up with an all-knowing look in our eyes that told everyone that we, of course, knew what we were doing all along, we departed our own many ways. Of course we knew that the shop was about to close. Of course we were about to head out anyway. Of course… 😎

Before we head into the sunset, let’s have a listen to these eloquent words:

I thought that since Kafka was a system optimized for writing, using a writer’s name would make sense. I had taken a lot of lit classes in college and liked Franz Kafka. Plus the name sounded cool for an open source project. So basically there is not much of a relationship 🎎
~ Jay Krepps (Lead designer of Apache Kafka)

A tad anticlimactic—we were expecting the origins of  the name chosen for what became Apache Kafka to be a bit more dramatic—but pretty cool on the whole eh? 🎠

(Add stuff here…)


(Add stuff here…)

(Add stuff here…)

Dedication 🌹


I fondly dedicate this essay to my muse. I revere her; my very DNA undeniably reveres her 🐱

(No muse, no essay; all I would otherwise do is sachet, and you ain’t gonna get no essay that way!)

Your Comment Here!

This site uses Akismet to reduce spam. Learn how your comment data is processed.