I think that the method which Mr. Spencer uses in his Data of Ethics will help to fix our opinion. Ideality in conduct is altogether a matter of adaptation.
~ William James (in The Varieties Of Religious Experience)
0. Intro đ
Setting The Stage
What a singular book it wasâthe masterpiece entitled The Varieties Of Religious Experience by the younger brother of Henry Jamesâin how it managed, along with The Principles of Psychology, to do nothing less than launch the enterprise itself of American psychology. But we are after a different kind of enterprise today, one having to do with the far more immediate varieties (of experience) that squarely revolve around the new currency of our societyâdata.
Enter the varieties of data experience.
Data, of course, comes in all kinds: Big data, streaming data, data in tables, you name it. (There’s a “data” variant for that.) And today we get to have fun with questions such as these:
- Is NoSQL really a prequel to SQL? (To jog your memory, “SQL” is pronounced “sequel” in most quarters.)
- Is it true that all your joinsâboth inner and outerâare actually belong to streaming?
- Of the three biggies in the world of distributed dataâConsistency, Availability and Partition-tolerance (aka CAP)âwhich two shall it be today?
Plenty there to get us going, eh.
Getting On The Stage
First, though, and in a nod to William James’ subtly-named masterwork The Varieties Of Religious Experience, be forewarned that we are about to take a decidedly pictorial walk through the contours of the landscape that makes up (a big part of) the data experience.
Why “pictorial”?, you ask.
Why not “pictorial”?, I say.
Heh. Let me level with you: That’s so because, in all seriousness, and speaking here to the “pictorial” aspects, I am inviting you to take in the aforementioned jaunt (through the data landscape) coming right up.
Behold “Exhibit A”: A picture I snapped with my iPhone on a recent morning walk down a rustic trail, one that was replete with magnificently speckled cows in the far distance. (Hence, a speck, or two)
1. It All Began One Fine Texas Morning â·
Look, the sinuous trail above sure was an inviting one. So I took the plungeâwell, not literally, as a dive into the concrete would’ve been a bit too jarring for my likingâand got going on a brisk walk, though not quite a dash. (We’ve done that elsewhere.)
Lo and behold, the varieties of the Texas flowers in bloom that greeted me was staggering. It got me thinking to the equally rich varieties in which data come bundled our way. We technologists love to get our hands on reams and reams of tables populated with rich data; streaming data has always been fair game.
But what about the tablesâyeah, those fabled tablesâin the picture coming right up?
2. Of Tables And Streams And Such đ
Yes, Akram, what about the tables in the pic above?
Well, I will reveal, in full candor, what’s up with that at the very end (in the tradition of Bertrand Meyer’s….) Surely you can wait? No?
A long time agoâsome time after the dinosaurs stopped roaming our planet and a bit before the arrival of the Internetâwe had this thing called The Yellow Pages, where one could purportedly find the listings of businesses. What a radical idea (for its age.) And for readers of a certain (youthful) age, I’m not making this up. Honest.
The reason I bring that up is the lovely jingle that went with The Yellow Pages phenom:
“So goodbye yellow brick road…”
Wait, and excuse moi. Not exactly that. It was actuallyâhere goes take two nowâmore like:
“Let your fingers do the walking.“
And today, we go one step further: Yep, leaving behind us all that talk of how one should “Let your fingers do the walking”, we’re going to let our pictures do the talking. Is that radical or what?
You decide because, from here on out, it’s (mostly) you and the picturesâmy (muffled) musings make some solitary appearances. But that’s about it.
Here we go.
3. Look Me Straight In The Eyes đż
And now we get up-close-and-personalâI’m talking eye-level gaze hereâwith those tables. (Remember, as always, that we’re on terra firma, we’re squarely in Programming Digressions territory, so everything’s on the tableâincluding the physical ones!)
Where have we seen that intrepid sleuth before, you know the one on the “mezzanine” table above, walking straight toward us, with an orange suitcase in one hand and an overcoat in the other?
Anyone?
While you figure that one out, a little bird has told me that the sleuth’s mission this time is to get to the bottom of relativity, as applied to data. For example
- Should tables right be considered as containersâto accumulate and be observed over timeâfor data at rest?
- Is the metaphor of streams as being one of data in motionâfor encoding a quantized view of the evolution of a table over timeâat just the right level?
That little bird also told me to look up: Chirpily good stuff.
4. An Assembly-Line Or Something? đ
Will the right-sized dataâI don’t care if it’s Big Data, or Streaming Dataâplease stand up now and make yourself known? Ah, therein lies the rub: Where is the sweet spot wherein lies the habitat of that oh-so-right-sized data?
But first, check this delightful writeup: Streaming Analytics.
It is quite extraordinary: Seldom do I see quality shine through in writing as I’ve seen in that writeup! To pick just one example of how “relatable” the narrative is, I have but to look to this lead-in sentence:
Data is a perishable commodity: It holds the most value at the time itâs produced or captured.
What I have in mindâin my remarking on that writeup being splendidly “relatable”âis simply this: Others go round and round (the mulberry bush) without so much as even trying to bring the subject down anywhere remotely close to Earth. But that’s not Streaming Analytics. Good stuff. Enjoy.
Meanwhile, back at the ranch here, it’s time to ponder: I’ve done my partâI’ve got all the ducks in a row. Now it’s up to you (and your imagination.)
5. Those Lovely Transactions! đ§
Transaction processing, good old transaction processing: Where would we be without such a crucial abstractionâone where we group together several reads and writes into a logical unit to simplify error handling and concurrency issuesâat our side (or at least in our computer systems.)
This all goes to prove that, sometimes at least, you can get what you want. (Why, thank you, Mr. Jagger.)
6. Of ACID And BASE đ°
Time to brush up on our high school Chemistry, amirite? (In particular that pesky pH scale for disambiguating or something between acidity and alkalinity.)
Relax.
We have a transaction (or two) to complete first. And this reeks of dĂ©jĂ vu, but oh well: See those dark brown tables, almost nested one inside the other? I have a feeling that this may be tables all the way down. But let’s not rush; we’ll wait, and find out for sure.
Oh, nearly forgot: We were going to see aboutâsurely you haven’t forgotten that we had a transaction (or two) to completeâthis fabled parable of distinguishing between acidity and alkalinity.
(If this talk of high school Chemistry is causing you to break in cold sweat, we’ve got you, too, coveredâThink of it this way: Tell acid from base is easier than navigating these COVID-19 pandemic times, now isn’t it?)
So this transactionâimmutable or otherwiseâwas going to happen at a reception where we were gonna meet a connection, Mr. Jagger, amirite?
Why yes, and here’s the scoop, complete with links for your enlightenment:
There you have it. And we are moving right on.
Allow me tell youâand this is especially for all those still inseparably glued to the Ahmad Tea box in the pic aboveâthat tea time is now officially over.
We got some buildings to stand up now.
Let’s stand up and deliver.
7. The Architecture Metaphor đ°
Check the edifice above, at once delicate and sturdy, laden with meaning: Is that dĂ©jĂ vu or what? I mean, for one thing,Â
- It’s those tables again, nested one inside another like Matryoshka dolls.
- Then there’s that serpentine, pliable rulerâ”S Ruler” I think it’s called.
- And finally, on the lowermost of those tables sits a glorious dandelion in full bloomâa gift for me from my momâencased ethereally, for ever and ever, with its wondrousness intact, unsullied.
Anyone see the data leitmotif in there? Here’s a hint from the play book of a sleuthâand it’s not Tintin this time:
“Data! Data! Data!” Sherlock Holmes cried impatiently. “I can’t make bricks without clay.“
~Arthur Conan Doyle (in The Adventure of the Copper Beeches)
You see, it’s elementary: It’s about relationshipsâbetween streams and tablesâwhich permeate and power our data-rich industry. Consider how
- Non-grouping operations consume streams and produce those dainty streams as “sinks”.
- Grouping operations, on the other hand, consume streams and spit out tables asâyou guessed it!â”sinks”.
That little birdâyou know that one with which we had crossed paths earlierâalso told me to look up, one more time: Chirpily good stuff.
Onward (and upstream, to be sure.)
8. It’s Nearly Curtain Time đŹ
Behold the motley crew that populates the topmost table above…
Now that’s what I call one big, happy family: Transaction processing rubbing shoulders withâmore like wedged-in between (1) a SQL cookout guide, and (2) a tome on streaming data goodness. (And that’s only the background!)
One more time: It’s those tables. Again.
Man, this is too weird for words. So what’s the unstated message? Is it that “Families that play together stay together“? (Actually, judging by the supremely cozy and ragtag “family” assembled in the pic above, that would be a reasonable assessment.)
You’re getting close, but not quite there, yet.
What is the secret, then?
Hint: The wondrously nestled tables are miming a message, amirite?
9. Hey, You On Top Of That Soap-Box đŁ
This is where Tintin gets to stand on his soap-box. Wait. That’s no soap-box! That’s a nifty Ahmad Tea-box…
While our pal Tintin ascends to the Pantheon, we mere mortals have plenty of coolness right here on our hands: I’m telling you, that guy E.F. Codd was a genius, way ahead of his times. He accomplished nothing less than lay the foundation for a declarative languageâSQLâthat powers databases the world over.
And exactly how did it spread like wildfire? In two words: Network effectsâthink network effects.
10. It’s Tables All The Way Down đą
Tintin descends, his mouth still wide agape with astonishment at having solved yet another mystery. (Where have we seen that exact phenom before?!)
Well, I leave you with that pointer (for further explorations.) Meanwhile, we get to rejoice in the heartwarming sight of the assemblage above: Now that’s what I call a big, happy family. That ain’t no diaspora; that won’t get me a patent anytime soon either, but that’s okay because I am satisfied in the knowledge of having sparked your imaginationâI did now, didn’t I?âto go forth and explore.
Ah, the punchline: The recurring patternâwe know a thing or two about that sort of thing, amirite?âin the bedecked tables has (hopefully) told the tale it was meant to. So in the spirit of software pioneer Bertrand Meyer’s grand finale “Afterword” to his classic bookârecall how a whole generation was introduced to object technology through this bookâentitled Object-Oriented Software Construction, we reveal in full candor that
It’s tables all the way down.
Till next time, I leave you with this thought: Do you like “pictorial” essaysâsuch as this oneâmore than those that have less pictures (and more, um, words)?
Either way, our jaunt through the data landscape has officially come to a close. Stay safe, especially so during these COVID-19 pandemic times.
And remember that voting is democratic and patriotic: Please vote.