This book targets data scientists and engineers. We chose these two groups because they have the most to gain from using Spark to expand the scope of problems they can solve. Spark’s rich collection of data-focused libraries (like MLlib) makes it easy for data scientists to go beyond problems that fit on a single machine while using their statistical background. Engineers, meanwhile, will learn how to write general-purpose distributed programs in Spark and operate production applications. Engineers and data scientists will both learn different details from this book, but will both be able to apply Spark to solve large distributed problems in their respective fields. 

The second group this book targets is software engineers who have some experience with Java, Python, or another programming language. If you are an engineer, we hope that this book will show you how to set up a Spark cluster, use the Spark shell, and write Spark applications to solve parallel processing problems (italicized by me for emphasis). If you are familiar with Hadoop, you have a bit of a head start on figuring out how to interact with HDFS and how to manage a cluster, but either way, we will cover basic distributed execution concepts.

Its inner workings are complex, resting as they do on a mixture of distributed systems theory, practical engineering, and common sense. And to the uninitiated, Hadoop can appear alien.  

But it doesn’t need to be like this. Stripped to its core, the tools that Hadoop provides for working with big data are simple. If there’s a common theme, it is about raising the level of abstraction—to create building blocks for programmers who have lots of data to store and analyze, and who don’t have the time, the skill, or the inclination to become distributed systems experts to build the infrastructure to handle it.

The book is divided into five main parts: Parts I to III are about core Hadoop, Part IV covers related projects in the Hadoop ecosystem, and Part V contains Hadoop case studies. You can read the book from cover to cover, but there are alternative pathways through the book that allow you to skip chapters that aren’t needed to read later ones.

Tom is now a respected senior member of the Hadoop developer community. Though he’s an expert in many technical corners of the project, his specialty is making Hadoop easier to use and understand. 

Given this, I was very pleased when I learned that Tom intended to write a book about Hadoop. Who could be better qualified? Now you have the opportunity to learn about Hadoop from a master—not only of the technology, but also of common sense and plain talk.

This hands-on book targets users who have some practical experience with Hadoop and understand the basic concepts of MapReduce and HDFS. Manning’s Hadoop in Action by Chuck Lam contains the necessary prerequisites to understand and apply the techniques covered in this book.  

Many techniques in this book are Java-based, which means readers are expected to possess an intermediate-level knowledge of Java. An excellent text for all levels of Java users is Effective Java, Second Edition by Joshua Bloch (Addison-Wesley).

5. Professional Hadoop Solutions (Wrox), by Boris Lublinsky et al 🐙

Once comfortable with the Hadoop paradigm, you’ll be able to appreciate the gem of a book we’ve got in this next title: Professional Hadoop Solutions (Wrox), by Boris Lublinsky et al. The authors have assembled a first-class collection of design expertise narratives.

In my mind, the key to understanding the value in this book lies in appreciating the following observation, which the authors make in the introductory chapter

They go on to explain cogently the reasons why great emphasis is placed on MapReduce code throughout the book. So if you approach this book with the mindset that the narratives will directly revolve around MapReduce, you’ll glean quite a bit of value out of this book. Their explanation of the MapReduce paradigm, as well as its nuts-and-bolts mechanisms, really are top notch.

The standout chapters are the following:

• Processing Your Data with MapReduce
• Customizing MapReduce Execution
• Hadoop Security
• Building Enterprise Security Solutions for Hadoop Implementations

The Appendix toward the end of Professional Hadoop Solutions is especially rich and useful. Overall, I’m glad to have found this book!

6. Data Scientists at Work (Apress) by Sebastian Gutierrez ☕

And now let’s segue from Hadoop to a foray into Data Science kingdom proper 🏰

But first a fair warning is in order about this next book: Once you start reading it, you’re going to have a terribly hard time putting it down or, for that matter, doing anything else before you’ve read it all! Such was my experience of reading (and re-reading) this page-turner of a book: Data Scientists at Work (Apress) by Sebastian Gutierrez.

Consider this… We have these marvelous frameworks—in Spark, Hadoop, Storm and others—but surely they were not created in some ethereal vacuum. Right, these frameworks were of course created in the service of genuine business needs, and to solve pressing problems that folks were facing. So if you’re looking for the scoop on this nexus (i.e. the potent symbiosis between the aims of Data Science and what Big Data has to offer), this is the book for you.

The corpus of this book is made up of in-depth interviews of 16 gifted data scientists. What makes these interviews incredibly engaging is the spectacularly good job done by the interviewer (the author of this book), Sebastian Gutierrez. His academic training is from MIT—where he earned a BS in Mathematics—and he is a data entrepreneur who has founded three data-related companies.

The pointed and evocative questions asked throughout the book could only have come from someone who knows the pragmatics of the Data Science field inside-out! And therein lies the immense value of this book: Detailed answers by 16 top data scientists as they shed light on the human side of data science, their thoughts on how this field is evolving, where it’s headed, plus plenty of straight-from-the-trenches stories about their work.

While the quality of the interviews is uniformly excellent, the standout interviews in my mind are the ones with these data scientists who are doing stellar work

• Claudia Perlich 

• Eric Jonas 

• Daniel Tunkelang 

To give you a flavor of the interviews—each of which is given its own chapter—ever so briefly, here is something from Claudia, who is the Chief Scientist at Dstillery. She teaches a high-level overview course on data mining for the NYU Stern MBA program to, in here own words, “…give people a good understanding of what the opportunities are and how to manage them instead of really teaching them how to do it”. She has taught at NYU, MIT, Wharton, and Columbia. In response to the interview question in the book (“What about this work is interesting and exciting for you?”), Claudia noted

Some 20 years ago, when I was finishing grad school—at that time, I earned an MS degree in electrical engineering from Texas A&M University—we didn’t call work such as my dissertation (Noise-tolerant Software Method for Traffic Sign Recognition) Data Science. But in several ways, while I was reading the fine interviews in this book, I sure was reminded of the algorithms I worked out back then: Various AI programming techniques (neural networks primarily, such as the Back-propagation Neural Network and the Adaptive Resonance Theory model, aka ART2). Good stuff, and enough reminiscing, for that matter 🙂

So Data Scientists at Work is a fantastic book overall, if this sort of thing piques your interest.

7. MapReduce Design Patterns: Building Effective Algorithms and Analytics for Hadoop and Other Systems (O’Reilly), by by Donald Miner and Adam Shook 🍯

Segueing right back to Hadoop now, the title of the next book is decidedly open-ended—MapReduce Design Patterns: Building Effective Algorithms and Analytics for Hadoop and Other Systems (O’Reilly), by by Donald Miner and Adam Shook. Given the open-ended title, allow me to elaborate on the gist of this fine book…

The authors are clearly experts in the Hadoop ecosystem, and what they’ve put together is more than what you’ll find in the endearing O’Reilly “cookbook” series. Thus, they don’t call out specific problems and accompanying solutions. Instead, they share the lessons that they have learned along the way to becoming experts in the Hadoop ecosystem. Note, too, that this book is mostly about the analytics side of Hadoop and MapReduce.

And they assume that you’re already familiar with how Hadoop and MapReduce work, so they don’t dive into the details of the APIs which they use in this book—Those topics have already been covered thoroughly in other books, and they focus on analytics. In their own words

One thing I appreciated a lot was the way the authors answer the question, “So why should we use Java MapReduce in Hadoop at all when we have options like Pig and Hive?”. They point out two core reasons for spending time explaining how to implement something in hundreds of lines of code when the same can be accomplished in a couple lines with, say, Pig and Hive. In their own words

Remaining mindful of the fact that the title of this book is admittedly open-ended, I mention here the table of contents to give you a flavor of the topics covered

• Chapter 1. Design Patterns and MapReduce
• Chapter 2. Summarization Patterns
• Chapter 3. Filtering Patterns
• Chapter 4. Data Organization Patterns
• Chapter 5. Join Patterns
• Chapter 6. Metapatterns
• Chapter 7. Input and Output Patterns
• Chapter 8. Final Thoughts and the Future of Design Patterns

With the caveats noted above, MapReduce Design Patterns: Building Effective Algorithms and Analytics for Hadoop and Other Systems is a book absolutely worth exploring!

8. Real-Time Analytics: Techniques to Analyze and Visualize Streaming Data (Wiley), by Byron Ellis 🐝

Finally, let’s segue to the land of real-time, streaming data 🙂

This next book is impeccably written in an eminently thoughtful style—Real-Time Analytics: Techniques to Analyze and Visualize Streaming Data (Wiley), by Byron Ellis. The author is the CTO of Spongecell, and he has a Ph.D. in Statistics from Harvard University.

No doubt, with enough determination and time, one can do online searches and cobble together a solution to handle real-time, high-volume mega data. But that begs the question, and I’m not questioning anyone’s tenacity here: Is that really the ideal strategy? And that’s where the book shines—What makes it stand out is the care and thought that have clearly been poured into making this book a one-stop resource for crafting end-to-end solutions for effectively grappling with real-time, high-volume mega data.

Much as I alluded to above, this book is impeccably written. The author has clearly honed his writing skills—quite likely while preparing his dissertation for the Ph.D. that he earned from Harvard University 🙂

Clearly written books are a heaven-send, and this superb book is one. In that vein, the author notes with razor-sharp precision the aim of this book

The author weaves into the narratives a lot of pragmatic advice; he has clearly been in the development trenches and done it all. As with the prior book, I mention here the table of contents to give you a flavor of the topics covered in Real-Time Analytics: Techniques to Analyze and Visualize Streaming Data

Part I: Streaming Analytics Architecture
Chapter 1: Introduction to Streaming Data Sources of Streaming Data, Why Streaming Data Is Different, Infrastructures and Algorithms, Conclusion
Chapter 2: Designing Real-Time Streaming Architectures Real-Time Architecture Components Features of a Real-Time Architecture Languages for Real-Time Programming A Real-Time Architecture Checklist Conclusion
Chapter 3: Service Configuration and Coordination Motivation for Configuration and Coordination Systems Maintaining Distributed State Apache ZooKeeper Conclusion
Chapter 4: Data-Flow Management in Streaming Analysis Distributed Data Flows Apache Kafka: High-Throughput Distributed Messaging Apache Flume: Distributed Log Collection Conclusion
Chapter 5: Processing Streaming Data Distributed Streaming Data Processing Processing Data with Storm Processing Data with Samza Conclusion
Chapter 6: Storing Streaming Data Consistent Hashing “NoSQL” Storage Systems Other Storage Technologies Choosing a Technology Warehousing Conclusion  

Part II: Analysis and Visualization 
Chapter 7: Delivering Streaming Metrics Streaming Web Applications Visualizing Data Mobile Streaming Applications Conclusion
Chapter 8: Exact Aggregation and Delivery Timed Counting and Summation Multi-Resolution Time-Series Aggregation Stochastic Optimization Delivering Time-Series Data Conclusion
Chapter 9: Statistical Approximation of Streaming Data Numerical Libraries Probabilities and Distributions Working with Distributions Random Number Generation Sampling Procedures Conclusion
Chapter 10: Approximating Streaming Data with Sketching Registers and Hash Functions Working with Sets The Bloom Filter Distinct Value Sketches The Count-Min Sketch Other Applications Conclusion
Chapter 11: Beyond Aggregation Models for Real-Time Data Forecasting with Models Monitoring Real-Time Optimization Conclusion Introduction Overview and Organization of This Book Who Should Read This Book Tools You Will Need What’s on the Website Time to Dive In

All this makes Real-Time Analytics: Techniques to Analyze and Visualize Streaming Data a book that shouldn’t be missed 😉

9. Big Data: Principles and Best Practices of Scalable Realtime Data Systems (Manning), by Nathan Marz 🐾

Last, but certainly not the least—continuing now in the spirit of frameworks that enable us developers to tackle real-time, streaming data—is a book by Nathan Marz: Big Data: Principles and Best Practices of Scalable Realtime Data Systems (Manning). The author happens to be the originator of the Lambda Architecture approach to programming in the world of Big Data, and he deploys his considerable knowledge of this approach in explaining the details.

This book is dives deep into the concepts underlying Lambda Architecture—which is what the author dubbed the approach that he formalized during his years working at the startup BackType—along with, importantly, many illustrative examples which are nicely supplemented by code snippets. The author puts it succinctly when he notes that

This book is the result of my desire to spread the knowledge of the Lambda Architecture and how it avoids the complexities of traditional architectures. It is the book I wish I had when I started working with Big Data. I hope you treat this book as a journey—a journey to challenge what you thought you knew about data systems, and to discover that working with Big Data can be elegant, simple, and fun.

As an aside—confessing here my fondness for Clojure, the Lisp that runs on the JVM—I couldn’t help but resonate with the following sentiments echoed by Nathan Marz in the Acknowledgments section of Big Data: Principles and Best Practices of Scalable Realtime Data Systems, where he notes that

Rich Hickey has been one of my biggest inspirations during my programming career. Clojure is the best language I have ever used, and I’ve become a better programmer having learned it. I appreciate its practicality and focus on simplicity. Rich’s philosophy on state and complexity in programming has influenced me deeply.

In sum, this is a worthwhile book, nicely structured into theory and illustration chapters.

In the end, and as I mentioned at the outset, I invite your comments—Having now read my brief take each on the books above…

• Do you find that your experience of reading any of these books was different? 
• Perhaps some qualities that I did not cover are the ones that you found the most helpful as you learned Scala and its ecosystem. 
• Did I omit any of your favorite Big Data book(s)? 
• I’ve covered only a partial list of the Big Data books that I’ve read, limited as you can imagine I am by the time available…

As with my prior post, which contains a set of book vignettes—those pertaining to the finest and most useful books on Scala in print—my aim here, too, in sharing these brief reviews remains the same, albeit on a different subject (Big Data) this time: I hope these vignettes will help you in selecting your resources well, and help you in your journey to grokking the Big Data solution space!

Bon voyage, and I leave you with an obligatory photo of a section of one of my bookshelves—one that’s, um, rather biased toward Big Data material in a statistically significant way, eh 😉