Google’s search engine makes it wonderfully easy to locate stuff on the web, whether it’s in a news article, a corporate website, or a video on YouTube. But that only begins to describe Google’s ability to find information. Inside the company, engineers use several uniquely powerful tools for searching and analyzing its own massive trove of data.
One of those is Dremel, a tool that helps Google’s employees analyze data stored across thousands of machines, at unusually fast speeds. What’s more, Dremel lets the Google team to manipulate all of this data using a language very similar to SQL, short for Structured Query Language, the standard way of grabbing information from databases.
Like most of its custom-built tools, Dremel is only available inside Google. But now, the rest of the world can hack data a little more like Google does, thanks to Quest, a Dremel-like query engine created by Theo Vassilakis, one of the lead developers of Dremel at Google, and Toli Lerios, a former engineer at Facebook. The tool is one of a growing number of that seek to mimic the way web giants like Google and Facebook rapidly analyze enormous amounts of online information stored across hundreds or even thousands of machines. This includes everything from a project called Drill, from a company called MapR, to a sweeping open source platform called Spark.