People, Person, Computer, Electronics, LCD Screen, Laptop, Pc

The Drilling Rigs of the Big Data Age

2017 was the year the phrase data is the new oil went mainstream. It is true that we are awash in all kinds of data, and the value chain is clear: large amounts of data, machines/software used to analyze that data, insights derived from it, money generated from those insights.

I’ve met with thousands of companies in this space. Here are some trends that are creating the need for a new class of companies that will serve as the drilling rigs of the big data age:

  • The ratio of communications vs compute has risen dramatically over the last 10 years. Engines built for data centers need to adapt for this
  • The amount of data we need to process can no longer be stored on a single server. It must be stored on hundreds or thousands of servers, which further increases the amount of communication required. The use of solid state storage has led to a 100-fold increase in performance, putting more demands on network infrastructure
  • Microservices are driving new ways for applications to be built and managed
  • The amount of data AI/ML also needed in order to function is very large
  • Moore’s Law (the observation that the number of transistors in a dense integrated circuit doubles approximately every two years) is going away
  • Edge data centers are on the rise for security as well as to deliver real-time, low latency application performance

As a result of these trends, we see five big areas of opportunities:

1. There will be a new Intel.

Over the next decade, we will see a flowering of silicon architectures engineered for specific domains, as we are already seeing with GPUs applied to machine learning. One example from our portfolio is Fungible, led by Dr. Pradeep Sindhu, Juniper Networks founder, which is building a specialized silicon architecture called a data processing unit (DPU) that can move data rapidly and connect different applications engines in the modern data center.

2. There will be a new Akamai.

We have moved beyond the era of CDNs, where edge data centers will require full-stack application platforms, which guarantee reliability and security while delivering real-time processing data-first apps with low latency. Entrepreneurs with software, systems and networking experience will be ideally suited to create these companies forming the new category of a data delivery network (DDN).

3. There will be a new Oracle.

The era of big data and multi-cloud requires NoSQL vs. SQL relational databases and converged platforms which can support all data, on one platform, and on every cloud.

4. There will be a new Business Objects/Tableau.

Visualization of data takes on a whole new level when the underlying datasets are multi-modal and huge. Analytics software needs to run directly within modern data platforms such as Apache Hadoop and the cloud. It has to analyze large volumes of data without moving it, filling the gap between self-service BI and advanced analytics for use cases like cyber security, IOT, and customer intelligence.

5. There will other industry pioneers in management and automation.

As microservices emerge as the metaphor for devops, automation becomes the lifeblood. Tools like Kubernetes, Mesos, and Swarm automate container clusters.There is need for a platform that automates the deployment and operations of data services at scale.

At the risk of pushing the oil analogy too far, the age of big data presents many wildcat opportunities to build the new super majors.

Originally published at on February 8, 2018.

The Drilling Rigs of the Big Data Age was originally published in Mayfield Viewpoints on Medium, where people are continuing the conversation by highlighting and responding to this story.

Read more >

Published on Feb 21, 2018

Other people viewed

Senior Solutions Engineer - Irvine

MapR Irvine California United States Irvine, California, United States
You are a customer-facing, hands-on sales engineer or big data architecture consultant with a track record of successful value-based and solution selling. You collaborate with account managers and the partner ecosystem in your territory to drive p...

Manager/Director, Technical Support - India

MapR Hyderabad Telangana India Hyderabad, Telangana, India
You are a customer-facing, hands-on Engineer and Manager who has a track record of working with customers in a support organization (technical guru preferred over manager). You collaborate with the account team, and the partner ecosystem in your t...

Distributed Systems Engineer

MapR Hyderabad Telangana India Hyderabad, Telangana, India
In this exciting position, you will provide the highest level of technical expertise and advice to MapR's rapidly growing customer base. You will provide assistance to these enterprises throughout the Pre- and Post-Sales process. On average, your ...

Principal Software Engineer, MapR Data Platform

MapR Santa Clara California United States Santa Clara, California, United States
Looking for a fast-paced environment where you can learn push the boundaries of performance and scale? MapR Data Platform is built for Cloud, Containerized and on-prem deployments and provides scale and integrated security. It contains a robust No...