Table of Contents
Introduction
Chapter 1: The Big Data Business Opportunity
The Business Transformation Imperative
The Big Data Business Model Maturity Index
Big Data Business Model Maturity Observations
Summary
Chapter 2: Big Data History Lesson
Consumer Package Goods and Retail Industry Pre-1988
Lessons Learned and Applicability to Today's Big Data Movement
Summary
Chapter 3: Business Impact of Big Data
Big Data Impacts: The Questions Business Users Can Answer
Managing Using the Right Metrics
Data Monetization Opportunities
Summary
Chapter 4: Organizational Impact of Big Data
Data Analytics Lifecycle
Data Scientist Roles and Responsibilities
New Organizational Roles
Liberating Organizational Creativity
Summary
Chapter 5: Understanding Decision Theory
Business Intelligence Challenge
The Death of Why
Big Data User Interface Ramifications
The Human Challenge of Decision Making
Summary
Chapter 6: Creating the Big Data Strategy
The Big Data Strategy Document
Starbucks Big Data Strategy Document Example
San Francisco Giants Big Data Strategy Document Example
Summary
Chapter 7: Understanding Your Value Creation Process
Understanding the Big Data Value Creation Drivers
Michael Porter's Valuation Creation Models
Summary
Chapter 8: Big Data User Experience Ramifications
The Unintelligent User Experience
Understanding the Key Decisions to Build a Relevant User Experience
Using Big Data Analytics to Improve Customer Engagement
Uncovering and Leveraging Customer Insights
Big Data Can Power a New Customer Experience
Summary
Chapter 9: Identifying Big Data Use Cases
The Big Data Envisioning Process
The Prioritization Process
Using User Experience Mockups to Fuel the Envisioning Process
Summary
Chapter 10: Solution Engineering
The Solution Engineering Process
Solution Engineering Tomorrow's Business Solutions
Reading an Annual Report
Summary
Chapter 11: Big Data Architectural Ramifications
Big Data: Time for a New Data Architecture
Introducing Big Data Technologies
Bringing Big Data into the Traditional Data Warehouse World
Summary
Chapter 12: Launching Your Big Data Journey
Explosive Data Growth Drives Business Opportunities
Traditional Technologies and Approaches Are Insufficient
The Big Data Business Model Maturity Index
Driving Business and IT Stakeholder Collaboration
Operationalizing Big Data Insights
Big Data Powers the Value Creation Process
Summary
Chapter 13: Call to Action
Identify Your Organization's Key Business Initiatives
Start with Business and IT Stakeholder Collaboration
Formalize Your Envisioning Process
Leverage Mockups to Fuel the Creative Process
Understand Your Technology and Architectural Options
Build off Your Existing Internal Business Processes
Uncover New Monetization Opportunities
Understand the Organizational Ramifications
Introduction
Big data is today's technology hot topic. Such technology hot topics come around every four to five years and become the “must have” technologies that will lead organizations to the promised land—the “silver bullet” that solves all of our technology deficiencies and woes. Organizations fight through the confusion and hyperbole that radiate from vendors and analysts alike to grasp what the technology can and cannot do. In some cases, they successfully integrate the technology into the organization's technology landscape—technologies such as relational databases, Enterprise Resource Planning (ERP), client-server architectures, Customer Relationship Management (CRM), data warehousing, e-commerce, Business Intelligence (BI), and open source software.
However, big data feels different, maybe because at its heart big data is not about technology as much as it's about business transformation—transforming the organization from a retrospective, batch, data constrained, monitor the business environment into a predictive, real-time, data hungry, optimize the business environment. Big data isn't about business parity or deploying the same technologies in order to be like everyone else. Instead, big data is about leveraging the unique and actionable insights gleaned about your customers, products, and operations to rewire your value creation processes, optimize your key business initiatives, and uncover new monetization opportunities. Big data is about making money, and that's what this book addresses—how to leverage those unique and actionable insights about your customers, products, and operations to make money.
This book approaches the big data business opportunities from a pragmatic, hands-on perspective. There aren't a lot of theories here, but instead lots of practical advice, techniques, methodologies, downloadable worksheets, and many examples I've gained over the years from working with some of the world's leading organizations. As you work your way through this book, you will do and learn the following:
The beauty of being in the data and analytics business is that we are only a new technology innovation away from our next big data experience. First, there was point-of-sale, call detail, and credit card data that provided an earlier big data opportunity for consumer packaged goods, retail, financial services, and telecommunications companies. Then web click data powered the online commerce and digital media industries. Now social media, mobile apps, and sensor-based data are fueling today's current big data craze in all industries—both business-to-consumer and business-to-business. And there's always more to come! Data from newer technologies, such as wearable computing, facial recognition, DNA mapping, and virtual reality, will unleash yet another round of big data-driven value creation opportunities.
The organizations that not only survive, but also thrive, during these data upheavals are those that embrace data and analytics as a core organizational capability. These organizations develop an insatiable appetite for data, treating it as an asset to be hoarded, not a business cost to be avoided. Such organizations manage analytics as intellectual property to be captured, nurtured, and sometimes even legally protected.
This book is for just such organizations. It provides a guide containing techniques, tools, and methodologies for feeding that insatiable appetite for data, to build comprehensive data management and analytics capabilities, and to make the necessary organizational adjustments and investments to leverage insights about your customers, products, and operations to optimize key business processes and uncover new monetization opportunities.
Every now and then, new sources of data emerge that hold the potential to transform how organizations drive, or derive, business value. In the 1980s, we saw point-of-sale (POS) scanner data change the balance of power between consumer package goods (CPG) manufacturers like Procter & Gamble, Unilever, Frito Lay, and Kraft—and retailers like Walmart, Tesco, and Vons. The advent of detailed sources of data about product sales, soon coupled with customer loyalty data, provided retailers with unique insights about product sales, customer buying patterns, and overall market trends that previously were not available to any player in the CPG-to-retail value chain. The new data sources literally changed the business models of many companies.
Then in the late 1990s, web clicks became the new knowledge currency, enabling online merchants to gain significant competitive advantage over their brick-and-mortar counterparts. The detailed insights buried in the web logs gave online merchants new insights into product sales and customer purchase behaviors, and gave online retailers the ability to manipulate the user experience to influence (through capabilities like recommendation engines) customers' purchase choices and the contents of their electronic shopping carts. Again, companies had to change their business models to survive.
Today, we are in the midst of yet another data-driven business revolution. New sources of social media, mobile, and sensor or machine-generated data hold the potential to rewire an organization's value creation processes. Social media data provide insights into customer interests, passions, affiliations, and associations that can be used to optimize your customer engagement processes (from customer acquisition, activation, maturation, up-sell/cross-sell, retention, through advocacy development). Machine or sensor-generated data provide real-time data feeds at the most granular level of detail that enable predictive maintenance, product performance recommendations, and network optimization. In addition, mobile devices enable location-based insights and drive real-time customer engagement that allow brick-and-mortar retailers to compete directly with online retailers in providing an improved, more engaging customer shopping experience.
The massive volumes (terabytes to petabytes), diversity, and complexity of the data are straining the capabilities of existing technology stacks. Traditional data warehouse and business intelligence architectures were not designed to handle petabytes of structured and unstructured data in real-time. This has resulted in the following challenges to both IT and business organizations:
This blitz of new data has necessitated and driven technology innovation, much of it being powered by open source initiatives at digital media companies like Google (Big Table), Yahoo! (Hadoop), and Facebook (Hive and HBase), as well as universities (like Stanford, UC Irvine, and MIT). All of these big data developments hold the potential to paralyze businesses if they wait until the technology dust settles before moving forward. For those that wait, only bad things can happen:
The time to move is now, because the risks of not moving can be devastating.
The big data movement is fueling a business transformation. Companies that are embracing big data as business transformational are moving from a retrospective, rearview mirror view of the business that uses partial slices of aggregated or sampled data in batch to monitor the business to a forward-looking, predictive view of operations that leverages all available data—including structured and unstructured data that may sit outside the four walls of the organization—in real-time to optimize business performance (see Table 1.1).
Table 1.1 Big Data Is About Business Transformation
| Today's Decision Making | Big Data Decision Making |
| “Rearview Mirror” hindsight | “Forward looking” recommendations |
| Less than 10% of available data | Exploit all data from diverse sources |
| Batch, incomplete, disjointed | Real time, correlated, governed |
| Business Monitoring | Business Optimization |
Think of this as the advent of the real-time, predictive enterprise!
In the end, it's all about the data. Insight-hungry organizations are liberating the data that is buried deep inside their transactional and operational systems, and integrating that data with data that resides outside the organization's four walls (such as social media, mobile, service providers, and publicly available data). These organizations are discovering that data—and the key insights buried inside the data—has the power to transform how organizations understand their customers, partners, suppliers, products, operations, and markets. In the process, leading organizations are transforming their thinking on data, transitioning from treating data as an operational cost to be minimized to a mentality that nurtures data as a strategic asset that needs to be acquired, cleansed, transformed, enriched, and analyzed to yield actionable insights. Bottom-line: companies are seeking ways to acquire even more data that they can leverage throughout the organization's value creation processes.
Data can transform both companies and industries. Walmart is famous for their use of data to transform their business model.
The cornerstone of his [Sam Walton's] company's success ultimately lay in selling goods at the lowest possible price, something he was able to do by pushing aside the middlemen and directly haggling with manufacturers to bring costs down. The idea to “buy it low, stack it high, and sell it cheap” became a sustainable business model largely because Walton, at the behest of David Glass, his eventual successor, heavily invested in software that could track consumer behavior in real time from the bar codes read at Walmart's checkout counters.
He shared the real-time data with suppliers to create partnerships that allowed Walmart to exert significant pressure on manufacturers to improve their productivity and become ever more efficient. As Walmart's influence grew, so did its power to nearly dictate the price, volume, delivery, packaging, and quality of many of its suppliers' products. The upshot: Walton flipped the supplier-retailer relationship upside down.1
Walmart up-ended the balance of power in the CPG-to-retailer value chain. Before they had access to detailed POS scanner data, the CPG manufacturers (such as Procter & Gamble, Unilever, Kimberley Clark, and General Mills,) dictated to the retailers how much product they would be allowed to sell, at what prices, and using what promotions. But with access to customer insights that could be gleaned from POS data, the retailers were now in a position where they knew more about their customers' behaviors—what products they bought, what prices they were willing to pay, what promotions worked the most effectively, and what products they tended to buy in the same market basket. Add to this information the advent of the customer loyalty card, and the retailers knew in detail what products at what prices under what promotions appealed to which customers. Soon, the retailers were dictating terms to the CPG manufacturers—how much product they wanted to sell (demand-based forecasting), at what prices (yield and price optimization), and what promotions they wanted (promotional effectiveness). Some of these retailers even went one step further and figured out how to monetize their POS data by selling it back to the CPG manufacturers. For example, Walmart provides a data service to their CPG manufacturer partners, called Retail Link, which provides sales and inventory data on the manufacturer's products sold through Walmart.
Across almost all organizations, we are seeing multitudes of examples where data coupled with advanced analytics can transform key organizational business processes, such as:
Customers often ask me:
To help address these types of questions, I've created the Big Data Business Model Maturity Index. This index provides a benchmark against which organizations can measure themselves as they look at what big data-enabled opportunities may lay ahead. Organizations can use this index to:
Organizations are moving at different paces with respect to how they are adopting big data and advanced analytics to create competitive advantages for themselves. Some organizations are moving very cautiously because they are unclear where and how to start, and which of the bevy of new technology innovations they need to deploy in order to start their big data journeys. Others are moving at a more aggressive pace to integrate big data and advanced analytics into their existing business processes in order to improve their organizational decision-making capabilities.
However, a select few are looking well beyond just improving their existing business processes with big data. These organizations are aggressively looking to identify and exploit new data monetization opportunities. That is, they are seeking out business opportunities where they can either sell their data (coupled with analytic insights) to others, integrate advanced analytics into their products to create “intelligent” products, or leverage the insights from big data to transform their customer relationships and customer experience.
Let's use the Big Data Business Model Maturity Index depicted in Figure 1.1 as a framework against which you can not only measure where your organization stands today, but also get some ideas on how far you can push the big data opportunity within your organization.
Figure 1.1 Big Data Business Model Maturity Index
In the Business Monitoring phase, you deploy Business Intelligence (BI) and traditional data warehouse capabilities to monitor, or report on, on-going business performance. Sometimes called business performance management, business monitoring uses basic analytics to flag under- or over-performing areas of the business, and automates sending alerts with pertinent information to concerned parties whenever such a situation occurs. The Business Monitoring phase leverages the following basic analytics to identify areas of the business requiring more investigation:
The Business Monitoring phase is a great starting point for your big data journey as you have already gone through the process—via your data warehousing and BI investments—of identifying your key business processes and capturing the KPIs, dimensions, metrics, reports, and dashboards that support those key business processes.
The Business Insights phase takes business monitoring to the next step by leveraging new unstructured data sources with advanced statistics, predictive analytics, and data mining, coupled with real-time data feeds, to identify material, significant, and actionable business insights that can be integrated into your key business processes. This phase looks to integrate those business insights back into the existing operational and management systems. Think of it as “intelligent” dashboards, where instead of just presenting tables of data and graphs, the application goes one step further to actually uncover material and relevant insights that are buried in the detailed data. The application can then make specific, actionable recommendations, calling out an observation on a particular area of the business where specific actions can be taken to improve business performance. One client called this phase the “Tell me what I need to know” phase. Examples include:
The following steps will transition your organization from the business monitoring to the business insights stage.
The Business Optimization phase is the level of business maturity where organizations use embedded analytics to automatically optimize parts of their business operations. To many organizations, this is the Holy Grail where they can turn over certain parts of their business operations to analytic-powered applications that automatically optimize the selected business activities. Business optimization examples include:
The following steps will transition your organization from the Business Insights phase to the Business Optimization phase:
You should also consider the creation of a formal analytics governance process that enables human subject matter experts to audit and evaluate the effectiveness and relevance of the resulting optimization models on a regular basis. As any good data scientist will tell you, the minute you build your analytic model it is obsolete due to changes in the real-world environment around it.
The Data Monetization phase is where organizations are looking to leverage big data for net new revenue opportunities. While not an exhaustive list, this includes initiatives related to:
An example of the first type of initiative could be a smartphone app where data and insights about customer behaviors, product performance, and market trends are sold to marketers and manufacturers. For example, MapMyRun (www.MapMyRun.com) could package the customer usage insights from their smartphone application with audience and product insights for sale to sports apparel manufacturers, sporting goods retailers, insurance companies, and healthcare providers.
An example of the second type of initiative could be companies that leverage new big data sources (sensor data or user click/selection behaviors) with advanced analytics to create “intelligent” products, such as:
An example of the third type of initiative could be companies that leverage actionable insights and recommendations to “up-level” their customer relationships and dramatically rethink their customer's experience, such as:
The following steps will be useful in helping transition to the Data Monetization phase.
The Business Metamorphosis phase is the ultimate goal for organizations that want to leverage the insights they are capturing about their customers' usage patterns, product performance behaviors, and overall market trends to transform their business models into new services in new markets. For example:
In order to make the move into the Business Metamorphosis phase, organizations need to think about moving away from a product-centric business model to a more platform- or ecosystem-centric business model.
Let's drill into this phase by starting with a history lesson. The North American video game console market was in a massive recession in 1985. Revenues that had peaked at $3.2 billion in 1983, fell to $100 million by 1985—a drop of almost 97 percent. The crash almost destroyed the then-fledgling industry and led to the bankruptcy of several companies, including Atari. Many business analysts doubted the long-term viability of the video game console industry.
There were several reasons for the crash. First, the hardware manufacturers had lost exclusive control of their platforms' supply of games, and consequently lost the ability to ensure that the toy stores were never overstocked with products. But the main culprit was the saturation of the market with low-quality games. Poor quality games, such as Chase the Chuck Wagon (about dogs eating food, bankrolled by the dog food company Purina), drove customers away from the industry.
The industry was revitalized in 1987 with the success of the Nintendo Entertainment System (NES). To ensure ecosystem success, Nintendo instituted strict measures to ensure high-quality games through licensing restrictions, maintained strict control of industry-wide game inventory, and implemented a security lockout system that only allowed certified games to work on the Nintendo platform. In the process, Nintendo ensured that third-party developers had a ready and profitable market.
As organizations contemplate the potential of big data to transform their business models, they need to start by understanding how they can leverage big data and the resulting analytic insights to transform the organization from a product-centric business model into a platform-centric business model. Much like the Nintendo lesson, you accomplish this by creating a marketplace that enables others—like app developers, partners, VARs, and third party solution providers—to make money off of your platform.
Let's build out the previous example of an energy company moving into the home energy optimization business. The company could capture home energy and appliance usage patterns that could be turned into insights and recommendations. For example, with the home energy usage information, the company could recommend when consumers should run their high energy appliances, like washers and dryers, to minimize energy costs. The energy company could go one step further and offer a service that automatically manages when the washer, dryer, and other high-energy appliances run—such as running the washer and dryer at 3:00 a.m. when energy prices are lower.
With all of the usage information, the company is also in a good position to predict when certain appliances might need maintenance (for example, monitoring their usage patterns using Six Sigma control charts to flag out-of-bounds performance problems). The energy company could make preventive maintenance recommendations to the homeowner, and even include the names of three to four local service dealers and their respective Yelp ratings.
But wait, there's more! With all of the product performance and maintenance data, the energy company is also in an ideal position to recommend which appliances are the best given the customer's usage patterns and local energy costs. They could become the Consumer Reports for appliances and other home and business equipment by recommending which brands to buy based on the performance of different appliances as compared to their customers' usage patterns, local weather, environmental conditions, and energy costs.
Finally, the energy company could package all of the product performance data and associated maintenance insights and sell the data and analytic insights back to the manufacturers who might want to know how their products perform within certain usage scenarios and versus key competitors.
In this scenario, there are more application and service opportunities than any single vendor can reasonably supply. That opens the door to transform to a platform-centric business model that creates a platform or ecosystem that enables third party developers to deliver products and services on that platform. And, of course, this puts the platform provider in a position to take a small piece of the “action” in the process, such as subscription fees, rental fees, transaction fees, and referral fees.
Much like the lessons of Nintendo with their third-party video games, and Apple and Google with their respective apps stores, creating such a platform not only benefits your customers who are getting access to a wider variety of high-value apps and services in a more timely manner, it also benefits the platform provider by creating a high level of customer dependency on your platform (for example, by increasing the switching costs).
Companies that try to do all of this on their own will eventually falter because they'll struggle to keep up with the speed and innovation of smaller, hungrier organizations that can spot and act on a market opportunity more quickly. Instead of trying to compete with the smaller, hungrier companies, enable such companies by giving them a platform on which they can quickly and profitability build, market, and support their apps and solutions.