Entering a world filled with data

Of course, data collection itself isn’t new. But technological advances like chip and sensor technology, the Internet, cloud computing and the ability to store and analyse data have changed the quantity of data you can collect.

Activities that have been a part of everyday life for decades – shopping, listening to music, taking pictures, talking on the phone – increasingly happen wholly or in part in the digital realm and therefore leave a trail of data.

And the amount of data being generated every day is staggering. For example, users of Facebook upload around one billion pieces of content to the social network site every day. In industry, machinery and vehicles are fitted with sensors and trackers that record their every move, and whenever you call a call centre, an audio recording of your conversation is made and stored in a huge digital database.

Understanding the infinite ways to use big data

Eventually, every aspect of your lives will be affected by big data. However, there are some areas where big data is already making a real difference today – in business and in other areas. Let’s look at the main areas where big data is most widely used right now.

• Understanding and targeting customers: This is one of the most common uses of big data today. Here, big data is used to better understand customers and their behaviours and preferences.
• Understanding and optimising business processes: Big data is also increasingly used to optimise a wider range of business processes, including stock control, supply chain and delivery routes and HR processes.
• Optimising personal quantification and performance: Individuals can now benefit from the data generated from wearable devices such as smart watches and smart bracelets – data like calorie consumption, activity levels and sleep patterns.
• Improving healthcare and public health: The computing power of big data analytics enables scientists to decode entire DNA strings in minutes and allows them to find new cures and better understand and predict disease patterns. What’s more, big data analytics allow researchers to monitor and predict the developments of epidemics and disease outbreaks by integrating data from medical records with social media analytics.
• Improving sports performance: Most elite sports have now embraced big data analytics. Video analytics track the performance of every player in a football game, and sensor technology in sports equipment such as golf clubs allows you to get feedback (via your smartphone) on your game and how to improve it.
• Improving science and research: Science and research is currently being transformed by big data. Experiments with the Large Hadron Collider, for example, generate huge amounts of data. The CERN (European Organization for Nuclear Research) data centre has 65,000 processors to analyse its 30 petabytes (a petabyte is one quadrillion bytes) of data.
• Optimising machine and device performance: Big data analytics help machines and devices become smarter and more autonomous. For example, big data tools are used to operate Google’s self-driving car.
• Improving security and law enforcement: I’m sure you’re aware of the revelations that the National Security Agency (NSA) in the US uses big data analytics to foil terrorist plots (and maybe spy on us!). Others use big data techniques to detect and prevent cyber attacks and police forces use big data tools to catch criminals.
• Improving and optimising cities and countries: Big data is used to improve many aspects of where we live. For example, it allows cities to optimise traffic flows based on real-time traffic information as well as social media and weather data.
• Optimising financial trading: High-Frequency Trading (HFT) is an area where big data finds a lot of use today by using big data algorithms to make trading decisions.

Using big data in small businesses

Big data might seem like it’s something that only big business can make use of. When people first hear that massive volumes of information are being used to fight terrorism, cure cancer or predict the spread of Ebola, it sounds expensive, difficult and time-consuming. But that doesn’t have to be the case.

Huge datasets on everything from demographics to weather and consumer spending habits are freely available online for small businesses to use. Plus, the basic tools to make sense of the data are also free and becoming increasingly simple for anyone to use. For example, if you’re using Google’s AdWords to track what your customers are searching for online, you’re engaging in big data analysis, even if you don’t know it.

In many ways, big data is much better suited to small businesses than to big corporations – smaller companies tend to be more agile and able to act on the insights that data provides in a more timely fashion. In the end, even the most impressive data set and the most potent insights are worthless if you don’t act on them.

Plenty of small businesses are already using big data to better understand and target customers. Retailers can predict what products will sell, car insurance companies can understand how well their customers actually drive and detect potential fraud and takeaway companies can tailor their services to meet local customer preferences and demand. Social media has become a particularly valuable source of data for understanding customers, trends and markets.

Big data can also help improve business processes. Retailers are able to optimise their stock levels based on what’s trending on social media, what people are searching for on the web or even weather forecasts. Supply chains can be optimised so that delivery drivers use less gas and reach customers faster. And you can use data to understand and improve staff engagement or improve your hiring process.

There’s more detail on the many big data uses in Chapter 3 – and there are examples dotted throughout the book. Just look out for the Example icon.

Too often I see big data analysis being done in an infrequent, unstructured or ad hoc way (and that’s in businesses of all sizes). You really need an underlying strategy in order to get the most out of big data, and there’s more on that in Chapter 10. Without an underlying strategy, you may stumble across the odd valuable insight, but with proper planning and preparation, those insights are more frequent and more useful.

Understanding the value of insights

In today’s competitive business world, success often comes down to a company’s ability to learn faster than the competition and act on what they learn faster than the competition. The process of turning data into insights and actionable knowledge is the key to that success.

No matter how much data you gather, it’s worth very little unless you can turn it into insights and actionable knowledge. Data-based insights help you to make better business decisions. Crucially, you then need to act on those decisions. It’s this action that creates the fifth V of big data: value.

A key part of this process is making sure the right information is delivered to the right people at the right time. In order to aid decision making and ensure the necessary action is taken, insights need to be presented in a clear, concise and interesting way. People are less likely to act if they have to work hard to understand what the data is telling them.

Data and insights can also feed into the machines in your company, as well as your people. This applies to any machine or technology that’s a key part of how the business operates on a day-to-day basis, such as stock control systems or machinery on a production line. Connecting data and machines allows businesses to increase efficiency, improve product quality, cut costs and much more. Processes and systems can also be connected with data, so that you can improve how you do things based on what the data shows.

There’s more on focusing on insights and feeding data to your people, machines and systems in Chapter 7.

Building big data skills and competencies

There’s currently a skill shortage in big data, meaning there’s more demand for big data experts than there are available experts. This can make it hard for smaller businesses to recruit good data staff. But there are alternatives to hiring in-house staff. You can try training up your existing people, working with external data providers (of which there are now many, big and small) and partnering with other organisations, such as universities.

Whether you want to hire new people or boost your existing skills, I think there are six key skills required to successfully use big data in business:

• Analysing data
• Being creative
• Applying mathematics and statistics
• Understanding computer science
• Grasping the business side of things
• Communicating insights

There’s more on these skills in Chapter 8.

I think developing your existing staff is a brilliant place to start. So, as a first step, think about whether your existing people have the potential to meet some or all of these needs (with a little extra training in all likelihood). There are some brilliant big data-related courses out there, many of them online and many of them free. I list my favourites in Chapter 8.

If data is going to be a key part of your business, then it’s a good idea to consider hiring a data scientist to work in-house. The six skills I list are a good starting point when you’re searching for the right person, and I also list some helpful recruitment questions in Chapter 8. If you don’t have any tech experience at all then recruiting in the tech field can seem daunting – with these questions and by focusing on the core skills, you’ll be able to assess candidates with confidence.

In a competitive recruitment environment where demand outstrips supply, you may need to get a little creative when it comes to tapping into big data skills. It’s all about finding creative ways to pull the necessary skills together in whatever way works for you. For example, you may find someone with statistical and analytical skills who falls short on business insights, but your own people could help supplement those skills.

Getting the infrastructure in place

Your ultimate goal is to gather insights which will lead to better decision making and improved business performance. In order to do this, you’ll need to invest in some tools or services.

By infrastructure I mean the software and hardware that will enable you to turn big data into insights. There are four key elements to a big data infrastructure:

• Data collection: If you need to source new data, this may require new infrastructure investments, such as sensors, cameras, or systems to collect text or audio data. For example, if you want to collect machine data from your factory operations or vehicles, you’ll need to invest in sensors to collect the data.
• Data storage: You need to think about where to store all this data. The main storage options include: data warehouses, data lakes, distributed or cloud-based storage systems, company servers and hard disks.
• Data analysis/processing: Now you have your data, the next step is to sort and tidy up the data and analyse it to extract information.
• Data visualisation/communication: If the key insights aren’t clearly presented, they won’t result in action. For most small businesses looking to improve their decision making, simple graphics or visualisation platforms are more than enough to present insights from data.

I explore the main options for each element in Chapter 9, along with some of the most commonly used software packages.

The first step is to assess your existing infrastructure so, for each of these four elements, you need to consider what related technology or resources you already have in-house and how they might need to be improved or supplemented. For example, you may already be collecting useful customer data through your website or customer service department but don’t yet have the analytics in place to work with that data.

If you’re accessing someone else’s data (using Facebook or Twitter, for instance), then the data capture, storage and processing elements may not apply to you – or they may apply to a lesser degree (you may want to partner someone else’s data with some of your own internal data). In the last few years many businesses have sprung up offering cloud-based big data services to help other companies and organisations solve their data dilemmas. This type of big data as a service can cut costs because you pay only for the data you use.