ABOUT THE BOOK

Dozens of times per day, we all interact with intelligent machines that are constantly learning from the wealth of data now available to them. These machines – from smartphones and talking robots to self-driving cars – are remaking the world of the twenty-first century in the same way that the Industrial Revolution remade the world of the nineteenth.

AIQ is based on a simple premise: if you want to understand the modern world and where it’s heading, you have to know a little bit of the mathematical language spoken by intelligent machines. AIQ will teach you that language, but in an unconventional way: through stories rather than equations.

You will meet a fascinating cast of historical characters – from Isaac Newton to Florence Nightingale – who have a lot to teach you about data, probability and better thinking. Along the way you will see how these same ideas are playing out in the modern age of big data and intelligent machines. And rather than ushering in the dystopian future so familiar from science fiction, you will see how these technologies can help us overcome some of our built-in cognitive weaknesses, and give us all the chance to lead happier, healthier and more fulfilled lives.

INDEX

The page references in this index correspond to the printed edition from which this ebook was created. To find a specific word or phrase from the index, please use the search feature of your ebook reader.

Page numbers in italics refer to figures.

ACKNOWLEDGMENTS

Together we want to thank the two people most responsible for nurturing this book from its earliest stages: Lisa Gallagher of DeFiore & Company and Tim Bartlett of St. Martin’s Press. This is the first thing either of us have written that wasn’t for an academic audience, and we started with almost no sense of what writing and publishing a “real” book would actually entail. We are so grateful to Lisa for seeing and cultivating the potential in those first scribbled drafts, which now look so painfully clunky. We are equally grateful to Tim, both for taking a gamble on two data scientists foolhardy enough to try their hands at writing, and for giving us such unerring advice along the way. We are also indebted to Doug Young of Transworld for his valuable editorial feedback.

We also thank the many other people at DeFiore, St. Martin’s Press, and Macmillan who have been so helpful, including Robert Allen, Alan Bradshaw, Jeff Capshew, Laura Clark, Jennifer Enderlin, Tracey Guest, Leah Johanson, Linda Kaplan, Alice Pfeifer, Gabrielle Piraino, Jason Prince, Sally Richardson, Brisa Robinson, Mary Beth Roche, Robert Van Kolken, Laura Wilson, and George Witte. We give special thanks to India Cooper, whose magnificent editing efforts have put into stark relief the difference between a professional writer and two amateurs like us. Thanks also to Larry Finlay, Bill Scott-Kerr, and the rest of the Transworld team for their support.

We thank Ellen Zippi for her invaluable help in researching this book. We are also grateful to many of our colleagues for sharing stories and expertise, most especially Steven Levitt for introducing us to Lisa Gallagher, and David Madigan for drawing our attention to the two studies on bisphosphonate usage described in chapter 7. We thank Rosalind Eggo, Katherine Heller, and Mark Sendak for their time and trouble in agreeing to be interviewed. Thanks also to those family members who tirelessly read early drafts and gave their feedback: Catherine Aiken, Patricia and Josh Lowry, Anne and George Scott, and Brian Woods.

PERSONAL ACKNOWLEDGMENTS

I am grateful to my co-author, James Scott. Above all, I thank my family for their love and support: my wife, Anne Gron, and our children, Emma, Michael, and Sarah.

—Nick Polson

Thank you to Nick Polson. I owe Nick for so much in my career that I cannot possibly list it all here; this book is but the latest in a long string of projects and ideas that he has so generously shared with me. I expect that over the decades to come, I will look back and see Nick as the single most important influence on my professional life, and the best friend I ever had in this field. I also want to thank the three most important teachers I ever had: Bill Jeffreys, Jim Berger, and John Trimble. Without Bill and Jim, I would never have become a statistician. Without John’s kindness and generosity, I would never have known how to “tighten/sharpen/brighten” my way to better prose. I also thank my parents, who gave me so much—not least of which was their example. Finally, I am grateful to my wife, Abigail Aiken, for just about everything. I love you, and I could not have helped write this book without your support.

—James Scott

ABOUT THE AUTHORS

Nick Polson is Professor of Econometrics and Statistics at the Chicago Booth School of Business. He is a Bayesian statistician involved in research in machine intelligence, deep learning and computational methods. He regularly speaks to large audiences in the US, UK and the rest of Europe.

James Scott is Associate Professor of Statistics at the University of Texas. James is a statistician and data scientist who studies Bayesian inference and computational methods for big data. James lives in Austin, Texas with his wife, Abigail.

TRANSWORLD PUBLISHERS

61–63 Uxbridge Road, London W5 5SA

www.penguin.co.uk

Transworld is part of the Penguin Random House group of companies whose addresses can be found at global.penguinrandomhouse.com

First published in Great Britain in 2018 by Bantam Press

an imprint of Transworld Publishers

Copyright © Nicholas Polson and James Scott 2018

Cover design by David Baldeosingh Rotstein

Nicholas Polson and James Scott has asserted their right under the Copyright, Designs and Patents Act 1988 to be identified as the authors of this work.

Every effort has been made to obtain the necessary permissions with reference to copyright material, both illustrative and quoted. We apologize for any omissions in this respect and will be pleased to make the appropriate acknowledgements in any future edition.

A CIP catalogue record for this book is available from the British Library

Version 1.0 Epub ISBN 9781473554368

ISBN 9780593079775

This ebook is copyright material and must not be copied, reproduced, transferred, distributed, leased, licensed or publicly performed or used in any way except as specifically permitted in writing by the publishers, as allowed under the terms and conditions under which it was purchased or as strictly permitted by applicable copyright law. Any unauthorized distribution or use of this text may be a direct infringement of the author’s and publisher’s rights and those responsible may be liable in law accordingly.

1 3 5 7 9 10 8 6 4 2

CONTENTS

INTRODUCTION

WE TEACH DATA science to hundreds of students per year, and they’re all fascinated by artificial intelligence. And they ask great questions. How does a car learn to drive itself? How does Alexa understand what I’m saying? How does Spotify pick such good playlists for me? How does Facebook recognize my friends in the photos I upload? These students realize that AI isn’t some sci-fi droid from the future; it’s right here, right now, and it’s changing the world one smartphone at a time. They all want to understand it, and they all want to be a part of it.

And our students aren’t the only ones enthusiastic about AI. They’re joined in their exaltation by the world’s largest companies—from Amazon, Facebook, and Google in America to Baidu, Tencent, and Alibaba in China. As you may have heard, these big tech firms are waging an expensive global arms race for AI talent, which they judge to be essential to their future. For years we’ve seen them court freshly minted PhDs with offers of $300,000+ salaries and much better coffee than we have in academia. Now we’re seeing many more companies jump into the AI recruiting fray—firms sitting on piles of data in, say, insurance or the oil business, who are coming along with whopping salary offers and fancy espresso machines of their own.

Yet while this arms race is real, we think there’s a much more powerful trend at work in AI today—a trend of diffusion and dissemination, rather than concentration. Yes, every big tech company is trying to hoard math and coding talent. But at the same time, the underlying technologies and ideas behind AI are spreading with extraordinary speed: to smaller companies, to other parts of the economy, to hobbyists and coders and scientists and researchers everywhere in the world. That democratizing trend, more than anything else, is what has our students today so excited, as they contemplate a vast range of problems practically begging for good AI solutions.

Who would have thought, for example, that a bunch of undergraduates would get so excited about the mathematics of cucumbers? Well, they did when they heard about Makoto Koike, a car engineer from Japan whose parents own a cucumber farm. Cucumbers in Japan come in a dizzying variety of sizes, shapes, colors, and degrees of prickliness—and based on these visual features, they must be separated into nine different classes that command different market prices. Koike’s mother used to spend eight hours per day sorting cucumbers by hand. But then Koike realized that he could use a piece of open-source AI software from Google, called TensorFlow, to accomplish the same task, by coding up a “deep-learning” algorithm that could classify a cucumber based on a photograph. Koike had never used AI or TensorFlow before, but with all the free resources out there, he didn’t find it hard to teach himself how. When a video of his AI-powered sorting machine hit YouTube, Koike became an international deep-learning/cucumber celebrity. It wasn’t merely that he had given people a feel-good story, saving his mother from hours of drudgery. He’d also sent an inspiring message to students and coders across the world: that if AI can solve problems in cucumber farming, it can solve problems just about anywhere.

That message is now spreading quickly. Doctors are now using AI to diagnose and treat cancer. Electrical companies use AI to improve power-generating efficiency. Investors use it to manage financial risk. Oil companies use it to improve safety on deep-sea rigs. Law enforcement agencies use it to hunt terrorists. Scientists use it to make new discoveries in astronomy and physics and neuroscience. Companies, researchers, and hobbyists everywhere are using AI in thousands of different ways, whether to sniff for gas leaks, mine iron, predict disease outbreaks, save honeybees from extinction, or quantify gender bias in Hollywood films. And this is just the beginning.

We see the real story of AI as the story of this diffusion: from a handful of core math concepts stretching back decades, or even centuries, to the supercomputers and talking/thinking/cucumber-sorting machines of today, to the new and ubiquitous digital wonders of tomorrow. Our goal in this book is to tell you that story. It is partly a story of technology, but it is mainly a story of ideas, and of the people behind those ideas—people from a much earlier age, people who were just keeping their heads down and solving their own problems involving math and data, and who had no clue about the role their solutions would come to play in inventing the modern world. By the end of that story, you’ll understand what AI is, where it came from, how it works, and why it matters in your life.

What Does “AI” Really Mean?

When you hear “AI,” don’t think of a droid. Think of an algorithm.

An algorithm is a set of step-by-step instructions so explicit that even something as literal-minded as a computer can follow them. (You may have heard the joke about the robot who got stuck in the shower forever because of the algorithm on the shampoo bottle: “Lather. Rinse. Repeat.”) On its own, an algorithm is no smarter than a power drill; it just does one thing very well, like sorting a list of numbers or searching the web for pictures of cute animals. But if you chain lots of algorithms together in a clever way, you can produce AI: a domain-specific illusion of intelligent behavior. For example, take a digital assistant like Google Home, to which you might pose a question like “Where can I find the best breakfast tacos in Austin?” This query sets off a chain reaction of algorithms:

And that’s AI. Pretty much every AI system—whether it’s a self-driving car, an automatic cucumber sorter, or a piece of software that monitors your credit card account for fraud—follows this same “pipeline-of-algorithms” template. The pipeline takes in data from some specific domain, performs a chain of calculations, and outputs a prediction or a decision.

There are two distinguishing features of the algorithms used in AI. First, these algorithms typically deal with probabilities rather than certainties. An algorithm in AI, for example, won’t say outright that some credit card transaction is fraudulent. Instead, it will say that the probability of fraud is 92%—or whatever it thinks, given the data. Second, there’s the question of how these algorithms “know” what instructions to follow. In traditional algorithms, like the kind that run websites or word processors, those instructions are fixed ahead of time by a programmer. In AI, however, those instructions are learned by the algorithm itself, directly from “training data.” Nobody tells an AI algorithm how to classify credit card transactions as fraudulent or not. Instead, the algorithm sees lots of examples from each category (fraudulent, not fraudulent), and it finds the patterns that distinguish one from the other. In AI, the role of the programmer isn’t to tell the algorithm what to do. It’s to tell the algorithm how to train itself what to do, using data and the rules of probability.

How Did We Get Here?

Modern AI systems, like a self-driving car or a home digital assistant, are pretty new on the scene. But you might be surprised to learn that most of the big ideas in AI are actually old—in many cases, centuries old—and that our ancestors have been using them to solve problems for generations. For example, take self-driving cars. Google debuted its first such car in 2009. But you’ll learn in chapter 3 that one of the main ideas behind how these cars work was discovered by a Presbyterian minister in the 1750s—and that this idea was used by a team of mathematicians over 50 years ago to solve one of the Cold War’s biggest blockbuster mysteries.

Or take image classification, like the software that automatically tags your friends in Facebook photos. Algorithms for image processing have gotten radically better over the last five years. But in chapter 2, you’ll learn that the key ideas here date to 1805—and that these ideas were used a century ago, by a little-known astronomer named Henrietta Leavitt, to help answer one of the deepest scientific questions that humans have ever posed: How big is the universe?

Or even take speech recognition, one of the great AI triumphs of recent years. Digital assistants like Alexa and Google Home are remarkably fluent with language, and they’ll only get better. But the first person to get a computer to understand English was a rear admiral in the U.S. Navy, and she did so almost 70 years ago. (See chapter 4.)

Those are just three illustrations of a striking fact: no matter where you look in AI, you’ll find an idea that people have been kicking around for a long time. So in many ways, the big historical puzzle isn’t why AI is happening now, but why it didn’t happen long ago. To explain this puzzle, we must look to three enabling technological forces that have brought these venerable ideas into a new age.

The first AI enabler is the decades-long exponential growth in the speed of computers, usually known as Moore’s law. It’s hard to convey intuitively just how fast computers have gotten. The cliché used to be that the Apollo astronauts landed on the moon with less computing power than a pocket calculator. But this no longer resonates, because … what’s a pocket calculator? So we’ll try a car analogy instead. In 1951, one of the fastest computers was the UNIVAC, which performed 2,000 calculations per second, while one of the fastest cars was the Alfa Romeo 6C, which traveled 110 miles per hour. Both cars and computers have improved since 1951—but if cars had improved at the same rate as computers, then a modern Alfa Romeo would travel at 8 million times the speed of light.

The second AI enabler is the new Moore’s law: the explosive growth in the amount of data available, as all of humanity’s information has become digitized. The Library of Congress consumes 10 terabytes of storage, but in 2013 alone, the big four tech firms—Google, Apple, Facebook, and Amazon—collected about 120,000 times as much data as this. And that’s a lifetime ago in internet years. The pace of data accumulation is accelerating faster than an Apollo rocket; in 2017, more than 300 hours of video were uploaded to YouTube every minute, and more than 100 million images were posted to Instagram every day. More data means smarter algorithms.

The third AI enabler is cloud computing. This trend is nearly invisible to consumers, but it’s had an enormous democratizing effect on AI. To illustrate this, we’ll draw an analogy here between data and oil. Imagine if all companies of the early twentieth century had owned some oil, but they had to build the infrastructure to extract, transport, and refine that oil on their own. Any company with a new idea for making good use of its oil would have faced enormous fixed costs just to get started; as a result, most of the oil would have sat in the ground. Well, the same logic holds for data, the oil of the twenty-first century. Most hobbyists or small companies would face prohibitive costs if they had to buy all the gear and expertise needed to build an AI system from their data. But the cloud-computing resources provided by outfits like Microsoft Azure, IBM, and Amazon Web Services have turned that fixed cost into a variable cost, radically changing the economic calculus for large-scale data storage and analysis. Today, anyone who wants to make use of their “oil” can now do so cheaply, by renting someone else’s infrastructure.

When you put those four trends together—faster chips, massive data sets, cloud computing, and above all good ideas—you get a supernova-like explosion in both the demand and capacity for using AI to solve real problems.

AI Anxieties

We’ve told you how excited our students are about AI, and how the world’s largest firms are rushing to embrace it. But we’d be lying if we said that everyone was so bullish about these new technologies. In fact, many people are anxious, whether about jobs, data privacy, wealth concentration, or Russians with fake-news Twitter-bots. Some people—most famously Elon Musk, the tech entrepreneur behind Tesla and SpaceX—paint an even scarier picture: one where robots become self-aware, decide they don’t like being ruled by people, and start ruling us with a silicon fist.

Let’s talk about Musk’s worry first; his views have gotten a lot of attention, presumably because people take notice when a member of the billionaire disrupter class talks about artificial intelligence. Musk has claimed that in developing AI technology, humanity is “summoning a demon,” and that smart machines are “our biggest existential threat” as a species.

After you’ve read our book, you’ll be able to decide for yourself whether you think these worries are credible. We want to warn you up front, however, that it’s very easy to fall into a trap that cognitive scientists call the “availability heuristic”: the mental shortcut in which people evaluate the plausibility of a claim by relying on whatever immediate examples happen to pop into their minds. In the case of AI, those examples are mostly from science fiction, and they’re mostly evil—from the Terminator to the Borg to HAL 9000. We think that these sci-fi examples have a strong anchoring effect that makes many people view the “evil AI” narrative less skeptically than they should. After all, just because we can dream it and make a film about it doesn’t mean we can build it. Nobody today has any idea how to create a robot with general intelligence, in the manner of a human or a Terminator. Maybe your remote descendants will figure it out; maybe they’ll even program their creation to terrorize the remote descendants of Elon Musk. But that will be their choice and their problem, because no option on the table today even remotely foreordains such a possibility. Now, and for the foreseeable future, “smart” machines are smart only in their specific domains:

• Alexa can read you a recipe for spaghetti Bolognese, but she can’t chop the onions, and she certainly can’t turn on you with a kitchen knife.
• An autonomous car can drive you to the soccer field, but it can’t even referee the match, much less decide on its own to tie you to the goalposts and kick the ball at your sensitive bits.

Moreover, consider the opportunity cost of worrying that we’ll soon be conquered by self-aware robots. To focus on this possibility now is analogous to the de Havilland Aircraft Company, having flown the first commercial jetliner in 1952, worrying about the implications of warp-speed travel to distant galaxies. Maybe one day, but right now there are far more important things to worry about—like, to press the jetliner analogy a little further, setting smart policy for all those planes in the air today.

This issue of policy brings us to a whole other set of anxieties about AI, much more plausible and immediate. Will AI create a jobless world? Will machines make important decisions about your life, with zero accountability? Will the people who own the smartest robots end up owning the future?

These questions are deeply important, and we hear them discussed all the time—at tech conferences, in the pages of the world’s major newspapers, and over lunch among our colleagues. We should let you know up front that you won’t find the answers to these questions in our book, because we don’t know them. Like our students, we are ultimately optimistic about the future of AI, and we hope that by the end of the book, you will share that optimism. But we’re not labor economists, policy experts, or soothsayers. We’re data scientists—and we’re also academics, meaning that our instinct is to stay firmly in our lane, where we’re confident of our expertise. We can teach you about AI, but we can’t tell you for sure what the future will bring.

We can tell you, however, that we’ve encountered a common set of narratives that people use to frame this subject, and we find them all incomplete. These narratives emphasize the wealth and power of the big tech firms, but they overlook the incredible democratization and diffusion of AI that’s already happening. They highlight the dangers of machines making important decisions using biased data, but they fail to acknowledge the biases or outright malice in human decision-making that we’ve been living with forever. Above all, they focus intensely on what machines may take away, but they lose sight of what we’ll get in return: different and better jobs, new conveniences, freedom from drudgery, safer workplaces, better health care, fewer language barriers, new tools for learning and decision-making that will help us all be smarter, better people.

Take the issue of jobs. In America, jobless claims kept hitting new lows from 2010 through 2017, even as AI and automation gained steam as economic forces. The pace of robotic automation has been even more relentless in China, yet wages there have been soaring for years. That doesn’t mean AI hasn’t threatened individual people’s jobs. It has, and it will continue to do so, just as the power loom threatened the jobs of weavers, and just as the car threatened the jobs of buggy whips. New technologies always change the mix of labor needed in the economy, putting downward pressure on wages in some areas and upward pressure in others. AI will be no different, and we strongly support job-training and social-welfare programs to provide meaningful help for those displaced by technology. A universal basic income might even be the answer here, as many Silicon Valley bosses seem to think; we don’t claim to know. But arguments that AI will create a jobless future are, so far, completely unsupported by actual evidence.

Then there’s the issue of market dominance. Amazon, Google, Facebook, and Apple are enormous companies with tremendous power. It is critical