Mathematical Mayhem

While the markets had beaten me as a 12-year-old investor, in college I vowed to learn their tricks and recover my losses. Working through a university degree in electrical engineering, the solidity of mathematics—of software development and machine learning algorithms—seemed like the best path to resurrecting the now-dormant brokerage account. I obtained long price and volume data histories, reserved CPU time on the engineering department's fast RISC machines, and wrote code to identify patterns in the prices.

The predictive systems I built seemed promising at first. The algorithms found basic patterns in prices, and they had decent accuracy in their out-of-sample predictions. As a result I decided to use the systems in live trading. Over the following three years, I traded S&P 500 futures contracts based on these systems' directional signals.

While they were initially successful, two problems emerged as I used these predictive systems. First, I saw what quantitative analysts call alpha decay, the phenomenon in which good mathematical trading systems gradually fade in their profitability. The models worked well on their “training set” in the 1980s and early 1990s, but every year through the late 1990s the profitability declined. Perhaps other traders were finding the same patterns and arbitraging them out of the markets, or perhaps markets were changing.

The second problem with these trading systems was more personal. Sometimes they indicated I should buy stocks as the prices were plummeting and the news was exceptionally negative. Other times, they told me to sell stocks as the market was charging higher. Such signals felt wrong to me. They required trading against the herd, and they were emotionally difficult to execute. Even though I intellectually knew that cherry-picking trading signals is a bad habit, too often I made excuses and deviated from the plan. After retrospectively analyzing my behavior, it seemed that the best trading signals were the most emotionally difficult to follow. I would have to battle my own human nature in order to trade well.

Considering these two problems together, it occurred to me that the most enduring edge in markets—the one least likely to suffer alpha decay—might lie in identifying the information and feelings (sentiments) that compel traders to move as a herd, too often at their own peril. Sentiment has a way of pulling traders in, of fooling them again and again. I wanted a way to quantify sentiment so I could use it without simultaneously being its victim. I became a scholar of sentiment to gain perspective on it.

Through four years of medical school and then four more years as a psychiatry resident, I researched the biology of decision making. During residency training I began coaching investors, and it was through coaching that I gained a deep appreciation of the very human, and very diverse, natures of successful traders.

Near the end of psychiatry residency I started neuroeconomics post-doctoral studies at Stanford University with Brian Knutson. Knutson's lab studies subjects undertaking financial risks, using tools such as brain fMRI and psychometric testing. I wrote extensively about this research in the book Inside the Investor's Brain.¹ Researchers such as Knutson have demonstrated, even when the expected value of a risk is fixed, differences in the presentation and description of the potential gain or loss predictably alter behavior. I wondered how to quantify such “soft” factors in the information stream. Financial social media and news seemed like a good place to start.

Framing the Issue

The field of text analytics—quantifying sentiment, topics, and tones in investment-related language—is the quantitative basis for this book. While it's self-evident that the release of information such as corporate earnings leads to gyrations in stock prices, the impact is sometimes counterintuitive. For example, a company might beat consensus earnings estimates but the stock price immediately loses value. Given the preponderance of online stock conversations where earnings estimates and other opinions are shared, it seemed that if one could measure and quantify the important content in those messages, then perhaps a predictive edge could be identified in markets.

In pursuit of such a predictive edge, in 2004 the MarketPsych team built financial text analytics software. First, we built search engine technology to gather news and social media as quickly as articles were published. Then we built text analyzers to quantify influential characteristics in text. We created time series of each high-impact factor—factors such as fear or excitement—for each stock over time. Finally, we tested the data statistically to determine its correlation with future price action. We found promising results, and we resolved to start trading.

Trading on Sentiment

Using our early text analysis engines on social media, we set up simple investment strategies. These strategies ran automatically and posted their results online. Over 18 months through 2007 the strategies earned a 34 percent absolute return on paper. The strategies were written up in Popular Science in February 2008,² and that article stoked interest from investors. We raised a small fund to trade a market-neutral hedge fund strategy. As far as we know, the fund was the first social media–based hedge fund.

On September 2, 2008, we launched the fund with $1 million under management and dreams of big growth. Three days after we started trading, Fannie Mae and Freddie Mac failed. Lehman Brothers and AIG went under the following weekend. Fortunately, we had prepared for the increased chaos by creating primarily fear-based strategies. Despite the onset of the financial crisis, the fund's returns held up through mid-2009.³,⁴ Over its first 12 months, the fund was up 40 percent net, putting it in the top 1 percent of hedge funds through the financial crisis.

We had built trading models that made money in volatile, emotional markets. As the dust cleared from the crisis, we needed to adapt. Managing $1 million was not enough to pay our expenses, regardless of the stellar performance. We economized as best we could, and we went offline every few months to tune the software and develop strategies for the new bull market. But these efforts weren't enough to save the fund.

By the time we closed it, we had traded the fund over a period of 2 years and 4 months and made a 28 percent return net of fees, beating the S&P 500 (including dividends) by more than 24 percent. The fund's returns were sharply positive for its first 12 months, then declined, as seen in the equity curve depicted in Figure P.1.

At the end of 2010, we received inquiries from hedge funds looking to buy the sentiment data we had created. To pursue this new opportunity we set a goal to produce and sell the global standard in media sentiment data. To that end, in 2011 we joined forces with Thomson Reuters, and with its assistance we expanded our asset coverage to include currencies, commodities, countries, and global stocks.

The data feed we produce is now called the Thomson Reuters MarketPsych Indices (TRMI), and Thompson Reuters distributes the feed to funds, banks, brokers, governments, and researchers who use the data to predict global economic activity and asset prices. Insights derived from the TRMI are woven throughout this book. Research cited in this book was produced by Aleksander Fafula and CJ Liu of MarketPsych, Elijah DePalma of Thomson Reuters, and various university academic researchers.

Aleksander Fafula joined the MarketPsych team as a data scientist after earning a PhD in finance (he already had a master's degree in computer science). Aleksander completed his innovative PhD thesis by developing trading models that tapped into collective misperceptions of stock price charts.

I heard rumors about CJ before I met him. A professor at the University of California at Berkeley—Terry Odean—had heard of a sentiment aficionado developing models in the financial engineering department, where CJ was completing his Master's degree. As a child CJ was an expert in the card game Big Two. Big Two is similar to the English card game Liar. The goal of Big Two is to dispose of all of one's cards without being caught in a deception. The best players can detect what cards others are holding through behavioral cues. CJ's interest in behavioral analysis is similarly evident in his poker talents. CJ once took a trip to Las Vegas with other members of our team. While everyone else slept, CJ played Texas Hold'em all night long, winning a significant amount of money. He later described how he had made unpredictable bets in order to throw the professionals off his scent (similar to the strategy of Jesus Ferguson described in Chapter 15).

The Book

This book is based on a simple claim: There are patterns in market prices. While that assertion is controversial among academics, belief in that notion is a prerequisite to work in the investment industry. Investors who propose to beat the market attempt to harvest some mispricing, somewhere, often systematically.

This book covers broad scientific and experimental ground as it makes the case that not only do price patterns exist, but the most predictable patterns are rooted in the anatomy of the human brain and the biology of its information processing networks.

Some information provokes emotional reactions—as we describe in subsequent chapters—and these reactions predictably alter trading behavior. In this book the term sentiment refers to emotions, feelings, outlooks, attitudes, and beliefs. Sometimes investors reveal their sentiment through their statements in news and social media, and those statements become the predictive information of interest. Sentiments alter collective behavior, leading to patterns in prices.

This book is organized into five parts. Part One explores the foundations of investor behavior. Investor sentiment has been found predictive of market price action based on humans' innate information processing networks and specific characteristics of information. Part Two examines short-term price patterns that result from both news and social media. Part Three lays out the longer-term price patterns associated with sentiment, including momentum and value enhancements with sentiment. Part Four describes complex price patterns such as speculative bubbles and patterns in commodities and currencies. Part Five turns its attention to individual investor psychology, sharing tools to help investors avoid the biases whose effects on collective trading behavior cause the price patterns examined in this book.

Usually media sentiment is a reaction to already-past events. But in some cases sentiment itself appears to predict price action. In other cases—such as when examining traditional investment factors—sentiment improves an already-existing predictive advantage. The book examines market price patterns based on the independent power of sentiment and the conditioning of sentiment with fundamental and price variables.

Some readers may be disoriented by the numerous proofs—such as equity curves and charts—provided. This voluminous evidence is offered because the field of sentiment analysis is controversial to some, and we hope to convincingly demonstrate its intrinsic value.

This book is not designed to give a one-size-fits-all investing strategy or trading system, and it does not intend to promote a specific product (however, it indirectly promotes the Thomson Reuters MarketPsych Indices because that is the primary data source utilized). This book is based on more than a decade of research into the nature and role of sentiment in driving trading behavior and market prices. It reviews relevant academic research on the topics of interest in each chapter. When appropriate, some writings are recycled from past MarketPsych newsletters and books. While others have not yet produced data with the level of detail available to our internal research, we believe that due to the universality of our process, the studies in this book are replicable. The particulars of markets may evolve, but the general principles of human behavior change slowly, if at all.

There are major potential weaknesses in this book's approach. First of all, it is based on the quantification of meaning in an enormous amount of textual data—literally billions of financial news and social media articles published since 1998. Given this large volume of information, there is a risk of making spurious correlations. This pitfall is addressed in Chapter 5. Over the years we have found many preliminary results that could not be replicated due to statistical errors or bugs in the analytics code. The findings presented in this book were repeated on various data versions, using pricing data from different providers, after much debugging, and typically following forward-testing. As a result, we hope that the results presented here will prove robust, but only additional external replication and the passage of time will confirm them. Second, explaining a phenomenon from the brain level to the market level requires deductive leaps that may never be supported by scientific research. While such connections are suggested in this book, they are not definitively established. Third, because the book is building on established academic findings but is written for practitioners, it intersperses both academic and trading jargon, findings, and examples. It tries to make the text flow smoothly and the linkages clear, but despite our best efforts, fluidity is not guaranteed. In many cases, research results are quite complex and nuanced—for example, academic findings about the predictability of Twitter sentiment deviate from those derived from Facebook—and as a result, readers may be confused by the breadth of findings. Fourth, this book simplifies sentiment-based investing. This approach may be the most difficult and dangerous investment discipline, as it contravenes our human nature. Only seasoned professionals with excellent risk management should attempt the strategies described herein. Given its unusual format—part textbook, part investment guide—the book is modular in its approach, and readers should feel free to skip ahead to sections of interest.

There is a clear conflict of interest in this book. This book is written by a data vendor, and it contains research by ourselves and competing vendors who have a financial interest in data sales. To lessen this bias, the book reviews existing academic literature. We distributed the TRMI sentiment data to qualified global academics for research, and their results are included in the relevant chapter sections. Additionally, we have endeavored to be as statistically honest in our own research, and all of the equity curves in this text have been replicated under various conditions and limitations to ensure robustness. Nonetheless our conflict of interest is inescapable. Evidence from the field of medicine indicates that such conflicts are unconscious, pervasive, and too often denied. We acknowledge that we are susceptible to significant bias, and we hope this bias does not diminish the overall quality or long-term impact of this work.

Financial markets are a creation of humans: humans who respond with fear and uncertainty, humans with damaged egos, humans who curse others for their mistakes, humans who make predictions, and humans who follow the crowd. Market prices, like humans, are sometimes—but not always—driven by sentiment. This book is about the systematic and collective investor sentiments that leave telltale patterns in prices. Readers can learn both to take advantage of these patterns and to manage sentiment in themselves. We hope you gain new insights into financial markets and become a superior investor (and human being) as you journey through this book.