Netflix ran a million dollar competition for anyone who could make their Recommendation Engine ten percent better than it was. A software developer friend recently expressed with awe the high-caliber talent that competed for the Prize, from university professors to AT&T Labs engineers, close to 10,000 contestants representing thousands of teams participated from all over the world.
The function of their Recommendation Engine is to predict what movies you’ll like and then offer them to you when you’re browsing the Netflix website for something to watch. The more they recommend movies you like, the happier you’ll be with their service and continue to use them. This, in and of itself seems fairly innocuous.
Simple. Harmless. Right?
My friend followed the progression of the Prize. As he shared with adulation the performance numbers achieved by the many fine minds involved, I experienced a growing, rather gnawing feeling of unease. The Prize had been a mere million. Why did AT&T invest two of their top engineers on the competition? Why would Alberta Ingenuity, a research group funded by a billion dollar grant care about a million dollar prize? And beyond the money issue, what the hell did Craig Carmichael, a senior engineer for Lockheed Martin care about making better movie recommendations?
Then, the other day it hit me. I got that prickly sensation, like after barely missing a Mac truck. Awarded the Prize last September, AT&T didn’t really care about winning a million dollars. What the telecom giant, and so many others were, and are still trying to figure out is how humans think, and what motivates us to act.
The ramifications of Predictive Modeling—part of Machine Learning, aka AI are rather complex and somewhat profound. And as soon as I mention Artificial Intelligence everyone rolls their eyes. We’re all still waiting for that robot that does our menial tasks. But modern machine learning is about continually extrapolating information from billions of data points, and learning from behavior over time, effectively modeling the past to predict the future.
Netflix can now predict with impressive accuracy what you’ll think of any given movie or show. Improving their recommendation system has recently proven to boost client retention, serving Netflix, and their customers as well. But they’re not the only ones developing, utilizing and refining this cutting-edge technology. Google returned almost two million links to “predictive modeling,” and over 32 million for “machine learning.”
It’s the latest brass ring in the [Silicon] Valley. Analyzing as much information as possible, then identifying useful patterns amidst the massive amount of data out there, and then accurately projecting that pattern forward is the next Big Thing.
And possibly a very dangerous 'thing.'
Predictive Modeling isn’t new. In weather forecasting to financial projections, it’s used everyday. The credit default swap disaster is largely blamed on predictive models that showed the odds of default on a massive scale a mere one percent, and therefore was not taken seriously. The payoff was too good to pass up with such minor risk. The problem was the model was wrong. Even if the one percent figure was correct, the model didn’t account for people ignoring the risk. Predictive models can not yet account for abstractions, like greed and other human emotions.
With today’s compute power, modeling emotions may not be necessary for accurate predictions. Similar to the Neflix engine, many current models predict behavior of the masses by gathering and analyzing data from individuals. From our health records, to our financial histories, to our blogs, Tweets and FB posts, our lives are documented in intimate detail in databases everywhere. Location based algorithms in our cell phones now track where we go every minute of the day.
Perhaps you’ve recently changed your status on your Facebook page from married to single. Tracking your history on the site shows that months before you changed your status you explored singles groups, increased viewing profiles and ‘friending’ singles, joined a single parenting group...etc. Your pattern of behavior may seem exclusive to you, but analyzing the behavior of a million others who eventually changed their relationship status from married to single shows they did the exact same thing.
Armed with a model of how we are likely to behave gives advertisers, politicians, insurance companies, banks, brokers, anyone with an agenda the ability to target each of us individually, and a lot more effectively. Before you’ve decided to leave your spouse, ads for divorce lawyers and dating services designed to make single life attractive begin appearing on your FB page.
Your cell phone reveals you’ve stopped traveling to your office. The LinkedIn app on your Smart Phone shows daily job searches. Third day at home you get a call from your credit card company, a recording saying your card has been canceled. Their model determined the unemployed in your field are at high risk of defaulting on payments, effectively destroying your credit rating, along with 10,000 others that Wednesday, causing five percent of you to miss your mortgage payments and lose your homes.
On the gross or individual scale, in all probability machine learning will become more precise, and the utilization of accurate projections will get quite egregious—which is my predictive model based on personal observation and knowledge of our history. Quantifying, correlating and analyzing the data I’ve collect over the years, it appears we have a proclivity to exploit rather than elevate.
One last word of caution...
Societal shifts, in buying habits, attitude, belief or political systems begin with individuals and radiate outward, aka ‘Go Viral,’ the other big buzz in the Valley right now.