Comment

Weapons of Math Destruction

How Big Data Increases Inequality and Threatens Democracy
May 26, 2017JCLChrisK rated this title 4 out of 5 stars
You are prey. The predator is numbers. Numbers that have been carefully designed to turn you into prey. Numbers wielded by marketers, politicians, insurance companies, and so many others. The problem with these particular numbers is that they give those using them the illusion of knowing you when all they really manage is a proxy, a mathematical approximation that may or may not be accurate. And they are built into self-feeding, self-affirming, reinforcing loops that make them ever more restrictive and controlling. They don't simply feed on us, they increasingly define us. Cathy O'Neil has been a mathematics professor and has worked in the data science industry in a variety of businesses and roles. She knows how the numbers work and has seen them in action from multiple perspectives. At the start of her conclusion in Weapons of Math Destruction, she writes: "In this march through a virtual lifetime, we’ve visited school and college, the courts and the workplace, even the voting booth. Along the way, we’ve witnessed the destruction caused by WMDs. Promising efficiency and fairness, they distort higher education, drive up debt, spur mass incarceration, pummel the poor at nearly every juncture, and undermine democracy. It might seem like the logical response is to disarm these weapons, one by one. "The problem is that they’re feeding on each other. Poor people are more likely to have bad credit and live in high-crime neighborhoods, surrounded by other poor people. Once the dark universe of WMDs digests that data, it showers them with predatory ads for subprime loans or for-profit schools. It sends more police to arrest them, and when they’re convicted it sentences them to longer terms. This data feeds into other WMDs, which score the same people as high risks or easy targets and proceed to block them from jobs, while jacking up their rates for mortgages, car loans, and every kind of insurance imaginable. This drives their credit rating down further, creating nothing less than a death spiral of modeling. Being poor in a world of WMDs is getting more and more dangerous and expensive. "The same WMDs that abuse the poor also place the comfortable classes of society in their own marketing silos. . . . The quiet and personal nature of this targeting keeps society’s winners from seeing how the very same models are destroying lives, sometimes just a few blocks away." O'Neil has crafted a broad overview that introduces the complexity of the topic with numerous examples, and through it a call to wield those tools more ethically and morally. The book is highly accessible, intelligent without being difficult and entertaining without being frivolous. This is a book that deserves high readership and a topic that needs extensive discussion. "Predictive models are, increasingly, the tools we will be relying on the run our institutions, deploy our resources, and manage our lives. But as I’ve tried to show throughout this book, these models are constructed not just from data but from the choices we make about which data to pay attention to—and which to leave out. Those choices are not just about logistics, profits, and efficiency. They are fundamentally moral."