AI Ethics & Meta-Rationality

FEBRUARY 26, 2020

Ethical AI has become something of a buzzword of late; or, perhaps more cynically, a marketing term designed to ward off potential critics. But beneath the hype and misdirection lie genuine philosophical problems -- problems of justice, fairness, and value -- that will need at least provisional solution as the technology progresses. As a recent paper from DeepMind argues, these problems straddle both technical and normative domains: the kind of AI we build impacts the kinds of values we are able to pursue. The work of determining or negotiating these values, therefore, lies partly in the hands of the few technologists at the vanguard of AI research. Responsibly steering AI will require a clear understanding of the ideas, personality quirks, and incentives driving those at the helm.

A new article from the MIT Technology Review scrutinizes OpenAI through this lens. The article recounts how competitive pressures and safety concerns have lead the company to compromise on its nonprofit status and founding premise of transparency. But it also describes the near-religious culture that binds the company together. The company's charter is invoked "like scripture, an explanation for every aspect of the company's existence." The CEO, Greg Brockman, was married "in the office against a backdrop of flowers arranged in an OpenAI logo. [Chief Scientist Ilya] Sustkever acted as the officiant; a robot hand was the ring bearer." These behaviors aren't religious in the sense of being supernatural or irrational; rather, they suggest what the writer and computer scientist David Chapman calls eternalism: the tendency to interpret everything as having a fixed, definite meaning. Rationalist eternalism, on Chapman's account, is the tendency to ascribe such meanings by reference to one or another rational system, in much the same way that a religious person would by reference to a God. 

Chapman remarks that the OpenAI article gave him "flashbacks" to his time as a researcher in MIT's AI lab in 1988, because the "personalities and ways of thinking" were so familiar. Among these is a kind of loosely held rationalist eternalism, evinced by the way OpenAI researchers use AI metaphors (“What is your life a function of?” “What are you optimizing for?”) to describe their lives. To some degree, this is inevitable: the models AI researchers use to understand general intelligence stem from, and reinforce, a particular view of human nature. But when considering what goals to pursue with AI, narrow rationalism may be insufficient. As the cognitive scientist Keith Stanovich argues, most conceptions of rationality take goals or desires as given. In order to consider what goals are worth pursuing in the first place, he argues, "we will need a form of meta-rationality...Rationality will have to critique itself."

Chapman, too, has written extensively about meta-rationality. He defines it as the capacity to "us[e] rational systems effectively without taking them as fixed and certain. [Meta-rationality] evaluates, selects, combines, modifies, discovers, and creates rational methods." By recognizing the scope and limits of particular rational systems, the meta-rational decision-maker is able to avoid the pitfalls of narrow rationalism without rejecting rationality altogether. This kind of thinking is vital for AI alignment, because it allows us to consider the relationship between different AI approaches and different goals we might like to pursue. As Chapman writes, while rationality aspires to purpose-independence, meta-rationality "evaluates and coordinates purposes." 

The reciprocal relationship between AI models and views of human nature may lead AI practitioners to myopically adopt particular rational paradigms. For example, as Iason Gabriel writes in the DeepMind paper, the reinforcement learning paradigm of optimization "has a certain moral valence," because "it will be easier to align AI with moral theories that have the same fundamental structure of maximizing reward over time...than with other alternatives. Consequentialist moral theories, the most famous of which is act utilitarianism, fit the bill." In this way, the AI paradigm of utility maximization, the view of humans as rational utility maximizers, and the moral theory of utilitarianism mutually reinforce one another. Questions about what conception of "the good" is worth maximizing, in what context, may get ignored, even as they are necessarily being answered more or less well by any AI application.

Critiques of this approach from the standpoint of justice or rights usefully point out that narrow utility maximization based on past data (past conceptions of "the good") can perpetuate "entrenched discrimination." In the case of algorithms that predict recidivism rates, for instance, assuming the future will resemble the past can perpetuate disparities. But such critiques tend not to acknowledge that removing biases from these algorithms also compromises their accuracy with respect to predicting recidivism. It thus replaces one narrow purpose (avoiding disparities) for another (predicting recidivism). This may be appropriate, but it is not strictly more rational. Negotiating these questions depends on the capacity to "evaluate and coordinate purposes" without taking any as final. 

Meta-rationality has been little explored in the cognitive science literature; Chapman and Stanovich are among the few to discuss it under that name. But in a Letter exchange with the cognitive scientist John Vervaeke, Chapman points out that meta-rationality is synonymous with Vervaeke's conception of wisdom: "a process whereby rationality transcends itself" to aid "the enhancement of cognition (broadly construed)." While "meta-rationality" may sound obscure, few would dispute that we are in desperate need of wisdom. Indeed, according to the physicist Max Tegmark, the very fate of humanity depends on the outcome of the "wisdom race" between "the growing power of our technology" and "the wisdom with which we manage it." If, as it often seems, wisdom is losing that race, then meta-rationality may be a crucial part of closing the gap.