Artificial Intelligence researchers think we may create a “new species” of superintelligent machines, but caution against the US allowing China to gain an upper hand.
Two longtime researchers — one widely considered an AI “optimist” and the other an AI “doomer” — agreed to debate, mediated by The Post, where they discussed not trusting the China, how to avoid disaster and how “self-regulation is a joke.”
“I was wrong,” University of Washington professor and AI entrepreneur Oren Etzioni, told The Post. “I still retain some optimism. But I now see the things I thought were many, many years away [are] coming soon, at a pace that we may not be able to anticipate.”
Tegmark, founder of the Future of Life Institute, is surprised authorities haven’t stepped in, adding: “I never imagined that the people in America with the most power would let companies get so close to things that one could lose control over without intervening.
“Companies are just openly saying, well, we’re going to build super intelligence. [It] might be a new species.”
His concern is when we reach “recursive self-improvement,” or the point at which AI can start improving itself and reproducing: “The turning point is… when it becomes better than humans at AI research.”
However, both agreed the simplest way of keeping AI from going over the edge is by keeping it siloed, with different machines taking on different tasks.
Etzioni has long been an advocate of AI models with specific, narrow, domain intelligence, rather all-encompassing super-agents.
“I think what we need to do… is to design systems carefully and smartly so that we box them in,” he said.
Tegmark agreed: “By default, machines should be given the least amount of access and power that they need to do their job.
“You don’t give every janitor in the Pentagon the nuclear codes… A self-driving car… doesn’t actually know anything about bioweapons because it doesn’t need to, and it does not have the nuclear codes, and most importantly does not have superhuman persuasion abilities.”
The Trump administration has remained steadfast in its commitment against stepping in to curb AI, noting the top companies should stop themselves from creating tech they can’t control. However, Trump announced on Sept. 19 he would appoint a new AI adviser and create an “AI Force.”
AI companies such as Anthropic and OpenAI have been vocal in calling for more regulation over the last month, which smaller companies in the field say would likely stifle their progress.
Anthropic CEO Dario Amodei also put forward a plan for the top companies to pick who regulates them, which both Tegmark and Etzioni found ridiculous.
“Self-regulation is a joke,” Etzioni said. “We’ve seen this with the social media companies and what they’ve done. This is the big tobacco moment for high tech, [and] we don’t want to spend another 20 years to find out that AI companies didn’t exercise the appropriate level of self-discipline.”
Tegmark and Etzioni — who co-founded a company acquired by Anthropic earlier this year — are both in favor of a regulatory board, similar to the FDA, to ensure AI developers are making technologies that are helpful, rather than harmful, to humanity.
“Start treating AI companies like other companies where they just have to meet some basic standards before they can get rich off of them,” Tegmark said. “That will channel all the upside — the good things about capitalism — to make the companies themselves put a lot of effort into making the product safer.”
Tegmark said, after years of becoming more and more pessimistic about humanity’s future, he’s feeling “much more optimistic” now that concern about AI is going mainstream.
Earlier this month, he convened a Pro-Human Assembly in DC, where everyone from Richard Blumenthal to Bernie Sanders and Steve Bannon convened in agreement that we need to regulate AI.
“I was on the stage, and I got to introduce Bernie Sanders and Steve Bannon back to back, which had not been on my bingo card,” he said.
But Etzioni jumped in: “You’re right that it’s amazing that these two folks have come together. But, at the same time, can you imagine two worse demagogues?… I am terrified of exactly this thing that Max is talking about: is [AI regulation] becoming a political hobby horse, which politicians will cynically manipulate to their own ends?”
This was their ironic disagreement: even though they agree no regulating AI, a once-pessimistic Tegmark is optimistic about the path we’re headed down, while a once-optimistic Etzioni is more cautious.
Etzioni warned that, if we get too caught up in the populist backlash, we might jump to unhelpful, short-sighted solutions which ultimately backfire.
“I am terrified of what the democratic process does, particularly when it sets its mind to regulate something that we don’t understand, particularly when it does so out of fear,” Etzioni said, emphasizing that he believes democracy is the best political system, albeit an imperfect one.
“We need to have good solutions, not popular ones. I worry more about human stupidity… than about artificial intelligence.”
The biggest concern, Etzioni said, is that an American rush to regulate AI could empower China to surpass our capabilities.
“I don’t trust the Chinese at all,” Etzioni said. “Nothing would make them happier than us hamstringing ourselves while [they take] the front seat… We need to be mindful of the adversarial race with China, not delude ourselves that our interests are aligned.”
But Tegmark disagreed, and expressed his belief that China must be reaching the same conclusion that super-intelligent AI is against the interest of all nations, regardless of who develops them.
“We don’t have to trust China,” he said. “We don’t have to convince the Chinese government to prevent their own companies from doing something super crazy, causing control loss. They will want to do it anyway.”
For Tegmark, this is a problem that demands humanity-level solutions. But Etzioni has less faith that mankind can transcend our populist tendencies and international conflicts to solve this looming problem.
It was a fascinating inversion — the optimist being the pessimist, and the pessimist being the optimist — but that ultimately proves that AI has developed at a truly disorienting speed for everyone, including the experts.
