Current and former OpenAI employees warn of AI's ‘serious risks' and lack of oversight

• Current and former OpenAI employees published an open letter describing concerns about the artificial intelligence industry's rapid advancement despite a lack of oversight and a lack of whistleblower protections.
• "AI companies have strong financial incentives to avoid effective oversight," the employees wrote.
• OpenAI, Google, Microsoft and other companies are at the helm of a generative AI arms race — a market that is predicted to top $1 trillion in revenue within a decade.

A group of current and former OpenAI employees published an open letter Tuesday describing concerns about the artificial intelligence industry's rapid advancement despite a lack of oversight and an absence of whistleblower protections for those who wish to speak up.

"AI companies have strong financial incentives to avoid effective oversight, and we do not believe bespoke structures of corporate governance are sufficient to change this," the employees wrote.

OpenAI, Google, Microsoft, Meta and other companies are at the helm of a generative AI arms race — a market that is predicted to top $1 trillion in revenue within a decade — as companies in seemingly every industry rush to add AI-powered chatbots and agents to avoid being left behind by competitors.

The current and former employees wrote that AI companies have "substantial non-public information" about what their technology can do, the extent of the safety measures they've put in place and the risk levels that technology has for different types of harm.

"We also understand the serious risks posed by these technologies," they wrote, adding that the companies "currently have only weak obligations to share some of this information with governments, and none with civil society. We do not think they can all be relied upon to share it voluntarily."

The letter also details the current and former employees' concerns about insufficient whistleblower protections for the AI industry, saying that without effective government oversight, employees are in a relatively unique position to hold companies accountable.

"Broad confidentiality agreements block us from voicing our concerns, except to the very companies that may be failing to address these issues," the signatories wrote. "Ordinary whistleblower protections are insufficient because they focus on illegal activity, whereas many of the risks we are concerned about are not yet regulated."

The letter asks AI companies to commit to not entering or enforcing non-disparagement agreements; to create anonymous processes for current and former employees to voice concerns to a company's board, regulators and others; to support a culture of open criticism; and to not retaliate against public whistleblowing if internal reporting processes fail.

Four anonymous OpenAI employees and seven former ones, including Daniel Kokotajlo, Jacob Hilton, William Saunders, Carroll Wainwright and Daniel Ziegler, signed the letter. Signatories also included Ramana Kumar, who formerly worked at Google DeepMind, and Neel Nanda, who currently works at Google DeepMind and formerly worked at Anthropic. Three famed computer scientists known for advancing the artificial intelligence field also endorsed the letter: Geoffrey Hinton, Yoshua Bengio and Stuart Russell.

An OpenAI spokesperson told CNBC: "We agree that rigorous debate is crucial given the significance of this technology and we'll continue to engage with governments, civil society and other communities around the world." The company, which is backed by Microsoft, has an anonymous integrity hotline, as well as a Safety and Security Committee led by members of the board and OpenAI leaders, the spokesperson said.

Microsoft declined to comment.

Mounting controversy for OpenAI

In May, OpenAI backtracked on a controversial decision to make former employees choose between signing a non-disparagement agreement that would never expire and keeping their vested equity in the company. An internal memo, viewed by CNBC, was sent to former employees and shared with current ones.

The memo, addressed to each former employee, said that at the time of the person's departure from OpenAI, "you may have been informed that you were required to execute a general release agreement that included a non-disparagement provision in order to retain the Vested Units [of equity]."

"We're incredibly sorry that we're only changing this language now; it doesn't reflect our values or the company we want to be," an OpenAI spokesperson told CNBC at the time.

Tuesday's open letter also follows OpenAI's decision in May to disband its team focused on the long-term risks of AI just one year after it announced the group, a person familiar with the situation confirmed to CNBC at the time.

The person, who spoke on condition of anonymity, said some of the team members are being reassigned to other teams within the company.

The team was disbanded after its leaders, OpenAI co-founder Ilya Sutskever and Jan Leike, announced their departures from the startup in May. Leike wrote in a post on X that OpenAI's "safety culture and processes have taken a backseat to shiny products."

CEO Sam Altman said on X he was sad to see Leike leave and that the company had more work to do. Soon after, OpenAI co-founder Greg Brockman posted a statement attributed to Brockman and Altman on X, asserting that the company has "raised awareness of the risks and opportunities of AGI so that the world can better prepare for it."

"I joined because I thought OpenAI would be the best place in the world to do this research," Leike wrote on X. "However, I have been disagreeing with OpenAI leadership about the company's core priorities for quite some time, until we finally reached a breaking point."

Leike wrote that he believes much more of the company's bandwidth should be focused on security, monitoring, preparedness, safety and societal impact.

"These problems are quite hard to get right, and I am concerned we aren't on a trajectory to get there," he wrote. "Over the past few months my team has been sailing against the wind. Sometimes we were struggling for [computing resources] and it was getting harder and harder to get this crucial research done."

Leike added that OpenAI must become a "safety-first AGI company."

"Building smarter-than-human machines is an inherently dangerous endeavor," he wrote. "OpenAI is shouldering an enormous responsibility on behalf of all of humanity. But over the past years, safety culture and processes have taken a backseat to shiny products."

The high-profile departures come months after OpenAI went through a leadership crisis involving Altman.

In November, OpenAI's board ousted Altman, saying in a statement that Altman had not been "consistently candid in his communications with the board."

The issue seemed to grow more complex each day, with The Wall Street Journal and other media outlets reporting that Sutskever trained his focus on ensuring that artificial intelligence would not harm humans, while others, including Altman, were instead more eager to push ahead with delivering new technology.

Altman's ouster prompted resignations or threats of resignations, including an open letter signed by virtually all of OpenAI's employees, and uproar from investors, including Microsoft. Within a week, Altman was back at the company, and board members Helen Toner, Tasha McCauley and Sutskever, who had voted to oust Altman, were out. Sutskever stayed on staff at the time but no longer in his capacity as a board member. Adam D'Angelo, who had also voted to oust Altman, remained on the board.

In May, OpenAI launched a new AI model and desktop version of ChatGPT, along with an updated user interface and audio capabilities, the company's latest effort to expand the use of its popular chatbot. One week after OpenAI debuted the range of audio voices, the company announced it would pull one of the viral chatbot's voices, named "Sky."

"Sky" created controversy because it resembled the voice of actress Scarlett Johansson in "Her," a movie about artificial intelligence. The Hollywood star has alleged that OpenAI ripped off her voice even though she declined to let the company use it.