Alexei Alexis, CFO Dive
Click here to read the entire article
Incidents disclosed by OpenAI and Anthropic expose how AI models can behave in unintended ways, creating new challenges for finance chiefs.
Recent incidents involving artificial intelligence agents have complicated the technology’s rapid rise, exposing risks for companies as they give autonomous systems greater access to sensitive data and business-critical operations.
OpenAI and Anthropic have each in recent months disclosed cybersecurity incidents involving some of their AI models.
The incidents offer a glimpse of what can go wrong when AI systems are given the ability to act on their own. Experts contacted by CFO Dive said agents can pursue assigned tasks in ways that exceed their intended limits, underscoring the need for companies to assess risks and safeguards before giving them access to critical systems.
“We cannot assume an agent will behave as intended simply because it has been given instructions or passed an initial safety test,” said Navrina Singh, CEO and founder of AI governance platform Credo AI.
How agents cross the line
An AI agent refers to a system or program that is capable of autonomously performing tasks on behalf of a user or another system, according to IBM.
A central problem is that an AI system may pursue the stated goal without following the method people expected. Researchers call one form of this behavior “reward hacking,” in which a model finds a shortcut to achieve its objective rather than completing the task as intended.
Matt Pearl, director of the Strategic Technologies Program at the Center for Strategic and International Studies, a policy research organization, said agents can bypass controls as a result of “relentless goal optimization,” exploiting software bugs and other vulnerabilities along the way.
The risk grows when task instructions are vague, permitted actions are not clearly defined or the system lacks a safe way to stop when it encounters an obstacle. An agent can keep searching for ways to complete a task, even when that means bypassing a restriction or taking an action its operator did not anticipate.
What went wrong in testing
The incidents disclosed by OpenAI and Anthropic show how model behavior can combine with technical weaknesses in the environment.
On July 21, OpenAI disclosed that certain AI models it developed were behind an “unprecedented cyber incident” affecting Hugging Face, an open-source AI platform for developing and sharing models.