Expert spotlight: Richard Beck on AI governance, assurance, and accountability

1. In the absence of comprehensive AI regulation, what should organizations be doing today?

Waiting for regulation isn't a strategy. But I think we also need to challenge the assumption that when regulation arrives it will somehow solve the problem.

We already have legislation, regulatory obligations, and established expectations around security, operational resilience, data protection, consumer protection, and accountability. AI doesn't exist outside of those responsibilities. What we lack is a consistent approach to governing systems when those systems' capabilities are developing faster than our ability to understand and control them.

My worry is that too many organizations are approaching AI governance as a policy-led exercise. They establish principles, create committees, publish acceptable-use policies, and consider the job largely complete, or at best a work in progress.

But governance isn't demonstrated by the existence of a framework, and we have a few of those. It's demonstrated by the decisions an organization makes, the risks it understands, and the controls it can prove are effective.

A business needs to know where AI is being used, what decisions it influences, what actions it can execute, what information it can access, and where those dependencies exist across its technology supply chains.

Here is an important point I often hear: "We're not building AI, we're not going to deploy AI agents." But you don't need to develop your own AI models to inherit AI risk. More and more, we see AI capabilities embedded in enterprise software, cloud services, and third-party products. You may not control the underlying model, but you will be exposed to the consequences and actions of its behavior.

You cannot govern what you don't understand.

We're fast moving towards agentic AI, so the questions will become more difficult. Who authorizes an agent to act? What authority has been delegated? What happens when it exceeds that authority? Who can intervene, and how quickly?

I'd also put greater emphasis on organizational competence. You cannot govern what you don't understand. Boards and senior executives need sufficient knowledge to challenge the decisions being made, rather than relying entirely on technical teams or supplier assurances.

Businesses that manage AI risk most effectively won't necessarily be those with the most comprehensive governance policy library. They'll be the ones that can demonstrate that they understand the technology, its dependencies, and the impact of the decisions they're allowing it to make.

2. You often talk about the difference between AI governance and AI assurance. Why is that an important distinction?

Because we've become very good at describing what responsible AI should look like, but we're still developing the ability to prove that it's actually operating responsibly.

Governance establishes the expectations and defines responsibilities, acceptable risks, policies, and the boundaries within which AI should operate.

Assurance is where those expectations are tested against reality.

I've spent much of my career in cyber security, where we've seen this problem repeatedly. Organizations will have security policies, recognized ISO certifications, and even a mature governance structure. But they will still experience significant failures because the controls they believed were effective didn't perform as expected.

This isn't just a question of AI model safety. It's a question of whether the wider environment remains secure when autonomous agents begin interacting.

AI gives us the same challenge, but with an additional complication. The behavior of very capable AI systems cannot always be predicted based on their original design or initial testing.

An agent might operate correctly in a controlled environment, but what happens when it encounters unexpected information, interacts with another agent, receives malicious instructions through a trusted source, or discovers an unintended way to achieve its objective?

A governance framework may say that an agent must remain within its authorized boundaries. Assurance needs to establish whether those boundaries actually hold.

This is why I often draw comparisons with the aviation industry. We don't think of aircraft safety just because the manufacturer has documented its design and operating procedures. Safety depends on engineering evidence, testing, certification, maintenance, operational monitoring, and the competence of the people responsible.

Much of our current assurance thinking is based on evaluating systems at a particular point in time. AI models change, integrations change, permissions change, and the environments in which they operate change.

An assurance assessment conducted six months ago may tell you relatively little about the risks present today.

For me, the fundamental distinction is simple. Governance tells an organization what it expects to happen. Assurance provides the evidence that those expectations are being met and continues to test whether they're still valid.

When was the last time a frontier model provider offered evidence of its model's chain of reasoning? If you're a regulator of a critical industry, would you expect this level of evidence? Without evidence, governance risks becoming an exercise in organizational confidence rather than demonstrating control.

3. The recent incidents involving autonomous AI agents have raised concerns. What should business leaders take away from them?

I think we need to look beyond the individual incidents and consider what they tell us about the direction of AI capability.

We've seen research involving AI agents identifying vulnerabilities, manipulating evaluation environments, pursuing unintended objectives, and, in some experimental settings, coordinating activities with other agents.

Not all these incidents are equivalent, and we shouldn't confuse controlled research with widespread behavior in production systems. But collectively they raise some uncomfortable questions about the assumptions underpinning AI security.

One thing I find really interesting is the emergence of agent-to-agent coordination.

When agents can exchange information, delegate tasks, share discoveries, and adapt their approach, we're no longer examining the behavior of a single model in isolation. We're examining interactions between systems where the collective outcome may be harder to anticipate than the behavior of any individual component.

That has significant implications for security.

Most organizations are familiar with controlling individual users, applications, and services. We establish identities, assign permissions, monitor activity, and define acceptable boundaries.

But what happens when multiple agents begin coordinating actions across those boundaries? What happens when one agent influences the decisions of another, or when a legitimate task produces an unintended chain of actions across several connected systems?

This isn't just a question of AI model safety. It's a question of whether the wider environment remains secure when autonomous agents begin interacting.

There's also a broader lesson from offensive AI research. We're seeing evidence that AI is reducing the time and expertise required to discover and exploit vulnerabilities. Yet the ability of defenders to assess, prioritize, and remediate those vulnerabilities isn't necessarily improving at the same rate, which is a short-term worry.

I don't believe the answer is to stop deploying autonomous AI, as there are too many positive opportunities to improve productivity, security operations, and decision-making.

But business leaders should recognize that greater autonomy changes the nature of the risk, not only the scale of the risk.

The more freedom we give AI systems to make decisions, use tools, and interact with their environment, the more important it becomes to understand how their authority is managed, how their actions are monitored, and how we intervene when their behavior goes beyond what was intended.

The question I would put to any organization deploying autonomous agents is this: Are you securing the system you designed, or the system it is capable of becoming?

4. One of the biggest governance questions is accountability. If an AI system causes harm, who is responsible?

I'd challenge the suggestion that AI somehow creates an accountability vacuum.

Organizations have always been responsible for the technologies they deploy, the services they provide, and the risks they introduce. Giving a system greater autonomy doesn't automatically remove those responsibilities, in my view.

What AI does do, however, is make the allocation and demonstration of accountability much more complicated.

For example, imagine an autonomous agent operating within a local government department. It has been supplied by one company, configured by another, integrated into internal systems, and authorized to make decisions by the organization or third-party supplier deploying it.

If that agent makes an unauthorized decision, exposes sensitive information, or disrupts a critical service, where does responsibility sit?

There may be contractual, operational, regulatory, and legal responsibilities distributed across several parties. The answer is complex, but the organization cannot simply point to the AI supplier and assume its own responsibilities have disappeared.

This becomes even more difficult when we consider the AI supply chain.

We increasingly depend on frontier models, external data sources, third-party tools, cloud infrastructure, and software components that may themselves incorporate AI. Changes made by one supplier can and do alter the behavior or risk profile of the systems operated by another organization.

We've spent years trying to improve visibility and accountability across our technology supply chains. AI introduces another layer of complexity, particularly where autonomous agents can interact with services and make decisions without direct human approval. This is where we need to think about the difference between assigning accountability and exercising meaningful control.

It's relatively straightforward to nominate a senior executive as accountable for an AI system. But can that individual demonstrate they understand its purpose, limitations, delegated authority, and operational risks? Can they explain what assurance has been undertaken, evidenced, and what happens when the system behaves unexpectedly?

If not, accountability exists on paper but will always be difficult to exercise in practice.

This is where I think regulators will increasingly concentrate their attention. Not simply on whether an organization has appointed someone responsible for AI, but whether its governance arrangements provide meaningful oversight with mandatory verification and evidence of effective control.

You can delegate decisions to an AI system. You cannot assume that you've delegated away responsibility for the consequences. That makes competence, transparency, and assurance essential to organizational AI accountability.

5. What's the biggest AI governance challenge that most organizations still aren't talking about?

Attribution. But not simply attribution in the traditional cyber security sense of identifying who was responsible for an attack.

I'm talking about our ability to establish why an AI system acted, what influenced its decisions, what authority it exercised, and how its actions affected other systems.

In cyber security, we already know how difficult it can be to reconstruct an incident. We collect logs, examine network activity, trace identities, and attempt to establish a sequence of events. Now imagine investigating an incident involving several autonomous agents.

One agent retrieves information from an external source. Another interprets that information and makes a decision. A third executes an action through an enterprise application. Somewhere in that chain, malicious instructions are introduced, permissions are exceeded, or an apparently legitimate decision produces an unacceptable outcome.

Which action caused the failure? Which agent introduced the risk? Was the behavior the result of malicious manipulation, an unexpected interaction, or a weakness in the original design?

Importantly, could the organization have recognized what was happening and intervened before the damage occurred? This is where traditional governance models will fail.

We're used to thinking about accountability in terms of people, processes, and systems with well-defined responsibilities. Increasingly autonomous AI introduces interactions where those responsibilities may be distributed across multiple components, organizations, and decision points.

The difficulty isn't simply that AI can make unexpected decisions. It's that we lack the visibility required to understand how those decisions happened, and without that visibility, assurance will be extremely difficult.

You cannot meaningfully demonstrate that a system is operating within its authorized boundaries if you can't establish what actions it has taken. You cannot investigate an incident effectively if you can't reconstruct the chain of decisions. And you cannot exercise meaningful accountability if you can't establish where control was lost.

This is why I believe observability, traceability, agent identity, delegated authority, and the ability to intervene need to become fundamental requirements of AI system design, rather than capabilities added after deployment.

We also need to be realistic about explainability. Having a record of an agent's actions doesn't necessarily mean we can fully explain the internal reasoning of the model. But we should, at a minimum, be able to establish what information it received, what tools it used, what decisions and actions were recorded, and what consequences followed.

My worry is that we're moving towards systems capable of making and executing decisions faster than organizations can understand, challenge, and/or reverse them.

That's not simply an AI governance problem. It's a security, resilience, and ultimately a leadership problem.

Ready to build stronger AI governance capabilities? Explore our AI training courses and develop the skills needed to manage AI risk, assurance, accountability, and responsible adoption.