Data · AI · Law · Ethics

The magazine for decision-makers

·

AI Agents That Write Code: Without Governance, Speed Becomes a Risk

AI agents no longer just suggest code: they open pull requests, run tests and touch infrastructure. For those leading an organization, the question is not whether to adopt them, but under what rules.

by

·

·

3–4 minutes
Illustrazione: un flusso di agenti AI tra parentesi di codice attraversa due checkpoint di controllo e termina con uno scudo

In summary

  • What is changing: AI agents are moving from suggesting code to acting: they modify repositories, launch pipelines, call tools and APIs.
  • The risk: more autonomy means a larger surface for error, and responsibility still lies with the organization, not the agent.
  • The response: lightweight but explicit governance (perimeters, human control, traceability, clear accountability) that makes it possible to accelerate without losing control.

From assistant to agent

Until recently, AI in software development was a copilot: it proposed a line, and a developer decided whether to accept it. Today’s agents work differently. They receive a goal, break it down into steps, read the code, modify it, run the tests and open a pull request. Some can even make changes to configurations, dependencies and cloud environments.

The productivity gain is real and visible. But the nature of the question changes: it is no longer “is this suggestion correct?”, but “what can this system do, with what permissions, and who is accountable for what it does?”

Why this is a board-level issue, not just a matter for the technical team

An agent with too many privileges is the equivalent of a new hire holding the keys to every office. The main risks are concrete:

  • Excessive autonomy. OWASP ranks it among the top risks of LLM-based applications (Excessive Agency): an agent that can do more than it needs to will, sooner or later, do so.
  • Software supply chain. Suggested dependencies that do not exist or are compromised, libraries with incompatible licenses, code of uncertain provenance.
  • Data and secrets. Credentials, personal data or proprietary code ending up in a model’s context or in a log.
  • Diluted accountability. If nobody knows who approved a change, nobody is accountable for it. This is an audit problem, a security problem and, increasingly, a compliance problem.

The five cardinal points of governance

No bureaucratic apparatus is needed. What is needed is a few clear rules, applied consistently.

  1. Perimeter. Least privilege, isolated environments, access only to the tools and data required for the task. No production credentials by default.
  2. Human control where it matters. The agent proposes, a person approves, especially for irreversible actions: merges into main branches, releases, changes to infrastructure and permissions.
  3. Traceability. Every action taken by the agent must be logged and attributable: which agent, on whose behalf, with what instruction, with what outcome.
  4. Same quality standards. Code written by an agent goes through the same checks as code written by a person: code review, testing, security and dependency analysis. No fast lane.
  5. Explicit accountability. Every agent has a human “owner”, and usage policies are written down, known and reviewed periodically.

The reference framework

Anyone looking for a starting point does not need to invent anything. The ISO/IEC 42001 standard defines a management system for AI, the NIST AI Risk Management Framework offers a common language for mapping and measuring risks, and the European AI Act requires organizations to think in terms of risk levels, transparency and human oversight. Development tools are not in themselves “high-risk” under the regulation, but the software they produce may end up in systems that are. Upstream governance is the cheapest way to arrive prepared downstream.

What to do on Monday morning

  • Take inventory. Which AI agents and assistants are already in use across teams, with what permissions and on which repositories? The answer is often surprising.
  • Define three levels of autonomy. What the agent can do on its own, what requires approval, what is off-limits. One page, not a manual.
  • Measure. Delivery speed, but also defects, vulnerabilities and rollbacks in code produced with agents. Without metrics, governance becomes a matter of opinion.

The compass

Governance is not the brake on agentic AI: it is what allows you to press the accelerator. The organizations that can state precisely what an agent can do, and who is accountable for it, will be the ones able to entrust it with more.

Leave a Reply

Your email address will not be published. Required fields are marked *