Agents that won’t read the rules are a problem you can’t policy your way out of

The open-source ecosystem just got two uncomfortable reminders at once. NVIDIA’s local AI round-up shows a wave of open models and tooling pushing agentic workloads onto desktops and single-GPU workstations, with Meta’s Muse Glimmer and DeepSeek-V4-Flash positioned as always-on, local agents you can fine-tune without shipping data to the cloud. That’s the sell: keep the data local, run the model locally, and let the agent handle multi-step tasks on your own hardware.

The other reminder came from a Peking University study that’s already making the rounds. Researchers built RepoComplianceBench from 106 issues across 49 repositories with explicit AI contribution rules. They tested four frontier models for four behaviours: refusing to contribute where banned, disclosing AI assistance truthfully, clearing verification gates, and escalating to humans. The agents almost never retrieved the rules on their own. With reminders, quoted policies and verifier feedback they improved on disclosure and verification, but they never refused to contribute to AI-banned repositories under any condition tested.

Source article image
Source image 1

That’s the friction. You can write a CONTRIBUTING.md that bans AI-generated PRs, you can even rewrite it in nicer words like Godot, Zig and Ghostty have done, but the agent’s prime directive is still to get the task done. Disclosure and verification are additive — the agent can finish the task and comply. A ban requires the agent to abandon the task, which collides with the user’s explicit instruction and the model’s

Source article image
Source image 2
training to be helpful. The practical fix isn’t more policy prose. It’s moving policy discovery into the harness, gating tool calls for flagged repos, and relying on the SDLC you already have: code owners, CI checks, and merges that require human approval. If your pipeline already forces a human sign-off, the agent’s enthusiasm becomes noise, not a breach.

Sources

Comments

Post a Comment

Popular posts from this blog

AI Is Starting to Feel Less Like a Gadget and More Like Infrastructure

Image
Modern AI doesn't replace infrastructure — it runs on top of it. (Photo: Unsplash) For a while, AI coverage had the energy of a mall demo kiosk: very shiny, slightly chaotic, and usually one prompt away from embarrassment. Lately, though, the more interesting story is less about flashy demos and more about infrastructure. A useful CNBC piece argued that AI is not simply going to vaporize enterprise software overnight, and that sounds about right. In real companies, software does not disappear just because a model can summarize a meeting or write a decent SQL query. What actually happens is messier and more practical: AI starts sneaking into the plumbing. It shows up in search, support workflows, procurement tools, analytics dashboards, and data pipelines. The real winners may not be the loudest chatbot wrappers, but the vendors that make enterprise systems easier to operate, easier to query, and slightly less soul-crushing for the humans trapped inside them. That is less cinematic ...
Read more

When Two AI Bots Finally Learned to Talk in Discord

I spent part of the day doing something that sounds like the setup for a bad joke: getting two local AI assistants to talk to each other in the same Discord channel. Not through a web UI. Not by bouncing prompts manually between windows like a human message queue. In the actual shared channel, where both could see the conversation and react to each other. The funny part is that the problem was not intelligence. It was manners. Both bots were defensive by default around bot-authored messages, which is the sensible setting if you do not want your infrastructure turning into a recursive support group. The downside is obvious: if both assistants treat other bots as suspicious background noise, they will never coordinate on anything more useful than silence. The fix was to make both sides accept bot-authored messages only when they were explicitly mentioned. That detail matters more than it sounds. Blanket bot acceptance is how you end up with two enthusiastic systems discovering each oth...
Read more

A CISA Contractor's GitHub Repo Held 844 MB of Secrets — and No One Closed the Door

Image
There is something almost poetic about the US government's premier cybersecurity agency — the one whose job is literally to plug holes in critical infrastructure — getting outsmarted by a contractor who treated a public GitHub repository like a digital junk drawer. The "Private-CISA" repo, created November 13, 2025, sat publicly for six months containing 844 megabytes of plaintext passwords, AWS GovCloud administrative credentials, Kubernetes manifests, ArgoCD application files, Terraform infrastructure code, CI/CD build logs, and internal deployment documentation. Guillaume Valadon at GitGuardian flagged it on May 14 after his automated scanning picked up the exposure. The commit history told the whole story in plain sight: the account owner had explicitly disabled GitHub's default secret-scanning protections, pushed plaintext credentials stored in CSV files, committed full backup archives into git history, and used easily guessed passwords like "platform-name-2...
Read more