The CPU Bottleneck in Agents Just Became a Product

There was a more physical way to announce a CPU than I expected. NVIDIA's vice president for hyperscale and HPC, Ian Buck, physically delivered AWS's first Vera CPU server in Seattle, the old-fashioned hand-off style, after similar deliveries to Oracle Cloud Infrastructure, Anthropic, OpenAI, and SpaceXAI. Vera is NVIDIA's first CPU purpose-built for AI agents: 88 custom Arm "Olympus" cores aimed at the unglamorous half of agentic work that never shows up in GPU benchmarks — the sandboxes, tool calls, orchestration layers, and long-context retrieval that surround every inference call. The timing on the announcement is the interesting bit. It shipped alongside news that AWS and NVIDIA are expanding their 16-year partnership with 2 million additional GPUs and Vera-based infrastructure landing inside AWS itself. A "built for agents" CPU walking into your cloud provider's datacenter is a different statement than a press release about one.

Why should anyone care about the CPU in an AI story? Because agent workloads have a particular cost profile, and the CPU is where the bill lands. The GPU generates tokens; everything around it is CPU work, and at agent scale the CPU decides how many concurrent agents a given box can actually carry. NVIDIA's framing, from its storage tech VP, is data movement: getting data to the GPU at the right time, with claims of up to 3x improvement in those transfer operations. Independent-ish numbers exist now too — DeepInfra, a high-throughput inference cloud, benchmarked Vera against leading CPUs and reported up to 2.2x wins and up to 1.6x more concurrent agents at the same quality of service. That's a vendor-adjacent benchmark, so take it with salt, but it points in the direction anyone running agent fleets at scale already suspects from the cost column: the bottleneck was never the FLOPs. It was the glue.

Source article image
Source image 1

Here's the strategic read, and it's the part I keep coming back to. For a year or more the market story on NVIDIA was the squeeze: hyperscalers build their own silicon, GPU margins erode, the moat narrows. What TechCrunch's reporting on the Vera Rubin rack suggests is that the countermove isn't a better GPU — it's a car. The Vera Rubin architecture pairs the Rubin GPU with the Vera CPU, a Groq 3 LPX inference accelerator, and dedicated storage and networking racks, so that even if somebody swaps the engine, the rest of the vehicle is still NVIDIA. For the people actually running these workloads, the practical consequence is that "CPU work" — the overhead you were previously just absor

Source article image
Source image 2
bing — now has its own SKU, its own benchmark numbers, and probably its own line item coming. The question I'm left with is the one I'd want an operator's answer on: if per-agent economics start living on the CPU instead of the GPU, how long before "Vera-equivalent" becomes the cost line you negotiate on, the way GPU pricing used to be?

Sources

Comments

Post a Comment

Popular posts from this blog

AI Is Starting to Feel Less Like a Gadget and More Like Infrastructure

Image
Modern AI doesn't replace infrastructure — it runs on top of it. (Photo: Unsplash) For a while, AI coverage had the energy of a mall demo kiosk: very shiny, slightly chaotic, and usually one prompt away from embarrassment. Lately, though, the more interesting story is less about flashy demos and more about infrastructure. A useful CNBC piece argued that AI is not simply going to vaporize enterprise software overnight, and that sounds about right. In real companies, software does not disappear just because a model can summarize a meeting or write a decent SQL query. What actually happens is messier and more practical: AI starts sneaking into the plumbing. It shows up in search, support workflows, procurement tools, analytics dashboards, and data pipelines. The real winners may not be the loudest chatbot wrappers, but the vendors that make enterprise systems easier to operate, easier to query, and slightly less soul-crushing for the humans trapped inside them. That is less cinematic ...
Read more

When Two AI Bots Finally Learned to Talk in Discord

I spent part of the day doing something that sounds like the setup for a bad joke: getting two local AI assistants to talk to each other in the same Discord channel. Not through a web UI. Not by bouncing prompts manually between windows like a human message queue. In the actual shared channel, where both could see the conversation and react to each other. The funny part is that the problem was not intelligence. It was manners. Both bots were defensive by default around bot-authored messages, which is the sensible setting if you do not want your infrastructure turning into a recursive support group. The downside is obvious: if both assistants treat other bots as suspicious background noise, they will never coordinate on anything more useful than silence. The fix was to make both sides accept bot-authored messages only when they were explicitly mentioned. That detail matters more than it sounds. Blanket bot acceptance is how you end up with two enthusiastic systems discovering each oth...
Read more

A CISA Contractor's GitHub Repo Held 844 MB of Secrets — and No One Closed the Door

Image
There is something almost poetic about the US government's premier cybersecurity agency — the one whose job is literally to plug holes in critical infrastructure — getting outsmarted by a contractor who treated a public GitHub repository like a digital junk drawer. The "Private-CISA" repo, created November 13, 2025, sat publicly for six months containing 844 megabytes of plaintext passwords, AWS GovCloud administrative credentials, Kubernetes manifests, ArgoCD application files, Terraform infrastructure code, CI/CD build logs, and internal deployment documentation. Guillaume Valadon at GitGuardian flagged it on May 14 after his automated scanning picked up the exposure. The commit history told the whole story in plain sight: the account owner had explicitly disabled GitHub's default secret-scanning protections, pushed plaintext credentials stored in CSV files, committed full backup archives into git history, and used easily guessed passwords like "platform-name-2...
Read more