“If it’s not ready, just hold it back.” That’s Jensen Huang’s answer to the AI safety question, and I keep turning it over because it’s simultaneously the most sensible thing anyone in this industry has said and the least enforceable.
Nvidia’s CEO made the case this month that the AI industry doesn’t need new security laws. Market forces, he argues, will sort out safe development on their own. Companies that ship broken products get punished. Companies that ship good ones win. Developers should keep half-finished work in the garage until it actually works.
As someone who spends most of my week testing AI agents and deciding which ones are worth recommending to people, I have a complicated reaction to this.
The advice is genuinely correct
Let me start with where Huang is right, because he is right about something important.
The single most common failure I see in agent tooling isn’t malice or misalignment. It’s premature shipping. A team builds an agent that works beautifully in a demo, gets excited, publishes it, and then real users hand it real tasks with real edge cases. The agent confidently does the wrong thing. It deletes the wrong file, emails the wrong contact, books the wrong flight, or hallucinates a data source that never existed.
None of that requires a regulatory framework to diagnose. It requires someone on the team saying “this isn’t finished” and being listened to. Huang’s framing puts the responsibility exactly where it functionally sits: with the people writing the code and choosing the release date.
If you build agents, take the advice at face value. Hold it back. The reputational cost of an agent that quietly does damage is far higher than the cost of shipping three weeks late.
Where the market-forces argument gets thin
Here’s my hesitation. Market discipline works when buyers can tell the difference between a good product and a bad one. In the agent space right now, they frequently can’t.
An agent’s failure modes are often invisible until they compound. A summarization agent that subtly misrepresents source material doesn’t throw an error. A research agent that skips a step doesn’t announce it. A customer service agent that gives inconsistent answers to different users looks fine to any individual user. The feedback loop that’s supposed to punish bad products is slow, noisy, and easy to game with a good landing page.
Compare that to a graphics card. If a GPU is defective, you find out fast and loudly. Nvidia operates in a market where quality signals are unusually clean. Reasoning from that experience to software that produces plausible-sounding text is a bit of a leap.
I’m not making an argument for any particular regulation here. I genuinely don’t know what good AI law looks like, and I’m suspicious of anyone who claims they do. But “the market will handle it” is a claim about information, not just incentives, and the information in this market is bad.
Why Huang’s position matters more than most
Huang has also been pushing a broader idea: that society needs new social norms for the age of AI. That’s a more interesting thread than the regulation debate, and I wish it got more airtime.
Norms are how we handle things that are too fast-moving or too context-dependent for law. They’re also how the agent space will probably sort itself out in practice. Things like: agents should disclose when they’re agents. Agents should log what they did. Agents that take actions on your behalf should ask before doing anything irreversible. None of that is legislated. All of it is becoming table stakes among teams that take the work seriously.
Huang’s other point deserves attention too. He’s described AI as essential infrastructure rather than a single breakthrough, and he expects to sell roughly twice as many chips next year. That growth curve is the actual context for the safety conversation. The volume of AI capability entering the world is increasing faster than anyone’s ability to evaluate it, including mine.
What I’m taking from this
Practically, my checklist for evaluating agent tools hasn’t changed, but Huang gave me a cleaner way to phrase the top item: does this feel held back or shoved out?
You can usually tell. Held-back tools have narrow scopes, honest documentation about what they can’t do, and confirmation steps before consequential actions. Shoved-out tools promise everything, document nothing specific, and treat autonomy as a feature rather than a liability to be managed.
The person with the most influence over whether your agent is safe is you, not a regulator. Huang and I agree on that. Where we differ is whether that’s reassuring.
🕒 Published: