Building the Brakes Before the Engine: Inside AI's New Safety Watchdog

When a smartphone app crashes or behaves unexpectedly, developers simply push an update. But how do you issue a patch for an artificial superintelligence that might be capable of rewriting its own code? For a growing number of experts, the "patch it later" approach to AI safety is beginning to look like a dangerous gamble.

This exact concern has prompted researchers from the UK AI Security Institute and the alignment theory startup Timaeus to join forces and launch Sequent, a new nonprofit research organization. Their foundational premise is stark: when it comes to preparing for Artificial Superintelligence (ASI), current alignment efforts are simply "not on track."

Today, the AI industry largely relies on a reactive approach to safety. When a language model produces biased outputs or hallucinates, engineers tweak the system to fix those specific edge cases. While functional for current consumer AI, Sequent argues that this empirical, trial-and-error method fails to provide the fundamental, a priori confidence we will need when dealing with ASI. The stakes get astronomically higher if AI systems achieve "recursive self-improvement"—the ability to autonomously upgrade their own architectures. At that point, humanity cannot afford to wait for a failure to occur before fixing it.

To tackle this, Sequent is building a different kind of safety laboratory. With an initial fundraising goal of $100 million to $150 million and plans to scale to 40-80 full-time employees, the organization is placing a portfolio of "under-resourced bets." Instead of just stress-testing models, they are diving deep into learning theory, game theory, and scalable oversight. Their ultimate goal is to find principled, theoretical reasons to trust that an AI acting safely in a controlled training environment will continue to act safely when deployed into large-scale, long-horizon, real-world scenarios that humans cannot easily monitor.

Beyond technical research, Sequent is positioning itself as a vital independent watchdog. Because they are structurally separated from the massive commercial pressures of frontier AI labs, they retain the freedom to speak out. As the founders explicitly noted, if the race toward superintelligence starts cutting dangerous corners, "we might need to yell."

As billions of dollars pour into making AI faster and smarter, Sequent represents a crucial counterweight. It is an ambitious attempt to ensure that before we ignite the engine of superintelligence, we have mathematically sound brakes ready to deploy.

Key Points

• Sequent is a new nonprofit founded by top researchers to tackle the safety of Artificial Superintelligence (ASI).
• Founders argue that the 'reactive' safety measures used by major AI labs will be insufficient for self-improving AI.
• The organization seeks to raise up to $150M to explore parallel research paths like game theory and scalable oversight.
• Sequent aims to find principled, theoretical proofs that AI will remain safe in uncontrolled, real-world environments.
• The group intends to act as an independent watchdog, ready to call out dangerous practices in the broader AI industry.

Why It Matters

As AI systems approach the threshold of recursive self-improvement, empirical trial-and-error safety methods become obsolete. Developing theoretically sound alignment techniques now is essential to maintaining control over future superintelligent systems.

---

Sources:

• Import AI 461: "Alignment is not on track"; FrontierCode; and synthetic research interns — Import AI (Jack Clark)