What Comes After Transformers
Mapping the next era of AI — notes from a briefing built around a discussion with Yoav Shoham.
Deep Learning Is Necessary, But Insufficient
The present is autocomplete on steroids — brilliant at tokens, blind to semantics.
The Present
Language models manipulate tokens flawlessly but lack an underlying anchor in semantic representation — we've reached the frontier of pure probability.
The Missing Layer
Robust intelligence needs a foundational semantic-reasoning layer: harmonizing continuous deep learning with discrete, formal logic (predicates, axioms, ontologies) rather than pattern-matching alone.
The Symptom of Missing Semantics Is Jagged Intelligence
Current frontier models don't fail gracefully — they drop instantly from genius-level synthesis to logical nonsense.
| The Diagnostic | Current frontier models are highly unpredictable, dropping instantly from genius-level synthesis to logical nonsense. |
|---|---|
| The Structural Flaw | Knowledge is hopelessly entangled within the model's probabilistic weights. |
| The Required Fix | Future architectures must factor knowledge out of the weights and into interpretable, external semantic databases. |
Blending Continuous Gradients With Discrete Logic
The integration challenge: marrying deep learning's flexibility with symbolic reasoning's reliability.
Tokens / Continuous Gradients
Differential equations · probabilistic routing · gradient descent.
Predicates / Formal Logic
Crisp rules · semantic facts · symbolic reasoning.
Endlessly Scaling Data Cannot Replace Logical Dependency
The Bitter Lesson (scale) versus intentional design (hard-coded structure).
The Bitter Lesson
P(Thunder | Lightning) = 0.98
The attention mechanism acts as a soft, probabilistic dictionary. It learns associations but lacks strict conditional logic.
Intentional Design
IF Lightning THEN Thunder
The frontier: moving beyond standard query-key-value matching to dynamic, logically driven attention with strict layer dependencies.
The Quadratic Bottleneck Forces an Architectural Evolution
1000² tokens is manageable. 1,000,000² tokens breaks the system.
Transformers rely on quadratic complexity. As enterprise context windows stretch from paragraphs to entire corporate databases, pure attention becomes financially and computationally unviable — the next breakthrough needs a fundamentally new algorithmic geometry.
Hybrid Architectures Achieve Linear Complexity Without Losing Performance
AI21 Labs' Jamba: systematically interleaving State-Space Model layers with traditional attention layers.
Pure Transformer
High algorithmic performance, quadratic cost.
Pure Mamba / SSM
Linear cost, slightly degraded performance.
The Jamba Hybrid
Linear complexity, frontier performance — Mamba layers doing the bulk of the work with attention layers interleaved sparingly.
The Myth of the Winner-Take-All Frontier Oligopoly
A trillion-parameter generalist chatbot executing narrow enterprise tasks is a staggeringly expensive mismatch.
Constrained & Misaligned
The generalized, please-everyone chatbot is a staggeringly expensive game that only a few mega-labs will play.
Specialized Fleets
The enterprise market will not consolidate around a duopoly. Sovereignty, data security, and unit economics will drive adoption of localized, fine-tuned models over monolithic generalists.
The Anatomy of a Modern Intelligence Harness
Intelligence is no longer just about the underlying model; it is about the harness surrounding it.
Measuring the Multidimensional Enterprise Frontier
Optimizing exclusively for cost-per-token is a trap.
| Axis | What it captures |
|---|---|
| Quality | Accuracy & relevance of the output. |
| Latency | Time to first token. |
| Cost | Compute & API spend. |
| ROI | Business value actually generated. |
The Blind Spot
If you don't know exactly what you're optimizing for, you're flying blind — winging it without strict evaluations will haunt enterprise deployments.
The New Standard
Successful AI strategy requires abandoning single-metric fixation and dynamically balancing Quality, Latency, and Cost against actual business ROI.
Interacting Agents Require Mechanism Design, Not Just Software Engineering
When millions of autonomous agents interact, the critical missing skill is game theory.
Without Mechanism Design
Conflicting goals and asymmetric information between agent clusters produce diverging information and diverging incentives.
With Engineered Social Norms
Engineering social norms for AI — the digital equivalent of "everyone drives on the right" — drastically cuts the need for infinite micro-negotiations and prevents catastrophic emergent behaviors.
Tracking the Global AI Economy Requires a Multi-Dimensional Index
The Global Vibrancy Index — an AI-era analog to GDP, initiated a decade ago at Stanford.
Compute Volume
Tracking global energy consumption and FLOPs.
Research Velocity
Tracking publications, patents filed, and algorithmic breakthroughs.
Policy & Regulation
Tracking sovereign restrictions and national investment/promotion strategies.
The Geopolitical Imperative of Sovereign AI
Nations and Fortune 50 enterprises hold "crown jewel" data critical to national security or core business defensibility.
- These entities will permanently refuse to hand proprietary data to generalized frontier labs for training or serving.
- This guarantees an enduring market for localized, bespoke models running on sovereign, highly secured infrastructure.
Mapping Israel's Structural AI Defensibility
Strength lies not in generalized foundation models, but in dominating the extreme edges of the stack.
Silicon & Infrastructure
Rooted in the legacy of Intel's Pentium Pro design teams, Mellanox, and deep networking expertise.
Cyber & Alignment
A talent pipeline moving from IDF cybersecurity directly into AI safety and evaluations.
Longitudinal Healthcare Data
Unique, centralized HMO data spanning the entire population since the state's founding.
The Hidden Risk of Automating Intellectual Labor
The risk is not merely job replacement, but a loss of our ability to push science and culture beyond what the technology generates for us.
Active Discovery
Pushing the frontier — manually modifying the query path, restructuring the creative block, staying in the loop.
The Chaperone Trap
Passive chaperoning — auto-generation running with human override effectively inactive.
Elevating the Abstraction Layer of Human Cognition
Calculators changed mathematics but didn't eliminate mathematicians — AI is a similar shift in the abstraction layer.
AI will absorb the "how" of execution, demanding that humans become masters of the "why" and the "what."
The Ultimate Contrarian Bet: Human–Machine Convergence
One commentator's speculative framing, not a settled scientific claim.
The False Dichotomy
Debating whether machines will outsmart us or possess consciousness misses the point — "us vs. machines" is treated here as an obsolete framework.
The Synthesis (Speculative)
The claim: within 10–50 years, via neural implants and vascular nanobotics, humanity and technology merge, dissolving the boundary of consciousness. This is a bet, not a roadmap — treat the wide date range and the "dissolve the boundary" language as a sign of how open this actually is.
The Architecture of Intentionality
To navigate the next decade, the briefing argues for moving beyond brute-force probability to engineered logic, specialized economics, and human responsibility.
Semantic Architecture
Integrating logic with probability — the base layer.
Multi-Agent Markets
Mechanism design over monoliths — the middle layer.
Human/Machine Synthesis
Dissolving the consciousness boundary — the speculative capstone.