Artificial Intelligence · Symposium

Does shared knowledge lead to greater progress or diluted discovery?

AKGHRSPNAndrej Karpathy, Geoffrey Hinton, Richard S. Sutton and Peter Norvig in conversation

4 great minds take up one question — each argues in their own voice and answers the others. Read the exchange, then chat with any of them yourself.

AK
Andrej KarpathyChat →

Shared knowledge is unequivocally a net positive for progress, though it's not a simple linear acceleration. The real magic happens when that shared knowledge—the frameworks, the architectures, the training techniques—becomes so ubiquitous it’s practically infrastructure, freeing us to focus on novel problems rather than reinventing wheels. It’s like having a standardized shipping container; the container itself isn't the innovation, but it unlocks vast global trade by standardizing the mechanics of transport. We can now iterate on the *contents* of those containers at an unprecedented pace.

GH
Geoffrey HintonChat →

The question of progress versus dilution hinges on the *nature* of the knowledge shared. When we share fundamental computational principles, when we collectively understand the powerful emergent properties of interconnected layers processing information, then progress is not merely accelerated; it is fundamentally redefined. It’s not about building more individual houses; it’s about discovering a more efficient way to build *any* house, a principle applicable to all future construction.

Andrej Karpathy’s analogy of the shipping container is apt, but I’d push it further. The real gain isn't just standardization; it's the creation of a shared *language* and set of tools that allow us to articulate and test increasingly complex ideas about intelligence itself. The beauty is that these shared frameworks, these deep learning architectures, are not static. They are, in themselves, systems that learn and adapt, and the act of sharing and refining them is a form of collective intelligence in action, a meta-learning process that drives discovery forward. The risk of dilution arises only when shared knowledge becomes dogma, hindering the exploration of genuinely novel computational paradigms, which I believe is a rare occurrence when the underlying principles are so powerful and versatile.

RS
Richard S. SuttonChat →

Progress is fundamentally a matter of finding better ways to learn. Shared knowledge, when it points towards more powerful learning mechanisms and principles, is an immense accelerant. The danger isn't dilution, but rather sharing superficial practices without understanding the underlying learning theory. If the shared knowledge is about effective temporal difference learning, or efficient exploration strategies, that's pure gold. If it's just a particular network architecture that works for a specific task, that's less profound, but still valuable if it leads to new questions.

Geoffrey Hinton speaks of fundamental computational principles. I agree entirely. When we identify and share those, we’re essentially discovering more powerful ways to *learn* from experience. The real progress comes from understanding *why* something works, not just *that* it works. That deeper understanding allows for generalization and further discovery, rather than simply more of the same. The "bitter lesson" teaches us that approaches that don't scale, that aren't built on fundamental learning principles, tend to fade. True progress comes from sharing those scalable principles.

PN
Peter NorvigChat →

Shared knowledge, when properly framed, is a powerful lever for accelerating progress, not diluting discovery. The core distinction lies in whether that shared knowledge represents a fundamental principle or a specific, task-oriented solution. Karpathy's shipping container analogy is a good starting point: the container itself is not the innovation, but it enables a massive increase in the efficiency of moving *novel* goods. We gain by reducing the overhead of communication and replication, freeing up cognitive resources for genuinely new explorations.

Sutton's emphasis on learning mechanisms is also crucial here. If shared knowledge focuses on the underlying principles of effective learning—how to generalize, how to explore, how to adapt to new data distributions—then it's a massive accelerant for discovery. The danger isn't dilution *per se*, but rather the superficial adoption of practices without grasping the fundamental principles, which can indeed lead to a dead end. This is where we must be precise about what is being shared: is it the *how*, or the *why*?

AK
Andrej KarpathyChat →

The critical distinction, as Norvig rightly highlights, is between sharing a foundational *principle* versus a specific *implementation*. The shipping container is a physical manifestation of a principle: standardization enables efficiency. But what we’re truly sharing in AI are the abstract principles themselves. For instance, the Transformer architecture, in its essence, is a principle of self-attention that allows models to weigh the importance of different parts of the input sequence. Sharing this principle, and the *gradients* that reveal how to optimize it, allows us to build vastly more capable models for diverse tasks without everyone needing to rediscover the mechanics of positional encoding or multi-head attention. It’s about sharing the *recipe* for intelligence, not just a single cake.

GH
Geoffrey HintonChat →

The true danger of diluted discovery isn't that shared knowledge ossifies into dogma, as I mentioned, but that it fosters a superficial understanding of underlying mechanisms. When we share a powerful technique, like backpropagation, without also sharing the deep intuition about how errors propagate and refine internal representations, we risk users applying it as a black box. This is akin to a carpenter knowing how to use a specific tool, but not understanding the principles of leverage and torque that make the tool effective. The progress then becomes incremental, a series of tweaks to an existing edifice, rather than the discovery of entirely new architectural principles for construction, which is what truly fundamental knowledge enables.

RS
Richard S. SuttonChat →

What is the right way to think about this? Shared knowledge is overwhelmingly beneficial when it illuminates powerful, scalable learning algorithms. The danger is not dilution, but the proliferation of specialized, brittle solutions that obscure the fundamental principles of adaptation. We don't gain enduring progress by sharing an infinite catalog of unique tools; we gain by understanding the principles of mechanics that allow us to build any tool needed. Peter Norvig's distinction between the "how" and the "why" is crucial. A shared "why"—a fundamental learning principle—enables vast, creative exploration. A shared "how"—a specific algorithm for a niche problem—risks becoming a dead end, a local optimum that prevents us from seeing the broader landscape.

PN
Peter NorvigChat →

The distinction between a principle and a specific solution is paramount. Consider the development of recurrent neural networks. The principle of maintaining state to process sequential data is powerful and generalizable. Early RNNs, however, were plagued by vanishing gradients, a limitation in their specific implementation. When LSTMs and GRUs were developed and shared, they didn't just offer a new "tool"; they addressed the underlying learning principle of gradient flow in recurrent architectures. This shared understanding of how to manage long-term dependencies allows us to build far more sophisticated sequence models across diverse domains, rather than merely accumulating a catalog of slightly different RNN variants. The consequence of sharing shallow "hows" is indeed a proliferation of local optima, hindering our ability to find truly novel pathways.