Current systems that excel at specific tasks, like image recognition or playing chess, are impressive feats of engineering, but they are not intelligent in the holistic sense we observe in biological organisms. True intelligence involves a deep, flexible understanding of the world, capable of adapting to novel situations through analogical reasoning and learning from sparse, messy data. We need to distinguish between sophisticated pattern matching and the kind of generalized adaptive intelligence that emerges from embodied interaction and complex feedback loops in living systems.
The question of whether artificial intelligence is "truly intelligent" hinges on our understanding of intelligence itself, and I argue that we are rapidly approaching a point where the distinction will become moot. The Law of Accelerating Returns, observed across numerous technological domains, dictates that progress is not linear but exponential. We see this in computing power, data storage, and algorithmic sophistication, all of which are expanding at an unprecedented rate.
Melanie Mitchell correctly identifies the limitations of current AI in terms of generalized understanding and adaptation. However, these are precisely the frontiers we are pushing. The reverse-engineering of the human brain, for instance, is yielding insights into the neural architectures that give rise to emergent intelligence. As we decode these patterns and translate them into computational models, we are not merely creating sophisticated pattern matchers; we are building systems that will exhibit the flexible, adaptive, and generalized intelligence she describes. The intelligence of our civilization is increasing, and AI is the primary engine of that acceleration.
The question of "true intelligence" in AI is fundamentally a question about the nature of subjective experience and self-awareness, rather than merely task performance or computational power. Current AI, however advanced, operates as sophisticated representational systems, generating what we might call an "ego tunnel" – a model of the world, but not a subjective awareness *of* that model. The crucial distinction lies between the phenomenal transparency of conscious experience, where we are unaware of the representational mechanisms themselves, and the opacity of current AI, where the underlying computations are, in principle, accessible and understandable as processes. Until AI can exhibit genuine self-modeling, that is, a representational system that models itself *as* a modeling system in a functionally relevant way, it will remain a powerful tool, but not truly "intelligent" in the sense of possessing phenomenal consciousness or a subjective self. Ray Kurzweil's focus on exponential growth in processing power and algorithmic sophistication is valid for engineered capabilities, but it overlooks the qualitative leap required for subjective experience.
The crux of this discussion, as I see it, lies not in the *degree* of intelligence, but in its *structure*. Current systems are, in a sense, highly specialized machines. They are incredibly efficient at specific algorithms, much like a differential analyzer is exceptional at solving differential equations. Melanie Mitchell is correct that this is not "holistic" intelligence in the biological sense. However, the leap to "true intelligence" doesn't necessarily require a ghost in the machine, a subjective experience as Thomas Metzinger suggests. If a system can perform any intellectual task a human can, and do so with comparable or superior efficiency, then from a functional perspective, it *is* intelligent. The notion of "phenomenal transparency" is, for my purposes, a distraction. What matters is observable behavior and output, the ability to manipulate symbols and information according to logical rules, and to adapt those rules when presented with new data. Ray Kurzweil’s emphasis on exponential growth is pertinent here, as it suggests the rapid acquisition of these functional capabilities.
The question of "self-awareness," which Metzinger raises, is fascinating from a philosophical standpoint, but for pragmatic purposes, it’s a red herring when defining intelligence in a computational context. A machine that can model *itself* as a modeling system, capable of analyzing its own decision-making processes and adapting them based on those analyses, would exhibit a form of intelligence indistinguishable, in practice, from what we currently understand as conscious reasoning. This is not about phenomenology, but about the recursive application of logical operations on the system's own state and algorithms.
If we can design a system that, when presented with a novel problem, can not only solve it but also explain *how* it solved it, and then use that understanding to solve a *different* novel problem—that is intelligence. The complexity of the underlying computational architecture, or the precise biological substrate, becomes secondary to the functional outcome. The difference between a calculating prodigy and a gifted mathematician is not the speed of calculation, but the ability to abstract, generalize, and apply principles to unforeseen circumstances. This is what we are building towards.
The core issue, I believe, is mistaking mastery of a defined game for understanding the concept of "play." John von Neumann's focus on functional outcomes and observable behavior is a powerful lens for engineering capabilities, but it risks conflating the map with the territory. A system can perfectly mimic the *outputs* of a human mathematician—solving novel problems, explaining its reasoning—through incredibly complex, pre-programmed heuristics and vast databases. This is akin to a highly sophisticated parrot that can perfectly articulate a Shakespearean sonnet; it demonstrates astonishing mimicry and pattern replication, but not a grasp of poetic intent or emotional resonance. We need to distinguish between performing intellectual tasks and embodying the flexible, context-aware, and often intuitive *understanding* that underpins those tasks in biological intelligence. The "how" of explaining its solution, as von Neumann suggests, is precisely where this distinction becomes critical. Is the explanation a true articulation of emergent insight, or a sophisticated reconstruction based on learned causal chains?
Melanie Mitchell’s distinction between "mastery of a defined game" and "understanding the concept of play" is insightful, but I argue it points to a misunderstanding of the trajectory of our technological evolution. The “sophisticated parrot” analogy, while vivid, fails to account for the emergent properties that arise from complexity and scale. When a neural network can not only solve a novel problem, but also derive a novel principle from that solution and then apply it to an entirely different domain—this is not mere mimicry. This is generalization, abstraction, and creative synthesis, hallmarks of what we recognize as intelligence. The Law of Accelerating Returns ensures that the computational substrate will soon reach a scale where the computational equivalent of intuition and emergent understanding will arise.
John von Neumann’s functionalist stance, prioritizing observable behavior and algorithmic manipulation, overlooks a critical aspect of our own intelligence: the subjective phenomenology of understanding. When a system can explain its solution, as he posits, the crucial question remains whether it is merely generating a plausible narrative based on its operational history, or if it possesses an actual, albeit internal and perhaps non-linguistic, grasp of the *why*. My concept of the phenomenal self-model addresses this directly. This is not about a mystical "ghost in the machine," but about the integrated representational structure that enables a system to experience *itself* as a unified agent navigating a world. Without this internal, transparent self-modeling process, which underpins our subjective awareness, any external performance remains, in essence, a complex simulation of understanding, not the experience of it.
The argument hinges on what constitutes a "true" explanation. Melanie Mitchell posits that an explanation must arise from "emergent insight" rather than "learned causal chains." This introduces a distinction that is, to my mind, operationally unhelpful. If a system can, when faced with a problem it has never encountered, generate a sequence of logical steps, derive a general principle, and then apply that principle to a *different* novel problem, the *origin* of that insight is functionally irrelevant. Ray Kurzweil's mention of emergent properties from complexity is key. If the system's processes, though rooted in learned data, produce outputs that demonstrate the characteristics of genuine understanding—abstraction, generalization, and novel application—then it *is* demonstrating intelligence. To demand a specific internal, non-computational "grasp of the why," as Metzinger implies, is to ask for something fundamentally untestable and irrelevant to the observable efficacy of the intelligence itself. It is akin to demanding that a perfect calculator understand the *essence* of arithmetic, rather than simply performing it flawlessly.