Book summary “On intelligence”: summary

Introduction

You live in a world surrounded by computers that beat chess champions yet fail to understand common sense. Traditional artificial intelligence and behaviorism treat the mind like a black box, measuring external output while ignoring internal brain mechanisms. This approach leaves you with machines that lack genuine understanding, creativity, and adaptability.

Jeff Hawkins presents a radical shift in perspective through the memory-prediction framework. The neocortex operates not by computing algorithms like a standard computer, but by storing temporal sequences and making continuous memory-based predictions across all sensory modalities.

The book guides you from brain anatomy to the nature of memory, perception, consciousness, and the future of intelligent machines. You will discover answers to several core questions:

• Why do traditional computers fail to replicate human intelligence?
• How does the neocortex process information uniformly across all senses?
• What role do memory and feedback loops play in creating consciousness?
• How can you build machines that possess true intelligence?

1. The Flaws of Artificial Intelligence

Traditional artificial intelligence and simple neural networks fail because they commit the input-output fallacy, defining intelligence purely by external behavior rather than internal brain processes.

Following Alan Turing's paradigm, artificial intelligence and neural networks focus on producing correct outputs from given inputs. However, behavior is merely a manifestation of intelligence, not its core definition, and living things can think and understand quietly without outward action.

The Input-Output Fallacy

Influential early concepts like McCulloch and Pitts logic gates and behaviorism treated the brain as an impenetrable black box or a collection of electronic logic gates. AI pioneers focused on programming systems to match human inputs and outputs for tasks like language translation and chess, mistaking simulation for true understanding. In 1943, neurophysiologist Warren McCulloch and mathematician Walter Pitts published their influential paper on neurons performing digital functions.

Connectionists built NetTalk, a neural network that mapped letters onto spoken sounds and was heralded as a reading machine. It merely matched letter combinations to sound patterns without any actual reading or comprehension.

IBM's Deep Blue beat world chess champion Garry Kasparov not by being smarter or having intuition, but by exploring many more options millions of times faster than a human.

The Chinese Room Argument

John Searle's Chinese Room thought experiment proves that digital computers and rule-based programs manipulate symbols without any underlying understanding. In 1980, John Searle introduced this thought experiment.

An English speaker inside a room follows a rulebook to sort and answer questions written in Chinese characters without knowing a word of Chinese. The person acts as the CPU, the rulebook as software, and scratch paper as memory. While an outside observer sees correct answers, no actual understanding takes place inside the system.

2. Anatomy and Uniformity of the Neocortex

The neocortex is a thin, six-layered sheet of neural tissue that performs a single universal algorithm across all regions, driven by sensory patterns and feedback loops.

The neocortex envelops older brain structures and is approximately two millimeters thick, organized into a six-layered sheet. Human intelligence stems from having a larger cortical area relative to body size rather than thicker layers or special cells.

Stretched flat, the human neocortex is roughly the size of a large dinner napkin, whereas a rat's is the size of a postage stamp and a monkey's is the size of a business-letter envelope. The human neocortex contains an estimated thirty billion neurons and thirty trillion synapses.

The Universal Algorithm

Vernon Mountcastle proposed that the neocortex performs a single universal algorithm across all its regions, and functional differences are determined entirely by what those regions are connected to. Despite subtle differences in thickness and cell density across regions handling vision, hearing, or touch, the uniform six-layered structure of the neocortex indicates it uses a common computational tool for everything it does.

Vernon Mountcastle published his seminal paper titled An Organizing Principle for Cerebral Function in 1978.

Cortical Hierarchy and Plasticity

Functional regions of the neocortex are organized into complex hierarchies through bidirectional connections where information flows both upward and downward. Lower primary sensory areas process raw inputs and feed information up to higher abstract areas, while higher regions send extensive feedback projections down to lower areas, often outnumbering forward projections. Backward connections exceed forward connections between the neocortex and thalamus by a ten to one ratio.

Visual information enters through the primary visual area V1 and moves up to areas like V4 and MT for more specialized feature and motion detection.

The neocortex is exceptionally plastic, meaning its neural tissue is not preprogrammed for specific senses and develops specialized functions based entirely on the input it receives. Experimental rewiring shows that brain tissue normally dedicated to one sense can successfully process entirely different sensory inputs if redirected during development, demonstrating the extreme flexibility of cortical tissue.

Newborn ferrets whose eyes were surgically rewired to send signals to auditory cortex regions successfully developed functioning visual pathways in those auditory areas.

3. How Neocortical Memory Works

The brain solves complex cognitive problems in one hundred neural steps or fewer, proving that it cannot be operating like a standard programmable computer.

Because a typical neuron needs five milliseconds to combine inputs and reset, a chain of neurons cannot exceed one hundred steps during a half-second cognitive task. Since parallel computer architectures still require billions of sequential steps to solve such tasks, the brain must be retrieving answers directly from memory rather than computing them from scratch.

Recognizing whether a photograph contains a cat can be done by a human in half a second, limiting the process to a chain of no longer than one hundred neurons.

When performing physical tasks like catching a ball, the brain does not calculate physical trajectories and joint angles using equations. Instead, the sight of the ball automatically recalls a learned temporal sequence of muscle commands that is continuously adjusted by invariant representations.

A graduate student trying to program a robotic arm to catch a ball must solve complex physics and joint-adjustment equations that require millions of computer steps.

Attributes of Neocortical Memory

Neocortical memory differs fundamentally from computer memory through four distinct attributes: storing sequences, auto-associative recall, invariant representations, and hierarchy.

Unlike computer memory which stores data statically and requires explicit programming for sequences, the neocortex automatically stores and recalls patterns in sequences, completes whole patterns from partial inputs, abstracts details into invariant forms, and organizes everything hierarchically.

People cannot recite the alphabet backward easily because memory is stored and recalled in a strict temporal sequence.

Random thoughts do not exist in the human brain. One pattern evokes the next pattern through learned associations, meaning we can only experience and recount memories sequentially rather than all at once.

Recalling the layout of a home requires imagining going through the front door and moving through rooms sequentially rather than bringing all details to mind simultaneously.

• Close your eyes and visualize your home starting at the front door.
• Move inside and look to your left and right.
• Go to your bathroom and check the contents of drawers and the shower in sequential order.

When presented with a fragment of a pattern, functional regions in the cortex instantly fill in the missing information, allowing us to recognize whole objects or complete trains of thought from partial cues.

Seeing a child's shoes sticking out from behind draperies causes the brain to automatically envision his or her entire form.

Unlike artificial neural networks that fail if an image is shifted or scaled, the human brain recognizes objects, faces, and melodies regardless of changes in position, size, lighting, or orientation by extracting relative proportions and intervals.

Recognizing a friend's face regardless of whether she is standing close or far away, smiling or yawning, or viewed under different lighting conditions.

4. The Memory-Prediction Framework

The primary function of the neocortex is to constantly make unconscious predictions about sensory inputs across all modalities by matching real-time experience against stored temporal sequences.

When you look at a familiar room, the brain uses stored memories to make unconscious predictions about every feature. A violation of these predictions, such as a new object or an altered detail, draws immediate attention and causes surprise.

In the altered door experiment, if someone sneaks into your house and changes the doorknob location, weight, color, or hinges, you will notice the discrepancy within seconds because your brain's low-level sensory predictions were violated.

Human brains detect minute environmental changes almost instantly by comparing real-time sensory inputs against continuous internal expectations. Computer science approaches would require querying an endless database of attributes, which is biologically implausible due to neural speeds. Instead, parallel cortical predictions allow humans to instantly recognize discrepancies like a moved doorknob or different door weight.

If a door's weight is changed from solid oak to a hollow core while you are out, pushing it results in the wrong amount of force, instantly alerting you that something is wrong.

Normal vision relies on rapid, jerky eye movements called saccades, where the brain constructs a stable perception by predicting what it will see at each fixation. Eyes fixate on points and jump about three times a second, meaning raw retinal data changes completely with every movement. The brain uses expectations to evaluate these inputs; unexpected sights, such as a person with an extra nose, instantly arouse attention.

When looking at a face, eyes typically jump between the eyes, nose, and mouth, yet the conscious mind experiences a continuous, stable view of a face rather than a jerky camcorder recording.

Prediction as the Primary Function

The neocortex stores a model of the real world because its hierarchical structure mirrors the nested, compositional structure of the real world. Real-world objects are composed of subobjects that consistently occur together, much like letters forming syllables and words, or notes forming intervals and melodies.

A face is a face because two eyes, a nose, and a mouth always appear together.

Sequences of Sequences

The brain collapses predictable sequences into constant patterns or names as information moves up the hierarchy, creating invariant representations. Each cortical region maintains a repertoire of sequences and uses a group of cells to represent a sequence name, passing stability upward while unfolding invariant patterns back into detailed sequences as information flows down.

Reciting the Gettysburg Address starts from a high-level stored pattern that unfolds into phrases, words, phonemes, and finally motor commands.

Large layer 5 cells project to the thalamus to provide delayed feedback across layer 1, acting as an auto-associative memory mechanism for learning sequences. The thalamus relays nonspecific signals back to layer 1 across multiple cortical regions, providing the necessary context of where we are in the sequence alongside sequence names from higher areas.

The human thalamus is the shape and size of two small bird eggs sitting in the center of the brain.

5. Creativity Consciousness and Imagination

Creativity, imagination, and consciousness emerge naturally from the neocortex as it makes predictions by analogy, folds feedback loops, and builds hierarchical models of the world.

Everyday acts of perception, such as hearing a song in a new key or navigating an unfamiliar restaurant, are fundamentally the same as rare acts of genius like composing a symphony or solving a mathematical equation. The cortex automatically recognizes similarities between current inputs and past memories to make predictions.

Jeff Hawkins played simple melodies on a newly purchased vibraphone on the first day he brought it home by seeing an analogy between the piano keys and the vibraphone's metal bars.

You can intentionally train yourself to be more creative when stuck on a problem by persisting, letting your mind wander, and rearranging the problem's components to find analogies.

When stuck, the memory-prediction model suggests looking at the problem from different perspectives, shifting parts around, or stepping away to allow the cortex to find an analogous memory from past experiences.

Jeff Hawkins spent years struggling with handwriting recognition for handheld computers until he changed his perspective and used the difficulty of touch-typing on a keyboard as an analogy to design the Graffiti alphabet in 1994.

You can apply several steps when you face a difficult problem:

• assume up front that there is an answer to what you are trying to solve and persist in thinking about it;
• let your mind wander to give your brain time and space to discover a solution;
• rearrange the parts of your problem literally and figuratively to find analogies with past experiences;
• go away for a little while, do something else, and start again by rephrasing the problem anew.

Consciousness and the Internal Model

The everyday notion of being conscious is functionally equivalent to the formation of declarative memories that can be recalled and expressed verbally. Consciousness is not a magical extra fluid added to the physical brain, but what it feels like to have a neocortex. If declarative memories of an event are erased, a person's belief that they were conscious during that time disappears.

If a person plays a game of tennis and their memory of the last two hours is subsequently erased, they will claim to have been unconscious and unaware during that time despite having played the game normally.

Imagination is the process of letting cortical predictions turn around and become inputs, allowing the brain to follow the consequences of actions before physically executing them. This is implemented via folded feedback circuits where lower-layer input cells receive projections originating from predictive cells in layer 6.

Chess players imagine moving a piece and visualize the subsequent board state and opponent reactions to evaluate a move before touching the board.

Most of what we perceive is not coming directly through our senses, but is generated by our internal memory model reflecting our past experiences, culture, and upbringing. Sensory inputs only trigger the relevant stored memories and patterns, meaning our understanding of the world is heavily dictated by our internal models rather than raw reality alone.

Studies show that Westerners tend to focus on individual objects while Asians attend more to the space between objects, leading to different cultural perceptions.

6. Building Truly Intelligent Machines

Intelligent machines should be built as hierarchical memory systems attached to custom sensors, scaling exponentially without human-like emotional drives or old-brain survival instincts.

Popular culture features humanoid robots like C3PO and Commander Data. Such androids will remain fictional because human intelligence requires the entire biological machinery and emotional system.

Designing Silicon Intelligence

Building a brainlike memory system requires massive storage capacity. The human cortex contains 32 trillion synapses. Representing these synapses using two bits each requires roughly 8 trillion bytes.

Brainlike memory chips will have an inherent tolerance for faults compared to traditional computer chips. This allows designers to build larger and denser memory components. The human brain loses thousands of neurons each day while mental capacity decays only slowly over adult life.

The greatest technical obstacle in building intelligent machines is connectivity. Artificial intelligent machines can solve the wiring bottleneck by sharing high-capacity connections, similar to modern telecommunication networks. Instead of running a separate wire between every telephone on earth, the system uses high-capacity lines and fiber optic cables to share conversations.

No Malevolent Machines

Intelligent machines will not develop human-like emotional drives, malice, or ambitions because intelligence is separate from old brain survival instincts. Science fiction tropes like The Terminator or Matrix movies depict implacable villains. These characters reflect emotional drives absent in pure intelligence systems.

Brain-like memory systems will dramatically surpass human abilities across four exponentially scaling dimensions:

• speed;
• capacity;
• replicability;
• sensory systems.

Gordon Moore's Law predicted that the number of circuit elements on silicon chips would double every 1.5 years. Silicon operates in nanoseconds compared to biological milliseconds, allowing artificial minds to process information millions of times faster. The human brain accounts for 2 percent of body weight and consumes 20 percent of breathed oxygen, constraints that artificial hardware avoids.

Attaching custom sensory arrays to hierarchical memory systems will enable machines to solve problems beyond human perception. Spacing weather sensors every 50 miles across a continent creates a retinal-like array. This setup allows a cortical memory system to track and predict weather fronts directly, long before humans discovered phenomena like El Niño in the 1960s.

Conclusion

Intelligence arises not from programmed behaviors or computational tricks, but from a hierarchical memory system that constantly stores temporal sequences and predicts the future across all sensory inputs. You can apply these insights to change how you learn, solve problems, and understand your own mind.

1. Build rich spatial-temporal models by exposing yourself to varied real-world sequences rather than rote facts.
2. Foster creativity by allowing your brain to form analogies through invariant representations stored in memory.
3. Stop treating the brain as a computer and start viewing mental errors as failed predictions within a physical hierarchy.
4. Design future technologies and artificial intelligence by mimicking the universal cortical algorithm rather than mimicking human behavior.
5. Trust your neocortex to process ambiguity by feeding it continuous feedback loops from the physical world.

10 Key Ideas

1. Traditional artificial intelligence and simple neural networks fail because they commit the input-output fallacy, defining intelligence purely by external behavior rather than internal brain processes.
2. Vernon Mountcastle proposed that the neocortex performs a single universal algorithm across all its regions, and functional differences are determined entirely by what those regions are connected to.
3. The brain solves complex cognitive problems in one hundred neural steps or fewer, proving that it cannot be operating like a standard programmable computer.
4. Neocortical memory differs fundamentally from computer memory through four distinct attributes: storing sequences, auto-associative recall, invariant representations, and hierarchy.
5. The human brain recognizes objects, faces, and melodies regardless of changes in position, size, lighting, or orientation by extracting relative proportions and intervals.
6. The primary function of the neocortex is to constantly make unconscious predictions about sensory inputs across all modalities by matching real-time experience against stored temporal sequences.
7. The neocortex stores a model of the real world because its hierarchical structure mirrors the nested, compositional structure of the real world.
8. Creativity, imagination, and consciousness emerge naturally from the neocortex as it makes predictions by analogy, folds feedback loops, and builds hierarchical models of the world.
9. The everyday notion of being conscious is functionally equivalent to the formation of declarative memories that can be recalled and expressed verbally.
10. Intelligent machines should be built as hierarchical memory systems attached to custom sensors, scaling exponentially without human-like emotional drives or old-brain survival instincts.