Richard Sutton

Source

Table of Contents

What's Wrong with Artificial Intelligence

Source November 12, 2001

I hold that AI has gone astray by neglecting its essential objective --- the turning over of responsibility for the decision-making and organization of the AI system to the AI system itself. It has become an accepted, indeed lauded, form of success in the field to exhibit a complex system that works well primarily because of some insight the designers have had into solving a particular problem. This is part of an anti-theoretic, or "engineering stance", that considers itself open to any way of solving a problem. But whatever the merits of this approach as engineering, it is not really addressing the objective of AI. For AI it is not enough merely to achieve a better system; it matters how the system was made. The reason it matters can ultimately be considered a practical one, one of scaling. An AI system too reliant on manual tuning, for example, will not be able to scale past what can be held in the heads of a few programmers. This, it seems to me, is essentially the situation we are in today in AI. Our AI systems are limited because we have failed to turn over responsibility for them to them.

Please forgive me for this which must seem a rather broad and vague criticism of AI. One way to proceed would be to detail the criticism with regard to more specific subfields or subparts of AI. But rather than narrowing the scope, let us first try to go the other way. Let us try to talk in general about the longer-term goals of AI which we can share and agree on. In broadest outlines, I think we all envision systems which can ultimately incorporate large amounts of world knowledge. This means knowing things like how to move around, what a bagel looks like, that people have feet, etc. And knowing these things just means that they can be combined flexibly, in a variety of combinations, to achieve whatever are the goals of the AI. If hungry, for example, perhaps the AI can combine its bagel recognizer with its movement knowledge, in some sense, so as to approach and consume the bagel. This is a cartoon view of AI -- as knowledge plus its flexible combination -- but it suffices as a good place to start. Note that it already places us beyond the goals of a pure performance system. We seek knowledge that can be used flexibly, i.e., in several different ways, and at least somewhat independently of its expected initial use.

With respect to this cartoon view of AI, my concern is simply with ensuring the correctness of the AI's knowledge. There is a lot of knowledge, and inevitably some of it will be incorrrect. Who is responsible for maintaining correctness, people or the machine? I think we would all agree that, as much as possible, we would like the AI system to somehow maintain its own knowledge, thus relieving us of a major burden. But it is hard to see how this might be done; easier to simply fix the knowledge ourselves. This is where we are today.

Verification, The Key to AI

Source November 15, 2001

It is a bit unseemly for an AI researcher to claim to have a special insight or plan for how his field should proceed. If he has such, why doesn't he just pursue it and, if he is right, exhibit its special fruits? Without denying that, there is still a role for assessing and analyzing the field as a whole, for diagnosing the ills that repeatedly plague it, and to suggest general solutions.

The insight that I would claim to have is that the key to a successful AI is that it can tell for itself whether or not it is working correctly. At one level this is a pragmatic issue. If the AI can't tell for itself whether it is working properly, then some person has to make that assessment and make any necessary modifications. An AI that can assess itself may be able to make the modifications itself.

The Verification Principle:

An AI system can create and maintain knowledge only to the extent that it can verify that knowledge itself.

Successful verification occurs in all search-based AI systems, such as planners, game-players, even genetic algorithms. Deep Blue, for example, produces a score for each of its possible moves through an extensive search. Its belief that a particular move is a good one is verified by the search tree that shows its inevitable production of a good position. These systems don't have to be told what choices to make; they can tell for themselves. Image trying to program a chess machine by telling it what kinds of moves to make in each kind of position. Many early chess programs were constructed in this way. The problem, of course, was that there were many different kinds of chess positions. And the more advice and rules for move selection given by programmers, the more complex the system became and the more unexpected interactions there were between rules. The programs became brittle and unreliable, requiring constant maintainence, and before long this whole approach lost out to the "brute force" searchers.

Although search-based planners verify at the move selection level, they typically cannot verify at other levels. For example, they often take their state-evaluation scoring function as given. Even Deep Blue cannot search to the end of the game and relies on a human-tuned position-scoring function that it does not assess on its own. A major strength of the champion backgammon program, TD-Gammon, is that it does assess and improve its own scoring function.

Another important level at which search-based planners are almost never subject to verification is that which specifies the outcomes of the moves, actions, or operators. In games such as chess with a limited number of legal moves we can easily imagine programming in the consequences of all of them accurately. But if we imagine planning in a broader AI context, then many of the allowed actions will not have their outcomes completely known. If I take the bagel to Leslie's office, will she be there? How long will it take to drive to work? Will I finish this report today? So many of the decisions we take every day have uncertain and changing effects. Nevertheless, modern AI systems almost never take this into account. They assume that all the action models will be entered accurately by hand, even though these may be most of the knowledge in or ever produced by the system.

Finally, let us make the same point about knowledge in general. Consider any AI system and the knowledge that it has. It may be an expert system or a large database like CYC. Or it may be a robot with knowledge of a building's layout, or knowledge about how to react in various situations. In all these cases we can ask if the AI system can verify its own knowledge, or whether it requires people to intervene to detect errors and unforeseen interactions, and make corrections. As long as the latter is the case we will never be able to build really large knowledge systems. They will always be brittle and unreliable, and limited in size to what people can monitor and understand themselves.

"Never program anything bigger than your head"

And yet it is overwhelmingly the case that today's AI systems are not able to verify their own knowledge. Large ontologies and knowledge bases are built that are totally reliant on human construction and maintainence. "Birds have wings" they say, but of course they have no way of verifying this.

Verification

Source 11/14/2001

If the human designers of an AI are not to be burdened with ensuring that what their AI knows is correct, then the AI will have to ensure it itself. It will have to be able to verify the knowledge that it has gained or been given.

Giving an AI the ability to verify its knowledge is no small thing. It is in fact a very big thing, not easy to do. Often a bit of knowledge can be written very compactly, whereas its verification is very complex. It is easy to say "there is a book on the table", but very complex to express even a small part of its verification, such as the visual and tactile senations involved in picking up the book. It is easy to define an operator such as "I can get to the lunchroom by going down one floor", but to verify this one must refer to executable routines for finding and descending the stairs, recognizing the lunchroom, etc. These routines involve enormously greater detail and closed-loop contingences, such as opening doors, the possibility of a stairway being closed, or meeting someone on the way, than does the knowledge itself. One can often suppress all this detail when using the knowledge, e.g., in planning, but to verify the knowledge requires its specification at the low level. There is no comparison between the ease of adding unverified knowledge and the complexity of including a means for its autonomous verification.

Note that although all the details of execution are needed for verification, the execution details are not themselves the verification. There is a procedure for getting to the lunchroom, but separate from this would be the verifier for determining if it has succeeded. It is perfectly possible for the procedure to be fully grounded in action and sensation, while completely leaving out the verifier and thus the possibility of autonomous knowledge maintainence. At the risk of being too broad-brush about it, this is what typically happens in modern AI robotics systems. They have extensive grounded knowledge, but still no way of verifying almost any of it. They use visual routines to recognize doors and hallways, and they make decisions based on these conclusions, but they cannot themselves correct their errors. If something is recognized as a "doorway" yet cannot be passed through, this failure will not be recognized and not used to correct future doorway recognitions, unless it is done by people.

On the other hand, once one has grounding, the further step to include verification is less daunting. One need only attach to the execution procedures appropriate tests and termination conditions that measure in some sense the veracity of the original statement, while at the same time specifying what it really means in detail. What is a chair? Not just something that lights up your visual chair detector! That would be grounded knowledge, but not verifiable; it would rely on people to say which were and were not chairs. But suppose you have routines for trying to sit. Then all you need for a verifier is to be able to measure your success at sitting. You can then verify, improve, and maintain your "sittable thing" recognizer on your own.

There is a great contrast between the AI that I am proposing and what might be considered classical "database AI". There are large AI efforts to codify vast amounts of knowledge in databases or "ontologies", of which Doug Lenat's CYC is only the most widely known. In these efforts, the idea of people maintaining the knowledge is embraced. Special knowledge representation methods and tools are emphasized to make it easier for people to understand and access the knowledge, and to try to keep it right. These systems tend to emphasize static, world knowledge like "Springfield is the capital of Illinois", "a canary is a kind of bird", or even "you have a meeting scheduled with John at 3:30", rather than the dynamic knowledge needed say by a robot to interact in real time with its environment. A major problem is getting people to use the same categories and terms when they enter knowledge and, more importantly, to mean the same things by them. There is a search for an ultimate "ontology", or codification of all objects and their possible relationships, so that clear statements can be made about them. But so far this has not proven possible; there always seem to be far more cases that don't fit than do. People are good about being fluid with there concepts, and knowing when they don't apply.

Whatever the ultimate success of the symbolic "database AI" approach, it should be clear that it is the anti-thesis of what I am calling for. The database approach calls for heroic efforts organizing and entering an objective, public, and disembodied knowledge base. I am calling for an AI that maintains its own representations, perhaps different from those of others, while interacting in real time with a dynamic environment. Most important of all, the database approach embraces human maintainence and human organization of the AI's knowledge. I am calling for automating these functions, for the AI being able to understand its knowledge well enough to verify it itself.

Mind is About Information

Source 11/19/2001

What is the mind? Of course, "mind" is just a word, and we can mean anything we want by it. But if we examine the way we use the word, and think about the kinds of things we consider more mindful than others, I would argue that the idea of choice is the most important. We consider things to be more or less mindful to the extent that they appear to be making choices. To make a choice means to distinguish, and to create a difference. In this basic sense the mind is about information. Its essential function is to process bits into other bits. This position has two elements:

  • Mind is Computational, not Material
  • Mind is Purposive

Mind is Computational, not Material

The idea that the mind's activities are best viewed as information processing, as computation, has become predominant in our sciences over the last 40 years. People do not doubt that minds have physical, material form, of course, either as brains or perhaps as computer hardware. But, as is particularly obvious in the latter case, the hardware is often unimportant. Is is how the information flows which matters.

I like to bring this idea down to our basest intuition. What things are more mindlike and less mindlike? A thermostat is slightly mindlike. It converts a gross physical quanitity, the air temperature of your home, to a small deviation in a piece of metal, which tips a small lump of mercury which in turn triggers a fire in your furnace. Large physical events are reduced and processed as small ones, the physical is reduced to mere distinctions and processed as information. The sensors and effectors of our brains are essentially similar. Relatively powerful physical forces impinge on us, and our sensors convert them to tiny differences in nerve firings. These filter and are further processed until signals are sent to our muscles and there amplified into gross changes in our limbs and other large physical things. At all stages it is all physical, but inside our heads there are only small physical quanities that are easily altered and diverted as they interact with each other. This is what we mean by information processing. Information is not non-physical. It is a way of thinking about what is happening that is sometime much more revealing and useful than its physical properties.

Or so is one view, the view that takes a material physical reality as primary. The informational view of mind is just as compatible with alternative philosophical orientations. The one I most appreciate is that which takes the individual mind and its exchanging of information with the world as the primary and base activity. This is the so-called "buttons and lights" model, in which the mind is isolated behind an interface of output bits (buttons) and input bits (lights). In this view, the idea of the physical world is created by the mind so as to explain the pattern of input bits and how they respond to the output bits. This is a cartoon view, certainly, but a very clear one. There is no confusion about mind and body, material and ideal. There is just information, distinctions observed and differences made.

Mind is Purposive

Implicit in the idea of choice, particularly as the essense of mindfulness, is some reason or purpose for making the choices. In fact it is difficult even to talk about choice without alluding to some purpose. One could say a rock "chooses" to do nothing, but only by suggesting that its purpose is to sit still. If a device generated decisions at random one would hesitate to say that it was "choosing." No, the whole idea of choice implies purpose, a reason for making the choice.

Purposiveness is at heart of mindfulness, and the heart of purposeness is the varying of means to achieve fixed ends. William James in 1890 identified this as "the mark and criterion of mentality". He discussed an air bubble rising rising in water until trapped in an inverted jar, contrasting it with a frog, which may get trapped temporarily but keeps trying things until it finds a way around the jar. Varying means and fixed ends. In AI we call it generate and test. Or trial and error. Variation and selective survival. There are many names and many variations, but this idea is the essense of purpose, choice, and Mind.

Mind Is About Conditional Predictions

March 21, 2000

Simplifying and generalizing, one thing seems clear to me about mental activity---that the purpose of much of it can be considered to be the making of predictions. By this I mean a fairly general notion of prediction, including conditional predictions and predictions of reward. And I mean this in a sufficiently strong and specific sense to make it non-vacuous.

For concreteness, assume the world is a Markov Decision Process (MDP), that is, that we have discrete time and clear actions, sensations, and reward on each time step. Then, obviously, among the interesting predictions to make are those of immediate rewards and state transitions, as in "If I am in this state and do this action, then what will the next state and reward be?" The notion of value function is also a prediction, as in "If I am in this state and follow this policy, what will my cumulative discounted future reward be?" Of course one could make many value-function predictions, one for each of many different policies.

Note that both kinds of prediction mentioned above are conditional, not just on the state, but on action selections. They are hypothetical predictions. One is hypothetical in that it is dependent on a single action, and the other is hypothetical in that it is dependent on a whole policy, a whole way of behaving. Action conditional predictions are of course useful for actually selecting actions, as in many reinforcement learning methods in which the action with the highest estimated value is preferentially chosen. More generally, it is commonsensical that much of our knowledge is beliefs about what would happen IF we chose to behave in certain ways. The knowledge about how long it takes to drive to work, for example, is knowledge about the world in interaction with a hypothetical purposive way in which we could behave.

Now for the key step, which is simply to generalize the above two clear kinds of conditional predictions to cover much more of what we normally think of as knowledge. For this we need a new idea, a new way of conditioning predictions that I call conditioning on outcomes. Here we wait until one of some clearly designated set of outcomes occurs and ask (or try to predict) something about which one it is. For example, we might try to predict how old we will be when we finish graduate school, or how much we will weigh at the end of the summer, or how long it will take to drive to work, or much you will have learned by the time you reach the end of this article. What will the dice show when they have stopped tumbling? What will the stock price be when I sell it? In all these cases the prediction is about what the state will be when some clearly identified event occurs. It is a little like when you make a bet and establish some clear conditions at which time the bet will be over and it will be clear who has won.

A general conditional prediction, then, is conditional on three things: 1) the state in which it is made, 2) the policy for behaving, and 3) the outcome that triggers the time at which the predicted event is to occur. Of course the policy need only be followed from the time the prediction is made until the outcome triggering event. Actions taken after the trigger are irrelevant. [This notion of conditional prediction has been previously explored as the models of temporally extended actions, also known as "options" (Sutton, Precup, and Singh, 1999; Precup, thesis in preparation).

Let us return now to the claim with which I started, that much if not most mental activity is focused on such conditional predictions, on learning and computing them, on planning and reasoning with them. I would go so far as to propose that much if not most of our knowledge is represented in the form of such predictions, and that they are what philosophers refer to as "concepts". To properly argue these points would of course be a lengthy undertaking. For now let us just cover some high points, starting with some of the obvious advantages of conditional predictions for knowledge representation.

Foremost among these is just that predictions are grounded in the sense of having a clear, mechanically determinable meaning. The accuracy of any prediction can be determined just by running its policy from its state until an outcome occurs, then checking the prediction against the outcome. No human intervention is required to interpret the representation and establish the truth or falsness of any statement. The ability to compare predictions to actual events also make them suitable for beling learned automatically. The semantics of predictions also make it clear how they are to be used in automatic planning methods such as are commonly used with MDPs and SMDPs. In fact, the conditional predictions we have discussed here are of exactly the form needed for use in the Bellman equations at the heart of these methods.

A less obvious but just as important advantage of outcome-conditional predictions is that they can compactly express much that would otherwise be difficult and expensize to represent. This happens very often in commonsense knowledge; here we give a simple example. The knowledge we want to represent is that you can go to the street corner and a bus will come to take you home within an hour. What this means of course is that if it is now 12:00 then the bus might come at 12:10 and it might come at 12:20, etc., but it will definitely come by 1:00. Using outcome conditioning, the idea is easy to express: we either make the outcome reaching 1:00 and predict that the bus will have come by then, or we make the outcome the arrival of the bus and predict that at that time it will be 1:00 or earlier.

A natural but naive alternative way to try to represent this knowledge would be as a probability of the bus arriving in each time slot. Perhaps it has one-sixth chance of arriving in each 10-minute interval. This approach is unsatisfactory not just because it forces us to say more than we may know, but because it does not capture the important fact that the bus will come eventually. Formally, the problem here is that the events of the bus coming at different times are not independent. If may have only a one-sixth chance of coming exactly at 1:00, but if it is already 12:55 then it is in fact certain to come at 1:00. The naive representation does not capture this fact that is actually absolutely important to using this knowledge. A more complicated representation could capture all these dependencies but would be just that -- more complicated. The outcome-conditional form represents the fact simply and represents just what is needed to reason with the knowledge this way. Of course, other circumstances may require the more detailed knowledge, and this is not precluded by the outcome-conditional form. This form just permits greater flexibility, in particular, the ability to omit these details while still being of an appropriate form for planning and learning.

Subjective Knowledge

April 6, 2001

I would like to revive an old idea about the mind. This is the idea that the mind arises from, and is principally about, our sensori-motor interaction with the world. It is the idea that all our sense of the world, of space, objects, and other people, arises from our experience squeezed through the narrow channel of our sensation and action. This is a radical view, but in many ways an appealing one. It is radical because it says that experience is the only thing that we directly know, that all our sense of the material world is constructed to better explain our subjective experience. It is not just that the mental is made primary and held above the physical, but that the subjective is raised over the objective.

Subjectivity is the most distinctive aspect of this view of the mind, and inherent in it. If all of our understanding of the world arises from our experience, then it is inherently personal and specific to us.

As scientists and observers we are accustomed to prasing the objective and denigrating the subjective, so reversing this customary assessment requires some defense.

The approach that I am advocating might be termed the subjective viewpoint. In it, all knowledge and understanding arises out of an individual's experience, and in that sense is inherently in terms that are private, personal, and subjective. An individual might know, for example, that a certain action tends to be followed by a certain sensation, or that one sensation invariably follows another. But these are its sensations and its actions. There is no necessary relationship between them and the sensations and actions of another individual. To hypothesize such a link might be useful, but always secondary to the subjective experience itself.

The subjective view of knowledge and understanding might be constrasted with the objective, realist view. In this view there are such things as matter, physical objects, space and time, other people, etc. Things happen, and causally interact, largely independent of observers. Occasionally we experience something subjectively, but later determine that it did not really, objectively happen. For example, we felt the room get hot, but the thermometer registered no change. In this view there is a reality independent of our experience. This would be easy to deny if there were only one agent in the world. In that case it is clear that that agent is merely inventing things to explain its experience. The objective view gains much of its force because it can be shared by different people. In science, this is almost the definition of the subjective/objective distinction: that which is private to one person is subjective whereas that which can be observed by many, and replicated by others, is objective.

I hasten to say that the subjective view does not deny the existence of the physical world. The conventional physical world is still the best hypothesis for explaining our subjective data. It is just that that world is held as secondary to the data that it is used to explain. And a little more: it is that the physical world hypothesis is just that, a hypothesis, an explanation. There are not two kinds of things, the mental and the physical. There are just mental things: the data of subjective experience and hypotheses constructed to explain it.

The appeal of the subjective view is that it is grounded. Subjective experience can be viewed as data in need of explanation. There is a sense in which only the subjective is clear and unambiguous. "Whatever it means, I definitely felt warm in that room." No one can argue with our subjective experience, only with its explanation and relationship to other experiences that we have or might have. The closer the subjective is inspected, the firmer and less interpreted it appears, the more is becomes like data, whereas the objective often becomes vaguer and more complex. Consider the old saw about the person who saw red whenever everybody else saw green, and vice versa, but didn't realize it because he used the words "red" and "green" the wrong way around as well. This nonsense points out that different people's subjective experiences are not comparable. The experience that I call seeing red and the experience you call seeing red are related only in a very complicated way including, for example, effects of lighting, reflectance, viewpoint, and colored glasses. We have learned to use the same word to capture an important aspect of our separate experience, but ultimately the objective must bow to the subjective.

The appeal of the objective view is that it is common across people. Something is objectively true if it predicts the outcome of experiments that you and I both can do and get the same answer. But how is this sensible? How can we get the same answer when you see with your eyes and I with mine? For that matter, how can we do the "same" experiment? All these are problematic and require extensive theories about what is the same and what is different. In particular, they require calibration of our senses with each other. It is not just a question of us using the same words for the same things -- the red/green example shows the folly of that kind of thinking -- it is that there is no satisfactory notion of same things, across individuals, at the level of experience. Subjective experience as the ultimate data is clear, but not the idea that it can be objectively compared across persons. That idea can be made to work, approximately, but should be seen as following from the primacy of subjective experience.

At this point, you are probably wondering why I am belaboring this philosphical point. The reason is that the issue comes up, again and again, that it is difficult to avoid the pitfalls associated with the objective view without explicitly identifying them. This fate has befallen AI researchers many times in the past. So let us close with as clear a statement as we can of the implications of the subjective view for approaches to AI. What must be avoided, and what sought, in developing a subjective view of knowledge and mind?

All knowledge must be expressed in terms that are ultimately subjective, that are expressed in terms of the data of experience, of sensation and action. Thus we seek ways of clearly expressing all kinds of human knowledge in subjective terms. This is a program usually associated with the term "associationism" and often denigrated. Perhaps it is impossible, but it should be tried, and it is difficult to disprove, like a null hypothesis. In addition to expressing knowledge subjectively, we should also look to ways of learning and working with subjective knowledge. How can we reason with subjective knowledge to obtain more knowledge? How can it be tested, verified, and learned? How can goals be expressed in subjective terms?

Notes:

  • McCarthy quote.
  • Relate to logical positivism.
  • Then Dyna as a simple example, and which highlights what is missing.

Fourteen Declarative Principles of Experience-Oriented Intelligence

  1. all goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a single externally received number (reward). “the reward hypothesis” thus life is a sequential decision-making problem, also known as a Markov decision process. “learning is adaptive optimal control”
  2. a major thing that the mind does is learn a state representation and a process for updating it on a moment-by-moment basis. the input to the update process is the current sensation, action, and state (representation). “state is constructed”
  3. all action is taken at the shortest possible time scale, by a reactive, moment-by-moment policy function mapping from state to action. anything higher or at longer time scales is for thinking about action, not for taking it. “all behavior is reactive”
  4. all efficient methods for solving sequential decision-making problems compute, as an intermediate step, an estimate for each state of the long-term cumulative reward that follows that state (a value function). subgoals are high-value states. “values are more important than rewards”
  5. a major thing that the mind does is learn a predictive model of the worldʼs dynamics at multiple time scales. this model is used to anticipate the outcome (consequences) of different ways of behavior, and then learn from them as if they had actually happened (planning).
  6. learning and planning are fundamentally the same process, operating in the one case on real experience, and in the other on simulated experience from a predictive model of the world. “thought is learning from imagined experience”
  7. all world knowledge can be well thought of as predictions of experience. “knowledge is prediction” in particular, all knowledge can be thought of as predictions of the outcomes of temporally extended ways of behaving, that is, policies with termination conditions, also known as “options.” these outcomes can be abstract state representations if those in turn are predictions of experience.
  8. state representations, like all knowledge, should be tied to experience as much as possible. thus, the bayesian and POMDP conceptions of state estimation are mistaken.
  9. temporal-difference learning is not just for rewards, but for learning about everything, for all world knowledge. any moment-by-moment signal (e.g., a sensation or a state variable) can substitute for the reward in a temporal-difference error. “TD learning is not just for rewards”
  10. learning is continual, with the same processes operating at every moment, with only the content changing at different times and different levels of abstraction. “the one learning algorithm”
  11. evidence adds and subtracts to get an overall prediction or action tendency. thus policy and prediction functions can be primarily linear in the state representation, with learning restricted to the linear parameters. this is possible because the state representation contains many state variables other than predictions and that are linearly independent of each other. these include immediate non-linear functions of the other state variables as well as variables with their own dynamics (e.g., to create internal “micro-stimuli”).
  12. a major thing that the mind does is to sculpt and manage its state representation. it discovers a) options and option models that induce useful abstract state variables and predictive world models, and b) useful non-linear, non-predictive state variables. it continually assesses all state variables for utility, relevance, and the extent to which they generalize. researching the process of discovery is difficult outside of the context of a complete agent.
  13. learning itself is intrinsically rewarding. the tradeoff between exploration and exploitation always comes down to “learning feels good.”
  14. options are not data structures, and are not executed. they may exist only as abstractions.

some of these principles are stated in radical, absolutist, and reductionist terms. this is as it should be. in some cases, softer versions of the principles (for example, removing the word “all”) are still interesting. moreover, the words “is” and “are” in the principles are a shorthand and simplification. they should be interpreted in the sense of Marrʼs “levels of explanation of a complex information-processing system.” that is, “is” can be read as “is well thought of as” or “insight can be gained by thinking of it as.”

a complete agent can be obtained from just two processes:

  • a moment-by-moment state-update process, and
  • a moment-by-moment action selection policy.

everything else has an effect only by changing these two. a lot can be done purely by learning processes (operating uniformly as in principle 10), before introducing planning. this can be done in the following stages:

  • (a) a policy and value function can be learned by conventional model-free reinforcement learning using the current state variables
  • (b) state variables with a predictive interpretation can learn to become more accurate predictors
  • (c) discovery processes can operate to find more useful predictive and non-predictive state variables
  • (d) prediction of outcomes, together with fast learning, can produce a simple form of foresight and behavior controlled by anticipated consequences

much of the learning above constitutes learning a predictive world model, but it is not yet planning. planning requires learning from anticipated experience at states other than the current one. the agent must disassociate himself from the current state and imagine absent others.

The Definition of Intelligence

July 9, 2016

John McCarthy long ago gave one of the best definitions: "Intelligence is the computational part of the ability to achieve goals in the world”. That is pretty straightforward and does not require a lot of explanation. It also allows for intelligence to be a matter of degree, and for intelligence to be of several varieties, which is as it should be. Thus a person, a thermostat, a chess-playing program, and a corporation all achieve goals to various degrees and in various senses. For those looking for some ultimate ‘true intelligence’, the lack of an absolute, binary definition is disappointing, but that is also as it should be.

The part that might benefit from explanation is what it means to achieve goals. What does it mean to have a goal? How can I tell if a system really has a goal rather than seems to? These questions seem deep and confusing until you realize that a system having a goal or not, despite the language, is not really a property of the system itself. It is in the relationship between the system and an observer. (In Dennett's words, it is a ‘stance’ that the observer take with respect to the system.)

What is it in the relationship between the system and the observer that makes it a goal-seeking system? It is that the system is most usefully understood (predicted, controlled) in terms of its outcomes rather than its mechanisms. Thus, for a home-owner a thermostat is most usefully understood in terms of its keeping the temperature constant, as achieving that outcome, as having that goal. But if i am an engineer designing a thermostat, or a repairman fixing one, then i need to understand it at a mechanistic level—and thus it does not have a goal. The thermostat does or does not have a goal depending of the observer. Another example is the person playing the chess computer. If I am a naive person, and a weaker player, I can best understand the computer as having the goal of beating me, of checkmating my king. But if I wrote the chess program (and it does not look very deep) I have a mechanistic way of understanding it that may be more useful for predicting and controlling it (and beating it).

Putting these two together, we can define intelligence concisely (though without much hope of being genuinely understood without further explanation):

Intelligence is the computational part of the ability to achieve goals. A goal achieving system is one that is more usefully understood in terms of outcomes than in terms of mechanisms.

The Bitter Lesson

Source March 13, 2019

The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore's law, or rather its generalization of continued exponentially falling cost per unit of computation. Most AI research has been conducted as if the computation available to the agent were constant (in which case leveraging human knowledge would be one of the only ways to improve performance) but, over a slightly longer time than a typical research project, massively more computation inevitably becomes available. Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation. These two need not run counter to each other, but in practice they tend to. Time spent on one is time not spent on the other. There are psychological commitments to investment in one approach or the other. And the human-knowledge approach tends to complicate methods in ways that make them less suited to taking advantage of general methods leveraging computation. There were many examples of AI researchers' belated learning of this bitter lesson, and it is instructive to review some of the most prominent.

In computer chess, the methods that defeated the world champion, Kasparov, in 1997, were based on massive, deep search. At the time, this was looked upon with dismay by the majority of computer-chess researchers who had pursued methods that leveraged human understanding of the special structure of chess. When a simpler, search-based approach with special hardware and software proved vastly more effective, these human-knowledge-based chess researchers were not good losers. They said that "brute force" search may have won this time, but it was not a general strategy, and anyway it was not how people played chess. These researchers wanted methods based on human input to win and were disappointed when they did not.

A similar pattern of research progress was seen in computer Go, only delayed by a further 20 years. Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale. Also important was the use of learning by self play to learn a value function (as it was in many other games and even in chess, although learning did not play a big role in the 1997 program that first beat a world champion). Learning by self play, and learning in general, is like search in that it enables massive computation to be brought to bear. Search and learning are the two most important classes of techniques for utilizing massive amounts of computation in AI research. In computer Go, as in computer chess, researchers' initial effort was directed towards utilizing human understanding (so that less search was needed) and only much later was much greater success had by embracing search and learning.

In speech recognition, there was an early competition, sponsored by DARPA, in the 1970s. Entrants included a host of special methods that took advantage of human knowledge---knowledge of words, of phonemes, of the human vocal tract, etc. On the other side were newer methods that were more statistical in nature and did much more computation, based on hidden Markov models (HMMs). Again, the statistical methods won out over the human-knowledge-based methods. This led to a major change in all of natural language processing, gradually over decades, where statistics and computation came to dominate the field. The recent rise of deep learning in speech recognition is the most recent step in this consistent direction. Deep learning methods rely even less on human knowledge, and use even more computation, together with learning on huge training sets, to produce dramatically better speech recognition systems. As in the games, researchers always tried to make systems that worked the way the researchers thought their own minds worked---they tried to put that knowledge in their systems---but it proved ultimately counterproductive, and a colossal waste of researcher's time, when, through Moore's law, massive computation became available and a means was found to put it to good use.

In computer vision, there has been a similar pattern. Early methods conceived of vision as searching for edges, or generalized cylinders, or in terms of SIFT features. But today all this is discarded. Modern deep-learning neural networks use only the notions of convolution and certain kinds of invariances, and perform much better.

This is a big lesson. As a field, we still have not thoroughly learned it, as we are continuing to make the same kind of mistakes. To see this, and to effectively resist it, we have to understand the appeal of these mistakes. We have to learn the bitter lesson that building in how we think we think does not work in the long run. The bitter lesson is based on the historical observations that 1) AI researchers have often tried to build knowledge into their agents, 2) this always helps in the short term, and is personally satisfying to the researcher, but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.

One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.

The second general point to be learned from the bitter lesson is that the actual contents of minds are tremendously, irredeemably complex; we should stop trying to find simple ways to think about the contents of minds, such as simple ways to think about space, objects, multiple agents, or symmetries. All these are part of the arbitrary, intrinsically-complex, outside world. They are not what should be built in, as their complexity is endless; instead we should build in only the meta-methods that can find and capture this arbitrary complexity. Essential to these methods is that they can find good approximations, but the search for them should be by our methods, not by us. We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done.