Should We Give Credit to Kurzweil for Predicting AGI in 1999?

One argument I hear a lot nowadays is that we should defer to people who work in the AI industry on anything related to AI, not just how quickly model capabilities will improve but also the issue of AI-safety and sometimes even the broader impact AI will have on society, on the ground that they supposedly predicted AI, whatever that means exactly. I recently wrote a tweet in response to David Shor, who made a version of that argument, where I briefly explained why I disagree:

I have more to say on the topic on whether the predictive track-record of people in the industry, such as it is, is a good reason to defer to them on AI-related issues outside of the relatively narrow questions on which at least some of them have a proven track-record, but in this post I want to focus on another question.

In response to my tweet, several people brought up the argument Ray Kurzweil made in 1999, which predicted that we’d develop artificial human-like intelligence within 30 years.¹ They claimed that it showed that it wasn’t true that anyone who predicted that AGI in 1999 would happen around now had to have done so for bad reasons. However, not only do I think they are wrong, but I think Kurzweil’s argument is a perfect illustration of precisely the kind of sloppy thinking I was talking about. His argument roughly goes like this. First, using a simple model of the human brain (i. e. it contains 100 billion neurons, each of which has 1,000 connections on average and it’s capable of performing 200 calculations per second per connection), he estimates how much computational power it has. He then observes that the amount of computational power $1,000 can buy has been increasing more or less at a constant rate for decades and, by assuming that it will continue to be the case for a few decades, he infers that a $1,000 computer should have approximately the same amount of computational power as the human brain around 2020. Finally, once we have computers with as much computational power as the human brain, we’ll be able to harness that power to produce human-like intelligence in relatively short order, which Kurzweil thought would most likely happen by reverse-engineering the human brain and emulating it with a computer.

There is a whole family of related arguments, which vary in their details, but share the same basic structure:

1. A model M of the human brain captures all the biology that underlies the functions that constitute human-like intelligence and M implies that the brain has a computational power of x FLOPS.²

2. The computational power available for $1,000 or another relatively small amount of money has grown at a rate that is approximately constant and it will continue to grow more or less at this rate for several decades.

3. It follows that, by such and such a date (give or take a few years), anyone will be able to buy a computer with as much computational power as the human brain for a price similar to that of common durable consumer goods.

4. Since there is some kind of relationship between available computational power and the kind of algorithms we can implement, once we reach that point, it probably won’t take long before we find a way to use that computational power to produce artificial human-like intelligence.

This is clearly the structure of Kurzweil’s argument, but many other people have made a similar argument. For instance, Hans Moravec made a version of that argument even earlier than Kurzweil and also thought that $1,000 computers would reach the computational power of the human brain sometime in the 2020s, although he was more cautious and refrained from making a prediction about when exactly that would translate into artificial human-like intelligence. I think it’s clear he thought it wouldn’t take very long though.

This kind of argument is, to be blunt, worthless. First, the fact that M implies that the human brain has a computational power of x FLOPS doesn’t mean that the human brain actually has a computational power of x FLOPS, because that’s only true if M captures all the biology that underlies the functions that constitute human-like intelligence. Unless we can be confident that it’s the case, there is no reason to be confident that the estimate of the human brain’s computational power implied by M is even in the right ballpark. However, we still have such a poor understanding of how the human brain works that I don’t see how we could be confident that any model captures the relevant biology even approximately, so we have no reason to think that the estimate of the human brain’s computational power that can be derived from M is anywhere near the computational power required to emulate the human brain. For instance, it’s not clear from his back-of-the-envelope calculation what model of the human brain Kurzweil implicitly assumes to estimate its computational power (he just says that it’s a network of 100 billion neurons with 1,000 connections each that perform 200 calculations per second per connection), but whatever he had in mind exactly I don’t think anyone seriously believes it’s even remotely adequate as a model of the brain computational power.

Moravec uses a different method, but it’s just as inadequate, if not more. He focuses on the retina, assumes that its only functions are edge and motion detection, considers how much computational power is required for a computer program to perform the same functions and, using the ratio between the number of neurons in the entire human brain to that in the retina, scales that figure to estimate the computational power of the brain as a whole. It’s not difficult to see how this method could go wrong. For instance, the retina probably has other functions besides edge and motion detection, some of which may require considerably more computational power. Similarly, even if we assume that Moravec’s estimate of the computational power of the retina was approximately correct, it may not be representative of the rest of the brain. Of course, this method could actually overestimate the brain’s computational power, not just underestimate it. For instance, the computer programs Moravec used to estimate how much computational power was required to perform edge and motion detection were almost certainly not maximally efficient, so to the extent that the retina’s functions can be reduced to edge and motion detection and that it’s representative of the computational power of the rest of the brain Moravec’s back-of-the-envelope calculation would overestimate the computational power of the human brain.

The point is that, given how rudimentary our understanding of how the human brain works still is, it’s completely irrational to think that we are in a position to estimate how much computational power it has even very approximately. Back in 2020, someone at Coefficient Giving, then known as Open Philanthropy, published a paper where he performed various back-of-the-envelope calculations of that sort and found a range of estimates that span several orders of magnitude:

If we assume that by calculation he meant a floating-point operation, Kurzweil’s estimate of the human brain’s computational power is approximately equal to 10^16 FLOPS, so it’s about in the middle of the range of estimates in that paper, but to be clear I’m sure that you could make a case for much higher estimates. Despite what he assumed, even in 2026, 10^16 FLOPS is far more than what you can buy for $1,000. It depends on exactly what type of calculation we’re interested in and the level of precision, but for that price, it seems that you can get about 5 x 10^13 FLOPS today. If we assume that the computational power that you can get for $1,000 doubles every couple of years, this means that if we accept Kurzweil’s estimate, we are still more than 17 years away from the point he thought we were going to reach in 2020. If we assume that he underestimated the brain’s computational power by 5 orders of magnitude, putting it at 10^21 FLOPS, it means that we’re still almost 50 years away from it and that Kurzweil was off by about 60 years. To be clear, I have no idea what the human brain’s computational power is, but neither does anybody else and that’s the point here.

However, that’s not even the most problematic aspect of this kind of argument, which is rather that even if we could know with a high degree of confidence when a chip you can buy for $1,000 will buy you the same computational power as the human brain it wouldn’t be particularly informative about when AGI will happen. First, the fact that we have enough computational power to emulate the brain in practice doesn’t mean that it will be easy or quick to harness that computational power to produce human-like intelligence, because we still need to figure out a way to create the software to do it. Kurzweil understood that but he thought it would be easy because, as he explains in his essay, he was convinced that by now we’d have figured out how the human brain works in exquisite detail thanks to progress in scanning technology, nano-robotics, etc. But nothing even remotely like the kind of progress he imagined happened and we still have a very poor understanding of the human brain, so even if we had enough computational power to emulate it, we couldn’t produce human-like intelligence in that way.³ What this means is that we have to find some other strategy to produce it, which is what we’re currently doing to be clear (some aspects of the current paradigm are inspired by how we think the human brain works but for the most part we aren’t trying to figure out how the human brain works to create something that works in the same way), but I see no reason to think that, once we have enough computational power to produce human-like intelligence in principle, we’ll be able to do it easily or quickly.

It’s a bit like saying that, once you have all the ingredients necessary to make a specific dish, you will quickly figure out what the recipe is and be able to make it.⁴ Even if you have tasted the dish before, it may be very difficult and take a long time to figure out how to combine the ingredients to make it, so merely knowing that you have all the ingredients doesn’t tell you much. Here it’s important to distinguish the question of how much computational power is necessary to produce human-like intelligence from the question of how much compute will have to be expended to come up with a way to harness computational power to produce human-like intelligence.⁵ The evolutionary process that produced the human brain can be seen as a massive computation that was implemented through natural selection, mutation, etc. over billions of years. As a search procedure, it was almost certainly extremely inefficient, because it’s not a teleological process. In other words, evolution wasn’t trying to create human-like intelligence, it just stumbled on it after a few billion years of natural selection, mutation, etc. By contrast, we are going about creating human-like intelligence in a deliberate way, so our search procedure will almost certainly be far more efficient and much faster. But this doesn’t mean that it will be easy or particularly fast, because “faster than a few billion years” is a pretty low bar.

Even if we accept that our search procedure will become more efficient as we have access to more and more computational power, which seems reasonable because in practice the search problem is decomposed into several subproblems, progress on some of them can facilitate progress on some of the others and even holding the efficiency of the search procedure we use to solve one of them having more computational power makes it easier to solve it by brute force, this by itself doesn’t really us much about how long it will take to solve the problem. It will also depend on how much computational power the solution we are likely to stumble on first requires. As we have seen above, this is not the same thing as the computational power required to emulate the human brain, but it’s also not the same thing as the computational power necessary to generate human-like intelligence regardless of the particular way in which it’s done. And neither of those is the same thing as the computational power we, the apes who are trying to artificially produce human-like intelligence, are likely to need to do it.

It could be that, although evolution was a very inefficient search procedure, it produced a very efficient implementation of human-like intelligence. In that case, since we are unlikely to understand the human brain well enough for many decades (at least without the help of AGI), we’re either going to have to wait until then or we’ll have to find another way to do it and it will require more computational power than the human brain, which means that it will take longer than Kurzweil thought even if his estimate of the human brain’s computational power were approximately correct. Either way it will take longer than he thought to artificially produce human-like intelligence. It could also be that, even if the human brain is a very inefficient way to produce human-like intelligence and in principle it’s possible to come up with a much more efficient implementation, the implementations we’re likely to come up with at first will be even more inefficient than the human brain. In that case, even if Kurzweil’s estimate of the brain’s computational power were in the right ballpark, we could still be a long way from the kind of computational power we’ll need in practice to produce human-like intelligence because we’re going to do it in a way that will require a lot more computational power than that. Of course it could also be that, regardless of how accurate Kurzweil’s estimate of the human brain’s computational power is, not only is there a way to produce human-like intelligence with the kind of computational power we already have or that we’ll soon have.

The point is that we just don’t know and nothing in Kurzweil’s argument or, as far as I know, in the variants of that argument that other people have proposed does anything to explain why AGI should be developed quickly once computers with the same computational power as the human brain become relatively cheap. Yet many AGI-pilled people seem to find this kind of argument compelling, which I find extremely frustrating because it’s obviously a terrible argument. Of course, that is not to say that we don’t have good reasons today to think that AGI will come relatively soon, but that’s a different question. Unlike in 1999, we now have models that, while not quite AGI yet, come much closer than what anyone would have predicted even not so long ago. But this happened because scaling laws, which were only discovered less than 10 years ago, have continued to hold up pretty well. It’s true that most of the ingredients except compute, as well as new architectures like transformer and diffusion models, already existed in 1999, which superficially can be seen as vindicating Kurzweil. However, nobody could have predicted scaling laws back then, let alone known they would hold up so well, because they are empirical laws that were only discovered because people trained a lot of model and examined the relationship between model complexity, data size and compute.⁶ In fact, even today more than a decade into the deep learning revolution, we still have a very poor understanding of why it works so well. Not only should this make people wary of assuming that it could have been predicted 30 years ago, let alone that it was predicted, but it should also make them less confident about what’s going to happen next.

1. In this post, when I say that something exhibits human-like intelligence, what I mean is that, for any stimuli of the relevant sort, there could be a normal adult human with a similar distribution of cognitive responses. This definition is vague and would require further elaboration, but for the purposes of this post, it will be enough.

2. More sophisticated versions of the argument also try to use M to estimate other relevant characteristics of the brain, such as memory, bandwidth, etc., but let’s keep things simple to capture the essence of this type of argument.

3. Despite the fact that its nervous system only contains 302 neurons and that its full connectome was mapped decades ago, we still can’t reliably simulate the behavior of the C. elegans worm, because there is a lot more to figuring out how even such a simple nervous system works than just knowing which neurons are connected to which. Unless AGI can help us with that, but in that case it obviously won’t have been developed by emulating the human brain, we are probably decades away from being able to do that for the human brain even in the most optimistic scenario.

4. The credit for this analogy should go to my wife, who came up with it the other night when I was complaining to her about Kurzweil’s argument instead of letting her sleep.

5. As I use those terms, computational power is a rate, it’s the number of operations that a device can perform in a unit of time, whereas compute is a number of operations that were performed regardless of how long it took.

6. To be clear, I also think that people are overconfident that scaling laws and the possibility of recursive self-improvement mean that AGI is right around the corner and that the arguments they make on the topic aren’t as good as they think, but that’s a story for another time.