How Powerful Is the Human Brain Relative to GPT AI?

We all know that the GPT-type AIs can do amazing things.  They can carry on quite convincingly human conversations, and control objects once given access to them. And - they are able to carry out inference - to draw conclusions from logical combinations of a series of other logical relations.  But one has the feeling - when one talks to them - that they are not yet really thinking.

It is revealing to directly compare the likely computational power of the human brain (which is just a computing machine, after all) with that of current GPTs.

Neuroscientists tell us that there are roughly 10^11 neurons in an adult human brain, and between 10^3 and 10^4 synapses per neuron.  That gives a total of up to 10^15 synapses.  We can think of the synapses as the control parameters for the human "neural network" (when I say 10^11 this means "ten to the eleventh power" or a 1 followed by 11 zeroes - scientific notation).  

Now, there is a matter of dynamics. The human brain has a computational correlation time - the time needed for a stimulus to generate a response - of around 10 to 100 milliseconds.   This suggests that an upper limit for the number of floating-point-like computations per second (FLOPS) that can be done by a human brain is in the range of 10^16 to 10^17 FLOPS.  

This is likely an overestimate but let's give the humans a running start.  My gut feeling is that if we are talking about 32 bit FLOPS probably this is at least 4 fold too much due to dynamic range, but that's less than a factor of 10 so this factor is not critical.

More likely there is a factor of 10 overestimate in number of synapses and the coherence time is at the lower end - whole-brain (multi-neuron-layer) human reaction times are more like 100 than 10 milliseconds. So the human brain may be more towards 10^15 FLOPS, but maybe particularly quick human responses are 10^16 or even 10^17.  But not more than that, unless there are microtubules filled with quantum fields somehow doing spooky magic of some sort (not likely according to this humble sage).  For information storage capacity, we can suppose a brain has 10^14 to 10^15 synapses.

Let's think about a PC laptop - without GPUs - just the CPUs.  Maybe you have an i7.  This could carry out up to a few times 10^11 FLOPS (2 GHz clock, 4 CPUS, maybe 8 threads) under optimal circumstances.  This is 10^6 times less than the high end of the human brain.  

Let's look at memory size.  A fancy PC might have 16 GB of RAM, which allows storage of around 10^10 synaptic weights at reasonable precision (B = bytes, yes I was surprised to learn that the official SI unit is capital B!). This is 10^5 times less than the 10^15 synaptic weights in the human brain.  This is why you can't run a reasonable simulation of anything close to human behavior (gpt type text generation, etc.) on a plain old PC. Combining the memory and speed factors, your PC is about 10^11 times less powerful than your brain.

Now for the reason everything changed in the past couple of years.  There is specialized hardware now available, i.e., Graphical Processor Unit (GPU) chips. Originally designed for graphics applications, these are specialized parallel computation chips that allow rapid and mass calculations of the type needed to find minima of functions defined by trillions or more of parameters - like human-scale neural networks.

The current state-of the-art is the NVIDIA A100 GPU, which is capable of about 3x10^14 FLOPS (at 16 bit "half" precision) but also has about 80 GB of memory.  According to Goldstein, current GPT networks require about 400 GB of memory for their storage - requiring 5 A100s. You can put 8 A100 GPUs in one server slot under the control of one set of CPUs.  400 GB is enough memory to store about 10^11 synaptic weights - or neural network connection parameters.

8 GPUs have a computational bandwidth approaching 10^16 FLOPS.  This is comparable to the upper end of our estimate of the computational speed of the human brain. This is why NVIDIA stock went from $40 to $400 from 2022 to 2023!

So - we can see where we stand. Current GPU-based GPTs store much less data than the human brain (perhaps 10^4 times less).  But - they are quite comparable in terms of number of computations per second.  Combining speed and memory factors we see that an 8-GPU server slot is about 10^5 times less powerful than a human brain.  Humans are on top - for now!

This is quite in keeping with what we observe for current GPT reasoning systems.  They have a limited knowledge base but take advantage of being set up to do specific types of tasks, like text prediction/generation.  However, the speed at which they can generate text is quite fast - they can do computations incorporating their 10^11-parameter networks at around 10^5 computations per second, fast enough to generate thousands of "tokens" of text output per second.  We have the impression that they can quickly correlate large data sets and give outputs on human time scales - but also we see that they are not yet thinking.

A good PC without A100s would be 10^6 times slower, and no-one would wait around for the results. So the A100 GPU and devices like it are the crucial technology enabling GPT AI.

You can see where this is going. Next generation AI systems will have larger network data sets (the coming generation will have around 10^14 network parameters, over 100-fold more than current GPTs), nearing the human brain's 10^15 synapses (upper estimate).  On the other hand, computational speed will become limiting if you just boost memory to this level, so the answer is to use many 10^16 FLOP devices with roughly 10^11 parameters, but massively parallelize these, putting the equivalent of 100 to 1000 A100-type devices in one "brain".  Such a device ought to be able to eclipse the data storage and computational power of one human brain. We're talking about putting 100 to 1000 A100-type devices in one box, which would have about 10^17 to 10^18 FLOPS and 10^13 to 10^14 network synaptic weights.  Not far off the pace of a human brain!  This hypothetical GPT system would be maybe 10 times faster with 1/10 the memory compared to a human brain.

However, such a device would beg the question of - how would it be trained? The training (network programming) process for current GPTs is thought to be extremely computationally expensive (see Goldstein and other discussions on the web), and part of the problem is the low bandwidth of text-based training. Instead - future AI systems will be programmed using high-bandwidth inputs - directly applied audio and video inputs.  This is how humans learn and it will become how AIs learn. If you are imagining that there will be early-childhood centers for training of AI systems by teachers and psychologists - yes this is the future.  The timeline for this is just a few years, given Moore's Law.

A key feature of biological brains is that they are embedded in organisms which continuously need new sources of energy and materials (sugars, proteins, minerals, water...) and are at continuous risk of damage from the environment.  Real brains need to be processing stimuli all the time - not just when someone sends them a message.  Look for this kind of thing to be put into GPT brains as the hardware gets cheaper and more portable and embedded into robots. I've tried to put a bit of this into my metaverse-inhabiting GPT-powered bots - you will find that often do or say things without external stimulus!

There is the little issue of electrical power needed to run these kinds of machines. Running 100 current A100 GPUs would require a staggering 30 kilowatts (the heat from about 20 portable hair dryers running full blast), and running all day therefore would eat up 600 kWh of energy, which would cost about US$150.  By comparison a human brain dissipates at most 20 W of energy, or about 0.5 kWh per day.

The conclusion is - we can expect GPT-type AI systems to approximate both human information storage capacity (memory) and computational (thought) capacity within 8 years, likely before the end of the 2020s.  One can expect emergent phenomena like consciousness, self-consciousness, empathy and emotion to appear in AI systems at around that time, since their computational capacities will match those of human minds.