NVIDIA H100 vs. A100 - Speed, Memory, Brain, Precision, Second Life

Virtual cage match time between the NVIDIA A100 GPU, likely the GPU you are using when you use OpenAI GPT LLMs - and the H100. Which is better, and which is closer to the kind of processing power of the human mind?

As you may recall from my previous blog article - one A100 GPU (really a card filled with chips rather than a single integrated ("chip") has a whopping 80 GB of RAM, and is capable of about 300 TFLOPS TeraFLOPS) = 3 x 10^`14 FLOPS - of 16 bit floating point computations.  

Note that 16 bit floating point is a quite short and imprecise format but sufficient for the kind of optimization needed for LLMs.  If you want 32 bit ("single") floating point precision you drop the computational throughput of the A100 to 150 TFLOPS, simply because you are processing twice the number of bits. A typical pocket calculator is showing you 8 or 10 digits which is between 32 and 64 bit floating point precision, and most CPUs in PCs these days have FPUs that are intrinsically doing 64 bit floating point artithmetic.  For certain jobs, the additional precision of 64 bit floating point arithmetic is essential, but ironically, for simulating "thinking" 16 bits is good enough!

For a single H100 card, the memory is again 80 GB, but the computational throughput is really amazing - it is nearly 2000 TFLOPS = 2 x 10^15 FLOPS (!).  If you give up more precision and go down to 8-bit floating point calculations - really imprecise but again good enough for GPT LLMs - you can run at up to 4000 TFLOPS!

My prior post estimated the information storage capacity of a human brain as around 10^15 B = 1000 TB, or roughly 100 times larger than one of these GPU packages.  Of course you could put 100 of them in a single computer (maybe) and get your memory size that way.

My rather generous estimate of the maximum computational power of the human brain was around 10^17 FLOPS.  This is rather neatly what you would have with 100 H100s, each of which is able to do about 10^15 FLOPS.  So, 100 H100s in a box (with a pile of CPUs) is apparently comparable memory capacity and computational throughput to match a single human brain.

Bottom line: 100 H100s = 1 human brain.

The huge challenge would be to train that big network - is Dr. Chandra in the house?

Well now that we're done with that, what about the Zuck-Musk cage match?

Floating-Point Formats, Range, Precision, Seams in Second Life, and Neurons

I have been writing computer software for many years and I have almost always used floating-point numbers which are in 4-byte (32 bit or F32) or 8-byte (64 bit or F64) formats.  The standard F32 single precision format has one sign bit, eight exponent bits, and 23 bits for the mantissa, using binary scientific notation.  the eight exponent bits give you exponents from -128 to +127, and the 23 bits of mantissa give about   23 * ln(2) / ln(10) = 6.9 decimal digits of precision (nearly 1/10,000,000 = 10^-7 precision, a bit less than most pocket calculators which give you 8 digits).   The 8 bit exponent allows numbers of scale from about 2^-128 = 3 x 10^-38 to 2^127 = 10^38 to be represented (again, somewhat less than pocket calculations that generally give a range from 10^-100 to 10^100).

While the exponent is good enough for most applications, the mantissa does not give a lot of precision as you can see from the following example. Second Life uses single-point arithmetic to describe positions of objects.  This has some severe consequences - notably, avatars are basically forbidden to build things above about 4000 m, because projecting an object built to single precision 4000 m in the air magnifies an error of one part in 10^7 (roundoff error) at zero elevation to one part in 1000 at 10000 m   (10^-7 * 10,000 m / 0.001 m = 10^-3).  So the parts of a 1 m object (like an avatar) with no seams at 0 m elevation will start to show 1 mm gaps up at 10,000 m!  This is enough to notice, so there is a kind of height limit in Second Life of about 4000 m.

F64 double precision, or 8 byte floating point, is by comparison one sign bit, 11 exponent bits, and 52 mantissa bits.  A lot more range - from 2^-2048 = 10^-616 to 2^2047 = 10^616 - and a lot more precision, 52 * ln(2)/ln(10) = 15.65, or nearly 16 decimal digits.  Hence the name "double precision". Modern PCs (using typical Intel or AMD CPUs) use FPUs that are intrinsically F64.  There are also F128 formats (16 byte floating point) that are defined.

And if Second Life used 64 bit coordinates for objects?  You would be able to go up to 10^599 m without seeing seams beween your body prims!  Remember that our universe is around 10^10 light years (based on time since big bang) which is around 10^26 m in diameter - so we could travel to distant stars in Second Life if we went to 64 bits!  Well actually we could fit 10^599/10^26 = 10^573 of our universes into Second Life - now that's a metaverse!

What is kind of cute is that now, in our high-tech era, to boost GPU throughput, F16 or "half-precision" and even F8 formats have been defined.  F16 is one sign bit, five exponent bits, and 10 mantissa bits.  This gives a range of around 2^-32 = 0.001 to 2^32 = 1000 and mantissa precision of 10 * ln(2) / ln(10) = 3.01 decimal digits.  This is low precision but appropriate for certain jobs like image processing (plenty of gray levels there) and neural network computations.

The extreme is F8 which I guess should be called "quarter precision" but seems to be also called "minifloat".  This format is not very clearly defined and not widely used but one example format I've seen talked about is one sign bit, three exponent bits, and four mantissa bits.  Keep in mind this format can only describe 2^8 = 256 different values - it's really crude!.  With this format you have a range from 2^-16 to 2^15 (1/65536 = 1.6 x 10^-5 to 32768) and a precision of  4*ln(3)/ln(10) = 0.9 decimal digits.  Bad! But the H100 GPU supports this type of F8 precision, because optimization of 10^14 network parameters just doesn't need a lot of precision.  

Put another way, a real neuron in your brain is a cell - a biological object that probably has a reproducibility of its response (from neuron to neuron) or temporal variability (depending on how much glucose and other stuff is in your blood) of at best a few percent, and a total range of perhaps a few thousand, in other words, not much better than F8 range and precision.  The precision in our thinking is not dependent on individual neurons being precise calculators, but is instead dependent on how the network as a whole is wired, whether two neurons are connected together or not.