That exchange is quite a good prediction all things considered.
> Note that "minor" implementation issues like die space, routing, and gate delays, especially of 128-bit adders & shifters are non-trivial, so people aren't going to rush out and build 128-bitters for fun, just as people matched timing dates of their 64-bitters to their expected markets.
I think we're stuck with 64 bit for quite a while. The circuit size jump from 64 bit to 128 bit is significant. There's no fixed scalar or apples to apples comparison (that I'm aware of). Just for adders though, 2 bits requires 2 full adders, 4 bits requires 4 adders, 8 requires 8, etc.
Another way to look at this is that 16 bit gives addressable memory up to 65k, and quite a few programs had to deal with paging in architectures like the 8086/8088. 32 bits gave us up to 4GB addressable memory, and not it's now not uncommon that a program such as a web browser exceeds this. 64 bits would give us up to 18 exabytes of addressable memory. I'm not aware of any common programs breaking into the terabyte category (even in most research), let alone petabyte and then exabyte.
Even iterating over that many numbers becomes a large computational task. Just a quick test program:
// gcc -O3 count.c -o count
#include <stdint.h>
int main(){
uint64_t z = 0;
for(uint64_t i = 0; i < UINT64_MAX; i++) z += i;
return (int)(z % 2);
}
Using uint32_t and UINT32_MAX, it returns almost instantly. For uint64_t and UINT64_MAX you will be waiting a long time. 128 bit values? Even longer. Maybe many many cores could break that memory up, but then it makes sense to have a 64 bit system with some kind of ability to occasionally change page.
Dunno, a 128bit integer add is essentially 2x as expensive as a 64 bit one, a multiply is 4x. We've already had SSE in the early 2000s that could do computations like this in a single cycle (tho on multiple 32 bit numbers, not 128-bit ones).
I think on x86, we're still limited to 48 bits of address space on 64 bit systems, not sure if this has changed, but even 64 bits is so vast (16exabytes) , that you'd need a supercomputer with a single address space to fill it, but at least the latter is practically realizable.
> I'm not aware of any common programs breaking into the terabyte category (even in most research)
Wouldn't that be the commercial LLMs? ChatGPT, Claude etc are estimated to be in the 2-10 TB range, and a large portion of the population of developed countries are using those apps commonly. Not on their own systems, but if we're in the 1995 academic perspective, multiuser systems are assumed.
I imagine the strongest reason we won't need 128-bit memory addressing is because horizontal scaling is easier. If OpenAI had needed a single addressing plane to cover all their users, 128-bit might be required. Similar to how the internet hobbles along fine with 32-bit addressing by just adding a layer of indirection to the internet with NAT.
I guess the other example of 128-bit addressing is ZFS. What are the biggest ZFS file systems? And who has more data? S3? Once you get bigger than 64-bit you want to scale out horizontally anyway and not just put it all into one flat addressable plane.
> Wouldn't that be the commercial LLMs? ChatGPT, Claude etc are estimated to be in the 2-10 TB range, and a large portion of the population of developed countries are using those apps commonly. Not on their own systems, but if we're in the 1995 academic perspective, multiuser systems are assumed.
I think each part is generally treated like an individual large program - which makes sense because otherwise you have large parts of the model sitting there doing nothing. You want all of your silicon running hot, RAM sitting doing nothing for a period of time is a waste.
> If OpenAI had needed a single addressing plane to cover all their users, 128-bit might be required. Similar to how the internet hobbles along fine with 32-bit addressing by just adding a layer of indirection to the internet with NAT.
I think even then, 128 bit memory for OpenAI et al would be more of a headache than it is worth. Economically it's not worth building the 128 bit systems. The fact that servers and consumer CPUs have a lot of re-use within their designs reduces the costs for everybody. Sony for example stopped producing their 128 bit CPU [1], where it was mostly about pushing more data between CPUs and VPUs. Now we have specifications like PCIe where you can just adjust your pipe width to increase throughput.
> I guess the other example of 128-bit addressing is ZFS. What are the biggest ZFS file systems? And who has more data? S3? Once you get bigger than 64-bit you want to scale out horizontally anyway and not just put it all into one flat addressable plane.
Yeah exactly. The other thing to consider is that it doesn't matter how many bits wide you can go, the bottleneck is the smallest width. With ZFS, at some point you likely need to send it over a network, and I think fibre is up to 64 bits via parallel transceivers. If that points outwards, it'll probably be single bit by the time it gets to you.
So reflecting on that, I think the core assumption that didn't pan out is that the memory / CPU ratio will grow because we'll need more memory, hence requiring CPUs to address more than 16 EiB of data (16 EiB = 16384 PiB = 16777216 TiB of data, what 64 bits can address).
But in practice, we've produced a lot more compute. The memory / CPU ratio has increased, but most growth has been from more CPUs (and generally not shared-memory ones, but ones with their own completely separate memory space).
IPv6 is 128 bit, so we do have 128 bit ways of addressing computers, but I'd say we're still a long way from a CPU needing to address that much memory as a common case. The speed of light limits how far away memory can be from the CPU for good performance, so short of some drastically new memory technology, it seems unlikely we'll need it soon for any ordinary type of computing device.
Also arguably we already have 256 or 512 bit CPUs, what matters most for memory throughput is not the register width but essentialy the L1 cache line width (eg what in old CPUs was the databus width).
As for address width, we're not even close to get full 64 bit pointers from CPUs anyway, more like 48 or 52 bits.
Wide pointers (eg 128 bit general registers) would make sense for carrying capabilities for memory safety though I guess.
Personally I'm disappointed 128-bit floating point (quadruple precision) never properly made it into CPU's (sure it appeared in some niche ones here and there, but not in what we actually use today). After all, in the 1980's they had 80-bit ones, it's not even that far off, and they had millions times less transistors then.
While probably niche and applications that need higher precision using their own custom types anyway, they'd allow cool stuff like easy to program fractals with much higher detail than now. But also, less precision loss in many applications.
That's the thing, bigger datatypes means less effective memory throughput. From that perspective 16-bit or even 8-bit floats are often more useful than 80 or 128 bit floats. Same problem with 64 bit pointers and why it's often better to store narrower indices instead of full pointers, data can be packed more tightly and accessed more efficiently.
The point about large word lengths is you get higher data throughput, and faster processing, because everything is going through fat pipes into a big parallel machine with multiple levels of cache and vectorisation.
The issue is the utility of floats at different precisions. 128-bit floats have some benefits for high-end scientific applications, but the extra cost and complexity over 64-bit hardware would only make sense for specialised scientific supercomputing. So far it just hasn't been worth it.
The problem is for most practical uses where you need fast calculations the 64 bit precision is enough. For example there is hardly any physical calculation that would need more precision. 64 bit float can be used to measure the Earth-Sun distance to 30 micrometre precision. 128 bits does not just add anything generally useful. For monetary calculations you should be using decimals instead of binary anyway.
Doubles are precise to 1E-16, and the most precisely known fundamental constants have uncertainties on the 1E-12 level. This does not give a comfortable headroom, so the CODATA adjustment that determines the constants from measurements uses quadruple-precision numbers in FORTRAN.
In most cases where I needed higher precision, I just went to fixed point... There are also free libraries with arbitrary precision (although at a significant performance hit).
precision loss in floating point is often exponential.
If 64 bit isn't enough, then very quickly 128 is also not enough.
If precision is important then you will very often want systems that represent every number as an interval [a, b] meaning that the true value is between those 2 numbers. This makes you able to detect loss of precision due to e.g. d = a / (b - c) where b-c can result in a number close to zero which makes uncertainty grow. If you use this formula iteratively then precision is lost completely no matter how many bits there are in your floating point variables.
That exchange is quite a good prediction all things considered.
> Note that "minor" implementation issues like die space, routing, and gate delays, especially of 128-bit adders & shifters are non-trivial, so people aren't going to rush out and build 128-bitters for fun, just as people matched timing dates of their 64-bitters to their expected markets.
I think we're stuck with 64 bit for quite a while. The circuit size jump from 64 bit to 128 bit is significant. There's no fixed scalar or apples to apples comparison (that I'm aware of). Just for adders though, 2 bits requires 2 full adders, 4 bits requires 4 adders, 8 requires 8, etc.
Another way to look at this is that 16 bit gives addressable memory up to 65k, and quite a few programs had to deal with paging in architectures like the 8086/8088. 32 bits gave us up to 4GB addressable memory, and not it's now not uncommon that a program such as a web browser exceeds this. 64 bits would give us up to 18 exabytes of addressable memory. I'm not aware of any common programs breaking into the terabyte category (even in most research), let alone petabyte and then exabyte.
Even iterating over that many numbers becomes a large computational task. Just a quick test program:
Using uint32_t and UINT32_MAX, it returns almost instantly. For uint64_t and UINT64_MAX you will be waiting a long time. 128 bit values? Even longer. Maybe many many cores could break that memory up, but then it makes sense to have a 64 bit system with some kind of ability to occasionally change page.Dunno, a 128bit integer add is essentially 2x as expensive as a 64 bit one, a multiply is 4x. We've already had SSE in the early 2000s that could do computations like this in a single cycle (tho on multiple 32 bit numbers, not 128-bit ones).
I think on x86, we're still limited to 48 bits of address space on 64 bit systems, not sure if this has changed, but even 64 bits is so vast (16exabytes) , that you'd need a supercomputer with a single address space to fill it, but at least the latter is practically realizable.
> I'm not aware of any common programs breaking into the terabyte category (even in most research)
Wouldn't that be the commercial LLMs? ChatGPT, Claude etc are estimated to be in the 2-10 TB range, and a large portion of the population of developed countries are using those apps commonly. Not on their own systems, but if we're in the 1995 academic perspective, multiuser systems are assumed.
I imagine the strongest reason we won't need 128-bit memory addressing is because horizontal scaling is easier. If OpenAI had needed a single addressing plane to cover all their users, 128-bit might be required. Similar to how the internet hobbles along fine with 32-bit addressing by just adding a layer of indirection to the internet with NAT.
I guess the other example of 128-bit addressing is ZFS. What are the biggest ZFS file systems? And who has more data? S3? Once you get bigger than 64-bit you want to scale out horizontally anyway and not just put it all into one flat addressable plane.
> Wouldn't that be the commercial LLMs? ChatGPT, Claude etc are estimated to be in the 2-10 TB range, and a large portion of the population of developed countries are using those apps commonly. Not on their own systems, but if we're in the 1995 academic perspective, multiuser systems are assumed.
I think each part is generally treated like an individual large program - which makes sense because otherwise you have large parts of the model sitting there doing nothing. You want all of your silicon running hot, RAM sitting doing nothing for a period of time is a waste.
> If OpenAI had needed a single addressing plane to cover all their users, 128-bit might be required. Similar to how the internet hobbles along fine with 32-bit addressing by just adding a layer of indirection to the internet with NAT.
I think even then, 128 bit memory for OpenAI et al would be more of a headache than it is worth. Economically it's not worth building the 128 bit systems. The fact that servers and consumer CPUs have a lot of re-use within their designs reduces the costs for everybody. Sony for example stopped producing their 128 bit CPU [1], where it was mostly about pushing more data between CPUs and VPUs. Now we have specifications like PCIe where you can just adjust your pipe width to increase throughput.
> I guess the other example of 128-bit addressing is ZFS. What are the biggest ZFS file systems? And who has more data? S3? Once you get bigger than 64-bit you want to scale out horizontally anyway and not just put it all into one flat addressable plane.
Yeah exactly. The other thing to consider is that it doesn't matter how many bits wide you can go, the bottleneck is the smallest width. With ZFS, at some point you likely need to send it over a network, and I think fibre is up to 64 bits via parallel transceivers. If that points outwards, it'll probably be single bit by the time it gets to you.
[1] https://en.wikipedia.org/wiki/Emotion_Engine
So reflecting on that, I think the core assumption that didn't pan out is that the memory / CPU ratio will grow because we'll need more memory, hence requiring CPUs to address more than 16 EiB of data (16 EiB = 16384 PiB = 16777216 TiB of data, what 64 bits can address).
But in practice, we've produced a lot more compute. The memory / CPU ratio has increased, but most growth has been from more CPUs (and generally not shared-memory ones, but ones with their own completely separate memory space).
IPv6 is 128 bit, so we do have 128 bit ways of addressing computers, but I'd say we're still a long way from a CPU needing to address that much memory as a common case. The speed of light limits how far away memory can be from the CPU for good performance, so short of some drastically new memory technology, it seems unlikely we'll need it soon for any ordinary type of computing device.
the cases where its useful, it's in vectorized instruction sets, etc.
You're conflating 128-bit registers with 128-bit memory addressing. This article is about the latter.
Also arguably we already have 256 or 512 bit CPUs, what matters most for memory throughput is not the register width but essentialy the L1 cache line width (eg what in old CPUs was the databus width).
As for address width, we're not even close to get full 64 bit pointers from CPUs anyway, more like 48 or 52 bits.
Wide pointers (eg 128 bit general registers) would make sense for carrying capabilities for memory safety though I guess.
Because of LLMs we are back to memory being king (KV cache).
But there was another thing - clusters of thousands/millions of machine instead of one big iron with all the memory.
One interesting thing with wider addresses isn't necessarily increasing itself, but features built upon the expansion. CHERI is one example of that.
Yeah, or cryptographically secure non guessable addresses.
Personally I'm disappointed 128-bit floating point (quadruple precision) never properly made it into CPU's (sure it appeared in some niche ones here and there, but not in what we actually use today). After all, in the 1980's they had 80-bit ones, it's not even that far off, and they had millions times less transistors then.
While probably niche and applications that need higher precision using their own custom types anyway, they'd allow cool stuff like easy to program fractals with much higher detail than now. But also, less precision loss in many applications.
That's the thing, bigger datatypes means less effective memory throughput. From that perspective 16-bit or even 8-bit floats are often more useful than 80 or 128 bit floats. Same problem with 64 bit pointers and why it's often better to store narrower indices instead of full pointers, data can be packed more tightly and accessed more efficiently.
The point about large word lengths is you get higher data throughput, and faster processing, because everything is going through fat pipes into a big parallel machine with multiple levels of cache and vectorisation.
The issue is the utility of floats at different precisions. 128-bit floats have some benefits for high-end scientific applications, but the extra cost and complexity over 64-bit hardware would only make sense for specialised scientific supercomputing. So far it just hasn't been worth it.
The problem is for most practical uses where you need fast calculations the 64 bit precision is enough. For example there is hardly any physical calculation that would need more precision. 64 bit float can be used to measure the Earth-Sun distance to 30 micrometre precision. 128 bits does not just add anything generally useful. For monetary calculations you should be using decimals instead of binary anyway.
Doubles are precise to 1E-16, and the most precisely known fundamental constants have uncertainties on the 1E-12 level. This does not give a comfortable headroom, so the CODATA adjustment that determines the constants from measurements uses quadruple-precision numbers in FORTRAN.
In most cases where I needed higher precision, I just went to fixed point... There are also free libraries with arbitrary precision (although at a significant performance hit).
precision loss in floating point is often exponential.
If 64 bit isn't enough, then very quickly 128 is also not enough.
If precision is important then you will very often want systems that represent every number as an interval [a, b] meaning that the true value is between those 2 numbers. This makes you able to detect loss of precision due to e.g. d = a / (b - c) where b-c can result in a number close to zero which makes uncertainty grow. If you use this formula iteratively then precision is lost completely no matter how many bits there are in your floating point variables.