LLM inference is a two-phase process. The first phase is prompt processing aka prefill. It's compute-heavy but requires relatively low memory bandwidth. The second phase is token generation aka decode, which doesn't require much in the way of FLOPs but wants as much memory bandwidth as possible.
This announcement is for a system to do the first phase on Helios and the second phase on WSE.
It’s disaggregated so you Know it’s good
I'm not sure I understand. It's Helios but with WSE attached? And it uses one or the other depending on some criteria?
LLM inference is a two-phase process. The first phase is prompt processing aka prefill. It's compute-heavy but requires relatively low memory bandwidth. The second phase is token generation aka decode, which doesn't require much in the way of FLOPs but wants as much memory bandwidth as possible.
This announcement is for a system to do the first phase on Helios and the second phase on WSE.
They're a bit late to the party.
lol when is the music gonna stop?