Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and the technical floor is low.
The level to get into FPGA design is insanely high, you've got to read timing diagrams, you need yo know combinatorial logics and good sense of boolean algebra, you need to have an asynchronous signal based mindset which is vastly different from CPU/GPU, you need to know netlist and you need to endure the time it takes for the EDA to finish generating it. Yosys is still years behind Xilinx
There is a reason GPUs are called accelerators; it sacrifices and does not try to really specialize on one particular thing. Otherwise we will all be using DSPs
I didn't expect the 2,000 connection sweep to stay flat, since all of them are sharing one stream. What does per user latency look like at that end of the sweep?
But the problem is not that your model is fast.
Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and the technical floor is low.
The level to get into FPGA design is insanely high, you've got to read timing diagrams, you need yo know combinatorial logics and good sense of boolean algebra, you need to have an asynchronous signal based mindset which is vastly different from CPU/GPU, you need to know netlist and you need to endure the time it takes for the EDA to finish generating it. Yosys is still years behind Xilinx
There is a reason GPUs are called accelerators; it sacrifices and does not try to really specialize on one particular thing. Otherwise we will all be using DSPs
I didn't expect the 2,000 connection sweep to stay flat, since all of them are sharing one stream. What does per user latency look like at that end of the sweep?