6 points | by popopanda 4 hours ago ago
2 comments
Need one for vision models too tbh. Token/s doesn't really map easily
Yeah, tokens/s varies a lot based on workload. However, I’ve calibrated the estimator against public benchmarks, so it stays within a 30% error margin!
I'll definitely explore how to estimate vision models next!
Need one for vision models too tbh. Token/s doesn't really map easily
Yeah, tokens/s varies a lot based on workload. However, I’ve calibrated the estimator against public benchmarks, so it stays within a 30% error margin!
I'll definitely explore how to estimate vision models next!