I've had to repeatedly tell our AWS account reps that we're not even a little interested in the Trainium or Inferentia instances unless they have a provably reliable track record of working with the standard libraries we have to use like Transformers and PyTorch.
I know they claim they work, but that's only on their happy path with their very specific AMI's and the nightmare that is the neuron SDK. You try to do any real work with them and use your own dependencies and things tend to fall apart immediately.
It was just in the past couple years that it really became worthwhile to use TPU's if you're on GCP and that's only with the huge investment on Google's part into software support. I'm not going to sink hours and hours into beta testing AWS's software just to use their chips.
IMO AWS once you get off the core services is full of beta services. S3, Dynamo, Lambda, ECS, etc are all solid. But there are a lot of services they have that have some big rough patches.
RDS, Route53, and Elasticache are decent, too. But yes, I've also been bitten badly in the distant past by attempting to rely on their higher-level services. I guess some things don't change.
I wonder if the difference is stuff they dogfood versus stuff they don't?
I once used one of their services (I forget which, but I think it was there serverless product) that “supported” Java.
… but the official command line tools had show-stopper bugs if you were deploying Java to this service, that’d been known for months, and some features couldn’t be used in Java, and the docs were only like 20% complete.
But this work-in-progress alpha (not even beta quality because it couldn’t plausibly be considered feature complete) counted as “supported” alongside other languages that were actually supported.
(This was a few years ago and this particular thing might be a lot better now, but it shows how little you can trust their marketing pages and GUI AWS dashboards)
I'm assuming you're talking about Lambda. I don't mess with their default images. Write a Dockerfile and use containerized Lambdas. Saves so many headaches. Still have to deal with RIE though, which is annoying.
But yes, the less of a core building block the specific service is (or widely used internally in Amazon), the more likely you are to run into significant issues.
Hmm is it actually that bad? Keep in mind r2 is only stored in one region which is chosen when the bucket is first created so that might be what you're seeing
But I've never really looked too closely because I just use it for non-latency critical blob storage
Personally, EMR has never shaken off the "scrappy" feeling (sometimes it feels OK if you're using Spark), and it feels even more neglected recently as they seem to want you on AWS Glue or Athena. LakeFormation is... a thing that I'm sure is good in theory if you're using only managed services, but in practice is like taking a quick jaunt on the Event Horizon.
Glue Catalog has some annoying assumptions baked in.
Frankly the entire analytics space on AWS feels like a huge mess of competing teams and products instead of a uniform vision.
AWS keeps making grand statements about Trainium but not a single customer comes on stage to say how amazing it is. Everyone I talked to that tries it says there were too many headaches and they moved on. AWS pushes it hard but “more price performant” isn’t a benefit if it’s a major PITA to deploy and run relative to other options. Chips without a quality developer experience isn’t gonna work.
Seems AWS is using this heavily internally, which makes sense, but not observing it getting traction outside that. Glad to see Amazon investing there though.
The inf1/inf2 spot instances are so unpopular that they cost less than the equivalent cpu instances. Exact same (or better) hardware but 10-20% cheaper.
We're not quite seeing that on the trn1 instances yet, so someone is using them.
Heh, I was looking at an eks cluster recently that was using Cast AI autoscalar. Scratching my head as there was a bunch of inf instances. Then I realized it must be cheap spot pricing.
Not just AWS, looks like Anthropic uses it heavily as well. I assume they get plenty of handholding from Amazon though. I'm surprised any cloud provider does not invest drastically more into their SDK and tooling, nobody will use your cloud if they literally cannot.
Well AWS says Anthropic uses it but Anthropic isn’t exactly jumping up and down telling everyone how awesome it is, which tells you everything you need to know.
If Anthropic walked out on stage today and said how amazing it was and how they’re using it the announcement would have a lot more weight. Instead… crickets from Anthropic in the keynote
AWS has built 20 data centers in Indiana full of half a million Trainium chips explicitly for Anthropic. Anthropic is using them heavily. The same press announcement that Anthropic has made about Google TPUs is the exact same one they made a year ago about Trainium. Hell, even in the Google TPU press release they explicitly mention how they are still using Trainium as well.
I met a AWS engineer a couple of weeks ago and he said Trainium is actually being used for Anthropic model inference, not for training. Inferentia is basically defected Trainiums chips that nobody wants to use.
With GCP announcing they built Gemini 3 on TPUs the opposite is true. Anthropic is under pressure to show they don’t need expensive GPUs. They’d be catching up at this point, not leaking some secret sauce. No reason for them to not boast on stage today unless there’s nothing to boast about.
Anthropic is not going to interrupt their competitors if their competitors don't want to use trainium. Neither would you, I, nor anyone else. The only potential is downside. There's no upside potential for them at all in doing so.
From Anthropic's perspective, if the rest of us can't figure out how to make trainium work? Good.
Amazon will fix the difficulty problem with time, but that's time Anthropic can use to press their advantages and entrench themselves in the market.
> I'm surprised any cloud provider does not invest drastically more into their SDK and tooling
I used to work for an AI startup. This is where Nvidia's moat is - the tens of thousands of man-hours that has gone into making the entire AI ecosystem work well with Nvidia hardware and not much else.
It's not that they haven't thought of this, it's just that they don't want to hire another 1k engineers to do it.
>I'm surprised any cloud provider does not invest drastically more into their SDK and tooling, nobody will use your cloud if they literally cannot.
Building an efficient compiler from high-level ML code to a TPU is actually quite a difficult software engineering feat, and it's not clear that Amazon has the kind of engineering talent needed to build something like that. Not like Google which have developed multiple compilers and language runtimes.
Heavens to Betsy,
I don’t know if you can hear me,
But try supporting these things if you actually want them to be successful. About the 3rd day into trying to roll your own LMI container in sagemaker because they haven’t updated the vLLM version in 6 months and you can’t run a regular sagemaker endpoint because of a ridiculous 60s timeout that was determined to be adequate 8 years ago. I can only imagine the hell that awaits the developer that decides to try their custom silicon.
SRAM: 32 MiB * 8 = 256 MiB (ignoring 2 MiB * 8 = 16 MiB of PSUM which is not really general-purpose nor DMA-able)
Interconnect: 2560 GB/s (I think bidirectional, i.e. Jensen Math™)
----
At 3nm process node the FLOP/s is _way_ lower than competition. Compare to B200 which does 2250 BF16, x2 FP8, x4 FP4. TPU7x does 2307 BF16, x2 FP8 (no native FP4). HBM also lags behind (vs ~192 GiB in 6 stacks for both TPU7x and B200).
The main redeeming qualities seem to be: software-managed SRAM size (double of TPU7x; GPUs have L2 so not directly comparable) and on-paper raw interconnect BW (double of TPU7x and more than B200).
the real news is: "and teases an Nvidia-friendly roadmap"
The sole reason amazon is throwing any money at this is because they think they can do to AI what they did with logistics and shipping in an effort to slash costs leading into a recession (we cant fire anyone else.) The hubris is magnanimous to say the least.
but the total confidence is very low...so "Nvidia friendly" is face saving to ensure no bridges they currently cross for AWS profit get burned.
These product might be great, but seriously, who's choosing those names? Trainium, Inferentia? It's like let's just take the words from what they do, and put a little Latin twist on them? I know naming things is one of the great problems in computer science, but really they could come up with something a little better.
Amazon aside, interesting future here with NVLink getting more and more folks using it. Intel is also onboard with NVlink. This is like an PCI -> AGP moment, but Nvidia's AGP.
AMD felt like they were so close to nabbing the accelerator future back in HyperTransport days. But the recent version Infinity Fabric is all internal.
There's Ultra Accelerator Link (UALink) getting some steam. Hypothetically CXL should be good for uses like this, using PCIe PHY but lower latency lighter weight; close to ram latency, not bad! But still a mere PCIe speed, not nearly enough, with PCIe 6.0 just barely emerging now. Ideally IMO we'd also see more chips come with integrated networking too: it was so amazing when Intel Xeon's had 100Gb Omni-Path for barely any price bump. UltraEthernet feels like it should be on core, gratis.
NVLink Fusion sounds like a total trap where you pay to become Jensen's slave. It may make sense for Intel because they're desperate. It's not a good look for AWS to put themselves in the same category.
UltraEthernet feels like it should be on core, gratis.
I've been saying for a while that AMD should put a SolarFlare NIC in their I/O die. They already have switchable PCIe/SATA ports, why not switchable PCIe/Ethernet? UEC might be too niche though.
I've had to repeatedly tell our AWS account reps that we're not even a little interested in the Trainium or Inferentia instances unless they have a provably reliable track record of working with the standard libraries we have to use like Transformers and PyTorch.
I know they claim they work, but that's only on their happy path with their very specific AMI's and the nightmare that is the neuron SDK. You try to do any real work with them and use your own dependencies and things tend to fall apart immediately.
It was just in the past couple years that it really became worthwhile to use TPU's if you're on GCP and that's only with the huge investment on Google's part into software support. I'm not going to sink hours and hours into beta testing AWS's software just to use their chips.
IMO AWS once you get off the core services is full of beta services. S3, Dynamo, Lambda, ECS, etc are all solid. But there are a lot of services they have that have some big rough patches.
RDS, Route53, and Elasticache are decent, too. But yes, I've also been bitten badly in the distant past by attempting to rely on their higher-level services. I guess some things don't change.
I wonder if the difference is stuff they dogfood versus stuff they don't?
I once used one of their services (I forget which, but I think it was there serverless product) that “supported” Java.
… but the official command line tools had show-stopper bugs if you were deploying Java to this service, that’d been known for months, and some features couldn’t be used in Java, and the docs were only like 20% complete.
But this work-in-progress alpha (not even beta quality because it couldn’t plausibly be considered feature complete) counted as “supported” alongside other languages that were actually supported.
(This was a few years ago and this particular thing might be a lot better now, but it shows how little you can trust their marketing pages and GUI AWS dashboards)
I'm assuming you're talking about Lambda. I don't mess with their default images. Write a Dockerfile and use containerized Lambdas. Saves so many headaches. Still have to deal with RIE though, which is annoying.
A big problem for a when three AWS teams launch the same thing. Lowers confidence in dogfooding the “right” one.
Or when your AWS account rep is schmoozing your boss trying to persuade them to use something that is officially deprecated, lol.
My understanding is that AWS productizes lots of one-offs for customers (like Snowball), so that makes sense
Amazon Connect is a solid higher level offering. But only because it is a productized version of Amazon Retail’s call center
I'd add SQS to the solid category.
But yes, the less of a core building block the specific service is (or widely used internally in Amazon), the more likely you are to run into significant issues.
Lightsail fortunately behave like core services.
True with Cloudflare too. Just stick with Workers, R2, Durable Objects, etc...
Not even sure about R2 with it's unpredictable latencies.
Hmm is it actually that bad? Keep in mind r2 is only stored in one region which is chosen when the bucket is first created so that might be what you're seeing
But I've never really looked too closely because I just use it for non-latency critical blob storage
Kinesis is decent
That's heartening to know. I find running Kafka less pleasant.
Checkout redpanda
>But there are a lot of services they have that have some big rough patches.
Enlight us...
Personally, EMR has never shaken off the "scrappy" feeling (sometimes it feels OK if you're using Spark), and it feels even more neglected recently as they seem to want you on AWS Glue or Athena. LakeFormation is... a thing that I'm sure is good in theory if you're using only managed services, but in practice is like taking a quick jaunt on the Event Horizon.
Glue Catalog has some annoying assumptions baked in.
Frankly the entire analytics space on AWS feels like a huge mess of competing teams and products instead of a uniform vision.
This. 100 times this.
Agree, Google put a ton of work into making TPUs usable with the ecosystem. Given Amazon’s track record I can’t imagine they would ever do that.
There might be enough market pressure right now to make them think about it, but the stock price went up enough from just announcing it so whatever
Amazon has no interest in making their platform interoperable.
spoiler alert, they don't work without a lot of custom code
AWS keeps making grand statements about Trainium but not a single customer comes on stage to say how amazing it is. Everyone I talked to that tries it says there were too many headaches and they moved on. AWS pushes it hard but “more price performant” isn’t a benefit if it’s a major PITA to deploy and run relative to other options. Chips without a quality developer experience isn’t gonna work.
Seems AWS is using this heavily internally, which makes sense, but not observing it getting traction outside that. Glad to see Amazon investing there though.
The inf1/inf2 spot instances are so unpopular that they cost less than the equivalent cpu instances. Exact same (or better) hardware but 10-20% cheaper.
We're not quite seeing that on the trn1 instances yet, so someone is using them.
Heh, I was looking at an eks cluster recently that was using Cast AI autoscalar. Scratching my head as there was a bunch of inf instances. Then I realized it must be cheap spot pricing.
Not just AWS, looks like Anthropic uses it heavily as well. I assume they get plenty of handholding from Amazon though. I'm surprised any cloud provider does not invest drastically more into their SDK and tooling, nobody will use your cloud if they literally cannot.
Well AWS says Anthropic uses it but Anthropic isn’t exactly jumping up and down telling everyone how awesome it is, which tells you everything you need to know.
If Anthropic walked out on stage today and said how amazing it was and how they’re using it the announcement would have a lot more weight. Instead… crickets from Anthropic in the keynote
AWS has built 20 data centers in Indiana full of half a million Trainium chips explicitly for Anthropic. Anthropic is using them heavily. The same press announcement that Anthropic has made about Google TPUs is the exact same one they made a year ago about Trainium. Hell, even in the Google TPU press release they explicitly mention how they are still using Trainium as well.
Can you link to the press releases? The only one I'm aware of by Anthropic says they will use Tranium for future LLMs, not that they are using them.
This is the Anthropic press release from last year saying they will use Trainium: https://www.anthropic.com/news/anthropic-amazon-trainium
This is the AWS press release from last month saying Anthropic is using 500k Trainium chips and will use 500k more: https://finance.yahoo.com/news/amazon-says-anthropic-will-us...
And this is the Anthropic press release from last month saying they will use more Google TPUs but also are continuing to use Trainium (see the last 2 paragraphs specifically): https://www.anthropic.com/news/expanding-our-use-of-google-c...
There is no press release saying that they are using 500k trainium chips. You can search on amazon's site.
In the subheading of https://www.aboutamazon.com/news/aws/aws-project-rainier-ai-... Unless there are other Project Ranier customers that it's counting.
I met a AWS engineer a couple of weeks ago and he said Trainium is actually being used for Anthropic model inference, not for training. Inferentia is basically defected Trainiums chips that nobody wants to use.
> Anthropic isn’t exactly jumping up and down telling everyone how awesome it is, which tells you everything you need to know.
You can’t really read into that. They are unlikely to let their competitors know if they have a slight performance/$ edge by going with AWS tech.
With GCP announcing they built Gemini 3 on TPUs the opposite is true. Anthropic is under pressure to show they don’t need expensive GPUs. They’d be catching up at this point, not leaking some secret sauce. No reason for them to not boast on stage today unless there’s nothing to boast about.
Yes, but Google benefit from people using their TPUs, while Anthropic gains nothing unless AWS throws money at them for saying it.
This.
Anthropic is not going to interrupt their competitors if their competitors don't want to use trainium. Neither would you, I, nor anyone else. The only potential is downside. There's no upside potential for them at all in doing so.
From Anthropic's perspective, if the rest of us can't figure out how to make trainium work? Good.
Amazon will fix the difficulty problem with time, but that's time Anthropic can use to press their advantages and entrench themselves in the market.
I am not sure, I would imagine enthusiastic quotes will lead to huge discounts and in that scale that matters
Striking a deal with a competitor (AZURE) does though.
> I'm surprised any cloud provider does not invest drastically more into their SDK and tooling
I used to work for an AI startup. This is where Nvidia's moat is - the tens of thousands of man-hours that has gone into making the entire AI ecosystem work well with Nvidia hardware and not much else.
It's not that they haven't thought of this, it's just that they don't want to hire another 1k engineers to do it.
>I'm surprised any cloud provider does not invest drastically more into their SDK and tooling, nobody will use your cloud if they literally cannot.
Building an efficient compiler from high-level ML code to a TPU is actually quite a difficult software engineering feat, and it's not clear that Amazon has the kind of engineering talent needed to build something like that. Not like Google which have developed multiple compilers and language runtimes.
Heavens to Betsy, I don’t know if you can hear me, But try supporting these things if you actually want them to be successful. About the 3rd day into trying to roll your own LMI container in sagemaker because they haven’t updated the vLLM version in 6 months and you can’t run a regular sagemaker endpoint because of a ridiculous 60s timeout that was determined to be adequate 8 years ago. I can only imagine the hell that awaits the developer that decides to try their custom silicon.
Single-chip specs according to:
https://awsdocs-neuron.readthedocs-hosted.com/en/latest/abou...
https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/...
Eight NeuronCore-v4 cores that collectively deliver:
HBM: 144 GiB HBM @ 4.9 TB/sec (4 stacks)SRAM: 32 MiB * 8 = 256 MiB (ignoring 2 MiB * 8 = 16 MiB of PSUM which is not really general-purpose nor DMA-able)
Interconnect: 2560 GB/s (I think bidirectional, i.e. Jensen Math™)
----
At 3nm process node the FLOP/s is _way_ lower than competition. Compare to B200 which does 2250 BF16, x2 FP8, x4 FP4. TPU7x does 2307 BF16, x2 FP8 (no native FP4). HBM also lags behind (vs ~192 GiB in 6 stacks for both TPU7x and B200).
The main redeeming qualities seem to be: software-managed SRAM size (double of TPU7x; GPUs have L2 so not directly comparable) and on-paper raw interconnect BW (double of TPU7x and more than B200).
Anyone considering using trainium should view this Completely Factual Infomercial: https://x.com/typedfemale/status/1945912359027114310
Pretty accurate in my experience, especially re: the neuron sdk. Do not use.
The Block floating point (MXFP8/4 stuff) is interesting; the AI stuff is really pushing basic data types that haven't moved for decades.
https://en.wikipedia.org/wiki/Block_floating_point
Not a single mention of any benchmarks or performance.
They say 4x more, but not 4x faster, 4x more memory, but not 4x more than what!?
4x more units, clearly
yea, they "officially" dont release benchmarks even if we asked the AWS reps
Interesting that in the article, they do not say what the chip actually does. Not even once.
A bunch of 128x128 systolic arrays at its heart. More details:
https://newsletter.semianalysis.com/p/amazons-ai-self-suffic...
Nice article; I hate to think what the DC bus bars look like! ~50v at ~25kW/rack; 500A bus bars - I guess split, but still!
Training. It's in the name.
Ironically these chips are being targeted at inference as well (the AWS CEO acknowledged the difficulties in naming things during the announcement).
Perhaps they should do their training on their Inferentia chips and see how that works out?
The same thing happened to AMD and Gaudi. They couldn't get training to work so they pivoted to inference.
The I stands for Inference then?
Vector math
Probably because the only task this chip has to perform is to please shareholders hence there is no need to explain anything to us peasant developers.
Time to go squat on trainium4.com [0]
[0] https://www.godaddy.com/domainsearch/find?domainToCheck=trai...
the real news is: "and teases an Nvidia-friendly roadmap"
The sole reason amazon is throwing any money at this is because they think they can do to AI what they did with logistics and shipping in an effort to slash costs leading into a recession (we cant fire anyone else.) The hubris is magnanimous to say the least.
but the total confidence is very low...so "Nvidia friendly" is face saving to ensure no bridges they currently cross for AWS profit get burned.
When they're in short supply can we name them Unobtranium?
These product might be great, but seriously, who's choosing those names? Trainium, Inferentia? It's like let's just take the words from what they do, and put a little Latin twist on them? I know naming things is one of the great problems in computer science, but really they could come up with something a little better.
at some point the cost of transferring will dwarf the cost you pay to NVIDA. I bet that is their bet
Amazon aside, interesting future here with NVLink getting more and more folks using it. Intel is also onboard with NVlink. This is like an PCI -> AGP moment, but Nvidia's AGP.
AMD felt like they were so close to nabbing the accelerator future back in HyperTransport days. But the recent version Infinity Fabric is all internal.
There's Ultra Accelerator Link (UALink) getting some steam. Hypothetically CXL should be good for uses like this, using PCIe PHY but lower latency lighter weight; close to ram latency, not bad! But still a mere PCIe speed, not nearly enough, with PCIe 6.0 just barely emerging now. Ideally IMO we'd also see more chips come with integrated networking too: it was so amazing when Intel Xeon's had 100Gb Omni-Path for barely any price bump. UltraEthernet feels like it should be on core, gratis.
NVLink Fusion sounds like a total trap where you pay to become Jensen's slave. It may make sense for Intel because they're desperate. It's not a good look for AWS to put themselves in the same category.
UltraEthernet feels like it should be on core, gratis.
I've been saying for a while that AMD should put a SolarFlare NIC in their I/O die. They already have switchable PCIe/SATA ports, why not switchable PCIe/Ethernet? UEC might be too niche though.
Amazon post: https://www.aboutamazon.com/news/aws/trainium-3-ultraserver-...