My reading is that this is entirely Go’s fault. In Go, there was a union of two different types without metadata to distinguish the two. The disambiguation was done by looking to see if the data looked like a certain pointer.
I think the reason it is a problem only in 32 bit systems is that the assignment setting the lower half of the data is a 32 bit op, which can overflow the 16 bits used, but won’t affect the other half of a 64 bit value. (Corrections welcome.)
It's 64 bits on both but only 32 bits were read on 32-bit systems. Two 64-bit values that were different were considered the same because only 32 bits were checked.
Go was born into 64-bit era, I remember seeing many more bugs that only occur on 32-bit systems. I'm pretty certain the observation in the article is correct that no one at Google tests Go on anything apart from 64-bit ARM/x86 Linux
(That particular build dashboard just shows post-commit tests. '386' there is 32-bit x86, 'arm' is 32-bit ARM.)
I can't speak to what Google runs in production or what Google tests internally, but the Google-run CI for the Go project definitely does test 32-bit, and those tests essentially match the tests run against 64-bit.
Reason why it's super hard for juniors at the moment. For "agents" to produce good results, the operator must have the experience to direct it in the right direction and properly validate the output.
I've had some weird bug investigations where it has suggested wrong fixes confidently multiple times before arriving at the actual fix. Only with extra details from manual testing did we arrive at the real bug. The ~5 first suggested fixes would've hidden the root cause even more.
Very interesting read, these bugs that only happen after a lot of time in production are the worse to troubleshoot
So a stronger type system in the boundary between Go and Linux would have caught this?
My reading is that this is entirely Go’s fault. In Go, there was a union of two different types without metadata to distinguish the two. The disambiguation was done by looking to see if the data looked like a certain pointer.
I think the reason it is a problem only in 32 bit systems is that the assignment setting the lower half of the data is a 32 bit op, which can overflow the 16 bits used, but won’t affect the other half of a 64 bit value. (Corrections welcome.)
It's 64 bits on both but only 32 bits were read on 32-bit systems. Two 64-bit values that were different were considered the same because only 32 bits were checked.
Go was born into 64-bit era, I remember seeing many more bugs that only occur on 32-bit systems. I'm pretty certain the observation in the article is correct that no one at Google tests Go on anything apart from 64-bit ARM/x86 Linux
FWIW, every commit for the Go project is tested on multiple 32-bit architectures, including pre-commit tests and post-commit tests.
You can see a list here:
https://build.golang.org/
(That particular build dashboard just shows post-commit tests. '386' there is 32-bit x86, 'arm' is 32-bit ARM.)
I can't speak to what Google runs in production or what Google tests internally, but the Google-run CI for the Go project definitely does test 32-bit, and those tests essentially match the tests run against 64-bit.
Very cool site. Makes me want to do some system programming project just to make a site with all the tests like this.
interesting that they build for solaris as well
Isn't this something that software developers do everyday?
The way I look at these types of difficult debugging problems has completely changed.
If it's a system I want to understand more deeply and I have the time, I try to force myself to do it by hand. If not, the agent does it.
The trap is that it's easy to convince myself that I never have the time.
Not being a user of agents, my question is 'does the agent always find the root cause and remove the defect'?
The way I got good at finding the hard ones was finding hard ones. Without that practice, I would be helpless.
I've found agents very effective at both idebtifying root causes from vague descriptions, and explaining both the problem and the fix to me.
It's not a panacea though. Sometimes it comes up with bad causes and/or solutions.
Reason why it's super hard for juniors at the moment. For "agents" to produce good results, the operator must have the experience to direct it in the right direction and properly validate the output.
I've had some weird bug investigations where it has suggested wrong fixes confidently multiple times before arriving at the actual fix. Only with extra details from manual testing did we arrive at the real bug. The ~5 first suggested fixes would've hidden the root cause even more.
Not directly related to your embedded Go bug, but maybe helpful:
What really sky-rocketed my debugging skills in embedded devices was learning to utilize the Python plugin interface of 'gdb'.
I wonder what other 32 bit bugs are waiting to be discovered.