For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues.
Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys.
Optimizers like Muon and Adam play around with weights' first, or second order singular values to train models.
Last statement is a bit sus... Muon computes matrix sign function which can be defined as setting singular values to 1, though you can also define it without SVD. Muon itself doesn't use SVD because it uses a faster method to compute matrix sign. Adam doesn't do anything related to SVD or singular values. Also not sure what you meant by "second order singular values"
Just going to sound really pedantic here, but RGB does not capture the entire colour space. In fact, it only captures about 35% of the colours the human eye can perceive.
You seem to be conflating "RGB" with one particular RGB color space: sRGB. That's a common enough conflation to make, but not appropriate when you're trying to be pedantic.
The SVD seems to come up everywhere in my work in computer vision. I find myself continuously using the various C++/Eigen SVD implementations. Actually I should speak in the past tense. Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases. SVD truly is an amazing tool.
It comes up anywhere that youre working with data that has some sort of correlation structure.
In image processing, the SVD makes it possible to talk about all the rich spatial correlations in the image, and pick out the strongest ones and discard noise.
This is also why it's so ubiquitous in compression algorithms, and of central importance in stuff like quantum information.
> Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases.
I find this so annoying. I had to PR some Claude-generated gaussian elimination routine last month and making sure it got the pivoting logic correct was a waste of my time.
For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues.
Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys.
Optimizers like Muon and Adam play around with weights' first, or second order singular values to train models.
Last statement is a bit sus... Muon computes matrix sign function which can be defined as setting singular values to 1, though you can also define it without SVD. Muon itself doesn't use SVD because it uses a faster method to compute matrix sign. Adam doesn't do anything related to SVD or singular values. Also not sure what you meant by "second order singular values"
Just going to sound really pedantic here, but RGB does not capture the entire colour space. In fact, it only captures about 35% of the colours the human eye can perceive.
https://www.oceanopticsbook.info/view/photometry-and-visibil...
You seem to be conflating "RGB" with one particular RGB color space: sRGB. That's a common enough conflation to make, but not appropriate when you're trying to be pedantic.
Doesn't matter: there's no RGB model that captures the colour space. That exactly the reason CIE exists.
Okay, but that was a really useful metaphor if incomplete in a lot of ways. It made me say “oh”.
The SVD seems to come up everywhere in my work in computer vision. I find myself continuously using the various C++/Eigen SVD implementations. Actually I should speak in the past tense. Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases. SVD truly is an amazing tool.
It comes up anywhere that youre working with data that has some sort of correlation structure.
In image processing, the SVD makes it possible to talk about all the rich spatial correlations in the image, and pick out the strongest ones and discard noise.
This is also why it's so ubiquitous in compression algorithms, and of central importance in stuff like quantum information.
what work are you doing in computer vision that isn't entirely ML these days?
> Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases.
I find this so annoying. I had to PR some Claude-generated gaussian elimination routine last month and making sure it got the pivoting logic correct was a waste of my time.