“ Mathiness lets academic politics masquerade as science. Like mathematical theory, mathiness uses a mixture of words and symbols, but instead of making tight links, it leaves ample room for slippage between statements in the languages of words as opposed to symbols, and between statements with theoretical as opposed to empirical content. Because it is difficult to distinguish machines from mathematical theory, the market for lemons tells us that the market for mathematical theory might collapse, leaving only machines as entertainment that is worth little but cheap to produce.”
> In Calling Bullshit, the authors give the example of the Virginia Mason Quality Equation Q = [Ax(O+S)÷W] (Quality equals Appropriateness times the sum of Outcomes and Service divided by Waste), a formula used for improving operations management in healthcare.
I think it's best to think of this as an overextended metaphor, rather than a clumsy attempt to provide a rigorous relationship. The message to take away from this 'equation' is:
* more appropriateness means more quality;
* better outcomes means more quality;
* better service means more quality;
* less waste means more quality.
It's a small minority of people who give a shit about mathematics. The children who used to whine about, 'When are we every going to use this?' grow up to be adults who complain about not having learnt about how to complete their tax returns at school.
The equation above is an aide de memoire for people managing healthcare facilities. It's irrelevant to people who understand that it's an abuse of mathematical ideas.
The 'Fixing mathiness' section, however, is very good. This article is from a survey company, so this is about increasing the validity of the results you get from their product, but it speaks to the problem of inappropriate statistical methods being used to manufacture signals from noise.
I'd read it as a symptom of something going pretty badly wrong. For example, imagine a funny scenario where a business decides that it needs to use computers to improve efficiency. Some desktop towers are bought. The front-line employees use them as the physical foundation for a new table that they needed and report "mission accomplished" up the chain. The problem in this scenario is that while technically yes, real problems are being solved with the suggested tool someone here is badly, badly out of touch with what needs to be happening to see real improvement.
Similarly here, if there is a formula that stupid being used, even as a memory aid, it suggests to me that real lives could be saved if they embed a few real Ops Research specialists somewhere important instead of whatever they are doing.
>I think it's best to think of this as an overextended metaphor
It may be best, I agree, but it's not what people do. I've seen enough people with advanced degrees take things like FMEA risk priority numbers or adding up the numbers in a Pugh matrix very literally, not as a basis for discussion and insight.
>The children who used to whine about, 'When are we every going to use this?' grow up to be adults who complain about not having learnt about how to complete their tax returns at school.
I've seen my former classmates complain on Facebook about not having been taught things ("there should be a class in high school!") that were absolutely taught in the dumb state required classes.
I'm very skeptical of this view. Additive relationships and multiplicative relationships differ dramatically; the difference is related to linear Vs non linear phenomena. If all you mean is "positively correlated" and "negatively correlated" then you should just say that (or words to that affect).
Only if they are measured on a scale allowing negative values for outcomes and sufficiently high values for service. It would be unsurprising to have outcomes measured from zero and service topping out at 100%. The equation doesn't say anything remotely precise, it's just a bundle of vibes.
I could think of a bunch of simpler ways to state this equation without the baloney false math (similar to the paraphrasing you did).
No, the purpose of putting this into an "equation" is to imply false rigor, and simultaneously to imply that you should hire our people to implement our "healthcare quality framework". I mean, would people really think this is any sort of non-obvious advice if you simply stated, "To improve quality, you should improve appropriateness of treatments, improve outcomes and improve service, and reduce waste." I mean, no shit Sherlock. But once you put it into an "equation", you're deliberately, and falsely, implying "Oooh, science!"
Using an equation to represent this is dishonest. It assumes linearity and proportionality between variables which may not be the case. Also none of the terms are really measurable. You might as well write statements instead.
I mean try defining waste and quality.
Fundamentally, and to use a non-mathematical term appropriately in context, it is a load of bollocks. It is used to make simple ideas look like they are rigorously defined to people without the tools to interpret them. And that is dishonest.
As for inappropriate statistical methods, survey companies are a breeding ground for providing tools which the results of are not interpreted with any statistical rigour or language.
1. If I wanted to use PCA, is there a better approach than normalization? I am asking because you seem better at math than I (based on your comment history).
2. Not that PCA is my favorite technique, but you can apply PCA to data that are expressed in the same units.
3. Do you oppose most clustering techniques since one can apply them to mixed units data?
Woah that’s an unexpected and very hot take! What application of PCA are you familiar with that’s flawed? I learned it in the context of data science, where a) parameter normalization is a very important setup step, and b) it has directly observable empirical success.
Like, I believe you that it’s used for Mathiness bs by some people, but disregarding it in its entirety seems like disregarding linear regression, or, idk, long division. I didn’t realize that was an option!
a mitigating argument for mathiness is that we use math to describe shapes and relationships we can't physically see, and how do you contruct an analogy for a dynamic between factors (or narrative elements) without using changing quantities?
Is the analogy a useful abstraction, or does it provide consistency with lower or higher levels of abstraction, or have external consistency with the rest of maths? Probably not, but as an application that is sufficient for its purposes, some mathiness enables people to separate the things they talk about from just their personal animal interests.
Sure, some people want more from the math, and economics is a great gateway to math because it's probably one of the most sophiticated systems of bullshit ouside string theory, and it provokes the desire for rigour.
Math isn't evidence, it's the lens, and you can reject mathiness in anything by just declining to accept that lens.
Well put! I’m personally a big Marx fan so I won’t give you all of economics, but let’s say “lots” ;)
I feel strongly that math is a set of intellectual tools for dealing with quantities rigorously, so “lens” seems absolutely correct to me. As they say in formal logic: an argument can be sound (well-constructed, uses its tools properly) without being valid (accurate to the actual world) if it employs some bad premises. Which new economics does in spades by constructing inaccurate metrics to do their math with.
Which, hey, I don’t blame em. As the other top HackerNews thread rn on Efficiency and Metrics teaches us: when you’re dealing with the actual world, you can basically never have a flawless virtual metric.
what makes him so evil to me is that where we start with math or geometry and a notion of perfection that we can extrapolate from- one which humans are uniquely imbued to appreciate and emulate, he eschewed the quantiative and replaced it with viral language and sold it as somehow more material and "real."
with any contemplation at all the existence of geometry has necessary and unavoidable moral implications about the possible intent of a creator, and it's hard to see how constructing an ideology to dissolve our connection to those is anything other than a poison. he was a snake and a pimp who wrote a mind virus for self-enslavement as a way to separate people from the dignity of their own humanity. math is not a mystery cult either, and I sympathize with professionals seeing their discipline used for bullshit, but gatekeeping its tools only to initiates should be treated as suspect. to scholars i would say either enlighten people or fuck off, as they aren't equipped for what real gatekeeper personalities are capable of, and a little knowledge won't save you from them.
> In Calling Bullshit, the authors give the example of the Virginia Mason Quality Equation Q = [Ax(O+S)÷W]
I read somewhere (maybe in Thinking, Fast and Slow?), that formulas like this can actually be surprisingly effective, even though the units don’t make sense, because they encode an intuition but prevent you from putting your finger on the scale when applying it, by mixing in other biases. IIRC, the studies on this found that trying to tune these formulas by adding in weights tended to make them worse, which is especially surprising.
I’ll have to see if I can dig up the reference on this.
If at the end of the day you have to compare wildly different things, you have to compare wildly different things. Usually this will involve making a judgment call about how to weigh these things, or in other words, selecting units for the purpose of adding quantities of wildly different units.
I always often feel this way about folks who give probability based estimates of their certainty, instead of just describing their level of certainty with human language. Some caveats for things like 99%, which can be said in a way that makes it clear that it is actually just being used as and expression.
A funny way to describe 95% certainty, among a certain type of nerd at least, is a “critical failure.” IMO it is a nice way of expressing the fact that you’d be very surprised to be wrong, but then, D&D has a whole mechanic about 1-in-20 events happening occasionally. All without any numbers.
Agree with the underlying thesis of the post, but wonder if their criteria suffers the same issues as the thing they are being critical of.
I wonder what the author would think of the Drake Equation or quantum mechanic's superposition, as they seem to check a good deal of these boxes but are widely regarded.
I partially attribute this issue to this paper from 2012:
"The abstract that included the meaningless mathematics tended to be judged of higher quality. However, this “nonsense math effect” was not found among participants with degrees in mathematics, science, technology or medicine."
This makes me wonder if there is some useful way to apply lattice theory to software development estimation. Essentially, take the tasks and compose a lattice using a “definitely harder than” relation. While we are observably bad at predicting just how hard a task is, I believe we can do a much better job of answering is A definitely harder than B with a yes or no. And if we can’t then we provide no answer while building the lattice.
And then, well I have no idea. I’d have to build some examples and play with them.
Yes you can do pairwise comparison and ranking of subjective things (how long will this task take) to get a less arbitrary measure of it.
In the past I've done this for consumer product testing - if you ask someone to rate hair straightness out of 10... it's just really difficult and inconsistent. But you can show them two photos they can usually pick the best, even if it feels pretty arbitrary.
Bradley-Terry is the standard method, it's extremely easy to use.
I have never seen anyone try it for bug estimation though. I think it would be too much effort and tbh the industry hasn't even figured out that you need to be able to write down confidence for estimates yet. There's a long way to go before we start doing things properly.
What is bullshit is the attempt to appropriate a useful neutral adjective to mean more than it does. Mathy is just mathy. Physics is more mathy than psychology. Too mathy can be bad, sure.
In the past century Psychology transformed from speculative writing combined with clinical observation, to sophisticated statistical analysis and attempt to follow scientific methods, because many audiences including yourself believe that makes it more true.
I see it as attempting to provide a patina of quantifiability for an essentially qualitative or incalculable result. If it is actually used as a measure, then it will lead to horrible distortion. The "next article" on the site is about OKRs (objectives and key results) and how they can be used to motivate and also destroy motivation, inspire and also make people afraid, by how faithfully you stick to the measurement and how much you assume the Os and KRs are actually tied to business results. The example of the Virginia Mason Quality Equation is somewhat egregious because it is presented as an equation but nothing in it is properly quantifiable, the units are meaningless, and the authors say it's not to be calculated.
However, there is value in putting everything in a mnemonic so that you remember its parts, and what adds and subtracts from the results. Amazon shipping and FedEx Express have different types of quality. Amazon is next-day or better for virtually everything and unreliable. It comes with a guarantee to refund the cost of shipping, which is 0 in most cases. FedEx Express has varying service levels at increasing costs for performance and includes meaninful guarantees and insurance tied to the performance goal. If Amazon fails to deliver on time, it is frustrating but probably not typically enough to make the user quit using the service or attempt to sue, as they didn't pay for specific service and they are satisfied with the overall expediency of the service. If FedEx Express had Amazon levels of performance, they would lose to competitors who are willing to provide the requested service, regardless of whether they refund people's delivery charges. Putting it back in the equation, the Appropriateness of the service dominates the difference between Amazon Shipping and FedEx Express, where Amazon can tolerate more bad outcomes in their model and FedEx can tolerate more waste (higher fees). Amazon can impress you with same-day delivery you didn't ask for while FedEx has no need to do so, people want merely on-time delivery at the price level they requested.
I think Paul Romer, economics Nobel laureate, coined this term in 2015
https://www.aeaweb.org/articles?id=10.1257/aer.p20151066
“ Mathiness lets academic politics masquerade as science. Like mathematical theory, mathiness uses a mixture of words and symbols, but instead of making tight links, it leaves ample room for slippage between statements in the languages of words as opposed to symbols, and between statements with theoretical as opposed to empirical content. Because it is difficult to distinguish machines from mathematical theory, the market for lemons tells us that the market for mathematical theory might collapse, leaving only machines as entertainment that is worth little but cheap to produce.”
Following from "truthiness", coined by Stephen Colbert in 2005 to describe the state of politics.
https://en.m.wikipedia.org/wiki/Truthiness
Sophisticated marketing annd political campaigns include everything from bikini clad women to academic papers.
Using mathematical language is always a rhetorical choice to suggest precision and authority.
> In Calling Bullshit, the authors give the example of the Virginia Mason Quality Equation Q = [Ax(O+S)÷W] (Quality equals Appropriateness times the sum of Outcomes and Service divided by Waste), a formula used for improving operations management in healthcare.
I think it's best to think of this as an overextended metaphor, rather than a clumsy attempt to provide a rigorous relationship. The message to take away from this 'equation' is:
* more appropriateness means more quality; * better outcomes means more quality; * better service means more quality; * less waste means more quality.
It's a small minority of people who give a shit about mathematics. The children who used to whine about, 'When are we every going to use this?' grow up to be adults who complain about not having learnt about how to complete their tax returns at school.
The equation above is an aide de memoire for people managing healthcare facilities. It's irrelevant to people who understand that it's an abuse of mathematical ideas.
The 'Fixing mathiness' section, however, is very good. This article is from a survey company, so this is about increasing the validity of the results you get from their product, but it speaks to the problem of inappropriate statistical methods being used to manufacture signals from noise.
I'd read it as a symptom of something going pretty badly wrong. For example, imagine a funny scenario where a business decides that it needs to use computers to improve efficiency. Some desktop towers are bought. The front-line employees use them as the physical foundation for a new table that they needed and report "mission accomplished" up the chain. The problem in this scenario is that while technically yes, real problems are being solved with the suggested tool someone here is badly, badly out of touch with what needs to be happening to see real improvement.
Similarly here, if there is a formula that stupid being used, even as a memory aid, it suggests to me that real lives could be saved if they embed a few real Ops Research specialists somewhere important instead of whatever they are doing.
>I think it's best to think of this as an overextended metaphor
It may be best, I agree, but it's not what people do. I've seen enough people with advanced degrees take things like FMEA risk priority numbers or adding up the numbers in a Pugh matrix very literally, not as a basis for discussion and insight.
>The children who used to whine about, 'When are we every going to use this?' grow up to be adults who complain about not having learnt about how to complete their tax returns at school.
I've seen my former classmates complain on Facebook about not having been taught things ("there should be a class in high school!") that were absolutely taught in the dumb state required classes.
I'm very skeptical of this view. Additive relationships and multiplicative relationships differ dramatically; the difference is related to linear Vs non linear phenomena. If all you mean is "positively correlated" and "negatively correlated" then you should just say that (or words to that affect).
That equation predicts that great service can outweigh net negative outcomes.
Only if they are measured on a scale allowing negative values for outcomes and sufficiently high values for service. It would be unsurprising to have outcomes measured from zero and service topping out at 100%. The equation doesn't say anything remotely precise, it's just a bundle of vibes.
> It would be unsurprising to have outcomes measured from zero
Assuming that you can never make anybody worse off than they were before you showed up would be a much worse abuse.
See you wrote it in a clearly understandable way without abusing mathematics or giving any credence to mathematics being involved in the concept.
I could think of a bunch of simpler ways to state this equation without the baloney false math (similar to the paraphrasing you did).
No, the purpose of putting this into an "equation" is to imply false rigor, and simultaneously to imply that you should hire our people to implement our "healthcare quality framework". I mean, would people really think this is any sort of non-obvious advice if you simply stated, "To improve quality, you should improve appropriateness of treatments, improve outcomes and improve service, and reduce waste." I mean, no shit Sherlock. But once you put it into an "equation", you're deliberately, and falsely, implying "Oooh, science!"
Using an equation to represent this is dishonest. It assumes linearity and proportionality between variables which may not be the case. Also none of the terms are really measurable. You might as well write statements instead.
I mean try defining waste and quality.
Fundamentally, and to use a non-mathematical term appropriately in context, it is a load of bollocks. It is used to make simple ideas look like they are rigorously defined to people without the tools to interpret them. And that is dishonest.
As for inappropriate statistical methods, survey companies are a breeding ground for providing tools which the results of are not interpreted with any statistical rigour or language.
Source: annoyed mathematician.
This post essentially described Pincipal Component Analysis in its entirety.
Trying to find eigencomponents in a mixing of incompatible units means that your result is completely arbitrary to the scale factor of your units.
And "fixing" the dimensionality by choosing an arbitrary normalization factor only further increases its mathiness factor.
1. If I wanted to use PCA, is there a better approach than normalization? I am asking because you seem better at math than I (based on your comment history).
2. Not that PCA is my favorite technique, but you can apply PCA to data that are expressed in the same units.
3. Do you oppose most clustering techniques since one can apply them to mixed units data?
Woah that’s an unexpected and very hot take! What application of PCA are you familiar with that’s flawed? I learned it in the context of data science, where a) parameter normalization is a very important setup step, and b) it has directly observable empirical success.
Like, I believe you that it’s used for Mathiness bs by some people, but disregarding it in its entirety seems like disregarding linear regression, or, idk, long division. I didn’t realize that was an option!
so all of economics, basically.
a mitigating argument for mathiness is that we use math to describe shapes and relationships we can't physically see, and how do you contruct an analogy for a dynamic between factors (or narrative elements) without using changing quantities?
Is the analogy a useful abstraction, or does it provide consistency with lower or higher levels of abstraction, or have external consistency with the rest of maths? Probably not, but as an application that is sufficient for its purposes, some mathiness enables people to separate the things they talk about from just their personal animal interests.
Sure, some people want more from the math, and economics is a great gateway to math because it's probably one of the most sophiticated systems of bullshit ouside string theory, and it provokes the desire for rigour.
Math isn't evidence, it's the lens, and you can reject mathiness in anything by just declining to accept that lens.
Well put! I’m personally a big Marx fan so I won’t give you all of economics, but let’s say “lots” ;)
I feel strongly that math is a set of intellectual tools for dealing with quantities rigorously, so “lens” seems absolutely correct to me. As they say in formal logic: an argument can be sound (well-constructed, uses its tools properly) without being valid (accurate to the actual world) if it employs some bad premises. Which new economics does in spades by constructing inaccurate metrics to do their math with.
Which, hey, I don’t blame em. As the other top HackerNews thread rn on Efficiency and Metrics teaches us: when you’re dealing with the actual world, you can basically never have a flawless virtual metric.
what makes him so evil to me is that where we start with math or geometry and a notion of perfection that we can extrapolate from- one which humans are uniquely imbued to appreciate and emulate, he eschewed the quantiative and replaced it with viral language and sold it as somehow more material and "real."
with any contemplation at all the existence of geometry has necessary and unavoidable moral implications about the possible intent of a creator, and it's hard to see how constructing an ideology to dissolve our connection to those is anything other than a poison. he was a snake and a pimp who wrote a mind virus for self-enslavement as a way to separate people from the dignity of their own humanity. math is not a mystery cult either, and I sympathize with professionals seeing their discipline used for bullshit, but gatekeeping its tools only to initiates should be treated as suspect. to scholars i would say either enlighten people or fuck off, as they aren't equipped for what real gatekeeper personalities are capable of, and a little knowledge won't save you from them.
> In Calling Bullshit, the authors give the example of the Virginia Mason Quality Equation Q = [Ax(O+S)÷W]
I read somewhere (maybe in Thinking, Fast and Slow?), that formulas like this can actually be surprisingly effective, even though the units don’t make sense, because they encode an intuition but prevent you from putting your finger on the scale when applying it, by mixing in other biases. IIRC, the studies on this found that trying to tune these formulas by adding in weights tended to make them worse, which is especially surprising.
I’ll have to see if I can dig up the reference on this.
Adding quantities of wildly different units is totally unsound [1].
______________
[1] except in machine learning
OLS legitimately incorporates all unit conversions in the betas. That's gross but sound.
If at the end of the day you have to compare wildly different things, you have to compare wildly different things. Usually this will involve making a judgment call about how to weigh these things, or in other words, selecting units for the purpose of adding quantities of wildly different units.
I always often feel this way about folks who give probability based estimates of their certainty, instead of just describing their level of certainty with human language. Some caveats for things like 99%, which can be said in a way that makes it clear that it is actually just being used as and expression.
A funny way to describe 95% certainty, among a certain type of nerd at least, is a “critical failure.” IMO it is a nice way of expressing the fact that you’d be very surprised to be wrong, but then, D&D has a whole mechanic about 1-in-20 events happening occasionally. All without any numbers.
Mathiness + AI
Agree with the underlying thesis of the post, but wonder if their criteria suffers the same issues as the thing they are being critical of.
I wonder what the author would think of the Drake Equation or quantum mechanic's superposition, as they seem to check a good deal of these boxes but are widely regarded.
so much of economics is just mathiness
The term was introduced by Romer with respect to economics and predates the book referenced in the article: https://en.m.wikipedia.org/wiki/Mathiness
I partially attribute this issue to this paper from 2012:
"The abstract that included the meaningless mathematics tended to be judged of higher quality. However, this “nonsense math effect” was not found among participants with degrees in mathematics, science, technology or medicine."
https://www.cambridge.org/core/journals/judgment-and-decisio...
This makes me wonder if there is some useful way to apply lattice theory to software development estimation. Essentially, take the tasks and compose a lattice using a “definitely harder than” relation. While we are observably bad at predicting just how hard a task is, I believe we can do a much better job of answering is A definitely harder than B with a yes or no. And if we can’t then we provide no answer while building the lattice.
And then, well I have no idea. I’d have to build some examples and play with them.
Yes you can do pairwise comparison and ranking of subjective things (how long will this task take) to get a less arbitrary measure of it.
In the past I've done this for consumer product testing - if you ask someone to rate hair straightness out of 10... it's just really difficult and inconsistent. But you can show them two photos they can usually pick the best, even if it feels pretty arbitrary.
Bradley-Terry is the standard method, it's extremely easy to use.
I have never seen anyone try it for bug estimation though. I think it would be too much effort and tbh the industry hasn't even figured out that you need to be able to write down confidence for estimates yet. There's a long way to go before we start doing things properly.
What is bullshit is the attempt to appropriate a useful neutral adjective to mean more than it does. Mathy is just mathy. Physics is more mathy than psychology. Too mathy can be bad, sure.
In the past century Psychology transformed from speculative writing combined with clinical observation, to sophisticated statistical analysis and attempt to follow scientific methods, because many audiences including yourself believe that makes it more true.
You may not be familiar with the "truthiness" term/meme it is a reference to.
https://en.m.wikipedia.org/wiki/Truthiness
H/t Colbert.
I see it as attempting to provide a patina of quantifiability for an essentially qualitative or incalculable result. If it is actually used as a measure, then it will lead to horrible distortion. The "next article" on the site is about OKRs (objectives and key results) and how they can be used to motivate and also destroy motivation, inspire and also make people afraid, by how faithfully you stick to the measurement and how much you assume the Os and KRs are actually tied to business results. The example of the Virginia Mason Quality Equation is somewhat egregious because it is presented as an equation but nothing in it is properly quantifiable, the units are meaningless, and the authors say it's not to be calculated.
However, there is value in putting everything in a mnemonic so that you remember its parts, and what adds and subtracts from the results. Amazon shipping and FedEx Express have different types of quality. Amazon is next-day or better for virtually everything and unreliable. It comes with a guarantee to refund the cost of shipping, which is 0 in most cases. FedEx Express has varying service levels at increasing costs for performance and includes meaninful guarantees and insurance tied to the performance goal. If Amazon fails to deliver on time, it is frustrating but probably not typically enough to make the user quit using the service or attempt to sue, as they didn't pay for specific service and they are satisfied with the overall expediency of the service. If FedEx Express had Amazon levels of performance, they would lose to competitors who are willing to provide the requested service, regardless of whether they refund people's delivery charges. Putting it back in the equation, the Appropriateness of the service dominates the difference between Amazon Shipping and FedEx Express, where Amazon can tolerate more bad outcomes in their model and FedEx can tolerate more waste (higher fees). Amazon can impress you with same-day delivery you didn't ask for while FedEx has no need to do so, people want merely on-time delivery at the price level they requested.