This bit tangential to the thrust of the post but what I find so interesting about Simpson's paradox and statistical inference / machine learning in general is that the same techniques that served us so well in computer science, (i) abstraction (don't look past the screen) and (ii) decomposition, can lead you astray.
Consider an employers record - 2000 qualified and capable job seekers applied. Thousand men and a thousand women. Only 100 women were offered whereas 900 men were offered.
At this level of interface it looks like a strong case of gender bias.
However if you look past the interface you find all the women applied to the compiler theory department that had a vacancy of a hundred only whereas men applied for ML department that had lots of vacancies and were offered jobs. None of the departments had any gender bias in their selection of candidates.
Of course this is a caricature of an example, but hopefully gets the point across.
This is really interesting! I don't know that I would have expected the result, but it's laid out well. Easy to absorb the wrong paradox into one's mental model of the world...
This bit tangential to the thrust of the post but what I find so interesting about Simpson's paradox and statistical inference / machine learning in general is that the same techniques that served us so well in computer science, (i) abstraction (don't look past the screen) and (ii) decomposition, can lead you astray.
Consider an employers record - 2000 qualified and capable job seekers applied. Thousand men and a thousand women. Only 100 women were offered whereas 900 men were offered.
At this level of interface it looks like a strong case of gender bias.
However if you look past the interface you find all the women applied to the compiler theory department that had a vacancy of a hundred only whereas men applied for ML department that had lots of vacancies and were offered jobs. None of the departments had any gender bias in their selection of candidates.
Of course this is a caricature of an example, but hopefully gets the point across.
This is really interesting! I don't know that I would have expected the result, but it's laid out well. Easy to absorb the wrong paradox into one's mental model of the world...
I would hypothesize deeper societal structure causes both equality and gender differences.
Content that James Damore can cite in his next paper.