One of my teacher friends often says that people confuse "few people are able to do x" with "x reveals who is good" and I think about that a lot
5
Replies:
16
@grimalkina Good 'at what'? The UK has a new Prime Minister today, so he's shuffling his Cabinet, as though some people are good at anything. That hasn't been my experience. I tend to favour experts in a domain rather than the 'Good chap' theory that was popular with our Conservative governments.
@[email protected] Yes! As someone who's given programming instruction to people with a wide range of experience there's certain topics I can point at that people consider "advanced" which I can trivially teach to ten year olds (recursion, anyone?) while there are others which are considered trivial by "trained" programmers that actually require a wealth of background knowledge they've forgotten they ever learned - like doing almost anything involving a network.
@grimalkina Wow, that really crystallizes an important distinction.
@grimalkina I love it!
It seems related to another performance myth I've pushed back on from time to time: "Few people are able to do x" isn't the same as saying "People who can do x can do a superset of what everyone else can do."
The example I bring up is lawyers vs. janitors. The lawyer's skills may be rarer, but they aren't "janitors+." Most office cleaners I've met couldn't litigate a case, but I know a LOT of lawyers, and very few of them would last two weeks cleaning toilets all day.
Ashley wrote a newsletter the other day* that is about other things besides but makes a really fascinating point in a footnote; in most cases a forced curve in a class is statistically indefensible
*bonus: there is a Baby Cat in here 😅
https://graymatter.ghost.io/more-time-on-exams-may-not-even-help/
3
@grimalkina I hate grading on a curve so, *so* much, on multiple axes. (I am of the opinion that if everyone in the class learned 95% of the material then it’s A+ for everyone, and if nobody learned anything then… it’s time to reevaluate teaching methods and also everyone provisionally fails)
I occasionally got into impassioned arguments with my husband about this, since turns out that growing up in a strongly hierarchical society gives one *bery* different opinions on the point of grades.
@grimalkina as someone that got extra time as an accommodation, it didn't help in the slightest. strangely more time didn't make my dyslexia and adhd any less of an issue when dealing with being forced to sit still and concentrate on words
1
@mensrea right!! Sometimes we need to deeply rethink assessment design itself not just act like the same thing but more will change it
Many of our definitions of achievement are, once you start to look at them, massively tautological.
We hire the best people. How do we know? Because few people pass our interview loop. Why are they the best? Because they can pass our interview loop. And they do good here. Just a lot of post hoc.
Problem is, you will never see what you missed. False negatives are never measured.
2
@grimalkina The number of companies in this world that insist that they only hire the top one percent of all possible candidates is.... well, telling, at a minimum.
2
@mhoye @grimalkina
Those same people think an LLM is intelligent. They are incapable of discerning quality, because they rank their own mediocrity as top 1%, when in reality they are extremely mediocre, and very ignorant.
@mhoye @grimalkina Where all the children are above average
That's not a real mental model for achievement and skill and aptitude. A real mental model says: what skills do I care about here and why? How do they connect to outcomes, as a process? Can I get a view into the process that's most driven by the individual? How can I design for people to demonstrate skills in their truest form? What assumptions am I making about proxies for skill? Where have I reduced the measurement noise of other factors (stress, barriers) that distort my ability to see that?
3
@grimalkina A friend had an interview at Google that did not go well. The reason it did not go well was the recruitor got a pannel of back-end engineers for a front-end job. The recruitor knew he screwed up, the pannel did not. All they knew was this candidate sure seemed to not know very much about infrastructure.
@grimalkina like many things done in a business context, it's lazy heuristic dressed up as meticulous research
Tautological thinking isn't just happening at hiring interviews. It's happening *constantly* inside of your organization, I bet. Conversations about performance, assignment of leadership, career ladders, designing the shapes that technical people have to fit into to match boxes that nobody really bothered to think about.
1
