Post by @inquiline@assemblag.es

i just stumbled upon something where a researcher gave Claude a bunch of abstracts in genetics, to which the researcher had *randomly* assigned author names, and asked Claude to find patterns re research topics and author social identity. And Claude did. E.g., "Female authors" were more likely to produce research about "DEI, Ethics, and Social Dimensions of Genomics". It looks like there's no way to use these tools w/o *imposing cultural patterns*

DressToKILT

@[email protected] · 8 hours ago

@amydiehl what exactly is being tested here? That two identical resumes are perceived diffently based on the assumed gender of the name at the top?

If so, *why did AI* need to get involved?*

Cat Hicks

@[email protected] · 7 hours ago

@dresstokilt @amydiehl you are misspecifying the study. The thing being manipulated between gender conditions is how people are perceived *for using AI*, and showing that there is a gender penalty in the attributions a reviewer makes about the competency of women that is steeper than the attribution made about men. This is useful because it reveals that there is a gender*competency beliefs effect, e.g. if it were only about AI regardless of user we would expect to see the same penalty

DressToKILT

@[email protected] · 7 hours ago

@grimalkina @amydiehl I'm not misspecifying, I was asking a legitimate question because the original post was a headline and a paywalled article.

Cat Hicks

@[email protected] · 7 hours ago

@dresstokilt @amydiehl I'm explaining it to you because you asked. You are making an incorrect assumption about what's being measured. It's not a moral problem it's just a fact.

Cat Hicks

@[email protected] · 7 hours ago

@dresstokilt @amydiehl this is useful to understand because these emerging judgments about competency based on assumptions of effort are shaping our world right now and when they are unevenly deployed against some identities and not others, it will create penalties for those groups.

Cat Hicks

@[email protected] · 7 hours ago

@dresstokilt @amydiehl regardless of one's feelings about AI being appropriate for a task, it is useful to ask whether the same demonstrated behaviors are *described* by observers as having different meanings and being a signal of different aspects of a person. This is a classic gender effect: men's behavior is seen as more circumstantial, something explainable in the pressures of a moment, while women are more likely to be judged as innately flawed or lacking despite the same behavior.

DressToKILT

@[email protected] · 7 hours ago

@grimalkina @amydiehl "shaping our world right now" yes it's *only now* a bias problem.

"AI has the same bias as the people who programmed it" is less of a story than "dog bites man."

1+ more replies

Shannon Clark

@[email protected] · 20 hours ago

@amydiehl @mhoye I seriously wonder how people read resumes where the name is ambiguous.

(Eh very personal interest here - I’ve been misgendered based on my name nearly daily since I was a teenager. I’m a very cis male but daily get people asking if I’m using my wife’s card or being startled when I pick up for Shannon etc. I’ve long wondered if my resume gets penalized by bias like this study shows)

mhoye

@[email protected] · 20 hours ago

@Rycaut @amydiehl At a previous job we put a pile of work into stripping names and gender/cultural identifiers out of CVs before any human looked at them just to minimize bias. It's a lot of work. Worth doing but a lot of work.

May Likes Toronto

@[email protected] · 7 hours ago

@mhoye @Rycaut @amydiehl
Unfortunately, anonymizing resumes isn't enough and sometimes even backfires.
https://journals.sagepub.com/doi/10.1177/01461672241304593

Intentionally building diverse teams is what works best.

Sage Journals

On the Limits of Anonymization for Promoting Diversity in Organizations - Linda W. Chang, Edward H. Chang, 2026

Anonymization of job applicant resumes is a recommended strategy to increase diversity in organizations, but large-scale tests have shown mixed results. We cons...

tools for commensality 🧿

@[email protected] · 2 hours ago

@mayintoronto @mhoye @Rycaut @amydiehl

1 more replies

Shannon Clark

@[email protected] · 19 hours ago

@mhoye @amydiehl indeed I think that’s great but also hard once the process starts - I suspect however that most companies don’t do that work (and these days many use AI filters that likely have unknown biases to filter most applications before a human even sees them.)

It’s definitely a good idea if as a company you actually want to hire the best people and get past your own inevitable biases in that process. Hiring has been pretty broken however for a long time