Post
Today I'm at the UC Irvine research forum on developer experience with researchers from across industry and academia.
If you're curious about the future of developer science, what do YOU want me to ask about, talk about, and think about with this crowd?
https://isr.uci.edu/event/uc-irvine-research-forum-series-developer-experience/
@grimalkina waves from across campus
I'm going to take some running notes.
Please do not use this thread to yell at me, argue about AI. Thoughtful dialogue is welcome but not anger.
First up is Eirini Kalliamvakou, GitHub, talking about developer identity shifts. Since I studied AI threat from a social identity perspective in 2024 this is interesting to me! I think identity theory needs to be brought in far more than it has been in DevEx
For instance, we know that strongly internalizing an essentialist identity can be harmful for people's ability to face change. Those are interesting mechanisms we can test for and potentially help reduce people's distress by changing, imho.
Her (qualitative, ~22 or so samples) research belies the idea that AI superuser developers see themselves as managers. Thinks the metaphor more appropriate is creative director of code, doing a lot of active cognitive work. My thought: thank God someone is saying this
In my experience with hundreds (and in my research thousands) of developers, rarely do I find people who simply don't care about learning, who don't want to do interesting work, and who don't have a high need for cognition.
You can read my study that draws from a social identity lens on AI and uses social science theory about mechanisms here to predict who is experiencing more and less AI skill threat. I've followed on this work with a lot of practical use of this model with teams:
Next is Yanina Ledovaya, JetBrains -- who is also a psychologist!!! And I'm so excited to hear this talk: the structure of software revolutions, which draws from her experience teaching the philosophy of science and on thinking and mental representations
Starting strong with saying "the idea of more power or more velocity" is not capturing the sense of what's really going on -- disputes comparing this to the steam engine or electricity, and simply making claims about speed and efficiency
"the scary things we're all going through" can be productively looked at through the lens of past work about paradigm shifts and crises, for example this work
https://en.wikipedia.org/wiki/Technological_Revolutions_and_Financial_Capital
Now we're talking about the features of paradigm shift: crisis, incommensurability (new paradigm isn't just a better version of the old, it requires a new language). How can we map this onto software dev changes? Think about "normal science" and "anomalies"--social change takes years and decades
Trying to describe the behavior of probabilistic systems in the terms of the old language is continually failing... The mental objects developers need here are complex and different. Another shout out to identity crisis! "We need to know how to research it"
Now we're diving into the history of psychologists thinking about mental schemas and representations! It's so fun to see some psych in this space :)
This is getting to be too much mental systems ideas for me to post but I like this idea that we really need to think about the types of mental models we have for how these systems perform and that these old assumptions are where we're seeing breakdown. This is a social and learning challenge
"we lack the mental models to simulate the systems we have already deployed" phew
Soooooo interesting to see on her research once again that developers are seeking to use AI TO LEARN! No one believes us but the developers themselves! This is not a brainrot story.
Hearing now about SAP's DevEx approach -- huge program including 20k users of Claude who aren't in engineering at all. Some interesting data suggesting that managers using AI strongly predicts more team usage of it (wish there were a specific process investigated here though, that could be confounded with a lot of things)
Ok I took a break to give my own talk! 💞😌
Running behind now but Lizzie Matusov, founder and CEO of Quotient, gave a lovely talk through their AI maturity model that was also balanced on presenting risks. (Disclosure, I'm an advisor to Quotient)
I loved the work the team put into this: talking to hundreds of companies and getting very very concrete with what organization policies really look like. Most orgs overestimating their processes by quite a lot
This could not be a better pairing with my talk (on making research more reusable and impactful for communities), Olga Lvova from JetBrains sharing how their developer data is shared for others to analyze!
@grimalkina Has there been any research on improved communication of the theory used by creators of the software?
I recently read Peter Naur's "Programming as Theory Building" and have wondered about off beat ideas like "spoken programming for better theory transfer?"
I'm interested in any attempts at improving the transfer of theory to new team members.
@shapr similar as this thread I think
Me and the two psychologists from jetbrains had a couple of very interesting exchanges about this but all agreed we are often facing the assumption that all devs' mental models are homogenous, that individual survey measures are all EQUALLY accurate/biased vs software metrics (this is not true imho), and other gaps between our fields. Honestly I wish I had time to write a paper about this it's becoming a negative space I'm noticing
@grimalkina Reality checking versus superstition for the individual developer or dev team, perhaps?
At work we learn what we experience in the context we find it. Sometimes the context encourages understanding, and sometimes it encourages explaining away the "why" without checking.
This morning's CoffeeOps peer discussion highlighted operations "game days", rehearsing failure scenarios, as a way to exercise and transfer working knowledge, to work across teams in the response, and test assumptions. Sometimes the incident response books get confirmed, and sometimes rewritten.
@jmeowmeow can you restate the question for me? Is it something like, how can we encourage people to do more reflection and check their understanding? Or is it understanding the features that encourage "explaining away the why" ? Both?
@grimalkina More the first question: How do we encourage practitioners to test assumptions about what matters in their work?
And partly the second question, as you point out: What systems and cultures encourage explaining away the why and how of working routines?
Both of these questions touch on superstitious or unexamined beliefs about work routines.
@jmeowmeow got it thank you!
@grimalkina How do we make the positive changes we want to see palatable enough to leadership that we can start building incentives around them?
@grimalkina How do we make the positive changes we want to see palatable enough to leadership that we can start building incentives around them?
@mhoye reporting back on this, there was some conversation about developing the skills to be present and ask analytical and guiding questions in a conversation that starts with a leader saying "I only want x..." To play out the consequences of a or b approach. It was emphasized that these decisions are never moved at the end of the day by statistics but are moved by storytelling
@mhoye I would also say my talk touches on this too: I argue that community response and tangible impact for developers makes research highly "reusable" in part because people can see the many many personalized actions they can take in response. Paradoxically sometimes to get to this you have to ignore the leaders and develop research very closely to the real problems of practitioners, but then AFTER research that authenticity translates to a lot of actionable suggestions.
@mhoye I had a few but not many follow-up conversations. I think I was most interested in whether grad students would think about putting community based methods into practice but to my disappointment basically every conversation with a grad student was "I want to study x and I've already decided the method now tell me how to make it work" which basically to me is a failure state sequence, just speaking honestly.
Install bonfire.mavnn.eu
Get the full app experience
