A study asking people to recall code snippets from a lab task is a memorization task not a comprehension task
This is a very problematic confusion in software research and I do not like it
6
Replies:
16
@grimalkina OK, I'll bite ... How DO we measure comprehension of computer science concepts?
I guess it depends on how the responses are scored/graded.
When I interview candidates, I ask them to describe in their own words a concept that is relevant to the position. Not an obscure algorithm, or a standard fizz buzz - a real scenario and then how would you take this conversation.
1
@rhempel I do not believe it's one construct but many. It's just a vague term that means knowing stuff. You might think about designing a learning science based measure that assesses structural knowledge and cause and effect prediction performance along with some capacity for responding to feedback, like a dynamic test. Really what we usually mean by comprehension is a mix of content AND metacognitive skills. You would have to make situated choices about languages and contexts too.
1
@rhempel your practical approach makes sense to me. You're trying to get signal through concrete examples.
If I were commissioned to design a scalable validated assessment I'd start with those design questions I listed though. There ISN'T a really solid code comprehension measure that I've ever come across and the existing research is very tautological (they define it as the task they came up with to measure it). You could look at a cs concept inventory for a starting place
1
@rhempel but that's still very focused on what we call achievement (what you already learned) not aptitude (typically what we are trying to predict in hiring)
2
I wouldn't rush to label remembering code as a "memorization task, not a comprehension task." While being able to recall a piece of code doesn't mean you understand it, I do think that the ability to recall code can serve as a way to measure understanding. The main question seems to be whether the way we test recall gives enough evidence of understanding, rather than seeing recall and understanding as completely separate things.
1
@morbet I'm talking about whether it's a good operationalization or not, not whether sometimes it operates as a signal, if it's an unreliable signal, it's still not a good research methodology. This is not a controversial take of mine. Your description assumes a whole bunch of situated factors that are not controlled for in the experiment, that's why operationalization in an experiment matters, especially when you were talking about human abilities
1
@morbet and I think I am capable of correctly labeling methodologies in my own field. Thanks
1
@morbet here, the operationalization goes:
- have two groups of people do two different things with code, one of which promotes memorization of the specific chunk better than the other
- subject them to a recall test
- argue your recall test is a measure of their ability to comprehend code without ever demonstrating that this is so
We in social science do not simply accept say so methods
@grimalkina this reminds me of one of the first companies I said "No" too.
They had part of the job interview on an airgapped computer, in Python with basic tools.
They said no then reconsidered as they "felt it wasn't the most fair" - I agreed.
(This was 20 years ago)
@grimalkina they used to be called qwiklabs 😉
@grimalkina Oh god. I understand how a bunch of code I'm responsible for works but I couldn't replicate it from memory.
1
@jeremiah a good counter example to these kinds of methods!
1
@grimalkina My understanding of my job is that I'm paid to know how the code works and then when someone comes knocking, I pull them aside and we walk through the code together.
I then promptly forget the exact shape of that code because why I would keep that memorized? It's in the version control, I just need to know how to find it.
1
@grimalkina But this is a good reason for me to check the methodology section of papers I read now. Normally, I feel like I don't know enough to really evaluate a methods section, but keeping this in the back of my head will be useful. Present discourse has me consuming a lot of developer related papers.
1
LOADS of things impact whether you memorize content in a lab task (mostly attention) that are not the same as capacity to comprehend
1
@[email protected] Watch 90% of the participants with #adhd be marked as having zero comprehension. My wife was baffled for years that I could relay the content of conversations easily while being completely incapable of repeating the words that were actually said verbatim.
