Discussion
I get asked to advise on the board of a decent number of NSF proposals every year on human centered computer science questions. It is remarkable how similar they are. Lots of questions about measuring learning outcomes in CS or competency at programming and software work more broadly. I am not convinced there is a single unitary measure for this that we can quest after
@grimalkina I'm quite curious about the distinction between achievement measures vs aptitude measures! Thanks for that nugget, I have some reading to do.
@eqe we typically distinguish between the "three As": achievement (measuring what was accomplished in the past), ability (measuring a current capacity), aptitude (measuring potential... typically what we are trying to approximate but we use the previous two to do so. Wars fought, intellectually speaking, over how we do this, at)
Measuring skill and its development in a defensible and validated way is very, very hard. It is the type of work that requires you to bring in behavioral science, learning science, statistics, psychometrics, and multiple forms of testing theory. It is a set of topics I really enjoy but I really do see a lot of teams write proposals that sound unbelievably ambitious to me.
@grimalkina I agree with your assessment and approach. I saw too many instances of people—Management—assuming that a complex set of skills and understandings could be easily assessed or predicted.
@grimalkina I think it's interesting because every company has at least two processes that claim to measure skill and its development: hiring and performance evaluation. In my experience these are not scientific processes, but management will claim they are objective. They are not.
@welbog yeah, and much ink has been spilt on assessment in hiring! I do think we could make it better, some orgs have done so, but it's a real location for pseudoscience.
I think practitioners themselves are often drivers of pseudoscience in assessment though. Many "I'll know it when I see it" beliefs keep us using poor proxies and interview processes that we are familiar with and therefore think are accurate.
Install bonfire.mavnn.eu
Get the full app experience
I think it is probably possible to develop better measures of multiple core competencies than we have, but those will be achievement measures, not aptitude measures (critical distinction!). I think the groundwork for this is sparse, besides concept inventories which are really suited to measuring undergrad threshold clearing, little validated work exists to measure problem solving directly in software work