Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin I'm actually surprised that homework score appears to be such a good predictor of exam score. Are they determined in comparable settings in China (open-book, parents that could help, time pressure, ...)?
-
@metin This graph is interesting and mostly matches my expectation. If you spend the time to actually do the homework, you perform ok the exam (on average). No matter if you use AI or not.
@Segebodo @metin
This is a fascinating graph in terms of "is AI even useful"* At basically no point on the graph does using AI get the student better (or worse) exam results than not; the AI is not useful for improving exam scores
* At around 55 minutes homework time students get better homework scores without affecting their exam scores
* Phoning it in on homework with AI lets students get high homework scores, but then they do poorly on the exam; presumably this compromises the teachers' and parents' (and students' own) ability to monitor progress, in addition to whatever direct effect the AI may have on learning
-
@metin fascinating, but the chart prompts so many questions for me.
Why are there below average genAI homeworks at all?
Is it valid to compare exam scores from the high scoring homework groups this way? Aren't there presumably a lot of bad students hidden in the lower right red line? Or in other words: shouldn't we expect a reordering of true performance in the genAI group?
The more interesting takeaway is the consistent outperforming from non-AI using students. Or is that the main message? The chart sets a different focus with the way it is presented.
Q: "Why are there below average genAI homeworks at all?"
A: Some teachers penalise students for untruthful content (LLM "hallucinations") or if it's clear the student has not proofread what they've used from a LLM. Maybe that's why.
-
@metin fascinating, but the chart prompts so many questions for me.
Why are there below average genAI homeworks at all?
Is it valid to compare exam scores from the high scoring homework groups this way? Aren't there presumably a lot of bad students hidden in the lower right red line? Or in other words: shouldn't we expect a reordering of true performance in the genAI group?
The more interesting takeaway is the consistent outperforming from non-AI using students. Or is that the main message? The chart sets a different focus with the way it is presented.
@mklovenotcyber @metin my takeaway is: students using AI, who get higher scores homework assignments, get significantly lower scores on exams.
-
@mklovenotcyber @metin Do I see Generative AI users dropping the absolute best exam scores, the best homework scores, and the lowest homework times?
Seems like a win-win-win!
I guess I gotta read the paper?
@poleguy @mklovenotcyber @metin no: gen-ai exam scores are significantly *lower*.
-
@gisiger @aoanla @mklovenotcyber @metin But put another way, AI had no gain when being used to study if they spent the same amount of time for similar test scores
@wesley @gisiger @mklovenotcyber @metin I mean, the paper shows a much smaller negative effect for using AI whilst still doing homework (which might not even be significant) but yeah.
(The paper also notes that this is in contrast to various studies that suggest AI tutors can have a positive effect on student learning - generally its argument is that those are idealised, and that this real world study is not going to look as positive. This is a point I have also made to people proposing AI tutoring tools, so I am naturally sympathetic to it, but it's also hard to see other good explanations.)
-
@poleguy @mklovenotcyber @metin no: gen-ai exam scores are significantly *lower*.
@deborahh @mklovenotcyber @metin
See the rest of my conversation with @mklovenotcyber
Yes, the _mean_ is lower, and the side below the mean is significantly lower, but the very highest scores do look to be red, which seems to indicate they are from Gen AI users.
I have not yet read the paper, but when I do I'm interested to see if they have thoughts on this.
(In case anyone missed it, based on this tiny red peak value, I was _making a joke._)
-
@metin fascinating, but the chart prompts so many questions for me.
Why are there below average genAI homeworks at all?
Is it valid to compare exam scores from the high scoring homework groups this way? Aren't there presumably a lot of bad students hidden in the lower right red line? Or in other words: shouldn't we expect a reordering of true performance in the genAI group?
The more interesting takeaway is the consistent outperforming from non-AI using students. Or is that the main message? The chart sets a different focus with the way it is presented.
@mklovenotcyber @metin The way I interpret the chart is dividing students that use AI vs don't use AI. That doesn't mean that they use AI for everything, so average or lower-than-average homework scores might indicate students that use AI a little. That also could explain why the gap between AI and no-AI is narrower there.
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scoresThe best way to detoxify the brain from AI madness is to pick up an old copy of "C Programming Language" by Kernighan and Ritchie. And other thoughtful engineering books that started the revolution. Follow the path of wise architects, not billionaires.
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores -
@mklovenotcyber @metin wow that is exceptionally bad data visualization.
to me this is really hard to parse, especially since there are two separate "white" shapes only differeniated by border colour
@saxnot @mklovenotcyber @metin
Thanks. I thought it was just me. All of the graphs I have seen posted in this thread seem badly designed. I suppose I will have to look at the paper and see if one word is worth a thousand pictures.
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin What is the source of this specific figure? cause I cannot find it in exactly the same way in the linked study
-
A alf149@social.data.coop shared this topic
