Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin fascinating, but the chart prompts so many questions for me.
Why are there below average genAI homeworks at all?
Is it valid to compare exam scores from the high scoring homework groups this way? Aren't there presumably a lot of bad students hidden in the lower right red line? Or in other words: shouldn't we expect a reordering of true performance in the genAI group?
The more interesting takeaway is the consistent outperforming from non-AI using students. Or is that the main message? The chart sets a different focus with the way it is presented.
-
@metin fascinating, but the chart prompts so many questions for me.
Why are there below average genAI homeworks at all?
Is it valid to compare exam scores from the high scoring homework groups this way? Aren't there presumably a lot of bad students hidden in the lower right red line? Or in other words: shouldn't we expect a reordering of true performance in the genAI group?
The more interesting takeaway is the consistent outperforming from non-AI using students. Or is that the main message? The chart sets a different focus with the way it is presented.
@mklovenotcyber The chart is an illustrative accompanying image from the research paper. If you want to dive deeper, best thing to do is click on the included link and read the paper.
-
@metin fascinating, but the chart prompts so many questions for me.
Why are there below average genAI homeworks at all?
Is it valid to compare exam scores from the high scoring homework groups this way? Aren't there presumably a lot of bad students hidden in the lower right red line? Or in other words: shouldn't we expect a reordering of true performance in the genAI group?
The more interesting takeaway is the consistent outperforming from non-AI using students. Or is that the main message? The chart sets a different focus with the way it is presented.
@mklovenotcyber @metin the paper makes more subtle claims - that, essentially the "using AI" cohort consists of two sub-cohorts - roughly 80% of using AI students who get the AI to just do their homework for them (and then do badly in exams), and roughly 20% of using AI students who actually seem to do the homework (but use AI in other ways) (who do about as well in exams as students who don't use AI, compared to their homework scores).
In the graph, you'd imagine the "using AI to do their homework for them" students are the right side of the using AI line, whilst the left side are students doing homework themselves (and thus looking a lot more like non-AI students).
-
@mklovenotcyber The chart is an illustrative accompanying image from the research paper. If you want to dive deeper, best thing to do is click on the included link and read the paper.
@metin Haha, true. Maybe a split violin plot is in there somewhere. That would be the visual I'm interested now.
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin The chart's real punch is the flat exam line while homework goes up. Same shape as shipping with green CI and a broken product: the proxy got optimized, the outcome didn't. The open question is whether schools move the grade onto exams or onto process evidence the model can't sit for.
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin i’m not sure i fully understand why they plotted the data this way
scores are compared to an average of 100 without AI, aka whatever the pre AI average score was is 100? so the never used AI line is compared to a previous average of 100 and shows improvement from not having used AI before as well?
-
@metin i’m not sure i fully understand why they plotted the data this way
scores are compared to an average of 100 without AI, aka whatever the pre AI average score was is 100? so the never used AI line is compared to a previous average of 100 and shows improvement from not having used AI before as well?
@crash Yeah, the graph is confusing. It’s better to click on the link and read the paper.
-
@metin Haha, true. Maybe a split violin plot is in there somewhere. That would be the visual I'm interested now.
@metin there it is:
-
@metin can confirm it literally rots the brain. Before AI I could confidently code a small project.
After AI I cant even think in code anymore.Just using code as an example.
But this is why im having to relearn *everything* from scratch basically.
GenAI has no place in my household, even have to draw the line with filters from time to time when I it comes to the kids being on apps like Snapchat.
@rexyislive @metin I use AI more like a search engine. Learn by example is a good way to learn, but you need to build the thing yourself — the example is teaching material, the thing you build is still your own stuff.
-
@mklovenotcyber @metin the paper makes more subtle claims - that, essentially the "using AI" cohort consists of two sub-cohorts - roughly 80% of using AI students who get the AI to just do their homework for them (and then do badly in exams), and roughly 20% of using AI students who actually seem to do the homework (but use AI in other ways) (who do about as well in exams as students who don't use AI, compared to their homework scores).
In the graph, you'd imagine the "using AI to do their homework for them" students are the right side of the using AI line, whilst the left side are students doing homework themselves (and thus looking a lot more like non-AI students).
@aoanla @mklovenotcyber @metin
Yes, the Economist points this out, too:
“The drop in exam scores was concentrated among students who rushed their homework. Those who used AI but spent as long on assignments as non-users paid little penalty. What matters, then, is how pupils use the technology. Those whose exam results remained strong were not simply copying and pasting answers to save time. More likely they used the chatbots as a personal tutor, perhaps to explain difficult concepts or help solve specific problems.”
https://www.economist.com/graphic-detail/2026/08/18/does-ai-stop-children-from-learning ($)
-
@rexyislive @metin I use AI more like a search engine. Learn by example is a good way to learn, but you need to build the thing yourself — the example is teaching material, the thing you build is still your own stuff.
@forthy42 That's a very expensive search engine. I miss old search like Askjeeves or the way Google was in the 2000s
-
@forthy42 That's a very expensive search engine. I miss old search like Askjeeves or the way Google was in the 2000s
@rexyislive Indeed. The reason why I don't use Google as search engine is that it works no more.
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin @PapyrusBrigade I can't understand the graph... What are the axis?
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin who could've foreseen that " #SelfLobotomy " is harmful to one's faculties?
- There's a reason we don't allow people to use coke to be better drivers, and that's one of those reasons!
-
@rexyislive @metin I use AI more like a search engine. Learn by example is a good way to learn, but you need to build the thing yourself — the example is teaching material, the thing you build is still your own stuff.
@forthy42 @rexyislive @metin I sometimes use it as a tutor, a kind of a teacher. A means to self-paced learning on steroids if you will
I never blindly copy the output. I never let it fully complete a task. I let it give me a nudge into a direction that I define. And take it from there.
But that requires a lot of reflection, self-awareness and understanding of the underlying technology, bc giving a command "do X for me" seems so much more convenient
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin this is brutal and also very important. Thanks for sharing!
-
@metin there it is:
@mklovenotcyber @metin wow that is exceptionally bad data visualization.
to me this is really hard to parse, especially since there are two separate "white" shapes only differeniated by border colour
-
@rexyislive Indeed. The reason why I don't use Google as search engine is that it works no more.
@forthy42 @rexyislive I use duck duck go instead of Google and it's much better
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scoresGood time to not be a lazy idiot I guess, look like a super genius compared to your peers just by doing the basics.
Begs the question though, are people smarter for not using AI or are the smart people the ones already pre-disposed to avoid AI?
-
@metin there it is:
@mklovenotcyber @metin Do I see Generative AI users dropping the absolute best exam scores, the best homework scores, and the lowest homework times?
Seems like a win-win-win!
I guess I gotta read the paper?
