Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
-
@mklovenotcyber @metin the paper makes more subtle claims - that, essentially the "using AI" cohort consists of two sub-cohorts - roughly 80% of using AI students who get the AI to just do their homework for them (and then do badly in exams), and roughly 20% of using AI students who actually seem to do the homework (but use AI in other ways) (who do about as well in exams as students who don't use AI, compared to their homework scores).
In the graph, you'd imagine the "using AI to do their homework for them" students are the right side of the using AI line, whilst the left side are students doing homework themselves (and thus looking a lot more like non-AI students).
@aoanla @mklovenotcyber @metin
Yes, the Economist points this out, too:
“The drop in exam scores was concentrated among students who rushed their homework. Those who used AI but spent as long on assignments as non-users paid little penalty. What matters, then, is how pupils use the technology. Those whose exam results remained strong were not simply copying and pasting answers to save time. More likely they used the chatbots as a personal tutor, perhaps to explain difficult concepts or help solve specific problems.”
https://www.economist.com/graphic-detail/2026/08/18/does-ai-stop-children-from-learning ($)
-
@rexyislive @metin I use AI more like a search engine. Learn by example is a good way to learn, but you need to build the thing yourself — the example is teaching material, the thing you build is still your own stuff.
@forthy42 That's a very expensive search engine. I miss old search like Askjeeves or the way Google was in the 2000s
-
@forthy42 That's a very expensive search engine. I miss old search like Askjeeves or the way Google was in the 2000s
@rexyislive Indeed. The reason why I don't use Google as search engine is that it works no more.
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin @PapyrusBrigade I can't understand the graph... What are the axis?
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin who could've foreseen that " #SelfLobotomy " is harmful to one's faculties?
- There's a reason we don't allow people to use coke to be better drivers, and that's one of those reasons!
-
@rexyislive @metin I use AI more like a search engine. Learn by example is a good way to learn, but you need to build the thing yourself — the example is teaching material, the thing you build is still your own stuff.
@forthy42 @rexyislive @metin I sometimes use it as a tutor, a kind of a teacher. A means to self-paced learning on steroids if you will
I never blindly copy the output. I never let it fully complete a task. I let it give me a nudge into a direction that I define. And take it from there.
But that requires a lot of reflection, self-awareness and understanding of the underlying technology, bc giving a command "do X for me" seems so much more convenient
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin this is brutal and also very important. Thanks for sharing!
-
@metin there it is:
@mklovenotcyber @metin wow that is exceptionally bad data visualization.
to me this is really hard to parse, especially since there are two separate "white" shapes only differeniated by border colour
-
@rexyislive Indeed. The reason why I don't use Google as search engine is that it works no more.
@forthy42 @rexyislive I use duck duck go instead of Google and it's much better
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scoresGood time to not be a lazy idiot I guess, look like a super genius compared to your peers just by doing the basics.
Begs the question though, are people smarter for not using AI or are the smart people the ones already pre-disposed to avoid AI?
-
@metin there it is:
@mklovenotcyber @metin Do I see Generative AI users dropping the absolute best exam scores, the best homework scores, and the lowest homework times?
Seems like a win-win-win!
I guess I gotta read the paper?
-
@forthy42 @rexyislive @metin I sometimes use it as a tutor, a kind of a teacher. A means to self-paced learning on steroids if you will
I never blindly copy the output. I never let it fully complete a task. I let it give me a nudge into a direction that I define. And take it from there.
But that requires a lot of reflection, self-awareness and understanding of the underlying technology, bc giving a command "do X for me" seems so much more convenient
@Ketakater I used to do just that, in the beginning, but if you stop to think about the fact that it had to get the information from somewhere. We never know who really spent the time to write tutorials and explanations, or give them any sort of credit by default.
And a full chat session for learning just takes real world resources, way more than sitting and researching or googl8ng used to.
-
@forthy42 @rexyislive @metin I sometimes use it as a tutor, a kind of a teacher. A means to self-paced learning on steroids if you will
I never blindly copy the output. I never let it fully complete a task. I let it give me a nudge into a direction that I define. And take it from there.
But that requires a lot of reflection, self-awareness and understanding of the underlying technology, bc giving a command "do X for me" seems so much more convenient
@Ketakater @forthy42 @rexyislive @metin
Even at that, LLMs will subtly alter the direction of the nudge — towards median, but also towards various biases -
@mklovenotcyber @metin Do I see Generative AI users dropping the absolute best exam scores, the best homework scores, and the lowest homework times?
Seems like a win-win-win!
I guess I gotta read the paper?
@mklovenotcyber @metin I now really want to read the paper. I really want to know if they did anything beyond looking at mean shifts.
Did they, for example, look at where individuals ranked in the distributions and track where those individuals landed in the distribution after the adoption of AI.
Did the performers of the curve stay at the top?
Did any high performing individuals move up higher on tests?
Did any low performers suddenly shine on tests? -
@Ketakater I used to do just that, in the beginning, but if you stop to think about the fact that it had to get the information from somewhere. We never know who really spent the time to write tutorials and explanations, or give them any sort of credit by default.
And a full chat session for learning just takes real world resources, way more than sitting and researching or googl8ng used to.
@rexyislive also true of course
-
@Ketakater @forthy42 @rexyislive @metin
Even at that, LLMs will subtly alter the direction of the nudge — towards median, but also towards various biases@sabik @forthy42 @rexyislive @metin I know
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin This graph is interesting and mostly matches my expectation. If you spend the time to actually do the homework, you perform ok the exam (on average). No matter if you use AI or not.
-
Using 30 months of panel data on 26,811 Chinese students in grades 7-12, it has been studied how generative AI affects homework productivity and learning.
● X-Axis: Homework scores
● Y-Axis: Exam scores@metin I suppose this is one way of shoving bots toward 'human-level' performance...
-
@mklovenotcyber @metin Do I see Generative AI users dropping the absolute best exam scores, the best homework scores, and the lowest homework times?
Seems like a win-win-win!
I guess I gotta read the paper?
@mklovenotcyber@infosec.exchange @poleguy@mastodon.social @metin@graphics.social
Your interpretation of the exam-score graph is consistent with the assumptions I made of you upon seeing that you're pro-GenAI.
-
@metin @PapyrusBrigade I can't understand the graph... What are the axis?
