25 lines
4.2 KiB
Markdown
25 lines
4.2 KiB
Markdown
# research design
|
||
Bradley, we are interested in how large language models can support the development of computational thinking in computer science education. Specifically, we are interested in how large language models support students reflecting on their projects to gain conceptual understanding, well engaged in creating personally meaningful projects in a constructionist modality.
|
||
|
||
Theoretical framework
|
||
- Computational thinking as a form of conceptual understanding grounded in computation. We remain at the level of big ideas, but we are interacting with a notional machine as an object to think with. The large language model support students in debugging their conceptual understandings.
|
||
- Constructionist pedagogy: we want students to construct situated understandings of computer science, which they can readily apply to context they care about, and which they can use to understand themselves and the world around them.
|
||
- Knowledge building: constructing understandings through discussion. Just making projects does not necessarily set students up to understand the big ideas involved, especially now that AI is so capable. We adapt the knowledge building framework (but also look into knowledge in pieces) to hypothesize that conceptual understanding of a constructionist project will be greatly enhanced by discussing it with an agent capable of testing, challenging, and debugging students understandings.
|
||
|
||
### research design
|
||
|
||
The context for this research is a high school class using the making with code curriculum, and specifically the games project in which they implement a game using the retro games framework. This will be extended using the retro gamer package described above.
|
||
|
||
The key outcome we are interested in is conceptual understanding of reinforcement learning, which will be assessed using a set of scenario-based conceptual questions related to reinforcement learning, such as:
|
||
- Imagine you were training an agent to play a game with a specified character set. If you forgot to include one of the characters which is used in the game, how would it affect the trained agent’s performance? Explain your reasoning.
|
||
- imagine you are training an agent to play a game, which has a specified character set. You realize that only half of the characters in the specified characters that are actually used in the game. If you change the character set to only include the characters that are actually used, how would the training process change? Explain your reasoning.
|
||
- Imagine you are creating a game where the goal is to win, and partial success has no value. For example, a game where the goal is to escape from a maze. What would be the effect on agent training to add artificial rewards for completing sub goals in the game (such as reaching a milestone halfway to the exit)? Explain your reasoning.
|
||
Each question would be evaluated using a rubric which values conceptual understanding, even if there are specific misconceptions or misunderstandings about the implementation.
|
||
|
||
After finishing the games unit, All participants in the study will be given a traditional classroom lesson on reinforcement learning, which contains all the information needed to answer the conceptual questions. Participants will then be given a pretest of the conceptual questions.
|
||
|
||
Then participants will be randomly assigned into four cases and a 2 x 2 study design: one factor is whether students explore reinforcement learning through the retro gamer package. The second factor is whether students use a large language model to discuss reinforcement learning, specifically focusing on the conceptual questions. In the case where students use the LLM, but not retro gamer, they will discuss the concept based on the games they created. In the case where students use retro gamer and then the LLM, they will discuss the concepts in the context of retro games and retro gamer. For students are not receiving both interventions, the interventions will be replaced by additional lessons related to the games they made. In all cases the pretest is repeated one week later.
|
||
|
||
We hypothesis that retro gamer and the LLM mama mama will produce significantly higher scorers than the other cases, and that the effect will be mediated by more specific higher quality questions to the LLM, and by more follow up questions.
|
||
|