ChatGPT in lesson preparation - A Teacher Choices Trial
| ISRCTN | ISRCTN13420346 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN13420346 |
- Submission date
- 17/07/2024
- Registration date
- 25/07/2024
- Last edited
- 15/01/2025
- Recruitment status
- No longer recruiting
- Overall study status
- Completed
- Condition category
- Other
Plain English Summary
Background and study aims
Generative AI (GenAI) in one of its most well-known forms (ChatGPT) has only been available to the public since November 2022 and already has more than 100 million weekly active (global) users. ChatGPT is a large language model which generates human-like text responses to questions or prompts entered by users. It has been trained on data from the internet including websites, books, articles and manuals which it uses to predict the next word in a sequence. It has been designed to respond in an accessible and conversational manner allowing users to engage in natural language interactions on various topics and almost anyone can interact with the program once they have a log in. The Department for Education (DfE) in England recognised that the education sector was using GenAI with increasing regularity and issued a call for evidence on the topic in 2023. It showed that these tools are already being used for lesson planning, creating resources, and writing exam questions. Benefits of using GenAI were reported to include ‘freeing up teacher time, providing additional educational support, including for pupils and students with special educational needs and disabilities (SEND) and pupils and students for whom English is an additional language (EAL), and subject specific applications’. Although the Education Secretary has said that “AI will have the power to transform a teacher’s day-to-day work”, there is limited research on how teachers are actually using AI.
Who can participate?
Any state secondary school in England can take part as long as there is at least one teacher who teaches Years 7 and/or Year 8 science, and who completes the baseline survey. Any science teacher who teaches Years 7 and/or Year 8 science (including non-specialist science teachers and Early Career Teachers (ECT)) at a participating school and who is willing to be part of the trial can take part. Teachers are only considered eligible once they complete the baseline teacher survey for the trial.
What does the study involve?
This is a cluster trial with randomisation at school-level. Schools were randomly assigned to one of two arms with equal allocations. The two arms are the ChatGPT group and the Non-GenAI group. In the ChatGPT group, science teachers are asked to use ChatGPT to prepare lessons and resources for upcoming Year 7 and/or 8 science lessons. They will also receive access to the online ChatGPT guide to guide their lesson and resource preparation. In the Non-GenAI group, science teachers are asked not to use ChatGPT or any other GenAI tool in any lesson and resource preparation for their Year 7 and/or 8 science lessons. Teachers will be asked to complete a weekly diary during ten weeks of the 2024 summer term regarding the lessons they delivered in each week.
What are the possible benefits and risks of participating?
The primary hypothesis of the study is that using ChatGPT will beneficially reduce workload for teachers. We do not anticipate any side effects of trial participation.
Where is the study run from?
The National Foundation for Educational Research (https://www.nfer.ac.uk/)
When is the study starting and how long is it expected to run for?
December 2023 to July 2024
Who is funding the study?
The study funder is jointly funded by the Education Endowment Foundation (https://educationendowmentfoundation.org.uk/) and the Hg Foundation (https://www.thehgfoundation.com/)
Who is the main contact?
Palak Roy, Senior Trials Manager at NFER, p.roy@nfer.ac.uk
Contact information
Public
National Foundation for Educational Research
The Mere
Upton Park
Slough
SL1 2DQ
United Kingdom
 ORCID ID |
0009-0009-3951-5937 |
|---|---|
| Phone | +44 (0)1753 574123 |
| p.roy@nfer.ac.uk |
Scientific
National Foundation for Educational Research
The Mere
Upton Park
Slough
SL1 2DQ
United Kingdom
| ORCID ID |
0009-0000-7079-4844 |
|---|---|
| Phone | +44 (0)1753 574123 |
| r.staunton@nfer.ac.uk |
Principal Investigator
National Foundation for Educational Research
The Mere
Upton Park
Slough
SL1 2DQ
United Kingdom
| Phone | +44 (0)1753 574123 |
|---|---|
| h.poet@nfer.ac.uk |
Study information
| Study design | Two-armed cluster (school)-randomized controlled trial |
|---|---|
| Primary study design | Interventional |
| Secondary study design | Cluster randomised trial |
| Study setting(s) | School |
| Study type | Treatment |
| Participant information sheet | 45797_PIS_05Feb2024.pdf |
| Scientific title | A two-armed randomised controlled trial investigating the effect of ChatGPT use during lesson preparation on the time spent on lesson preparation by teachers of Year 7 and/or Year 8 science classes over a 5-week period |
| Study hypothesis | Primary hypothesis: Teacher lesson and resource preparation time for Year 7 & 8 science lessons over five weeks (second half of summer term 2024) will not be equal in ChatGPT and Non-GenAI groups. Secondary hypotheses: Teacher lesson and resource preparation time for Year 7 & 8 science lessons over five weeks (first half of summer term 2024) will not be equal in ChatGPT and Non-GenAI groups. The quality of lesson and resource materials used in Year 7 & 8 science lessons will not be equal in ChatGPT and Non-GenAI groups. The proportion of lessons for which teachers use ChatGPT will not be equal in the first five week period and the second five week period. |
| Ethics approval(s) |
Approved 09/01/2024, NFER Code of Practice Group (The Mere, Upton Park, Slough, SL1 2DQ, United Kingdom; +44 (0)1753574123; Enquiries@nfer.ac.uk), ref: EEAI |
| Condition | Teacher workload |
| Intervention | This is a two-armed randomised controlled trial. The randomisation is at the school level which is stratified by school size (i.e., the number of participating teachers per school). The stratification variable was the number of teachers, cut into two categories. Within each stratum, schools were randomised to one of two study groups through simple random assignment in R. Participating science teachers will be asked to implement their allocated approach for 10 weeks over the summer term of 2024 with their Year 7 and/or Year 8 classes. The two arms are: ChatGPT and Non-GenAI. In the ChatGPT group, teachers are asked to use ChatGPT when preparing for lessons and creating resources. In the Non-GenAI group, teachers are asked to not use any GenAI tools (ChatGPT or otherwise) when preparing for lessons and creating resources. For both groups, the teacher guide emphasises that teachers do not need to create any additional lesson resources or do any additional planning specifically for this project, over and above what they usually would. All teachers are asked to teach their lessons as normal. |
| Intervention type | Other |
| Primary outcome measure | Total hours spent in lesson and resource preparation measured using data collated from a weekly teacher diary over a five-week period (second half of summer 2024 term) |
| Secondary outcome measures | 1. Total hours spent in lesson and resource preparation measured using data collated from a weekly teacher diary over a five-week period (first half of summer 2024 term) 2. Quality of lesson and resource materials used measured using ranking of teachers’ lesson resources by an independent panel of teachers in the second five-week period 3. Proportion of science lessons where ChatGPT was used for lesson and resource preparation measured using a weekly teacher diary over a five-week period 4. Proportion of weeks when the ChatGPT teacher guide was consulted at least once measured using a weekly teacher diary in each five-week period |
| Overall study start date | 11/12/2023 |
| Overall study end date | 31/07/2024 |
Eligibility
| Participant type(s) | Learner/student |
|---|---|
| Age group | Mixed |
| Sex | Both |
| Target number of participants | 174 |
| Total final enrolment | 258 |
| Participant inclusion criteria | 1. School eligibility: Any state secondary school in England can take part as long as there is at least one teacher who teaches Years 7 and/or Year 8 science, and who completes the baseline survey 2. Teacher eligibility: Any science teacher who teaches at an eligible school, who teaches Years 7 and/or Year 8 science (including non-specialist science teachers and Early Career Teachers (ECT)), and who is willing to be part of the trial. Teachers are only considered eligible once they complete the baseline teacher survey for the trial. |
| Participant exclusion criteria | Participants will be excluded if they request to withdraw from the trial |
| Recruitment start date | 01/02/2024 |
| Recruitment end date | 12/03/2024 |
Locations
Countries of recruitment
- England
- United Kingdom
Study participating centre
Upton Park
Slough
Slough
SL1 2DQ
United Kingdom
Sponsor information
Research organisation
The Mere, Upton Park
Slough
SL1 2DQ
England
United Kingdom
| Phone | +44 (0)1753 574123 |
|---|---|
| info@eefoundation.org.uk | |
| Website | https://www.nfer.ac.uk/ |
"ROR" |
https://ror.org/044sxgs38 |
Funders
Funder type
Research organisation
Private sector organisation / Trusts, charities, foundations (both public and private)
- Alternative name(s)
- EducEndowFoundn, Education Endowment Foundation | London, EEF
- Location
- United Kingdom
No information available
Results and Publications
| Intention to publish date | 09/12/2024 |
|---|---|
| Individual participant data (IPD) Intention to share | Yes |
| IPD sharing plan summary | Available on request |
| Publication and dissemination plan | Teacher accessible output and trial report will be published on EEF website - https://educationendowmentfoundation.org.uk/projects-and-evaluation/projects/choices-in-edtech-using-generative-ai-chatgpt-for-ks3-science-lesson-preparation-2024-teacher-choices-trial |
| IPD sharing plan | The datasets generated during and/or analysed during the current study will be available upon request from FFT Education. The EEF data archive is managed by FFT Education (FFT) and held by the ONS within their Secure Research service. https://educationendowmentfoundation.org.uk/privacy-notices/privacy-notice-for-the-eef-data-archive |
Study outputs
| Output type | Details | Date created | Date added | Peer reviewed? | Patient-facing? |
|---|---|---|---|---|---|
| Participant information sheet | 05/02/2024 | 18/07/2024 | No | Yes | |
| Protocol file | version 1.0 | 23/07/2024 | No | No | |
| Statistical Analysis Plan | version 1.0 | 23/07/2024 | No | No | |
| Funder report results | 31/12/2024 | 15/01/2025 | No | No |
Additional files
Editorial Notes
15/01/2025: Funder report of results linked.
18/07/2024: Study's existence confirmed by Education Endowment Foundation (EEF).
