Evaluation of the Basic Maths Premium
| ISRCTN | ISRCTN59217701 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN59217701 |
| Secondary identifying numbers | P13021 |
- Submission date
- 25/10/2019
- Registration date
- 28/10/2019
- Last edited
- 20/03/2024
- Recruitment status
- No longer recruiting
- Overall study status
- Stopped
- Condition category
- Other
Plain English Summary
Background and study aims
The Basic Maths Premium (BMP) Pilot is an intervention developed by the UK Department for Education (DfE) that provides additional funding to support 16-18-year-old students studying maths after GCSE. The premium, worth up to £500 per eligible student, will be made available to post-16 educational settings in the most disadvantaged areas of England. The pilot will run from autumn 2018 for two years. Settings are randomly allocated to three funding models. Model T1 consists of a guaranteed payment of £500 for every student. Model T2 consists of a guaranteed payment of £250 for every student and a further £250 in the academic year 2020 to 2021 for every student who achieves in maths. Model T3 consists of a payment of £500 for every student who achieves in maths. The aims of this study are to assess the impact of different funding models on maths outcomes for students, to identify how the additional funding is used by institutions to understand the underlying mechanisms of change, and to build up an evidence base on which activities lead to improvements in teaching and learning.
Who can participate?
The intervention was offered to all post-16 educational institutions in Category 5 and 6 Achieving Excellence Areas (AEA). All institutions that have agreed to be part of the pilot and had at least one eligible student from cohort 2018/19 enrolled by November 2018 received the intervention and were included in the evaluation. Students enrolled in a 16 to 19 study programme for the first time without prior attainment of a maths GCSE grade 4 or above for 2018/2019 and 2019/2020 academic years were eligible for the additional funding attached to this pilot.
What does the study involve?
Schools are randomly allocated to funding model T1, T2 or T3. The impact of the intervention on maths GCSE is measured, compared with a matched control group. The study also looks at how the programme is implemented across different settings and how the funding is spent.
What are the possible benefits and risks of participating?
The expected benefits are higher attainment in maths. The main risk of participating is a displacement effect (i.e. higher attainment in maths is achieved at the cost of lower attainment in other subjects such as English).
Where is the study run from?
1. National Centre for Social Research (NatCen) (UK)
2. Schools in England
When is the study starting and how long is it expected to run for?
May 2018 to August 2023
Who is funding the study?
Education Endowment Foundation (UK)
Who is the main contact?
Dr Andi Fugard
Andi.fugard@natcen.ac.uk
Contact information
Public
35 Northampton Square
London
EC1V 0AX
United Kingdom
 ORCID ID |
0000-0001-6507-2054 |
|---|---|
| Phone | +44 (0)207 549 9578 |
| Andi.fugard@natcen.ac.uk |
Study information
| Study design | Multi-site three-armed cluster-randomized trial with matched comparison group |
|---|---|
| Primary study design | Interventional |
| Secondary study design | Cluster randomised trial |
| Study setting(s) | School |
| Study type | Other |
| Participant information sheet | http://www.natcen.ac.uk/taking-part/studies-in-field/maths-premium-pilot/ |
| Scientific title | Effect of additional funding on maths attainment in post-16 institutions: a three-armed cluster-randomised trial with matched comparison group |
| Study acronym | BMP |
| Study hypothesis | Question 1: What is the impact of the different funding models on maths attainment of eligible students in post-16 settings in England compared to business as usual? Hypothesis 1a: The 2018/2019 cohort of students (hereafter 'Cohort 18/19') in participating institutions that have been randomly assigned to model T1 will on average have better maths outcomes than comparison students in non-participating institutions. Hypothesis 1b: Cohort 18/19 students in participating institutions that have been randomly assigned to model T2 will on average have better maths outcomes than comparison students in non-participating institutions. Hypothesis 1c: Cohort 18/19 students in participating institutions that have been randomly assigned to Model T3 will on average have better maths outcomes than comparison students in non-participating institutions. H1d: Same as hypothesis H1a but for the 2019/2020 cohort of students (hereafter 'Cohort 19/20) H1e: Same as hypothesis H1b but for Cohort 19/20 students H1f: Same as hypothesis H1c but for Cohort 19/20 students H1g: Same as hypothesis H1a but for all participating students (Cohort 18/19 and Cohort 19/20 students) H1h: Same as hypothesis H1b but for all participating students (Cohort 18/19 and Cohort 19/20 students) H1i: Same as hypothesis H1c but for all participating students (Cohort 18/19 and Cohort 19/20 students) Question 2: What is the effectiveness of the different funding models relative to each other? H2a: There will be a different (higher or lower) impact on maths outcomes of Cohort 18/19 students in institutions assigned to model T1 compared to those in institutions assigned to model T3. H2b: There will be a different (higher or lower) impact on maths outcomes of Cohort 18/19 students in institutions assigned to model T2 compared to those in institutions assigned to model T3. H2c: There will be a different (higher or lower) impact on maths outcomes of Cohort 18/19 students in institutions assigned to model T1 compared to those in institutions assigned to model T2. H2d: Same as hypothesis H2a but for Cohort 19/20 students H2e: Same as hypothesis H2b but for Cohort 19/20 students H2f: Same as hypothesis H2c but for Cohort 19/20 students H2g: Same as hypothesis H2a but for all participating students (Cohort 18/19 and Cohort 19/20 students) H2h: Same as hypothesis H2b but for all participating students (Cohort 18/19 and Cohort 19/20 students) H2i: Same as hypothesis H2c but for all participating students (Cohort 18/19 and Cohort 19/20 students) Question 3: What are the effects of the different funding models on GCSE maths exam retake? H3a: Being assigned to model T1 will have a different (higher or lower) impact on the chances of Cohort 18/19 students attempting a GCSE maths re-take exam compared to comparison students in non-participating institutions. H3b: Being assigned to model T2 will have a different (higher or lower) impact on the chances of Cohort 18/19 students attempting a GCSE maths re-take exam compared to comparison students in non-participating institutions. H3c: Being assigned to model T3 will have a different (higher or lower) impact on the chances of Cohort 18/19 students attempting a GCSE maths re-take exam compared to comparison students in non-participating institutions. H3d: Same as hypothesis H3a but for Cohort 19/20 students H3e: Same as hypothesis H3b but for Cohort 19/20 students H3f: Same as hypothesis H3c but for Cohort 19/20 students H3g: Same as hypothesis H3a but for all participating students (Cohort 18/19 and Cohort 19/20 students) H3h: Same as hypothesis H3b but for all participating students (Cohort 18/19 and Cohort 19/20 students) H3i: Same as hypothesis H3c but for all participating students (Cohort 18/19 and Cohort 19/20 students) Question 4: How do these effects differ by student disadvantage status, prior attainment in GCSE maths and number of eligible students per institution? H4a: The effect of the T1, T2, and T3 combined25 on student attainment for Cohort 18/19 will differ (be higher or lower) for those students that have ever been eligible for free school meals compared to comparison students in non-participating institutions. H4b: The effect of the T1, T2 and T3 combined on student attainment for Cohort 18/19 participants will differ (be higher or lower) depending on the number of eligible students that institutions have. H4c: The effect of the T1, T2 and T3 combined on student attainment and attempt to retake GCSE maths for Cohort 18/19 participants will differ (be higher or lower) depending on student prior attainment in GCSE maths. H4d: Same as hypothesis H4a but for Cohort 19/20 students H4e: Same as hypothesis H4b but for Cohort 19/20 students H4f: Same as hypothesis H4c but for Cohort 19/20 students H4g: Same as hypothesis H4a but for all participating students (Cohort 18/19 and Cohort 19/20 students) H4h: Same as hypothesis H4b but for all participating students (Cohort 18/19 and Cohort 19/20 students) H4i: Same as hypothesis H4c but for all participating students (Cohort 18/19 and Cohort 19/20 students) |
| Ethics approval(s) | Approved 11/05/2018, NatCen's Research Ethics Committee (REC) (35 Northampton Square, London, EC1V 0AX, UK; Tel: +44 (0)20 7250 1866; Email: joanne.mcbride@natcen.ac.uk), ref: P13021 |
| Condition | Attainment in maths; inequalities in education |
| Intervention | The Basic Maths Premium (BMP) Pilot is an intervention developed by the Department for Education (DfE) that provides additional funding to support 16-18-year-old students studying maths after GCSE. The premium, worth up to £500 per eligible student, will be made available to post-16 educational settings in the most disadvantaged areas of England. The pilot will run from autumn 2018 for two years. The BMP Pilot will test three alternative funding models: Model T1: Consists of a guaranteed payment of £500 for every student enrolled by November 2018 without a Grade 4 or above in GCSE maths (only for Cohort 18/19 funding-eligible students); Model T2: Consists of a guaranteed payment of £250 for every student enrolled by November 2018 without a Grade 4 or above in GCSE maths, and a further £250 in the academic year 2020 to 2021 for every student who achieves in maths by Summer 2020 (only for Cohort 18/19 funding-eligible students); Model T3: Consists of a payment of £500 in the academic year 2020 to 2021 for every student enrolled by November 2018 who goes on to achieve in maths by Summer 2020 (only for Cohort 18/19 funding-eligible students). Institutions can choose how to utilise their additional funding. Regardless of how resources were assigned, institutions will be free to invest their resources in Cohort 18/19 and/or Cohort 19/20 students as they see fit. This implies that eligible students (Cohort 18/19) will not necessarily be the only beneficiaries of the intervention. Institutions were randomly allocated to model T1, T2 or T3 using Stata software. Given DfE’s commitment to offer all institutions in the pilot areas some form of intervention, it was not possible to randomly assign a proportion of the institutions in the pilot areas to a control group. Consequently, the estimation of the impact of the three treatment conditions compared to business as usual will need to rely on the establishment of a counterfactual through a quasi-experimental design (QED). The researchers will use the doubly-robust method to create a control group. In order to construct a control group, they will use a pool of institutions similar to those in the treatment group. The doubly robust-method mixes propensity score estimation with regression analysis. To estimate the impact of the programme, the researchers will initially estimate propensity scores modelling the selection process into treatment. Later on, a regression model will be constructed to identify the causal effect of the treatment in the outcome of interest. Their preferred regression models will include student-level characteristics as well as institution-level characteristics and institution random effects, accounting for the clustered nature of the data. Finally, and to produce their preferred estimate of the impact of the intervention (the doubly robust-estimator) the researchers will make use of both the estimated regression coefficients as well as the estimated propensity scores for all individuals in the sample. |
| Intervention type | Behavioural |
| Primary outcome measure | Level 2 attainment in maths, defined as achievement of a grade 4 or above in GCSE maths or attainment of a pass in a Level 2 Functional Skills (depending on the student’s prior attainment). This data will be taken from the National Pupil Database (NPD). All baseline data for the 18/19 cohort – collected from National Pupil Dataset in November 2018 All baseline data for the 19/20 cohort – collected from National Pupil Dataset in November 2019 All endline data for the 18/19 cohort – collected from National Pupil Dataset in November 2020 All endline data for the 19/20 cohort – collected from National Pupil Dataset in November 2021 |
| Secondary outcome measures | Whether participants attempted to resit their GCSE maths exam. This data will be taken from the National Pupil Database (NPD). All baseline data for the 18/19 cohort – collected from National Pupil Dataset in November 2018 All baseline data for the 19/20 cohort – collected from National Pupil Dataset in November 2019 All endline data for the 18/19 cohort – collected from National Pupil Dataset in November 2020 All endline data for the 19/20 cohort – collected from National Pupil Dataset in November 2021 |
| Overall study start date | 01/05/2018 |
| Overall study end date | 31/08/2023 |
| Reason abandoned (if study stopped) | COVID-19 meant the outcome data were not available (exam results from school students). |
Eligibility
| Participant type(s) | Mixed |
|---|---|
| Age group | Other |
| Sex | Both |
| Target number of participants | 50,000 |
| Total final enrolment | 47859 |
| Participant inclusion criteria | In order to be eligible to take part, institutions must meet the following criteria: 1. Post-16 institutions in Category 5 and 6 AEAs that have signed up for the pilot and have at least one student eligible to receive BMP funding by November 2018 In order to be eligible to take part, students must meet the following criteria: 1. Students without prior attainment of at least grade 4 in GCSE maths that enrol in their post-16 studies in the participating institutions in the 2018/2019 academic year (the “eligible students” for this pilot). These students are hereafter referred to as “Cohort 18/19” 2. Students without prior attainment of at least grade 4 in GCSE maths that will enrol in their post-16 studies in the participating institutions in the 2019/2020 academic year. These students are hereafter referred to as “Cohort 19/20” |
| Participant exclusion criteria | Does not meet inclusion criteria |
| Recruitment start date | 15/05/2018 |
| Recruitment end date | 31/07/2018 |
Locations
Countries of recruitment
- England
- United Kingdom
Study participating centre
London
EC1V 0AX
United Kingdom
Sponsor information
Charity
5th floor, Millbank Tower
21-24 Millbank
London
SW1P 4QP
United Kingdom
| Phone | +44 (0)207 802 1676 |
|---|---|
| info@eefoundation.org.uk | |
| Website | https://educationendowmentfoundation.org.uk |
"ROR" |
https://ror.org/03bhd6288 |
Funders
Funder type
Charity
Private sector organisation / Trusts, charities, foundations (both public and private)
- Alternative name(s)
- EducEndowFoundn, Education Endowment Foundation | London, EEF
- Location
- United Kingdom
Results and Publications
| Intention to publish date | 30/09/2023 |
|---|---|
| Individual participant data (IPD) Intention to share | No |
| IPD sharing plan summary | Stored in repository |
| Publication and dissemination plan | 1. A protocol and analysis plan is available on the website of the Education Endowment Foundation. See: https://educationendowmentfoundation.org.uk/projects-and-evaluation/projects/basic-maths-premium-pilot/ 2. Results will be published as a report to the EEF on the same webpage. |
| IPD sharing plan | The dataset(s) used in the above-mentioned analyses will be anonymised and archived in a repository managed by FFT Education Datalab: https://ffteducationdatalab.org.uk/. Data and results will be available around May 2023. |
Study outputs
| Output type | Details | Date created | Date added | Peer reviewed? | Patient-facing? |
|---|---|---|---|---|---|
| Protocol (other) | 04/09/2019 | 19/06/2023 | No | No | |
| Statistical Analysis Plan | 04/09/2019 | 19/06/2023 | No | No | |
| Funder report results | 28/02/2024 | 20/03/2024 | No | No |
Editorial Notes
20/03/2024: A link to a funder report was added.
19/06/2023: The following changes have been made:
1. The study has been terminated.
2. The protocol and SAP have been added to the study outputs.
31/05/2023: The following changes were made to the study record:
1. Total final enrolment added.
2. Contact details updated.
3. The overall study end date was changed from 30/07/2022 to 31/08/2023.
4. The intention to publish date was changed from 05/05/2023 to 30/09/2023.
25/10/2019: Trial's existence confirmed by ethics committee.
