Off-rolling and Progress 8 - doing the maths
Off-rolling has become a hot topic and rightly so. The aim of this blog (warning – it is a long one) is to augment the excellent work of others, particularly the rich detail of Education Datalab, with specific reference to Progress 8. Much has been said about how off-rolling has been used by schools to improve GCSE examination results, but in my view not enough about the impact it can have on the main performance measure for secondary schools.
National press attention went up a gear when The Guardian published 300 schools picked out in gcse off-rolling investigation in June 2018. The article stated that
More than 19,000 who were in year 10 in 2016 had vanished from the school roll by the start of year 11.
Half of this number had ‘disappeared without trace’.
30% of this number had SEN.
Pupil premium students, Looked After Children and some minority ethnic groups were ‘more likely to leave’.
London was a particular issue.
The Guardian did not publish the list of 300, presumably because it did not have it. It remains unpublished by any organisation. Ofsted’s response has been along the lines of ‘why alert a school that an inspection is imminent?’
The Times then followed in August 2018 with weak pupils expelled as heads game exam tables, a front page headline splash no less, the publication of which neatly coincided with the publication of GCSE exam results. The coverage went further than The Guardian in naming a number of schools which, at face value, had lost very high numbers of students. Only a small number of schools were named but a disproportionate number were again in London. Some key points from the coverage were
An increase in 2017 of approximately 50% of ‘the number of pupils removed in the months before exams’ from 9,000 to 13,000.
As many as one in four GCSE students at some schools are removed from rolls or not included in league table results.
Forty-four students starting GCSE courses at one school left before exams or did not later have their results included in the league tables.
30% increase in the ‘PRU population’ in the four-year period up to 2018 (12,895 in 2014, 16,732 in 2018).
Ofsted has published two recent articles of note
what is off-rolling and how does ofsted look at it on inspection
teachers say parents need help to resist off-rolling pressure
The first of which offers some very useful definitions and explanations. Off-rolling is defined as
Off-rolling is the practice of removing a pupil from the school roll without using a permanent exclusion, when the removal is primarily in the best interests of the school, rather than the best interests of the pupil. This includes pressuring a parent to remove their child from the school roll.
adding that
While it may not always be unlawful, Ofsted believes off-rolling is never acceptable.
The article also highlights some of the limitations in tackling it. They are correct to say that it is ‘not a black and white issue’, that ‘inspectors will need to gather evidence’ and ‘if pupils are leaving a school to serve the school’s interests, not their own, we will identify this and report on it wherever we find it’. That is not easy within the remit and capacity of a section 5 (‘full’) inspection, let alone a section 8 (‘short’) unless it was the sole line of enquiry. The inspectorate has made its intentions clear, even if to this observer’s eyes at least, little has happened so far.
The second article showed some interesting findings from surveying the teaching profession including
a quarter of teachers have seen off-rolling – when a child is removed from the school roll for the school’s benefit, rather than in the child’s best interests – happen in their schools.
Two-thirds of these teachers believe the practice is on the rise.
The issue is within year 10 more than any other year group, but they still represent only 30% of those who Education Datalab describe in this article as leaving state education. This is worth further investigation, particularly now schools are wise to the increased profile of the issue and what Ofsted will be looking for. Beyond this are many issues with admissions and in year fair access panels that are beyond the brief of this blog. Put simply though schools can have a lot of control over who comes in at the start of year 7, meaning the practice of controlling who will count towards a set of GCSE examination results goes right back to the start. The ‘year 10 problem’ is only a small part of the story. Schools can be very agile in shifting their sights from one performance measure to the next these days, focusing on a year group other than year 10 will not be beyond most who have engaged in off-rolling.
The issue is also not new, it is true that more students left during/soon after year 10 in 2006 than in 2018 (another Education Datalab article). The increase from 2013 to 2016 was approximately 30% and was particularly sharp from 2015 to 2016 (the first year of Progress 8, and according to The Times even sharper in the following year). A correlation? Absolutely. The issue may not be new but, as is indicated below, the impact on Progress 8 is far more significant.
Ofsted’s new framework provides much more scope for off-rolling to be tackled from this September. The old framework had one reference to it, the new one has seventeen references and a dedicated section. Put simply ‘if inspectors determine the school to be off-rolling according to our definition, then the leadership and management of the school are likely to be judged inadequate’. On the official slides, off-rolling is named as one of six key areas for the leadership and management judgement.
Schools will need to be ready with their information of those who have left and be prepared to discuss it. How much information schools need to produce is not clear. The expectations are the same as the previous handbook, records of ‘pupils taken off-roll’ are required. Will schools be asked for the previous year or for all the records since the last inspection? A key issue here is that the horse has bolted, at least as for all the thousands of students to whom it applies in recent years. It will be interesting what changes are made to Ofsted’s data report (IDSR) next year. Will it show potential issues with the school roll over time? Why wait for an inspection to ask the questions?
On top of that the three years a new school (including as a result of brokering or re-brokering) has before inspection allows for a lot of activity that may not be picked up by inspectors. This has not changed and unless some section 8 inspections are going to be triggered because of off-rolling concerns, and not just on the results of the most recent cohort, a lot of practice will not be picked up.
Beyond this there is no mention of acknowledgement or reward for those schools who take on students. Education Datalab’s article who’s left shows one example of one school where ‘doing the right thing and taking on pupils in need of a school place has severely damaged the school’s P8 measure’. The sanction for off-rolling is made clear, but there is nothing about incentives for a school which ‘on-rolls’. After all, why would a school risk ‘severe damage’ to its score? It is a rhetorical question but the implications are clear. In effect the question is ‘why run schools in the interests of students?’
So, let’s show that ‘severe damage’ in practice.
Any statistician will be able to poke a few holes in the detail below, but I hope readers will still see the gist of it. This is a basic set of figures and does not account for the adjustments made for the so-called ‘outliers’ as has been the case since 2018. More detail here. Although this was a welcome development in my view the differences are minor and do not discourage the practice of off-rolling.
This is an example of a hypothetical secondary school
180 students in each year group
Slightly above average percentage of pupil premium students (30%) in each of years 7 to 10.
In general it is a popular school, full across years 7 to 10.
The exception is year 11, where they are down to 165.
When this cohort was in year 10 it was full.
For the sake of argument 15 students were off-rolled between the end of year 10 and the relevant school census for year 11.
10 of these students qualified for pupil premium.
15 is not an unusual figure, falling well short of what The Times reported ‘as many as one in four’. I have not found the methodology for being ‘one of the 300’ but I doubt 15 students/8% would mean it was on the list, particularly if for one year only (potentially covering what a school might wish to define as a ‘cohort issue’). Note that 300 schools is very approximately 8% of the total number of secondary schools.
In 2018 the school’s published Progress 8 score was +0.20
This was defined as ‘above average’ in the performance tables.
The pupil premium Progress 8 score was +0.10. A ‘gap’ with the rest of the school (or ‘other’, as non-pupil premium are defined) but in line with the performance of ‘other’ students nationally so, at face value, not a problem.
In fact, this is around half a grade higher than pupil premium students nationally, but Ofsted is not so interested in that (and that is another story).
The full detail of the methodology is available elsewhere, but a grade 9 scores 9 points down to grade 1 scores 1 is most of what you need to know. The old grade A is now a grade 7, the old grade C now a grade 4. From now on the words ‘grades’ and ‘points’ are used interchangeably. A score of +0.20 means that, on average across the 165 students who finished year 11, they were a fifth of a grade ahead of their peers nationally based on how well their primary school results indicated they would do.
0.20 of a grade across the 8 elements of Progress 8 means 2 grades/points in total (0.2 multiplied by 10 rather than 8 because English and mathematics count double). 2 grades/points each for 165 students is 330 grades.
Now let’s look at the 15 students who ended up elsewhere after the end of year 10. At the time the school projected their average Progress 8 score would be -2.5 at the end of year 11. In my experience this is not unrealistic and, remember, 8% of the roll is not as high as many. Those who come into the school with the highest SATs results from primary are the ones who can end up with a score of -3.5 or -4 if their secondary school years go badly wrong. A student projected to achieve 5s and 6s at GCSE with an attendance issue and other problems can easily average 2s and 3s.
-2.5 Progress 8 score means, when multiplied by 10 again, a loss of 25 grades per student. When this is multiplied by 15 for all the students who ended up elsewhere it amounts to 375 grades thus more than wiping out the 330 grades from above. The school is now ‘in deficit’ of 45 grades. The projected negative average outcomes of 15 students has more than offset the positive average outcomes of the other 165. In a nutshell, this is why off-rolling has become such a problem. A very small number of students can offset the outcomes of a very large number.
The impact of this is that the Progress 8 score has moved from +0.2 (‘above average’) to -0.03 (‘average’). A small deficit but with a potentially morale damaging minus sign at the beginning.
If it was 30 students who were off-rolled, again averaging a Progress 8 score of -2.5, the whole school score is down to -0.23 and defined on the performance tables as ‘below average’. The same methodology can obviously be applied elsewhere, a school that was ‘below average’ is now ‘average’ and so on.
The implications of all this is profound as Ofsted well understands. In short off-rolling can move a school by two categories (from ‘below average’ to ‘above average’). This particular example is likely to be rare but not unique given the examples quoted in the national press. If an inspection follows this particular set of results what might have been a battle to avoid an ‘inadequate’ outcome suddenly becomes an outside chance of an ‘outstanding’ one. The reverse position is also possible. There is more to inspection than a single score (let’s cut a long story short), and the new framework indicates a welcome change of direction, but I do not think I am exaggerating the recent experience of plenty. If schools did not see the risk, why engage with the practice? This is particularly as, under Progress 8, the results of every school are linked. The 300 schools (only the potential worst offenders remember) have had an impact on the results of everyone else.
Under the previous main performance measure, 5 GCSEs at A*-C including English and mathematics, one student who did not achieve it could only offset one student who did. Now one student can offset the results of many. One student with a score of -4 can easily offset 20+ positive, yet relatively small, scores.
The effect is even more significant when looking at the pupil premium figure. Of the 15 who were off-rolled, 10 were pupil premium students. What effect does this have?
As stated above 30% of each year group across 7 to 10 qualifies for pupil premium.
In a typical year group of 180 students this means 54 students who qualify for PP, and 126 do not. This was the case for this particular hypothetical cohort.
With the 10/15 who have been off-rolled qualifying for pupil premium the year 11 cohort is down to 44 who qualify and 121 who do not.
As stated above the PP P8 score for the cohort was +0.1, multiplied by 10 means 1 grade per student and with 44 students that means 44 grades.
The 10 PP students who were off-rolled had the same average P8 score as all who were off-rolled, -2.5 (looking at national statistics the PP scores are likely to be worse than the school average, but anyway). Multiplying by 10 means 25 grades per student, and the total for all 10 students is 250.
The ‘net’ number of grades has now moved from +44 to -206. The average for the cohort of 54 students is now -0.38.
The emphasis on PP outcomes in the Ofsted framework means this school potentially has a serious problem, even if this is in line with the results of PP students nationally.
If the 10 students were of disproportionately high ability (according to their primary SATs results) then the ‘high on entry/PP’ group would really stand out. And it can. Those with high primary results and potentially low outcomes are at the highest risk of a very low P8 score, and therefore of being off-rolled if the school is so inclined,
Off-rolling on the scale described by Ofsted and the national press does not have minor effects on Progress 8 scores, performance tables and potential Ofsted inspection outcomes. It can be cataclysmic. They also skew the figures for every other school, their overall results and particularly when a school’s score is broken down into component parts. This is without any mention of qualification gaming and so on. The large majority of that has closed down, and some schools have seen their outcomes change very significantly as a result.
The practice of off-rolling, and the scale described by none other by Ofsted, is nothing less than statistical doping. If individual schools can manipulate their headline score to such an extent, what does that mean for the integrity of the whole system? What does it say about the new framework that it goes to great lengths to define what it will do about the practice, but not about those who take on (or keep hold of, or do not dissuade in the first place) those students?
Anecdotally I have often heard conversations about schools ‘having to limit its losses’ or ‘we’re not alone’, or worse than this, claiming a triumph based on nothing more than the manipulation of both qualifications and the school roll. It is what I call the ‘Lance Armstrong argument’. He still has his seven yellow jerseys from Tour de France ‘victories’ on display, but they do not mean very much to anyone else. Until the off-rolling issue is resolved Progress 8, and all which has depended on it, has a problem. 300 schools is not a small number and the issue is not restricted to them.
There have been many proposals and ideas to combat the problem. Ofsted has done much to show awareness and promised action both in terms of a different flavour to inspection and on schools who persist in the practice. They have had the figures, and their suspicions, for a very long time. The intentions of the new framework are clearly steps in the right direction. The jury is out on what the next academic year, and the first term in particular, will bring. It is no good focusing on ‘curriculum intent’ if a high proportion of a school’s students are not there to study it until the end.
One idea from me in the interim, publish the median as well as the mean. It is much harder to manipulate, offsets the outlier effect much more effectively and will still tell a lot about a school’s performance. It would ensure some redress so that all schools were run in the interests of students, and not the adults who run them.
