Talk:Ofqual exam results algorithm

The algorithm

This is very clearly written, subscripts would be nice. I wonder how many people would understand that CAG and k both depend on student, that is, if the students are numbered in rank order 1,2,3,...,n then CAG(1), CAG(2),..., CAG(n) are the corresponding centre assessment grades, but also k(1),k(2),...,k(n) in increasing order would be values of the random variable with distribution given by the formula on the right (for each fixed value of j).

A question is, how are the values k(1),k(2),..,k(n) meant to be chosen? If we know that the desired distribution (the averaged and corrected distribution on the right side of the equation for each j) gives a particlar proportion of each grade A*, A, B, .... then we can allocate the k(i) in order to those grades. I suppose in general as the area under the curve is 1, we would choose k(1),k(2),... so that vertical lines drawn on the graph with those horizontal coordinates divide the area into n+1 parts each with equal area 1/(n+1). So k(n+1-i) is the inverse image of i/(n+1) under the function which is the integral of the distribution. In other words, the k(i) chosen so that the cumulative distribution function evaluated at k(1),k(2),...,k(n) would be each of n/(n+1), (n-1)/(n+1),..., 1/(n+1). So 0<k(n)<k(n-1)<...<k(2)<k(1)<1.

I guess the unfairness was that grade inflation is controlled *for each individual student* and in *each individual course*. While the method of controlling grade inflation doesn't matter much for the final statistics, from the standpoint of fairness for individual students, it isn't right to say 'Here's the maximum grade you could ever have received based on previous performance of your school'.Createangelos (talk) 21:46, 23 August 2020 (UTC)

@Createangelos: thanks for your kind remarks. You are correct we need to expand on CAG s and how it was implemented to create unfairness. Superscripts are easy^3.. Where do you _want ? Start over So starting as we always do- what are the references we should use. I need some clear referenced text to work with. Can you draft up something here, and I will work it in under the algorithm box for you to correct.ClemRutter (talk) 23:28, 23 August 2020 (UTC)

OK, have to be careful as it is efficient and clear writing the way it is. I just worry that almost no-one reading it will understand it. Very condensed and intelligent writing but the notation is unusual. There were descriptions which ofqual had put online which could serve as references, they were unreadable. Createangelos (talk) 23:38, 23 August 2020 (UTC)

@Createangelos: I have a plan- a new section- ==Running the algorithm== which can start with text based on what you have written above . The opening line will start with schools were asked to provide a single dataset where students were named and given a predicted grade, and ranked from most able to the least able in that subject, in tiered subjects the staff had to merge the two exams. ... Ofqual had access to historic datasets, particularly the schools historic results. Hepi analysis provides a load of useful data that can be included- and the comments on this article. ClemRutter (talk) 10:35, 29 August 2020 (UTC)

I don't really get it. Here is my first attempt at a rewrite:

The formulas: for large schools with $n\geq 15$ $P_{kj}=(1-r_{j})C_{kj}+r_{j}(C_{kj}+q_{kj}-p_{kj})$ for small schools with $n<15$ $P_{kj}={\text{CAG}}$ The variables $n$ is the number of pupils in cohort $k$ is a specific grade $j$ indicates the school $C_{kj}$ is the historical grade distribution of grade at the school (centre) over the last three years, 2017-19. That tells us already that the history of the school is very important to Ofqual. The grades other pupils got in previous years is a huge determinant to the grades this year’s pupils were given in 2020. The regulator argues this is a plausible assumption but for many students it is also an intrinsically unfair one: the grades they are given are decided by the ability of pupils they may have never met. $q_{kj}$ is the predicted grade distribution based on the class’s prior attainment at GCSEs. A class with mostly 9s (the top grade) at GCSE will get a lot of predicted A*s; a class with mostly 1s at GCSEs will get a lot of predicted Us. $p_{kj}$ is the predicted grade distribution of the previous years, based on their GCSEs. You need to know that because, if previous years were predicted to do poorly and did well, then this year might do the same. $r_{j}$ is the fraction of pupils in the class where historical are data available. If you can perfectly track down every GCSE result, then it is 1; if you cannot track down any, it is 0. CAG is the centre assessed grade. $P_{kj}$ is the result, which is the grade distribution for each grade $k$ at each school $j$.

A few problems I have:

• What is $P_{kj}$? According to the Guardian, it is "the predicted grades for the school." The whole algorithm, according to the Guardian, seems to work only on distributions. But if $k$ is a a grade ("... of grades, $k$, ..."), then $P_{kj}$ by itself looks like a probability: The probability that grade $k$ occurs at school $j$. If this is so, then
  • there must be a second part of the algorithm, which twists the CAGs in a way that the grades then have probablities $P_{kj}$;
  • the "small cohort formula" $P_{kj}={\text{CAG}}$ makes no sense at all (equalling a grade and a probability).
• If $P_{kj}$ is not a probability (a number), what is it then? A whole timeline (a vector indexed by time)? or a complete distribution (but then it's even more unclear how the final grades for each pupil are arrived at).
• What's this $n$? I thought it's the number of pupils in the course - not the school, that makes no sense at all (are there schools with less than 15 pupils)? Or is "school" a term used for a course? The articel Guardian does not contain this symbol or, for that matter, the number 15 at all.

--User:Haraldmmueller 10:29, 30 August 2020 (UTC)

@Haraldmmueller: Excellent. That is a real improvement- I will copy the box over and we can investigate the 'difficulties later. HMG first refused to release the algorithm, and then weren't helpful in the format they used- the Guardian was the first freely available source and they were constrained by their type setting. It was the one I used. I introduced the symbol n- from the description in the text. It is perfectly possible that some private schools will have less than 15 pupils entered for an A level but it does seem likely that will be the total numbers who are entered for a 'specialist niche subject'. ClemRutter (talk) 18:15, 30 August 2020 (UTC)

Done. I have tweaked a few words to reduce the confusion too. n is clear, it is the number of students sitting an exam in that subject. There may be students doing a slightly different course but sitting the same subject exam, but not the same paper. These could be students who are doing Maths (Lower tier) (top grade 5) and Full Maths. A college may have 90 Maths students. A private school only six, and some private schools would only do lower tier. ClemRutter (talk) 19:10, 30 August 2020 (UTC)

Links to Controversy

As the articles were not merged, it would be good to review if some content should be in the other article. The material about the Royal Statistical Society seems to be about the controversy. And it does not describe side effects of the algorithm. Ca3tki (talk) 15:35, 25 August 2020 (UTC)

@Ca3tki:Agreed: Keep on monitoring this please. Yes, transfer it over if you think that is for the best.

Two caveats - is there an existing link that the narrative could be hooked on. We can expect Hell on earth next week when Collier and Thomas appear (or not) infront of the Education subcommittee where Thomas will need to explain his refusal of RSS help.

If the RSS proposes an alternative algorith which we want to document in detail, it will be good to already have the reference/link already in place. (Thats just a preference).

Noting the comments in the section above, do you have any references to the maths that I have missed. Page number etc. we need to take this to decribing in detail how each of the datasets, and those missing were used. --ClemRutter (talk) 18:42, 25 August 2020 (UTC)