English school exams are fatally flawed

Covid-19 has brought exams firmly into attention with each of the UK regions struggling to find fair ways of assessing students when many of them have been forced to either miss lessons or receive a lesser education online. 

In all this the assumption has been that we must hold final exams if we can as they are the best way of assessing students, an assumption that led Gove to drop both continual assessment and AS levels.

It is an assumption that is wrong.

Exams are inaccurate

This is clearly true. I do not think that anyone expects that if a class of children took the equivalent exams, say a Geography GCSE, five days in a row that they would all get the same marks every time.

The question is not, therefore, whether exams are accurate or not but how inaccurate they are.

How inaccurate are they?

Surprisingly nobody seems to know. The best that I can find on my Google search is an Office of Qualifications and Examinations Regulation paper from 2010 that states, "The results showed that, for the GCE and GCSE units analysed, at least 89 per cent of all candidates with a particular grade (other than the highest or lowest grade) have true scores either in that grade or immediately adjacent".

That does not strike me as being particularly accurate and that statistical analysis is supported by the number of successful appeals against grades. Statistics published by Ofqual show there were 1,240 appeals in 2019, up from 857 in 2018. A total of 675 of the appeals were upheld, up from 406 the previous year – an increase of 66 per cent. These may be relatively small numbers but that is not the point, they are just further evidence that exams do not always give the correct result. 

That exams are inaccurate is not an opinion, it's a simple fact, yet most commentators talk as if they were perfect.

Sadly even education professionals make this mistake. When I was a school governor an LEA Inspector was concerned that our Teacher Assessment grades varied significantly from the results of the subsequent exams and the firm assumption was that the exams were right and the teachers were wrong. I asked them how accurate the exams were and they had no idea, to be honest I am not even sure that they understood the question.

Exams are just one data point

Many people will remember dealing with inaccurate measurements in school science lessons and part of the solution is simple, repeat the experiment many times and assess that actual results from the spread of results achieved.

So, if the class did take the equivalent exams five days in a row we would have five results for each child to work with and could, for example, take a simple average (mean or median, there are arguments for using both methods) of the five results and use that. This would not be perfect, five is still a small number, but it would be many times better than what we actually do.

Continual Assessment may have issues with personal biases etc., and I would not want to deny that, but it does provide a lot more data to make the final assessment from than a single exam ever can and there are ways to address the other concerns which would make some form of Continual Assessment a fairer method.

This, of course, is how assessment works in most workplaces during the Annual Appraisal process with employees having to cite examples of their work throughout the year as well as seeking feedback from multiple sources.

Conclusion

One data point that is measured imprecisely can only ever be right by chance (like a stopped clock) and that is why exams as used in English schools are fatally flawed.