Notes on making good progress?: Chapter 8

In this series of posts I record my notes from Daisy Christodolou’s book “Making good progress? The future of Assessment for Learning” It is quite excellent. You can buy a copy here.

Improving summative assessments

The aim of summative assessments is for them to provide an accurate and shared meaning without becoming the model for every classroom activity. Rubrics of prose assessment statements are not particularly good at delivering reliability, and they can end up compromising the creative and original aspects of the task. Prose descriptors can be interpreted in many different ways. Judging in absolute terms is extremely difficult. Markers will overgrade and undergrade depending on the sample. We are much better at making comparative judgements than absolute ones.

very prescriptive rubrics end up stereotyping pupil’s responses to the task removing the justification for having them (grading creative and original work). Responses that are coached to meet the rubric pass and truly original work that doesn’t fails. Rubrics encourage coaching.

Comparative judgement offers the possibility of dropping rubrics by defining quality through exemplars not prose and by not relying on absolute judgement. It simply asks markers to make a series of paired judgements about responses. It relies on tacit knowledge of the subject expert – knowledge that is not easy to express in words.

Comparative judgement is criticised for offering little in the way of formative feedback. This is precisely the point. It decouples the grading process from the formative process. It allows classroom practice to be refocussed away from the rubric and towards helpful analyses of quality. One extremely useful resource that could be produced would be a set of annotated exemplar scripts.

Decisions about the difficulty and content of national summative exams are made by national exam boards. What if a school wants to summatively assess more frequently? To what extent can they be linked to the curriculum that the pupils are following? One solution is to outsource summative assessments, but there is still a gap between the remote standardised assessments like CEM and the formative assessments of classroom practice. It is not easy to create or interpret the results of school made curriculum-linked assessments. It can be difficult to tell if the test is difficult enough, or if it has the right spread of difficulty. Tests taken by small numbers of pupils don’t produce reliable grades. We can compare the results of teacher made tests to national assessments. The content studied over one term is simply not broad enough domain to sample from. Assessments have to sample from what pupils have learnt in that subject, not just in previous terms but in previous years.

A summative assessment can be linked to the curriculum and the most recent unit of study. However if a grade is awarded it will not be based solely on that unit and cannot be seen as reflecting performance on solely that unit. A student can make great strides with a unit but not be reflected on the summative unit as the assessment is not sensitive enough.

Summative assessments need to be far enough apart that pupils have the chance to improve on them meaningfully. However pupils will make relatively slow progress on the large domains that summative assessments are sampling. There are risks with using summative assessments too frequently.

Using scaled scores can overcome this to some extent. A scaled score converts raw marks which are not comparable (from different assessments) into ones that are. They show the continuum of achievement. Grades suggest that pupil performance falls into discrete categories when in fact it is continuous.

Notes on making good progress?: Chapter 7

In this series of posts I record my notes from Daisy Christodolou’s book “Making good progress? The future of Assessment for Learning” It is quite excellent. You can buy a copy here.

Improving formative assessments

Formative assessments should be:

  • Specific
  • Frequent
  • Repetitive
  • Recorded as raw marks

Specific questions allow teachers to diagnose exactly what a pupil’s strengths and weaknesses are, and they make it easy to work out what to do next, whereas open and complex questions like essays or real-world problems are not particularly well-suited to this. Short answer and MCQs can be very precise. MCQs, despite their reputation, are excellent for diagnosis and indicate what pupils might need to work on next. MCQs give a specific diagnosis of conceptual understanding and are labour saving.

Criticism of MCQs include that they are easy for pupils to answer, but the risk can be mitigated in several ways. You can increase the number of distractors, you can increase the number of questions, you can also include more than one right answer. Answers can be analysed at the level of the class. MCQs can target misconceptions very effectively. Misconceptions are an important part of a progression model often because they involve particularly tricky and fundamental concepts without which pupils cannot progress.

They are very easy to analyse. You can record not just whether the pupil got the question right or wrong but which distractors they chose. When the analysis is done on a topic that has been recently taught then it becomes much more helpful. We don’t necessarily need to re-teach topics but can ensure to highlight those misconceptions again if the curriculum is structured in a way to allow this. Explanatory paragraphs in the question bank for each MCQ make it very easy to give feedback. Once the feedback has been delivered the teacher can follow up with another set of similar questions to see if the pupil has understood this time around. MCQs with together with this kind of in-depth, specific and precise feedback, can form a vital part of a progression model in any subject.

Research shows that the act of recalling information from memory actually helps to strengthen the memory itself. That is, testing doesn’t just help measure understanding; it helps develop it. This is called the testing effect. This effect can certainly apply to summative tests too so long as they don’t force students away from retrieval and into problem-solving search. The power of the testing effect is that it introduces desirable diffficulties. self-testing is much more effective revision than re-reading. Re-reading makes pupils feel familiar with the content but doesn’t guarantee thought. Testing makes it clear if students have understood something.

Generally assessment should not take place too close to the period of study as we can’t make a valid inference about whether a pupil about whether students have learned the material. If a student gets the question right very soon after study we are not provided with a valid inference. Some of the questions set for recap at the start of the lesson or for homework or at the end of the lesson should cover previously learned material.

Recording grades frequently forces formative assessment into a summative model. We could simply stop recording formative assessment as this assessment aims to be responsive not reportive. If we do record marks these should not be converted to grades. When converting to grades you are asserting that the difficulty of the two assessments is the same and that you are trying to derive a shared meaning. Also, the aim of formative assessments is to set questions that are closely tied to what is being studied.

Notes on making good progress? by Daisy Christodoulou

In this series of posts I record my notes from Daisy Christodolou’s book “Making good progress? The future of Assessment for Learning” It is quite excellent. You can buy a copy here.

Notes on making good progress?: Chapter 6

In this series of posts I record my notes from Daisy Christodolou’s book “Making good progress? The future of Assessment for Learning” It is quite excellent. You can buy a copy here.

Life after ‘Life after levels’: creating a model of progression

The two assessment systems described in the previous two chapters suffer from the same flaws:

  • The expect the same assessment to produce two very different inferences
  • lead to overgrading and overtesting
  • lead to unhelpful feedback
  • lead to the measurement of formative progress with summative grades
  • inadvertently encourage a focus on short-term performance and discourage long-term learning.

The purpose of a grade is to describe performance not measure progress. No assessment system can succeed unless it is based on a clear and accurate understanding of how pupils make progress in difference subjects.

Assessments have to be selected and designed with reference to their purpose. Different assessments serve different purposes and have to be designed accordingly. We cannot rely on one assessment or style of assessment for the all the assessment information we need. Pupils who get better at decoding phonemes do become better readers; those who establish a clear sequence of historical events do get better at source analysis.

A good assessment system must not only clarify the current state and the goal state, which is can do through the use of summative assessments, but it must also establish a path between the two: the model of progression.

Textbooks have a role here. They can be used to communicate the the model of progression. In science they provide exemplars, it is noted in Kuhn’s work that scientists gain expertise by learning many examples. Textbooks offer an effective and detailed way of communicating a progression model.

Modern textbooks can look very different to the older ones, as they can now be online and do not have to feature just prose.

A progression model needs to be specific, not generic, and it needs to break complex skills down into small tasks that do not overload pupil’s limited working memories. As the model builds, pupils will be able to manage more complex tasks because they have memorised and automated the initial steps, but the model must start with the basics.

It will look different in different subjects and for different concepts within the same subject. Teachers are required to make decisions about what tasks are most likely to lead to the attainment of the end goal in that particular topic.

We need to clarify what the final aim of education is and we must use these aims not exam success to build our progression model. Exams are only samples of wider domains and because of this, there will always be ways of doing them well that do not lead to genuine learning. However if we set mastery of a domain as a goal the exams will be valid measures.

Goodhart’s law: when a measure becomes a target it loses it’s value as a measure.

If our end goal is success on an exam we will end up with a progression model which leads to exam success but not to the wider goals we really want.

Teaching to the test and exam prep does not correspond to the problems that students will face in real life so, if they have focussed excessively on these types of questions it will compromise the validity of the results.

If pupils are graded every term or every few weeks and dramatic improvements are expected then cramming and teaching to the test is likely the only methods that will provide short term improvements.

Memorising the right thing vs the the wrong thing is exemplified by memorising model essays or memorising lines of poetry. Memorising poetry  helps pupils move towards the end goals of the English curriculum in ways that are not achieved by memorising essays.

How should we make decisions about what knowledge is worth remembering and what isn’t? Daniel Willingham provides some pointers – how could these be applied in group 4 specifically bio?

Lessons should be viewed in the context of the progression model. Remembering somethings will create meaning, other not so much. The same lesson may or may not create meaning depending on the sequence it is part of. It is possible for a lesson to be highly effective if part of one sequence and ineffective if part of another.

In establishing a progression model we first have to establish what it is we want a pupil to be able to achieve. We have to define this in terms of the fundamental concepts we want them to master, not in terms of exam success.

Isobel Beck recommend a list of 400 words per year to be taught for the first 10 years of education. The research on teaching vocabulary also  suggests that pupils learn examples of vocabulary in context rather than definitions.

Subjects may not always be the best way to think about progression models. Some traditional subjects have arbitrary content. Some subjects have content that is in other subjects in different national systems.

Progression models should be focussed on concepts that we want students to acquire. The metaphor of marathon training for a progression model is particularly helpful

Burn, heretic, burn!

Once upon a time in the West, if you believed in the transubstantiation of bread and wine during Holy Communion and you lived in one part of Europe you got burned at the stake. If you denied this small fact and lived in a different part of Europe you also got burned at the stake. It didn’t matter if you agreed on 99% of the other details of your religion, you still killed those with slightly different views. Humans do that darnedest things to each other based on the most trivial of differences.

Thankful we are all humanists now to a greater or lesser extent (whether you accept it or not) and therefore it isn’t acceptable to burn each other. In his books Noah Yuval Harari charts the course of the three great humanist traditions of the 19th and 20th centuries: liberalism, communism and facism. All of these traditions placed mankind and the human experience at the centre of their creeds, as opposed to an almighty, thats what makes them humanist.

We now live in the area when liberalism has triumphed against the others, according to Harari. Even as a conservative you are a liberal, in the sense that you believe in the rights of the individual, freedom of the individual, and the equality of individuals. Democracy is the flowering of liberalism in politics. Everyone’s vote is equally valid.

Like all religions, humanism and, specifically for this thought trail, liberalism has its schisms. We humans love to be tribal and to argue. In someways it is what makes us human. Identifying who isn’t in our tribe helps us identify who is. We depend on our social interactions within our tribe.

Indeed Harari, likens the intellectual differences and squabbles between  humanist tribes to be not too dissimilar to the tribalism that erupted in Europe within Christianity, best exemplified by the Spanish Inquisition, which murdered hundreds of people over differences in the interpretations of the bible.

If you have spent any time on Twitter as a teacher you can’t possibly have avoided the prog/trad squabbles, rows and playground name calling, highlighted this week by closure of Debra Kidds account.

It’s a shame that the greatest CPD tool for teachers also highlights so much of our  worst social natures.

Despite the protestations of some, the debate between progressive education and traditional education (the prog/trad debate) doesn’t just exist on Twitter. It’s obfuscated because teacher training courses don’t teach education history (to my knowledge) and generally they aren’t balanced in discussing pedagogical approaches (again in my experience).

Any honest reading of the history of the ideas in education can trace the debate back to at least the early 1800s. Hirsch provides a decent overview in the appendix. In the wake of the American war of independence and the French revolution new ideas about the progression of humanity began to take hold. Nothing happens in a vaccum. As the ideas of the intellectual founding fathers of liberalism, communism and fascism spread, they were also to influence ideas about education.

I don’t intend to recount that history here, as much better has been written about it but with the, sometime vehement, differences in opinion between proponents on both sides of the debate, it is easy to forget that, ultimately, according to Harari, wether you identify as trad, prog, trad/prog, atradprog, we are all children of the great intellectual revolution of liberalism.

We all believe in the rights of the individual. We all believe in equality. We all believe in the individual freedoms of adult members of a civilised society. We just disagree on methodology and approaches of indoctrinating and raising adults into this society.

Those advocating a traditional approaches do not do so because they are sadists. They do so because they believe these are the best methods of reducing inequality, and helping all individual children fulfil their potential.

Those advocating progressive approaches do not do so because they have a hidden agenda to keep an elitist society propped up. They do so because they believe that these are the best methods for ensuring individual freedom and individual expression, as well as helping all individual children fulfil their potential.

And to be honest, I think most people would probably describe themselves as mods.

In a sense this debate is simply a practical outworking of the inherent tensions within liberalism: those of ensuring individual freedom and of ensuring equality. It’s hard, in any society, to have both.

So next time you feel like throwing a stone, just remember, you’ve got more in common that you think. It also might be worth remembering that without tone or body language the written word can be so easily misunderstood.

At least no teacher in the Twittersphere has literally burned another teacher at the stake for professing different views….yet.