Notes on making good progress?: Chapter 2

In this series of posts I record my notes from Daisy Christodolou’s book “Making good progress? The future of Assessment for Learning” It is quite excellent. You can buy a copy here.

Aims and methods

The generic skills approach implies that skills are transferable and that the best way to develop a skill is to practice that skill. It is based on the analogy of the mind acting as a muscle.

There are examples of curriculums that follow this model like the RSA opening minds curriculum; it is interesting to contrast this to the core knowledge curriculum mentioned by Hirsch and the DI models.

Instruction based on this model is organised around developing transferable skills through projects where students practice authentic performances.

However research from 50 years of cognitive science shows us that skill is domain specific and dependent on knowledge. The examples of studies into Chess grand masters is given. These were the earliest experiments but have been reliably repeated in other knowledge domains. There are multiple lines of evidence that suggest the same thing (A bit like evolutionary theory).

Complex skills depend on mental models which are specific to a particular domain. These models are built in long term memory. They can be drawn on to solve problems and prevent working memory from being overloaded.

Working memory is highly limited and relying on it to solve problems is highly ineffective. Learning is the process of aquiring these mental models. Performance is the process of using them.

Formative assessment should aim to assess how these mental models are being developed. Summative assessment measures the performance or act of using those mental models.

Even scientific thinking is domain specific. One cannot evaluate anomalies or the plausibility of a hypothesis without domain specific knowledge.

The adoption of generic skills theory leads to a practical flaw: lessons are too careless about the exact content knowledge that is included in them (this also ties in with Hirsch’s ideas that individualisation leads to a reduction in knowledge). If skills are not transferable we need to be very careful about what the content of a lesson is. Specifics matter.

The educational aim of developing generic skills is sound but we need to think about the method. We can develop critical thinking only by equipping learners with knowledge. Good generic readers or problem solvers have a wide background knowledge.

Aquiring mental models is an active process. project based lessons can work as they help students to make the knowledge their own, its just that sufficient care and attention to the content that is to be learned is applied.

Specific and focussed practice is what is needed to develop skill. As shown by the work of K. Anders Ericsson, there is a difference between deliberate practice and performance. Deliberate practice builds mental models, while performance uses them.

What is learning and how is it different from performance? Learning is the creation of mental models of reality in long term memory. Copying what experts do (performing) is not a useful way of developing the skill of the expert because they do not build the mental models. Instead these lessons will overwhelm working memory and will be counter-productive.

Generic skill models only allow feedback that is generic, it does not allow are feedback to tell students exactly how to improve. “think from multiple perspectives more” is not useful advice if kids don’t know how to do it.

Models of progression are needed to show the progress that students are making.

Peer and self assessment can be useful so long as they are used appropriately. The Dunning Kruger effect shows us that novices cannot judge their performance accurately (does this contradict Hattie’s claim about self reported grades that kids know where they are at?). Developing pupils ability involves developing their ability to pervcieve quality. We cannot expect them to self assess complex tasks and showing them excellent work is not enough to develop excellence. Particular aspects of work need to be highlighted by the teacher.

To develop skill we need lots of specific knowledge and practice at using that knowledge. This helps to close the knowing-doing gap.

Notes on making good progress?: Chapter 1

In this series of posts I record my notes from Daisy Christodolou’s book “Making good progress? The future of Assessment for Learning” It is quite excellent. You can buy a copy here.

Why didn’t Assessment for Learning transform our schools?

Formative assessment is when teachers use evidence of student learning to adapt instruction to meet student need. It’s focus is on what students need to do to improve, on their weaknesses. Feedback then, needs to be tailored thoughtfully to direct the student in how to improve and allow students to act on that feedback.

Formative assessment should be used to diagnose weakness and feedback should tell students how to improve explicitly. AfL is not just about teachers diagnosing weakness and being responsive it is about students responding to information about their progress. Could be a good model for appraisal too.

There is a tension then, between summative and formative assessment. One is about measuring student progress against the aims of education while formative assessment is about the students finding out what they need to do to improve.

If we can agree on the aims of assessment, there is still a discussion about methods. We either favour the generic-skill method which states that to get better at a particular performance you just need to practice that performance. So if you need to practice critical thinking you just practice it or if you want to get better at writing an essay, you just write lots of them. Or we favour the deliberate practice method. This method breaks the final skill down into its constituent parts and practices those. So footballers practice dribbling, passing, defending and shooting not just playing whole games all the time.

Summative assessment is about assessing progress against the aims of education. Formative assessment is about the methods you choose to meet those aims: generic or deliberate. Depending on which one of these you subscribe too will affect your formative assessment, and thus whether assessment tasks can be used formatively and summatively.

If you believe that skill acquisition is generic then formative assessment tasks will match the final summative task. You will write lots of essays, feedback can be given and a grade awarded. If you believe that the method of deliberate practice is better then you may need to design formative tasks that don’t look like the final task. These tasks cannot be used summatively because they don’t match the final task.

Interestingly, belief in generic skills leads down the road of test prep and narrow focus on exam tasks because this model suggests that to get better at the exams you do need you need to practice taking them.

In my mind the key questions for a school, curriculum level or department that wants to adopt the deliberate practice model should be:

  • What are the key skills being assessed in the final summative tasks (don’t forget that language or maths skills might be a large component of this?
  • What sub-components make up these skills?
  • What tasks can be designed to appropriately formatively assess the development of these sub-skills?
  • What does deliberate practice look like in my subject?
  • How often should progress to the final summative task be measured i.e. how often should we set summative assessments in an academic year that track progress?

Whole school support for EAL learners II

Imagine a normal primary school in an anglophone country like the UK or US. Now imagine taking a year 4/grade 3 or year 5/grade 4 child from that school and giving them an academic program aimed at year 12/grade 11 or year 13/grade 12 students. It could be AP, A Levels, IB DP. The course doesn’t matter here. Lets just assume that these children would be taking academic, pre-university courses in the the humanities/social science and the natural sciences. For the sake of argument, lets assume that these fictional children have the social and emotional skills of 17-18 year olds. Clearly I am not describing a real situation here.

From a purely academic point of view: what would happen? Would those children succeed? Would they have the background knowledge, understanding and vocabulary skills to access in class discussions? Or text books for that matter? Or even to understand what the teacher was talking about?

Now, I wonder, how would the teachers, tasked with teaching these children respond? What strategies could classroom practitioners employ to help their students achieve? How could the curriculum coordinators and Heads of Year respond to implement strategies to allow the children to access the curriculum? What would you do?

What makes an EAL student like a primary schooler?

Of course, this never happens in practice or does it? Is there any cohort of students in international schools that would somewhat match this description? I would contend that there are, to varying degrees, and in varying numbers, students who fit this description as EAL students.

Now clearly, an average 17 year old student, has cognitive abilities beyond that of an average 10 year old and certainly, we would hope, more advanced social and emotional skills. And indeed they probably do know more.

But how do we ensure that, when a high school accepts an older student who has never had any prior formal instruction in academic disciplines in the language of the school, and will ultimately sit exams in that new language, this child will be able to succeed.

Some might answer that schools shouldn’t admit students when they cannot meet their needs. I would agree. But I have seen schools that do admit students when they can’t meet their needs; usually when a child’s needs meet the economic needs of a school, the latter concerns tend to win.

My concern here really revolves around the question: If most major testing systems in the English Language (AP, IBDP etc) are norm referenced, then aren’t we simply propping up the performance of our native language speakers with the ultimately poorer performance of non-native speakers? Are our anglophone speakers succeeding on the back of the poorer performance of our EAL students (on an international level)?

Of course, in international schools, there is a lot of variance and there is certainly flexibility in the system. Most students who can’t access the full curriculum will be able to graduate from the school with some form of modified curriculum. But we need to ensure that students have as many options available to them when they leave us as possible. Going to an international school is a privilege and affords so many additional benefits to kids that they may not have had in there home country but we need to ensure that students are able to succeed after they leave us.

How do we solve these problems?

In practical terms when, as a coordinator, I have a cohort of students for the majority of whom English is a second language and many of whom have only been learning their academic subjects in English for a few short years, how do I put strategies in place to support them as best I can?

I have written here, here and here in the past about classroom strategies for teaching upper secondary curriculums to EAL students. I am an interested novice. But now as a coordinator I am concerned about curriculum level interventions.

The context will matter both in terms of the cohorts profile and the curriculums that can be offered as well as their flexibility. I coordinate the IB, which is a flexible system in the sense that, when combined with an American style High School Diploma, students have the option of taking IB certificates in as many or as few courses as they would like.

But I am blue-skying today and want to think about how to offer the full Diploma to as many of my students as possible in this imaginary cohort.

Making the Diploma accessible

There are ways to do this but it may require restrictions in certain areas, for example limiting extended essay subject selection to the students mother tongue or English B if the students level of English is so low that the team feels this would preclude them for taking the extended essay in another academic subject, like business studies or economics for example.

And what level of English is too low? Whats the cut off? Recently I have discussed, with colleagues, using lexile analysis to determine what the English grade reading level is of my EAL students as well as the lexile score. This is a measure of how dense a text is. The lexile score is useful for a number of reasons. It can be used to work out what the equivalent reading age in English is for the EAL students and it can compared to the lexile level of the textbooks used on the course, allowing teachers to the see the difference in where there kids are at and the material they need to present.

The lexile analysis of a biology textbook. The level ranges from Y13/G12 to post secondary!

Lexile analysis can be performed here. Teachers can set up their own accounts but I think this should be done centrally on a term by term basis or semester by semester basis and the information shared with students and their families, as well as teachers as part of a set of on going sharing of strategies and training on support EAL students in the academic classroom.

Hirsch (2016) claims that “Vocabulary size is the outward and visible sign of an inward acquisition of knowledge.”Lexile analysis therefore shows us not only what these students can read but what they know in English as well. Hirsch makes the case that the more domain specific knowledge students acquire, the more their vocabulary naturally increases. This is why, for Hirsch, knowledge rich elementary curriculums are so important. They ensure that students acquire vocabulary and this vocabulary acquisition is the magic formula for reducing inequality. Children from affluent families have more vocabulary when they start school (they oral life at home is richer) compared to their disadvantaged peers and knowledge curriculums help them to catch up.

In a sense our EAL students are like disadvantaged native language children; they certainly don’t benefit from homes where English is spoken and so they don’t benefit from expanding their knowledge and vocabulary in English when they leave school.

The matthew effect shows how learners who have knowledge will tend to acquire more at a faster rate and those with less will acquire knowledge more slowly. This is one of the important psychological principles often overlooked by commentators who claim if we teach knowledge then our kids will be competing with computers. Teaching knowledge is the only way to ensure that they can be life long learners; the more knowledge we have in our brains the quicker we gain new knowledge.  This is also known as the knowledge capital principle it takes knowledge to make knowledge.

Hirsch also claims that “High school is too late to be taking coherent content seriously” as part of his argument for knowledge rich elementary curriculums. Where does this leave our EAL students?

Evidence from cognitive science also shows us that knowledge is domain specific and that it doesn’t transfer readily. Thus students may now about the detailed components that make up the processes of photosynthesis in Korean, but they are unlikely to be able to transfer this knowledge from Korean into English. This creates real problems when it comes to supporting EAL students in the mainstream academic classrooms.

Taking all of the above int account, it seems that we need to begin by getting students exposed to speaking and thinking in English as much as possible.

Let me be clear here, as I have run into hot water on this one in schools. If the aim of a school is to have students graduate by passing English language academic exams for whatever greater purpose, then I think that in school, whenever possible, students need to be encouraged to speak English. I don’t say this because I am a cultural imperialist but because it is demonstrably the best way of getting students to learn the academic subjects, most of the time.

As an IBDP Coordinator this means, among other things, ensuring that students get as much time in the English acquisition classroom as possible. I would consider placing all the students into the English B HL class  at the start of their course. This would give them more hours in the acquisition classroom initially. As they progressed through the course we could look at their progress to see if they could afford to drop down to SL.

Clearly there is a balance to be struck here. Forcing kids to be taking an HL subject they might not be into could seriously backfire in terms of motivation and so continual communication with teachers, students and parents is essential.

To ensure that students felt like they were making progress (and therefore maintaining their motivation – psychology) I would consider having dedicated EAL support after school. This time would be given over to allow the students to do grade-levelled reading in English.

I also apply the IB research discussed in this post to ensure that their is ongoing monitoring of the learners progress, too often students are assessed at the beginning of the year and never again. Ongoing, regular assessment of learners progress is necessary here.

Since beginning to write this, I have been introduced to a piece of software that appears to be an answer to some of these questions.

I hope that ongoing posts on this topic will help me explore the strategies that can be put in place to ensure all learners succeed.

References

E.D. Hirsch (2016) Why knowledge matters: Rescuing our children from failed educational theories. Harvard Education press

Reflections from examining 2018

This season I marked 140 IB DP Biology HL Paper 2 Timezone 1 papers. It was unusual for a couple of reasons: 1) I managed to pass the qualification marking on the first attempt for the first time in six years! 2) I managed to complete my marking target within seven working days and nine days before the deadline – the first time I have managed to complete the work so quickly.

I felt that this years timezone 1 exam was very straightforward to mark. This was particularly evident in the data analysis responses where the mark scheme was much easier to interpret than I recall previous years being.

Qualification

To qualify for marking, normally there are practice scripts and qualifying scripts to mark. The practice scripts are a chance for you to view comments from the senior examining team, so when undertaking these it pays to go very slowly, really thinking about how the mark scheme applies in each question and when you have marked each question, checking your own marking against the comments by toggling on the annotations. Using this method you may become quickly aware of any small details in the comments that you have missed.

In the past when I have undertaken the qualifying scripts I have opted to mark them in bulk and then submit them in bulk, so I would only submit the scripts once I had marked all of the papers. This year, instead, I submitted each script after I had marked it. This gave me the advantage of being able to read the annotations on each of the qualifying scripts, check my tolerance and adjust my marking of each of the subsequent qualifying scripts. I think this may have been a primary reason why I qualified first time.

Student misconceptions on the paper

I marked 140 scripts and when you mark that many certain themes begin to emerge. This year worryingly a large proportion of candidates were conflating the mechanisms of global warming with holes in the ozone layer. This is not a new thing and it is a problem that I have noticed in previous years but this year the sheer number of candidates writing a confused response to the question on the mechanisms of global warming was staggeringly impressive.

In 2018, 18-year-old students are still writing that carbon dioxide creates holes in the ozone layer and this is what heats up the planet – or something similar. This needs to be addressed. A teacher or teachers somewhere must be teaching kids about the ozone layer.

Now I struggle to believe that this is the result of their biology teachers (who most likely will have studied this subject to sime depth and understand the science) and I am wondering if this is the result of colleagues in other subjects unrelated to science. We know that there is a lot of confusion about climate change in the media and that the scienitific debate is often misconstrued in the popular press. We also know that this is an issue of global importance and for that reason, other subject teachers may well address it. IB student could meet it in TOK, studies in language as well as geography and other teachers. I am wondering if there are some miseducated teachers out there who are confused on the issues of climate science and are confusing their kids. This would be a great area for practitioner research and opens up the question about the professional responsibilities of teachers who have a particular subject specialism: should teachers who are well educated on a particular topic be responsible for sharing that knowledge with colleagues who may also approach this topic in the own teaching?

(on a side note a colleague previously told me that XX and XY chromosomes were “a lie” in a discussion on LGBTQ+ issues in school).

Other misconceptions that became apparent were:

  • Candidates thought that water was an organic molecule
  • Candidates didn’t understand that DNA transcription/translation = protein synthesis = gene expression = expression in the phenotype.
  • Not understanding that linked loci are genes on the same chromosome not in the same place.

Common factual errors were:

  • Few candidates knew that glutamic acid is replaced by valine.

Reductionism and the problem of testimonial belief

For 10 weeks in term 3 I completed an online course on “Theory of Knowledge” from the University of Oxford’s department for continuing education. As part of this course, I have to submit two assignments. The second, was due at the end of the course and is copied below. The first can be read here.

What is the reductionist position as regards the epistemology of testimonial belief? Is such a view defensible, do you think?

In this essay, we will examine the nature of knowledge and the relationship of testimonial belief to it. We will look at the problem of testimony and the various ways of responding to this problem before addressing the question above.

A summary of the structure of knowledge

Some context is necessary here. I assume that justification, truth and belief are all necessary conditions for knowledge but in and of themselves are not sufficient conditions for an agent to claim knowledge. In addition, we need an understanding of the nature and type of the justification given. Normally we would require justification to be rational and based on evidence. In order to maximise true beliefs we are concerned with epistemic rationality: rational thinking and ways of thought that lead to the acquisition of a maximum number of these true beliefs. Epistemic rationality is either internal or external. If it is internal, the agent is aware of how they formed their beliefs and can justify them. If external, the agent may not be aware of how they formed these beliefs and is therefore not in a position to justify them consciously. However, if these beliefs were formed through epistemic norms, ways of acting and thinking that likely lead to the formation of true belief, we can still claim them as justified (Pritchard, 2014). This distinction is important when we consider testimonial knowledge and I will provide some examples later in the essay.

Testimonial knowledge

Testimonial knowledge is the knowledge gained by the transmission of information verbally, through reading or other activities where an agent is gaining knowledge from another agent. We depend on testimony for forming many of our beliefs. Most of what we claim to know through formal education is acquired through testimony. The knowledge that our parents impart to us is also testimonial. Testimony is therefore central to knowledge and can be a way of acquiring knowledge (I acquired true belief X through testimony) and also a way of justifying the knowledge an agent claims (Belief X is true because I was told or I read it). For example, I justify my belief that the moon orbits planet earth because I was told this in school. I also received this knowledge through the testimony of my teacher at the time.

The problem of testimony and the responses to it.

The problem with testimonial knowledge arises from our inability to independently justify knowledge that we gain through testimony. By independently verify, I mean that we cannot verify this knowledge in most cases without resorting to some other form of testimony. For example I know that the moon orbits the earth because I was told this by my teacher but if I wish to independently verify this, I normally would have to consult a textbook (a form of testimony). To illustrate this further, I could look for other means of justification: I could call NASA to ask them to verify this is the case but this would also mean I was relying on their testimony. Without actually acquiring a telescope and making empirical observations of the movements of the sun, moon and stars and making advanced calculations I would have no way of independently verifying this knowledge without resorting to more justification via testimony.

Reductionism and credulism both try to answer this problem. Reductionism claims that testimony based beliefs will always ultimately reside on non-testimonial evidence. Or, if we are to rightly hold a testimony based belief then we must also hold evidence that is not testimony based (Pritchard, 2014). This is an epistemically rational internalist position because reductionism requires an agent to know how they formed those beliefs and be able to explain how they formed those beliefs.

The reductionist position easily applies for local beliefs, things we can verify through our own perception and perhaps through our own empirical investigations, like a preschooler learning about the world through perception and empirical experience, for example, I know what a banana tastes like from experiencing it. Reductionism gets harder to apply with non-local beliefs, where we are simply unable to empirically verify a testimonial belief, for example, my belief that the moon is not made of cheese.

Credulism offers another response to the problem of testimonial knowledge. This position holds that we don’t always need independent grounds to justify a testimony based belief (Pritchard, 2014). Instead, it claims, such beliefs are justifiably held unless there is special reasons to doubt them. This is an example of external epistemic rationality where we don’t require an agent to be able to justify how they formed their beliefs so long as they have been following epistemic norms. In this case, an epistemic norm could be that being told something by an authoritative source is one way to maximise true belief. Holding to views acquired by testimony in this way is an entirely rational thing to do.

When credulism is modified thus we can begin to appreciate its advantages. For example, most of the knowledge that we learn at school and university is taught to us by teachers or experts in a particular field. Many of the things that we may wish to independently verify, we cannot. Would we say that something we learned in school or university was not knowledge? Intuitively not. We may regard facts acquired in this way as more robust than picking something up in a pub from a casual conversation. So we can have a methodology in terms of discriminating how reliable someone might be by their level of expertise.

The problem with credulism is that it can seem to make a virtue out of not knowing but of trusting (Pritchard, 2014). Perhaps we should be more sceptical of the information that we receive, after all, teachers can often make mistakes, or be misinformed themselves (I know from my experience of being one!)

Is reductionism defensible?

We can think of both reductionism and credulism as lying on a spectrum of justification. On one hand we have the reductionist who requires that every belief acquired through testimony needs to be independently verified and on the other hand we have the credulist who accepts that so long as these testimonial beliefs have been acquired through epistemic norms then there is no need for independent verification.

The reductionist position is the ideal because it forces agents to acquire more than one line of evidence to justify a true belief. Ideally agents should be able to justify those beliefs acquired through testimony via other means, be it through perception or empirical investigation but this ideal has some serious difficulties.

Firstly the process of independent verification of every belief acquired through testimony would take an extremely long time, enough to render the exercise impractical on an individual level. If an individual was responsible for independently verifying each one of their beliefs acquired through testimony, they would not be able to necessarily maximise their true beliefs. If we try to answer this by allowing many agents to independently verify different beliefs they hold in common, we run into the problem of relying on testimony from other agents again. Thus this doesn’t seem to be an epistemically rational way to maximise true belief.

The second problem that arises from the reductionists position in that it is not always possible to independently verify beliefs gained via testimony. Should we discount these beliefs as knowledge then? It seems that if we were to ignore any beliefs that we were not able to independently verify we would miss out a large number of true beliefs and would therefore be impoverished in what we know.

The third problem is in some cases it is not necessarily appropriate to independently verify our testimonial beliefs: “it is improper to place too many intellectual demands on people’s everyday beliefs. … if the reduction is possible, requiring it is overly demanding; the requirement to reduce hyper-intellectualizes testimonial justification. Young children, for instance, lack the intellectual capacity to consider complicated issues regarding the reliability of their parents or others who give them testimonially-based beliefs, and so it is improper to place epistemic demands on them.” (IEP, 2018)

Therefore whilst reductionism offers a seemingly strong answer to the problem of testimonial knowledge, it leaves us with the more problems regarding maximising our true beliefs.

Credulism too has problems associated with it. How do we know that another agent isn’t trying to decieve us? One proxy I sometime hear people use for knowing if a written argument is well founded is that the argument contains references. This shorthand is often used in informal academic online discussions within education but what if one agent is deceitful and simply puts many references so that readers will trust them?

In reality, most of our beliefs will be justified through testimony. We should strive like the reductionist to independently verify these beliefs where possible. Where we can’t we should accept those beliefs where we can be more confident of the source of the testimonial belief. In this way, our beliefs will dot across a spectrum, where each individual belief occupies a position between pure reductionism and pure credulism.

References

IEP (2018) https://www.iep.utm.edu/ep-testi/ accessed on 26th July 2018\

Pritchard, D. (2014) What is this thing called knowledge? 3rd edition. Routledge.