|COOKIES: By using this website you agree that we can place Google Analytics Cookies on your device for performance monitoring.|
How do examiners reach judgements?
If you have a question about this talk, please contact thenetwork.
To register please contact the Network Team on 01223 553846 or email@example.com
Examiner judgements are an essential part of the assessment process. In this session there will be two presentations looking at how examiners in different contexts arrive at their judgements, and what variables have an influence on the judgements that are made
Cognitive cherry picking: the patchwork process of examining A level essays (Victoria Elliott)
The assumption of the A level system is that examiners make their decisions in a rule-based, logical way. However, given the time constraints under which examiners work, and the amount of information which must be considered, it is unlikely that the cognitive process is as rational as it is conceived to be. Essay subjects, such as history and English (two subjects which are associated with difficulty of marking), are likely to form a particular challenge in terms of the limits of working memory. The fact remains that most examiners mark within accepted margins of reliability. How, then, do they make these decisions, given the amount of information and the limited time available?
This paper considers some of the data gathered during an in-depth study of examiners’ training and decision-making, utilising Verbal Protocol Analyses of live marking and recordings of training meetings. A wide range of cognitive processes are demonstrated in the data, which are used to varying degrees by different examiners, at different times within and between scripts, according to the most useful, and potentially the most economical method at any one time.
Influences on rater judgements of second language speech (Dr Talia Isaacs)
The overall goal of this presentation is to examine rater perceptions of second language (L2) speech to identify possible sources of rater bias and shed light on the focal construct. First, we will examine the effects of individual differences in rater cognitive variables (musical ability, phonological memory, and attention control) on raters’ judgements of L2 comprehensibility (ease of understanding), accentedness (degree of foreign accent), and fluency (speed of delivery and smoothness). If these cognitive variables are found to exert a measurable influence on the scores that raters assign, then this could pose a threat to the validity of the assessments. Next, we will examine the way that ‘comprehensibility’ is operationalised in current L2 speaking scales (e.g., Cambridge ESOL Common Scale for Speaking, IELTS pronunciation scale). We will then examine how a greater understanding of the linguistic dimensions that underlie listeners’ L2 comprehensibility ratings can elucidate our understanding of the construct at different levels of ability. Implications for rating scale validation and for rater training in high-stakes assessment settings will be discussed.
This talk is part of the Current Issues in Assessment series.
This talk is included in these lists:
Note that ex-directory lists are not shown.
Other listsAssessment Principles Andrew Chamblin Memorial Lecture 2013 PDN Postdoc Symposium Plenary talks
Other talks‘Mixing It Up: A Blended Approach to Practice-based Research Training in Music’ Playability of bowed string instruments Nodal length fluctuations for arithmetic random waves. The 2013 Stem Cells Discussion Forum: Working Towards Clinical Application Mass Spectrometry Stable and unstable equilibrium points in the quantum Gaudin model