Logo image
Back
And the winner is … : inter-rater reliability among scholarship assessors
Journal article   Peer reviewed

And the winner is … : inter-rater reliability among scholarship assessors

L. Johnston and P.J. Schluter
Studies in Higher Education, Vol.42(9), pp.1736-1749
2017
DOI: https://doi.org/10.1080/03075079.2015.1124849
url
Link to Published Version *Subscription may be requiredView

Abstract

With increasing competition for postgraduate research scholarships, awarding processes demand attention and scrutiny. We examine inter-rater reliability for two prestigious New Zealand scholarships, the Shirtcliffe Fellowship and the Gordon Watson Scholarship. For each scholarship, five assessors (three academic; two non-academic) independently evaluate all applicants over three domains: Academic Merit, Quality of Study Plans and Character/Leadership. Data from years 2009 to 2014 were extracted, comprising 12 separate assessment rounds. Good to excellent agreement was observed for each scholarship in each year. Agreement was significantly higher for the Academic Merit domain compared to the other domains. Moreover, agreement among academics was higher and less variable than non-academics for this Academic Merit domain. No such differences were noted in the other domains. While resource efficiencies could be made, reductions in committee size resulted in poorer applicant selection performance. Applicants and donors alike can be confident that the awardee for these scholarships is a top applicant.

Details

Metrics

24 Record Views
2 Times Cited - Web of Science

InCites Highlights

These are selected metrics from InCites Benchmarking & Analytics tool, related to this output

Citation topics
6 Social Sciences
6.238 Bibliometrics, Scientometrics & Research Integrity
6.238.166 Bibliometrics
Web Of Science research areas
Education & Educational Research
ESI research areas
Social Sciences, general
Logo image