A new stage in the Crowdsourcing Language Assessment project

In 2020, CECL members Magali Paquot, Rachel Rubin, and Nathan Vandeweert launched the “Crowdsourcing Language Acquisition” project (CLAP), which aimed to develop a novel method for generating fast and reliable assessments of texts in learner corpora.

The method is based upon an assessment technique called Comparative Judgement (CJ). In CJ, judges are shown two pieces of student work and asked to decide which is “better”. By aggregating the results of many such comparisons, the work can be ranked in a manner which reflects an emergent group construct of the target proficiency. Subsequently, a score can be assigned to each piece of work.

The results of this first stage of the project were recently published as Paquot et al. (2022) ‘Crowdsourced Adaptive Comparative Judgment: A Community-Based Solution for Proficiency Rating’, Language Learning 72:3. The study demonstrated that crowdsourced comparative judgements yield reliable and accurate rankings of English L2 essays, when those essays differ widely in quality.

Thanks to the award of an FNRS research grant to project leader Magali Paquot, we are now able to launch a new stage in our project. In this stage, we hope to broaden our existing findings to the evaluation of texts of relatively similar proficiency, or which differ in topic. The project will also allow us to grade a carefully selected subsection of texts from the ICLE corpus, and will move us several steps closer to a standardized methodology for generating grades for texts in other learner corpora.

Can you help us with this project?

We would like to invite you, your colleagues, or even your students (English proficiency at or above B2 level, and experience of writing Academic argumentative essays in English) to complete 5-10 comparisons of academic argumentative essays. After completing an initial Google Form, you will be taken to the experiment platform No More Marking. There, you will be shown pairs of academic argumentative essays written by learners of L2 English, and will be asked to choose which is the most advanced piece of L2 writing. You can expect this to take 3-5 minutes per comparison.

To begin participating, please follow the link below.

The link is: https://forms.gle/Y5JTZzhBhgnFSzC16

If you have any questions about the project, or would like to discuss the possible participation of student groups, please contact us at clap-admin@uclouvain.be.

Many thanks!

Peter Thwaites & Magali Paquot, UCLouvain.

Published on January 16, 2023