Measuring Impact: what counts as evidence in language education?

PRIA and the Centre for Excellence in Language Teaching (CELT) cordially invites you the first of the CELT 2024-2025 Methodology Workshops series

PRIA and the Centre for Excellence in Language Teaching (CELT) cordially invites you the first of the CELT 2024-2025 Methodology Workshops series. We kick start the programme with an event on Measuring impact and a discussion on what counts as evidence in language education.

Aim: The aim of this session is to equip participants from across the Faculty with tools to design reliable and ethical impact studies to suit their context and their purpose, and to engage in informed discussions with all stakeholders in Higher Education, levelling their teaching and scholarship expertise to arbiter what counts as evidence.

Abstract:

Educators and institutions’ concern about ‘evidence-based’ teaching practices has led to a demand for evaluation of impact in language education programmes. Measuring impact is associated with best and reflective practice but can also lead to performative scholarship, and a narrowing of teaching practices (Ding, 2016). Indeed, despite its documented ethical and reliability limitations, the gold standard in impact measurement is still associated with the quasi-experimental approach with randomised control trials, but can have a detrimental effect on teaching practices, practitioners’ agency and creativity, and leaners’ experience (Derewianka et al., 2024). Qualitative studies that involve students’ perceptions of impact are also common and similarly present strengths and weaknesses (Bassett and Macnaught, 2024). In this session, we aim to provide an in-depth and nuanced discussion of what might counts as evidence in language education, taking into account the different aims of impact studies and the contextual constraints in which they might be conducted. We see language educators and practitioners particularly well placed to carry out meaningful studies into student’ learning and the impact of our work in the various campus communities,

This session aims to critically unpack the notion of impact, to tease out the different types of impact we might want to evidence, the different purposes of impact studies, the contextual constraints, the various stakeholders likely to take an interest, and how this impacts the choices of methods and ethical considerations. The speakers will present a range of methods and will discuss their strengths and drawbacks and why they adopted these methods in their scholarship. Approaches discussed include perception and performance studies including text analysis, talk-around-text, and quantitative measurements. We hope participants contribute to the discussion with their own experience and approaches to measuring impact.

References

Bassett, M. and Macnaught, L., 2024. Embedded approaches to academic literacy development: a systematic review of empirical research about impact. Teaching in Higher Education, pp.1-19.

Derewianka, B., Harper, H., Parkin, B., Acevedo, C., Rose, D., Dare, B., Brisk, M.E. and Jones, P., 2024. What counts as ‘evidence’ in literacy education? The Australian Journal of Language and Literacy, pp.1-18.

Ding, A. 2016. Challenging Scholarship: A thought piece. The Language Scholar.

Speakers

Bee Bond is Professor of English for Academic Purposes (EAP). She has been working with her colleagues in the Language Centre to investigate the impact of insessional EAP classes on student learning and understanding of their disciplinary knowledge and communication genres. This project gained umbrella ethical approval to cover a range of approaches to considering impact and this had led to a collective consideration of what impact means and how/ if it is possible to measure it.

Clark Girdlestone is an EAP practitioner at the University of Leeds who leads the MSc International Business EAP in-sessional. Most recently, he presented the findings of an in-sessional impact study at the University of Leeds Student Education Conference 2024.

Laetitia Monbec is Lecturer in English for Academic Purposes and the Director of Scholarship at the Language Centre. Her thesis investigated the impact of an EAP module on students’ writing in their STEM disciplines.

Isabel Molina Vidal is a Spanish Lecturer at the University of Leeds. She is also completing her PhD on the use of text-based online chat to improve grammatical accuracy in advanced learners of Spanish with the School of Education at the University of Leeds. She has been awarded the Fellowship in Higher Education Academy FHEA. Her areas of interest are the use of technology (synchronous computer-mediated communication text chat) for language learning, and the teaching of grammar from a cognitive point of view.