How do you assess the real-world impact of the social sciences and humanities? A team of researchers at the Leverhulme Centre for Demographic Science (LCDS) delved into around 4,000 Research Excellence Framework 2021 (REF) Impact Case Studies to reveal the impact of social sciences and humanities research.
Published today by the British Academy and Academy of Social Sciences, researchers at the Leverhulme Centre for Demographic Science assessed how the social sciences and humanities research produced at universities in the UK provides real-world impact on the economy, policy, industry, technology and enriches everyday lives.
The project report and online interactive dashboard (accompanied by an open source code library) reveals the substantial impact of UK social sciences and humanities research, within the UK and globally. Combining complex statistical analysis and Large Language Models (LLMs) with rich qualitative explorations enabled LCDS researchers to tell the story of the myriad ways in which impact was achieved.
Project lead Professor Melinda Mills, Director of the Demographic Science Unit and LCDS said, ‘It was fascinating to uncover the breadth of real world impact from social sciences and humanities researchers within government and industry to the improvement of everyday lives. ’
The project team identified the impact of social sciences and humanities research using the UK’s Research Excellence Framework (REF) 2021. The REF is the UK’s system for assessing research excellence in UK Higher Education Institutions (HEIs), with the results used to allocate around £2 billion of public funding to university research each year. Demonstrating the real-world impact of research teams via the submission of Impact Case Studies is a key element of this evaluation process, making up a quarter of the total assessment, which made the last REF2021 the largest structured research impact assessment worldwide.
The project team at LCDS applied a novel mixed-method approach combining Large Language Models (LLMs) with qualitative assessment and validation to analyse the impact of all 4,000 social sciences and humanities research Impact Case Studies that were submitted by UK HEIs to REF2021 (all submissions to the Main Panels C for Social Sciences and D for Arts and Humanities as well as to Unit of Assessment (UoA) 4 for Psychology, Psychiatry and Neuroscience).
Given the textual nature of Impact Case Studies, Dr Charles Rahal and his team employed a LLM that would analyse the massive corpus of text (each around four to five pages long) within the 4,000 case studies. These models would then use language proximity in the written text of the case studies to categorise them into 83 distinct topics ranging from ‘Music’ to ‘Renewable Energy’, as outlined in the report figure below.
Results from the model were then described and refined by an extensive qualitative effort, looking at all case studies individually. Dr Charles Rahal, Senior Research Lecturer at the Demographic Science Unit and LCDS said, ‘A mixed-methods research design allowed us to maximise the benefits of Large Language Models in categorising complex topics, reinforced by a ‘human in the loop’ approach to sense-check the algorithmic clustering. This ensured the best possible outcomes for robust and reliable scientific inference.’
The qualitative analysis was led by Dr Sander Wagner and a team of researchers at LCDS who individually read all 4,000 Impact Case Studies to ensure that they were correctly classified and assigned by the algorithms, leading to the creation of an extra 84th topic. This qualitative analysis allowed them to weave a rich narrative which documents the topics emerging from the algorithmic mapping.
A qualitative team also conducted interviews with panel reviewers and authors of Impact Case Studies, which allowed readers to make sense of the algorithmic mapping of impact and bring the themes to life with concrete examples. The breadth and depth of the impact that was achieved by social sciences and humanities research is shown by detailed, in-depth accounts of the ten ‘Grand Themes’ which our models uncover, all based on the 84 topics, that are outlined in the report:
- Arts, Language and Design
- Archaeology, Exhibits and Galleries
- Education, Teaching and Skills
- Business, Economics and Management
- Employment
- Crime and Exclusion
- Family and Gender
- Governance and Law
- Health and Wellbeing
- Sustainability and Infrastructure
The report combined the case studies with extensive external data to explore several additional themes. UK Research and Innovation (UKRI) represented a quarter of funding acknowledgements (25.64%). This was striking given that the Economic and Social Research Council (ESRC) and Arts and Humanities Research Council (AHRC) only receive 2.4% and 1.3% of the UKRI budget respectively.
The research underpinning impact case studies was found to be highly interdisciplinary, meaning that researchers from several fields worked together to produce an underpinning research paper, and multidisciplinary meaning that research papers from different disciplines came together in creating a particular impact. Interdisciplinarity occurred mostly between closely related fields such as ‘Health Sciences’ and ‘Biomedical and Clinical Sciences’, whilst multidisciplinarity bridged greater disciplinary distances and was found often across disciplines such as humanities and medical sciences.
The researchers also found that larger departments tended to have higher ‘Grade Point Averages’ compared to smaller and newly created units, finding a diverse range of gender balances in the authorship underpinning the research both across different UoAs, and between the social sciences and humanities and STEM disciplines.
The report concludes that social sciences and humanities research in the UK is multidisciplinary and interdisciplinary, fuelled by a handful of core funders, and has a global impact that goes beyond the boundaries of economics, policy, and technology. The analysis also suggests that the breadth of impact from social sciences and humanities research warrants new impact type categorisations in the REF framework.
The case studies, topics, and impact of research that are outlined in the report can be explored further in the project’s online dashboard that brings together the quantitative and qualitative elements of this analysis. Users can view Impact Case Studies for each of the 84 topics, explore their top funders and global impact beneficiaries, and get an overview of the UK HEIs researching these different areas of the social sciences and humanities. The dashboard also provides access to the project’s enhanced REF database and rich, licensed bibliometric information used in the analyses and results, which users can download and use free of charge.
Dashboard developer Dr Douglas Leasure, Senior Researcher and Data Scientist at the Demographic Science Unit and LCDS said, ‘This new dashboard is a great resource for academics and UK institutions involved in the REF process, and it acts as a powerful tool for funders and policymakers to highlight how research in the social sciences and humanities improves our lives.’
Professor Melinda Mills (FBA, FAcSS) will present the main findings of the report and online dashboard today at a British Academy and Academy of Social Sciences online launch event.
Professor Melinda Mills said, ‘Our use of generative AI, combined with human coding and qualitative research, allowed us to discover over 80 core areas of impact, all of which can be explored on our interactive online dashboard. Highly impactful research is also clearly more inter- and multidisciplinary, with social sciences and humanities researchers often making impact within medical and physical science realms, highlighting the importance of this type of research within STEM and beyond.’