FAQs on the REF 2021 Impact case study database

In the REF 2021, impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.

HEIs were required to submit impact case studies that demonstrate the impacts their research has had beyond academia. The number of impact case studies each submitting unit was required to submit was based on the FTE of the submitted staff within that unit, with a minimum of two impact case studies being required for the smallest units. A total of 6,781 impact case studies were submitted to REF 2021.For access to the impact case studies please see the impact case study database at: https://results2021.ref.ac.uk/impact

Joint submissions

Higher Education Institutions were able to make submissions with one or more other UK HEI where this is the most appropriate way of describing research they have developed undertaken collaboratively. All impact case studies will have been submitted jointly, and could stem from research undertaken either collaboratively or at either of the submitting HEIs. The case studies are not designated to a particular HEI within the joint submission.

Multiple submissions

Institutions would normally make one submission in each unit of assessment (UOA) they submit in. They could, by exception, make multiple submissions (with prior approval) to one UOA. An HEI might have wanted to make more than one submission to a UOA if they had two bodies of research that fell within the scope of the assessment panel but were clearly academically distinct.

Case studies submitted in multiple submissions will be identified as belonging to multiple submissions, although it is not possible to filter case studies on a particular multiple submission.

Each HEI submitted a selection of impact case studies for assessment in the REF. An impact case study is a five-page document, describing the impact of research undertaken within the submitting unit. It also contains information about the research that underpins the impact that took place. Further information about the criteria for the submission of impact case studies can be found in the two key REF guidance documents:

Assessment framework and guidance on submissions (available at: https://ref.ac.uk/publications-and-reports/guidance-on-submissions-201901/)

Panel criteria and working methods (available at: https://ref.ac.uk/publications-and-reports/panel-criteria-and-working-methods-201902/)

The REF Impact case study database includes 6,361 documents (at 22 June 2022) submitted by UK Higher Education Institutions (HEIs) to the 2021 Research Excellence Framework (REF2021). The impact case studies were submitted in PDF format for assessment and publication, and additionally in Word format for extraction into the database, this enables advanced searching and tagging against select fields (see the ICS help documentation for more details on how this operates).

REF impact case studies generally follow a template as set by the REF criteria (see ‘Assessment framework and guidance on submissions’ Annex G for impact case study template and guidance ( https://ref.ac.uk/publications-and-reports/guidance-on-submissions-201901/) . This template has a Title and five main text sections, plus the name of the Submitting Institution and the Unit of Assessment.

Some submitting institutions omitted some of this information or modified the template. The name of the Submitting Institution and the Unit of Assessment has therefore been added as metadata tags.

In addition to the Title of the case study, the text sections of the template and the indicative lengths, as recommended in the REF criteria are:

In some case studies the sections vary considerably from the indicative word length, however all case studies were restricted to five pages in total.

Some case studies include non-text items such as institutional shields, photographs, other images, tables and embedded links.

All case studies were submitted as both PDF and in Word format within the defined template.  The sections of the impact case study in the database were extracted from the submitted word document.  The presentation of the case studies in the impact case study database follows broadly the format in which it was submitted, where transformation from a word document to HTML layout permitted.  Content of the case study was not changed during the transformation from the submitted word document to the database.

The PDF of each original document, as submitted, can be downloaded for comparison from the page displaying the individual case study.

Text has been removed (or ‘redacted’) from some case studies by the submitting institutions because it is commercially sensitive or otherwise needs to be restricted. Generally, this is indicated by the formula phrase “[text removed for publication]”, which was recommended in the REF guidance, but variants of this phrase do occur. 

REF impact case study titles in this database are drawn from the REF database. They are the titles inserted by the HEI into the REF submission system when they submitted their case studies to the REF.

HEIs were able to notify the REF team that certain case studies were ‘not for publication’. These have not been included in the database. For further information, please see: https://ref.ac.uk/guidance-and-criteria-on-submissions/guidance/excluding-parts-of-submissions-from-publication/

All REF submissions information, including downloadable submissions data, REF impact case studies, institution environment statements and unit environment statements can be used under the CC BY 4.0 licence. Use is permitted under these licence conditions: http://creativecommons.org/licenses/by/4.0/legalcode

The total number of Impact case studies submitted to the REF is 6,781.

The number of Impact case studies in the database is 6,361 (at 22 June 2022)

Redaction is the censoring or obscuring of part of a text.

Some case studies have parts of the text removed (redacted) for confidentiality reasons (for instance, commercial sensitivity). These case studies are included in the database but with this text removed.

Case studies are assigned to a single ‘Summary Impact Type’ by text analysis of the ‘Summary of the Impact’ and ‘Details of the impact’ (Sections1 and 4 of the Impact case study template). This is an indicative guide to aid text searching and is not a definitive assignment of the impact described.

In line with the approach used in REF 2014, there are eight Summary Impact Types. These follow the PESTLE convention (Political, Economic, Societal, Technological, Legal, and Environmental) widely used in Government policy development. For the purposes of introductory guidance in REF impact searching, Health and Cultural impact types (otherwise subsumed within Societal) have been added to the six standard categories. 

A text classification model was built using the REF 2014 impact case study database and the summary impact type assigned to those impact case studies (using the sections, 'summary impact type' and 'details of the impact').  This text model was trained using a random sample of two thirds of the 2014 case studies and tested against the remaining third.  This model was then used to classify the REF 2021 case studies using 'summary of impact' and 'details of impact' sections.

Most REF impact case studies relate at some level to more than one type of impact. Some case studies arguably cover all eight. Tagging supports rapid initial searching that will reveal a deeper and more diverse range of impact; user perspectives on this will vary.

Please note, there is an ongoing independent study of the Impact Case Studies (ICS) that will produce an enhanced database containing additional metadata fields based on text mining, reference linking, and geoparsing. This enhanced database will also inform a synthetic analysis of the ICS that has been commissioned as part of this study. We anticipate the enhanced dataset will be made publicly available in September 2022.

REF impact case studies are tagged with one or more UK locations on the basis of places referenced in the text of either Section 1 (Summary of impact) or Section 4 (Details of the impact) of the document (UK cities and towns, as found the Office for National Statistics website, https://www.ons.gov.uk/). This is an indicative guide to aid text searching. It is not a definitive identification of where UK impact has occurred as some text makes passing references to associated locations; other text references impact beneficiaries without a specific location.

REF Impact case studies were submitted with additional contextual information which was collected to be used in conjunction with the impact case study database.  As part of this data, submitting HEIs were able to add in global location(s) where the impact occurred.  This data was used to tag the impact case studies with global locations.

The 2021 REF impact case studies are assigned to one or more underpinning research subjects (up to a maximum of three). This is an indicative guide to aid text searching via a more fine-grained disciplinary structure than is immediately available in the 34 REF Units of Assessment.

To look at the disciplines associated with the underpinning research, the Australian and New Zealand Standard Research Classification (ANZSRC) 2008 Fields of Research (FoR) was used, based on a publicly available journal mapping, with cited reference lists extracted from the OpenAlex database. The classification system has three levels of detail: (i) Divisions (2-digit codes), (ii) Groups (4-digit codes) and (iii) Subjects (6-digit codes). 4-digit FoR (Groups) were used for this analysis. The 2008 version of ANZSRC was used to allow direct comparison between the 2014 and 2021 ICS data. To determine the subject categories, the public ERA (Excellence in Research for Australia) Journal mapping file from 2018 was used that maps journals to FoR codes (up to 3 per journal). OpenAlex data was used to calculate a weighting for each DOI mentioned in an ICS to every 4-digit FOR code based on the number of cited references, and the 3 highest weighted codes were used to tag the ICS against the research subjects. Where no FoR groups could be suggested for an ICS, these were manually assigned. Where there was only one DOI linked to an ICS, additional manual review was undertaken to verify the suggested FoR groups.

Search results for REF impact case studies are ordered on case study title, which may include numbers and other tags inserted by the HEI, unless a text search is used.

Future releases to the database will enable results to be ranked according to the number of occurrences of the search term in the document.

Some case studies submitted to REF were a continuation of work that was submitted to REF 2014.  Submitting HEIs were asked to identify case studies as ‘continued’ where the underpinning research was the same as previously described and where there was significant overlap in the impact described, so that the impact types and beneficiaries were broadly the same as previously described.  The database displays information about which case studies were identified as ‘continued’ and allows the users to filter on this basis if desired (see help documentation for further details).

Contextual data was collected through the submission system as additional metadata for each case study.  HEIs were asked to supply, where applicable, the additional contextual data which would be used in the impact case study database to enable research funders to track and evaluate the impact of their funding.  Data collected was in the following categories:

• name(s) of funder(s)
• Global Research Identifier of funder(s): https://www.grid.ac/
• name(s) of funding programme(s)
• grant number(s) • amount of grant (in GBP (Sterling))
• ORCID for each named researcher, where held
• name(s) of formal partner(s)
• country/countries where the impact occurred

Sets of case studies can be downloaded in the following formats:

• Excel spreadsheet
• Zipped PDF files

In the case of Excel, there is no limit to the number of case studies that may be downloaded in a single session.

In the case of PDF files it is recommended that you restrict your download to the set of case studies that would be returned from a single index selection or specific text search.

It is possible to download a zip file of all case studies for a particular HEI/UOA as these have been pre-prepared to aid performance.

NB there are currently technical limitations on downloading large exports of case studies.  If you find that your download fails please try again with a smaller set of results.

An excel file is available to download all tags to a spreadsheet (download all impact case study tags).  The spreadsheet includes identifiers for the case study and a row for each tag that it has been tagged with, including the top level category where applicable.

The impact case study tags spreadsheet can also be generated for any filtered views of case studies.

The case studies are published under a CC BY 4.0 licence. The following use is permitted under these licence conditions: http://creativecommons.org/licenses/by/4.0/legalcode

A more user friendly version is available here but does not replace the full legal code: http://creativecommons.org/licenses/by/4.0/

If you have identified content that may require removal or redaction please get in touch with us by email: ResearchPolicy@re.ukri.org

The text search on the impact case study database utilises a full text search capability.  This means that you are able to perform full text searches and exploit complex search features.  If you are seeing the syntax error message it means that your search term cannot be understood by the search engine.  If you are searching for a simple phrase you should try to wrap the whole phrase in double quotes (for example, "English language research".  Please see the impact case study help document for more information on the powerful search function.

The spreadsheet export of the impact case studies returns the raw data for the case studies as extracted from the database.  This content is formatted using markdown language, a simple formatting language for web-based content. This enables the database to display formatted case studies on pages in the database.  See https://www.markdownguide.org/ for further information on the usage of markdown.