Guidance on REF 2021 results

Guide to the REF results

The information on this page provides a guide to the results of the 2021 Research Excellence Framework (REF), published on 12 May 2022.

Further information

The following material is available, or is forthcoming soon, on the REF website:

Full results: the overall quality profile and the outputs, impact and environment sub-profiles awarded to each submission.
Analysis: summary data about each unit of submission and analysis of the REF results as a whole.
Submissions: the submissions made by all HEIs, including details of the outputs, the impact case studies and information about the research environment will be available in mid-June 2022. In addition, a searchable database of the submitted impact case studies will be available in mid-June 2022.
Overview reports: a report by each main panel and its sub-panels, providing an overview of the assessment and the state of research in their discipline areas will be available from mid-May 2022.
Further reports: including from the Equality and Diversity Advisory Panel (EDAP), the Interdisciplinary Research Advisory Panel (IDAP), and the REF Director will be available in June 2022.
Background information: including guidance to institutions on submissions and information about the expert panels

Brief overview of REF 2021

The REF is the system for assessing the quality of research in higher education institutions (HEIs). It was first conducted in 2014 and replaced the previous Research Assessment Exercise (RAE).

The REF was conducted jointly by the four higher education (HE) funding bodies: Research England (RE), the Scottish Funding Council (SFC), the Higher Education Funding Council for Wales (HEFCW), and the Department for the Economy, Northern Ireland (DfE). The REF was managed by the REF team, based at RE, on behalf of the four funding bodies and was overseen by the REF steering group, consisting of representatives of the four funding bodies.

The primary purpose of REF 2021 was to produce assessment outcomes for each submission made by HEIs. These outcomes deliver the wider threefold purpose of the exercise:

To inform the selective allocation of the four HE funding bodies’ grants for research to the institutions which they fund, with effect from 2022-23.
To provide accountability for public investment in research and produce evidence of the benefits of this investment.
To provide benchmarking information and establish reputational yardsticks, for use within the HE sector and for public information.

Submissions

Submissions to REF 2021 were made by 157 UK HEIs into the 34 discipline-based units of assessment (UOAs) that cover all research disciplines. Each submission made by an HEI in a particular UOA is called a submitting unit. A total of 1,878 submissions were made, which included a total of 76,132 full-time equivalent (FTE) staff within these submissions. The size and scope of each institution’s submissions ranged from 4 FTE staff submitted in a single UOA to more than 3,000 FTE staff submitted across 30 UOAs.

Each submission made by an institution into a UOA contained a standard set of information in relation to three elements: outputs, impact and the environment. Each submission was assessed in terms of these three elements, which were combined to produce an ‘overall quality profile’ awarded to each submission. The three elements assessed were:

The quality of research outputs.
The quality of the impact of research.
The quality of the environment to support research and impact.

Outputs

Outputs are the published or publicly available products of research, which can take many forms. These include books, monographs, chapters in books and journal articles as well as performances, exhibitions and other practice research outputs, software, patents, conference proceedings, translations, and digital and visual media.

Outputs account for 60% of the overall outcome awarded to each submission and are assessed against three criteria: originality, significance, and rigour.

A total of 185,594 outputs were submitted to REF 2021. The number of outputs each submitting unit had to submit was equal to 2.5 times the FTE of its submitted staff. Each staff member had to have at least one and no more than five outputs attributed to them in the submission. For REF 2021, to meet their total number of required outputs for submission, HEIs could also submit outputs produced by former staff who had left the institution, providing the output had been made publicly available while that former staff member was employed on an eligible basis by the submitting HEI.

Impact

For REF 2021, impact is defined as the effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.

HEIs were required to submit impact case studies that demonstrate the impacts their research has had beyond academia. The number of impact case studies each submitting unit was required to submit was based on the FTE of the submitted staff within that unit, with a minimum of two impact case studies being required for the smallest units. A total of 6,781 impact case studies were submitted to REF 2021.

Impact accounts for 25% of the overall outcome awarded to each submission. Impact case studies were assessed against two criteria: reach and significance.

Environment

For REF 2021, ‘environment’ means the environment for supporting research and enabling impact within each submitting unit. Environment accounts for 15% of the overall outcome awarded to each submission and is assessed against two criteria: vitality and sustainability.

HEIs were required to submit narrative evidence of the environment to support research and enable impact within each unit, alongside data on research income, research income in kind, and completed doctoral degrees. The maximum length of the narrative statement that each submitting unit provided was determined by the FTE of the submitted staff within that unit.

HEIs were also required to submit an institutional level environment statement as part of a pilot of the assessment of institutional environment. These institutional level narratives were used by the sub-panels to inform and contextualise their assessment, but were not scored as part of the REF 2021 sub-profiles. A standalone assessment of institutional environment statements is being separately piloted by the Institutional Level Environment Pilot Panel.

Staff

For REF 2021, HEIs were required to submit all staff with significant responsibility for research and could choose how to map their staff into UoAs.

Contractual status (meaning staff with a contract of employment of 0.2FTE or greater on either teaching and research, or research only contracts) was used to identify the pool of staff eligible for submission to REF 2021. Where this eligible pool accurately captured all staff with significant responsibility for research, an HEI submitted 100% of eligible staff.

In some cases, the eligible pool of staff based on contractual status included some staff without significant responsibility for research. In these cases, HEIs ran processes to identify which staff within the eligible pool did have significant responsibility and therefore needed to be submitted to REF 2021. In these cases, the percentage of staff with significant responsibility for research will be lower than 100% of all staff meeting the eligibility criteria.

In a small number of instances, Category A eligible staff are not returned to HESA due to internal employment structures. This means that a small number of submissions have a % eligible staff submitted that is greater than 100%.

Image showing the breakdown of the overall quality profile into its 3 components of outputs (60%), impact (25%) and environment (15%)

Back to the top

Assessment process

The REF is a process of expert review. During 2021 and 2022, the submissions were assessed by expert sub-panels for each of the 34 discipline-based UOAs. These expert sub-panels were made up of both academics and research users.

Each sub-panel assessed each submission in its UOA and produced a sub-profile for each of the three elements of the submission (outputs, impact and environment) using a scale ranging from 4* to unclassified. More information on the level definitions can be found below under 'understanding the REF results'.

The sub-panels reviewed submissions according to the following common principles and practices:

Each part of each submission was reviewed by panel members and/or assessors with appropriate expertise.
Research users on the panel had a full and equal role in assessing the impact element of submissions, jointly with academic members.
Where the sub-panel did not contain sufficient expertise to review an output, it was cross-referred to an appropriate sub-panel for advice. In total 6340 outputs were cross-referred between sub-panels.
Where an output was published in a language that the sub-panel was unable to assess, it was referred to a specialist adviser with appropriate expertise.
All the information in each submission was examined in sufficient detail to contribute to the formation of robust sub-profiles for the submission.
Judgements were made solely on the basis of the information submitted by institutions (and responses to audit queries), and in accordance with the published criteria.
Consistency in standards of assessment was assured within and between sub-panels through a range of mechanisms. These included initial calibration exercises at sub-panel, main panel and cross-main panel levels, main panel members’ attendance at sub-panel meetings, and keeping under review the standards being applied within each sub-panel and each main panel.
Each sub-panel recommended the sub-profiles (for outputs, impact and environment) and the overall quality profiles to their main panel on the basis of its collective judgement.
No individual played any role in the assessment of any submission in which they had a major conflict of interest, including submissions from any institution(s) they had been employed by since the start of the REF period.

The 34 sub-panels worked under the guidance of four main panels, who were responsible for overseeing the assessment, ensuring that the criteria and standards were consistently applied, and signing off the results recommended by the sub-panels. Each main panel included international members to provide assurance about the international benchmarking of standards.

The expert sub-panels assessed the submissions made into each UOA according to a standard set of criteria. For further details including the complete definitions and criteria, see the following REF publications:

At the end of the assessment, each main panel and its sub-panels produced an overview report setting out how they carried out the assessment and commenting on the state of research in their discipline areas. These will be available on the REF website from mid-May 2022.

Back to the top

Audit and data verification

The accuracy of information provided by institutions in their submissions was checked and verified, including:

A sample of staff submitted by every institution compared with the HESA staff return for the census date, 31 July 2020
A sample of outputs within each UOA
A sample of impact case studies within each UOA
A sample of claims from environment statements within each UOA
Environment datasets were compared with data held by the Higher Education Statistics Agency and other agencies.

Panels were also able to make audit requests for any areas of concern they identified within submissions. Any discrepancies that arose through the audit process were investigated. Where necessary, submissions were amended and panels were supplied with the correct information through the audit process.

Back to the top

Key differences to REF 2014

Following implementation of the recommendations of the independent review of REF 2014 by Lord Stern, there were a number of differences between REF 2014 and REF 2021. This means caution must be exercised in comparing outcomes and results between the two exercises.

Some of the key differences between the exercises include:

The requirement to submit all staff with significant responsibility for research to REF 2021, compared to the ability to select staff for submission in REF 2014. As a result of this, there was a 46% increase in the number of staff submitted to REF 2021.
A more flexible output requirement for each staff member for REF 2021, whereby each submitted staff member had to have at least one output and no more than five outputs attributed to them. In comparison, all submitted staff members had to have four outputs for REF 2014.
For REF 2021, institutions were able to submit outputs produced by former staff where the output was first made publicly available while the staff member was employed by the submitting HEI.
The weighting for impact was increased from 20% to 25% between REF 2014 and REF 2021. For REF 2021 impact was assessed solely on submitted impact case studies and information about the environment to enable impact was instead included within environment statements.
Changes were made to the number and make-up of a small number of UOAs for REF 2014.

Back to the top

Covid-19

A number of measures were put in place for REF 2021 to take account of the effects of the COVID-19 pandemic.

When the pandemic was first declared in early 2020, the exercise was paused in recognition of the fact that institutions needed to divert staff resource to other critical areas and to reassure institutions that they should do so without concern for the effect on REF preparations.

The exercise resumed in July 2020 after a four month pause. A series of mitigations and revisions to the guidance were put in place in the period leading up to the revised submission deadline, including:

An extension of the submission deadline to 31 March 2021.
An extension to the assessment period for impact case studies to 31 December 2020.
An allowance for submitting outputs where publication was delayed due to Covid-19.

Additional provision for removing the minimum of one output requirement for staff with individual staff circumstances that include the effects of Covid-19 on their ability to research productively.

Options to submit explanatory text on affected outputs and impact case studies.
Options to submit additional text to describe the effects of Covid-19 on the institution’s environment for supporting research and enabling impact.
An extension to deadlines for upload of narrative templates to the submissions system.
Allowing for a period of data corrections following the submission deadline.
Additional flexibility in the audit process

Due to the ongoing pandemic, the sub-panels pivoted to virtual meetings for much of the assessment period, returning to in-person and hybrid meetings as public health guidance allowed. Additional support and guidance was provided to sub-panels throughout the assessment phase to ensure that their assessment remained robust.

Back to the top

Understanding the REF 2021 results

What do the quality profiles show?

The overall quality profile for each submission shows the proportion of the submission judged to meet the quality for each of the starred levels in 1% increments.

This overall quality profile is calculated from the sub-profile for each of outputs, impact and environment. The sub-profiles show, respectively, the proportion of outputs, impact and environment at each starred level.

Further analysis of the results can be found on the REF website.

Summaries of the outcomes for each unit of assessment and comparative data can also be found on the REF website.

In the example below, 15% of the submission from this department was 4* overall, with 35% being 3* overall. Please note, the below example shows only output sub-profiles to one decimal place. In the published results overall quality profiles are rounded to the nearest whole number, sub-profiles for each of outputs, impact and environment are to one decimal place.

Image showing the profile breakdown as % of submission meeting the quality standard of 4 star, 3 star, 2 star, 1 star and unclassified.

What are the quality levels?

At the overall quality profile level, the starred levels denote the quality of the submissions as follows:

Quality level	Description
Four star	Quality that is world-leading in terms of originality, significance and rigour.
Three star	Quality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence.
Two star	Quality that is recognised internationally in terms of originality, significance and rigour
One star	Quality that is recognised nationally in terms of originality, significance and rigour.
Unclassified	Quality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment.

For output sub-profiles, the starred levels denote quality as follows:

Quality level	Description
Four star	Quality that is world-leading in terms of originality, significance and rigour.
Three star	Quality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence.
Two star	Quality that is recognised internationally in terms of originality, significance and rigour
One star	Quality that is recognised nationally in terms of originality, significance and rigour.
Unclassified	Quality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment.

For impact sub-profiles, the starred levels denote quality as follows:

Quality level	Description
Four star	Outstanding impacts in terms of their reach and significance.
Three star	Very considerable impacts in terms of their reach and significance.
Two star	Considerable impacts in terms of their reach and significance.
One star	Recognised but modest impacts in terms of their reach and significance.
Unclassified	The impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit

For environment sub-profiles, the starred levels denote quality as follows:

Quality level	Description
Four star	An environment that is conducive to producing research of world-leading quality and enabling outstanding impact, in terms of its vitality and sustainability.
Three star	An environment that is conducive to producing research of internationally excellent quality and enabling very considerable impact, in terms of its vitality and sustainability.
Two star	An environment that is conducive to producing research of internationally recognised quality and enabling considerable impact, in terms of its vitality and sustainability.
One star	An environment that is conducive to producing research of nationally recognised quality and enabling recognised but modest impact, in terms of its vitality and sustainability.
Unclassified	An environment that is not conducive to producing research of nationally recognised quality or enabling impact of reach and significance.

What were the criteria for assessment?

Outputs were assessed against three criteria: originality, significance and rigour.

Impact was assessed against two criteria: reach and significance.

Environment was assessed against two criteria: vitality and sustainability.

What does % eligible staff mean?

The percentage of eligible staff shows the percentage of staff determined to have significant responsibility for research, and who were therefore submitted, out of the total staff meeting the eligibility criteria for REF. More information is set out above under 'staff'.

Why are the profiles FTE weighted? What does this mean?

Average profiles (for example, at UOA and main panel level) are weighted by staff FTE to take into account the relative size of submissions in the overall outcome. They are produced by weighting the proportion of activity at each starred level for each submission by the FTE of Category A submitted staff. Presenting the data in this way provides a UK-wide picture of the quality of submitted research activity

Why are the sub-profiles suppressed for some submissions?

The sub-profiles are suppressed for submissions with a submitted headcount of three or lower.

What are joint submissions and what do the results for these show?

A joint submission is where two or more UK institutions make a joint submission in a UOA, where this is the most appropriate way of describing research they have developed or undertaken collaboratively. Panels assessed joint submissions in the same way as submissions from single institutions.

Joint submissions receive one overall quality profile, and these results are displayed for each of the institutions that contributed to that submission.

For joint submissions both the FTE of the individual HEI’s submission and the total FTE of the whole joint submission are shown. The % eligible staff applies to the joint submission as a whole.

What are multiple submissions and what do the results for these show?

Institutions normally make one submission in each UOA they submit in. In a small number of cases, institutions made more than one submission (multiple submissions) in the same UOA. Acceptable reasons for multiple submissions include instances where:

An institution involved in a joint submission wished to make an additional individual submission in the same UOA
For submissions to sub-panel 26 (Modern Languages and Linguistics) one submission was in Celtic Studies and the other in Modern Languages and Linguistics
HEIs merged after 1 July 2018 and wished to make separate submissions in all of the UOAs in which they submitted.
It was agreed in advance that multiple submissions could be made given the nature of the disciplines within the UOA

For multiple submissions, the results are displayed for each individual submission and each individual submission receives its own overall quality profile.