Submitting research outputs

Submitting outputs as PDF or via URL

The REF submission system has the facility for institutions to upload outputs as PDFs, or to provide access to outputs via a URL. This page provides further guidance.

Outputs being submitted electronically due to COVID-19 that were previously expected to be submitted physically or scanned

Where institutions were intending to submit outputs in physical form, but are now considering submitting electronic versions of these outputs, they should submit the final version of the output, where possible, according to Annex K of the ‘Guidance on submissions’. PDFs that have been converted from other formats (such as eBooks) are acceptable, subject to appropriate copyright and licensing permissions, so long as they include all information required by the panels for assessment (see books, monographs and other long-form outputs section below).

Where it is not possible to submit the final version of the output in electronic form, institutions may instead submit a proof, author accepted manuscript, or other equivalent version of the output. In this instance, institutions should ensure that all information required by the panels for assessment is included in the PDF (or URL, where indicated in Annex K of the ‘Guidance on submissions’), such as title page, full referencing information and index. In these cases, the submitting institution will need to describe the form of the output that is being submitted in the ‘media of output’ field in the submission system.

Journal articles and conference proceedings

For all journal articles and conference proceedings (type D and E outputs), the institution must provide Digital Object Identifiers (DOIs) wherever available. This will enable the REF team to access the published outputs directly via the publishers' websites. Research England will store these outputs and make them available to panel members for REF purposes.

There will be some type D and E outputs which it will not be possible for the REF team to access directly, and we will not attempt to source type D and E outputs flagged as 'pending publication'. In these instances the HEI will be required to upload a PDF of the output to the submission system. Reports will be available on the submission system identifying type D and E outputs that the REF team has not been able to source directly, and will therefore need to be uploaded by the HEI.

It is each HEI's responsibility to run the reports regularly and provide PDFs of all submitted type D and E outputs which cannot be sourced by the REF team. The REF team will not be contacting individual HEIs to request these outputs.

Books, monographs and other long-form outputs

Where institutions are uploading PDFs of books, monographs and other long-form outputs, the entirety of the text (from title page to the end of the index, including full referencing information and any bibliography) should be submitted, including, where applicable, any illustrations, maps, figures or similar images.

To facilitate the electronic submission of long-form outputs, we have increased the file size for output types A and B in the submission system to 500MB. To minimise file sizes while ensuring sufficient quality, we recommend 150dpi for any images included within PDFs. Where institutions have an output that exceeds this file size limit, they should contact usersupport@ref.ac.uk.

See ‘creating PDF outputs’ below for further guidance.

Chapters in books

Where institutions are uploading PDFs of chapters in books where the author of the chapter is also a (co-)editor of the book, the entirety of the text (from title page to the end of the index, including full referencing information and any bibliography) should be submitted, including, where applicable, any illustrations, maps, figures or similar images. To minimise file sizes while ensuring sufficient quality, the REF team recommends 150dpi for any images included within PDFs.

Where the author of the chapter is not also a (co-)editor of the book, HEIs should provide the chapter itself, the contents page of the volume, any referencing information relevant to the chapter (including where this is separate from the chapter in the book), the volume bibliography (where it exists) and any relevant illustrations, images, maps, figures, appendices or other apparatus. Full publication details should also be provided. Where the editor’s introduction sets out a case for the relationship of the submitted chapter to the volume this should also be submitted where available.

See ‘creating PDF outputs’ below for further guidance.

Websites and other outputs that can change over time

For outputs that are submitted as a URL or DOI, and which can change over time (e.g. websites or databases), the submitting institution should ensure that the URL or DOI resolves to the version of the output that first became publicly available, which must be on or before 31 December 2020 (or, exceptionally, between 1 January 2021 and 31 March 2021 where production of the output was delayed by COVID-19 as permitted under the ‘Guidance on revisions’ (2020/02)).

There are various options that institutions may use to achieve this requirement, and we do not specify any particular method. 

Examples of methods that institutions may wish to consider include:

• Creating a copy of the website on the same, or a different, web server. Where the website serves dynamic content from a database, the website should use its own copy of the database, and neither the web nor database content should be updated after the stated point in time.
• Placing a copy on a media storage device, for example USB, and returning it as a physical output.

These examples are not exhaustive, and HEIs are free to use other methods so long as they achieve the aim of demonstrating that the submitted version of the output was fixed at a point in time.

HEIs will need to maintain proof of the date at which the item became publicly available and of its content at that date, and should be able to make such evidence available for audit if requested to do so.

Where the HEI presents the point in time copy as a URL, it must be made available in such a way as not to require the input of a login and/or password and should preserve the anonymity of the reviewer.

HEIs may wish to link to media hosted on third-party sites where such media are normally streamed, rather than made available as a file download.  HEIs are permitted to do this, but should only do so where:

• The media type is video or audio
• It is impractical to host it on their own site
• The third-party site carries evidence of the last date when the video was uploaded/updated, or the HEI is able to evidence this
• The third-party site can be accessed without requiring a login account
• Access will preserve the anonymity of the reviewer.

Other output types

Output types other than D and E will not be accessed by Research England and must be provided in electronic format by HEIs or sent to the warehouse. In addition to journal articles and conference proceedings that the REF team cannot source, HEIs are encouraged to upload other types of outputs as PDFs to the REF submission system, or provide access to the output via a URL as applicable to the output type. The table in annex K of the Guidance on submissions indicates which output types can be provided in these formats. Where there is a choice between providing an output in physical or electronic format, an electronic copy should be provided wherever the output is already available in this format. Where the output can be converted or scanned into a PDF of sufficient quality, within the file size limit, it should be provided in this way as described below.

Creating PDF outputs

Where the output is not already available as a PDF, a PDF of sufficient quality should be created while minimising the file size as far as practicable. A PDF of 'sufficient quality' means that it will be easily readable whether viewed on screen or printed, and provides an accurate representation of the output.

To minimise file sizes, the output should be converted directly to PDF if possible, rather than scanned. If this is not possible, the output should be scanned avoiding high resolution and colour as far as possible while ensuring the PDF is of sufficient quality.

Where it is not possible to produce a PDF copy of the final version of the output in either of these ways, institutions may instead submit a proof, author accepted manuscript, or other equivalent version of the output as described above under ‘Outputs being submitted electronically due to COVID-19 that were previously expected to be submitted physically or scanned’.

For all output types excluding types A and B, a PDF output cannot be uploaded to the submission system if the file size exceeds 100MB (and we anticipate that most PDF files will be significantly smaller than this). If a PDF of sufficient quality cannot be produced within this file size limit, the HEI should submit the output in physical form instead, either in hard copy or in digital form (on CD, DVD or a USB). For output types A and B, the maximum file size for a PDF is 500MB. Where institutions have an output that exceeds the applicable file size limit, they should contact usersupport@ref.ac.uk.

Note that all outputs submitted as PDFs will be available for panel members to view electronically; however, individual panel members may choose to print the PDF to view as a hard copy, and this may be in black-and-white. This should be considered when producing PDFs.

Only one PDF may be uploaded for each output being submitted.

When to upload PDF outputs

HEIs should regularly run reports on type D and E outputs, and upload any such outputs that the REF team is unable to source, as well as any other outputs to be submitted as PDFs, as soon as practical throughout the submission phase. HEIs should not wait until the end of the submission period to upload large numbers of PDFs as this may cause strain on the submission system. Should the submissions system be adversely affected we may implement a queuing system for HEIs to upload outputs.

We expect to be able to start retrieving outputs from the majority of publishers by January 2021. From this date, where the REF team has been unable to source the type D and E outputs within three weeks of the HEI listing the output on the submission system, the HEI should upload the output. This applies up to February 2021. For any type D and E outputs added to the submission during March 2021 that the REF team are unable to source within one working day, the HEI should assume the REF team will not be able to source these outputs and must upload the PDFs themselves.

All PDF outputs must be uploaded by the submission deadline on 31 March 2021.

Depositing physical outputs

All outputs included in a submission that are not sourced by the REF team or provided electronically to the submission system must be provided as physical outputs for assessment. Physical outputs may be submitted in hardcopy form, in digital or other media on CD, DVD or USB, or as a combination of these.

Outputs deposited in digital or other media on CD, DVD or USB must be in commonly accessible formats. HEIs should ensure that digital files and media are free from viruses, worms, trojans and other digital threats.

Artefacts, devices, products and items of great rarity or value should not be deposited. A representation of the output and sufficient information about the research it embodies should be provided to enable the panel to assess it, without providing the item itself. However, institutions should be prepared to make the item available should panel members request to view it. In these instances the REF team will make the necessary arrangements with the institution.

Each physical output must be deposited as a separate item or package (whether in hardcopy, on CD, DVD, or USB or a combination of these). Multiple outputs must not be provided on a single CD, DVD or USB.

Where the same physical output, of any type, is attributed to two staff members in the same unit of assessment, the institution need only supply it once. If the institution submits the same physical output more than once in different units of assessments, a separate copy must be supplied for each unit of assessment, individually labelled.

Where reserve outputs have been identified for double-weighted outputs, these should also be deposited.

Further information on labelling, packaging and delivery of outputs can be found here.

Uploading Corroborating Evidence

Corroborating evidence for Impact case studies needs to be uploaded by 1 June 2021. HEIs will be able to upload corroborating evidence before the REF submission deadline of noon on 31 March 2021, and the facility to do so will remain open until noon on 1 June 2021.

The corroborating evidence must be uploaded as a ZIP file. The ZIP file should contain one or more documents in PDF format. No other document formats should be included in the ZIP file. The ZIP file must be no larger than 100MB.

Each item of corroborating evidence should be appropriately titled and numbered according to section 5 of the impact case study template, to indicate the relationship to the specific claims made in the impact case study. Where the evidence can be freely accessed online, a URL can be provided either directly in section 5 of the REF3 template, or embedded in a PDF document and submitted as part of the corroborating evidence.

PDFs that will be uploaded as corroborating evidence should be of sufficient quality, and should where possible be created according to the same methods as for PDF copies of outputs - see Creating PDF outputs.