If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Clinical outcome measures are largely chosen by clinicians rather than patients.
•
Core outcome sets can improve representation in studies for all key stakeholders.
•
The report outlines the methods and challenges of conducting European core outcome sets.
•
There are considerable barriers to developing European core outcome sets.
•
As a European vascular community, we should produce sets through collaboration.
Introduction
Most of the outcomes reported in the literature have been chosen by doctors, constituting “traditional” outcome measures such as mortality and re-intervention. Some of the key outcome measures important to patients, families, health providers and other stakeholders may have been overlooked. Core outcome sets, consisting of 6–15 outcomes, can improve representation of all key stakeholders, standardise outcome reporting, and improve future ability to pool results. The aim of this study was to outline the methods and challenges of conducting European core outcome sets.
Report
As an overview, development of core outcome sets follows a multistep iterative process: (1) Systematic review of the literature summarising existing outcome measures, (2) Focus Group meeting with patients and other stakeholders to establish missing outcome measures, (3) Development and piloting of Delphi survey, (4) Delphi consensus study for prioritisation of outcomes and establishing consensus, and (5) European consensus meeting to produce a core outcome set. The challenges include the varying ethical requirements for survey work across Europe and translation for surveys and consensus meetings.
Discussion
There is an increasing need for core outcome sets to complement clinical practice guidelines. As a European vascular community we need to produce these through collaborative efforts. Unfortunately, there are considerable barriers to doing so – the time and energy required to set up a core outcome study is not dissimilar to that of a multicentre randomised trial. Currently only one core outcome set exists for vascular surgery, for critical limb ischaemia, but this was developed in a single country.
As technology advances at an ever increasing pace, there is an increasing need to evaluate the clinical effectiveness of these new treatments as rapidly and as cost effectively as possible. This often means synthesising evidence from different observational studies or randomised trials. Traditionally, vascular surgeons have focused on assessing parameters such as mortality, complications, and time spent in hospital as detailed in a recent systematic review of clinical studies investigating treatments for abdominal aortic aneurysm.
Towards a core outcome set for abdominal aortic aneurysm: systematic review of outcomes reported following intact and ruptured abdominal aortic aneurysm repair.
However, the patients themselves, their families, nurses and community medical staff may value other outcomes more highly. To represent the needs of all stakeholders, particularly patients, the concept of a “core outcome set” has been developed (see Fig. 1).
Figure 1Flowchart illustrating overview of developing a core outcome set.
Core outcome sets have been developed in many areas of surgery but there is currently only one core outcome set in vascular surgery, developed for studies assessing treatments for those requiring amputation due to critical limb ischaemia.
This was developed in the United Kingdom over several years as it followed the standard methodology of systematic review, focus groups, and Delphi consensus survey. Given that this core outcome set was developed in one outpost of Europe, there are questions over whether it is applicable to the rest of mainland Europe. This report provides an overview of developing core outcome sets which are widely applicable across Europe.
provides a source of advice about “how to do it” and keeps a register of core outcome sets in progress or completed. In this sense, it is a parallel of the PROSPERO registry for systematic reviews.
Focus group meeting with patients and other stakeholders to establish important missing outcome measures.
•
Development of Delphi survey with piloting and iteration.
•
Delphi consensus study for prioritisation of outcomes and establishing consensus.
•
European consensus meeting to produce a core outcome set.
Systematic review to gather existing outcome measures
The starting point for all core outcome sets is a systematic review of the outcomes reported in the literature. Such reviews can yield thousands of references, so it is prudent to introduce restrictions, including date and extracting pre-specified outcomes only. Importantly, the purpose of this systematic review differs from that of a conventional one. Rather than extract, critique, and evaluate the evidence, its purpose is to assess the evolution of outcome measures reported, compiling a register of these outcomes.
Many vascular studies report a limited number of primary and secondary outcomes but also a multitude of others, while some studies do not pre-specify any outcomes and embark on a “fishing trip” to find out whether there are any associations or correlations of interest. The best quality studies identify their main outcomes ahead of time and these are the outcomes reported for almost all participants. Hence, focusing on pre-specified outcomes limits the work involved while improving outcome reporting. Often only retrieving studies conducted in the last 10–20 years will limit the workload without compromising on quality.
Given the volume of references which need to be read and extracted, a team of several reviewers may be needed with duplicate extraction limited to a random 10–20% of papers checking for consistency. The present authors recommend having reviewers from more than one country to reduce healthcare system effects and enable inclusion of multiple languages.
Outcome measure, outcome timing, outcome definition, and completion rate are key variables to extract. Timing of the outcome (e.g., in hospital, three years) is often overlooked; however, these durations are important to include in the future consensus study.
Focus group meetings with key stakeholders, mainly patients
Additional input should be sought from focus groups of stakeholders as most of the literature comes predominantly from clinicians and carries their view of important outcomes. Ideally, focus groups should be take place across several European countries. Focus groups with patients, their families, hospital managers or others to identify additional outcome measures may not require ethical approval as the purpose is listening in order to improve the quality of care rather than primary research. There are various resources on involving patients in research, including the COMET handbook and guidance on the National Institute for Health and Care Research website (although not specific to core outcome sets).
NIHR HTA. PPI (Patient and Public Involvement) resources for applicants to NIHR research programmes 2022. Available at: https://www.nihr.ac.uk/documents/ [Accessed 22 October 2022].
The full list of outcomes, from the literature review in addition to others identified as important by focus groups, should now be categorised according to a published taxonomy. This aims to classify which, rather than how, outcomes are measured, for example, cognitive functioning rather than “Montreal Cognitive Assessment”.
These outcomes now need to be prioritised by category and timing for inclusion as questions with Likert scale responses, for the Delphi consensus phase.
Ethical and institutional approval
The need for formal ethical approval for focus groups and Delphi consensus studies varies across Europe. For example, any study fulfilling the definition of research in the UK recruiting from healthcare settings requires formal ethical approval issued by the Health Research Authority (irrespective of whether the project is a survey).
In some countries, such as the Netherlands, survey work may be conducted under an ethical waiver, while in others ethical approvals are not required. Individual institutional ethical approval and or waiver can be issued in some European centres. Considerable time (at least six months) should be allowed for this prior to launching the Delphi consensus study.
European Delphi consensus study
The Delphi consensus study assesses the importance of all the identified outcome measures across stakeholder groups. The COMET handbook
offers advice on the set up of the Delphi, including recommendations to avoid bias such as domain randomisation. The Delphi should consist of domains of groups of similar outcomes with accompanying explanations detailing the outcome measure.
The Delphi consensus study should be piloted across all the stakeholder groups and across multiple countries in their respective (translated) languages. Troubleshooting is undertaken to ensure that the platform and Delphi perform optimally. Inspection of the results, for example, looking for bimodal distributions, is helpful to identify poorly performing questions.
There are several online platforms that have been used to deliver Delphi consensus studies, including DelphiManager, Qualtrics, and REDCap. An important consideration is the level of security and insurance that providers offer and ensuring this complies with the study sponsor's requirements ahead of time. Pen and paper could be the best method of delivery depending on access and acceptability of online platforms.
There should be at least two rounds of Delphi survey, with the potential to include additional rounds if required. Round 2 can be deployed in the same manner as round 1, or outcomes can be removed to increase likelihood of consensus. Criteria for inclusion in round 2 vary in the literature with no standard methodology. Again, the COMET handbook
sheds light on potential methods. For example, one approach would be to include any item that is rated 7–9 (on a 9 point Likert scale) by ≥ 50% participants and 1–3 by ≤ 15% of participants in at least one stakeholder group.
European consensus meeting
The Delphi consensus study provides a breakdown of outcome prioritisation for each stakeholder group. Although it is likely that consensus will be reached on most outcomes, it is also likely that some measures will be held with high importance by a single group but lack a majority. A consensus meeting consisting of stakeholder representatives allows a dialogue to outline the rationale underpinning the Delphi results and finalisation of the core outcome set. While such consensus meetings for clinicians and industry can be conducted in English, trans-European consensus meetings of patients is a greater challenge and requires simultaneous translation resources for online meetings. Another way to approach this problem might be for the European Society for Vascular Surgery (ESVS) to establish an expert patient panel. The exact mechanics of which have been the topic of debate,
with most using a nominal group technique in which all arguments are considered initially prior to a ranking exercise.
Discussion
Although the need for core outcome sets is clear, as an essential complement to clinical practice guidelines, the challenges of developing these for Europewide use are considerable. Apart from the challenges of time, language, cultural, and healthcare system differences, there is the issue of who will fund the development of these core outcome sets, as the platforms for the Delphi consensus phase and probably the consensus meetings require financial support.
Considerable time must be allowed to gain the necessary ethical and regulatory approval for all participating centres. This is exacerbated by the study design being somewhat novel (a survey involving patients, their relatives, industry, and clinicians), with many processes not streamlined for such a design. The time and energy required to set up a core outcome study is not dissimilar to that of a multicentre randomised trial. An example of these delays is the Abdominal Aortic Aneurysm Core Outcome Set (AAA COS) project, which registered back in 2020 with the Delphi survey due to go live two years later.
Powell JT. Development of a core outcome set for clinical studies reporting the outcomes of patients undergoing repair of degenerative infrarenal abdominal aortic aneurysms. Available at: https://www.comet-initiative.org/Studies/Details/1582 [Accessed 22 October 2022].
This is to be included in the 2024 AAA guideline renewal and therefore has received some support from the ESVS. Despite such challenges, the present authors encourage others to develop the core outcome sets needed for vascular surgery and are willing to offer advice based on their experience.
Contributions
Matthew Machin: Conceptualisation and design of the work, drafting and revising of the article, final approval of the submitted version, and agreement to be accountable for all factual aspects of the work and for ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Janet Powell: Conceptualisation and design of the work, critical revision and writing of the article, final approval of the submitted version, and agreement to be accountable for all factual aspects of the work and for ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Funding
None.
Conflicts of interest
None.
References
Machin M.
Ulug P.
Pandirajan K.
Bown M.J.
Powell J.T.
Towards a core outcome set for abdominal aortic aneurysm: systematic review of outcomes reported following intact and ruptured abdominal aortic aneurysm repair.
NIHR HTA. PPI (Patient and Public Involvement) resources for applicants to NIHR research programmes 2022. Available at: https://www.nihr.ac.uk/documents/ [Accessed 22 October 2022].
Powell JT. Development of a core outcome set for clinical studies reporting the outcomes of patients undergoing repair of degenerative infrarenal abdominal aortic aneurysms. Available at: https://www.comet-initiative.org/Studies/Details/1582 [Accessed 22 October 2022].
To submit a comment for a journal article, please use the space above and note the following:
We will review submitted comments as soon as possible, striving for within two business days.
This forum is intended for constructive dialogue. Comments that are commercial or promotional in nature, pertain to specific medical cases, are not relevant to the article for which they have been submitted, or are otherwise inappropriate will not be posted.
We require that commenters identify themselves with names and affiliations.
00055-7&id=gr1.jpg){kind=link}