The NRC Dataset. Developing a Minimum Dataset for Multidisciplinary Rehabilitation: a Modified E-Delphi Study

This methodology of this modified e-Delphi study has been informed by guidelines on 'how to develop core domain/outcome sets' and 'how to conduct Delphi studies' by the Initiative on Methods, Measurements and Pain Assessment in Clinical Trials (IMMPACT), the Core Outcome Measures in Effectiveness Trials (COMET) group, the Outcome Measures in Rheumatology (OMERACT) group and The Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) team. Our study has been peer reviewed by a steering committee of 15 expert clinicians and academics and Nottingham University Hospital NHS foundation Trust's research committee. It is registered on the COSMIN database and will be published in a peer reviewed journal and on the NRC website on completion.

The modified Delphi study process will consist of a minimum of two phases with 2-4 rounds per phase.

E-Delphi software will be used to collect study data to decrease sample attrition and ensure anonymity and minimise power dynamics between participants.

The principal data holder will be the National Rehabilitation Centre as part of Nottingham University Hospitals Trust. The NRC will hold all rights for publication of the data, study protocol, full study report and anonymised participant level data. Authorship will be granted to aforementioned chief investigator and key contributors to this protocol, each of whom will also be involved in review of the final study results.

Full Methodology

Recruitment of the expert panel:

1. The NRC dataset steering committee or one of their nominees will recruit participants through purposive snowball sampling using standardised email invites
2. Each nominee will receive a document outlining the study background, aims, objectives, participant eligibility and methodology.
3. If participants agree to partake in the e-Delphi study they will contact the CI directly.
4. Upon receipt of a potential participant's expression of interest the CI will invite them to propose further study participants, outlining the specialist areas that still required on the expert panel. The participants will also be provided with a consent form and participant information sheet.
5. If nominees don't reply to invitations a follow-up email will be sent 10 days later. If there is no further communication, no further contact will be attempted.
6. If participants feel they meet the eligibility criteria, they must return a signed consent form to the CI as evidence of their intention to partake in the study. Participants are free to withdraw from the study at any time. No form of incentive will be provided.
7. The CI will countersign consent forms to confirm that study participants have signed and completed the form correctly. The participant is then assigned a study ID and provided with a copy of their countersigned consent form. At this point, the participant is asked to enter their professional experience and key demographic details into a JISC questionnaire. This information will be used to ensure that the expert panel is sufficiently diverse and represent the case-mix of patients being treated at the NRC.
8. The research team will confirm that the participant meets the eligibility criteria after reviewing the answers given in the JISC questionnaire.
9. Once a heterogenous panel of experts has been recruited or no further individuals are being nominated to be included in the expert panel, recruitment will close and the study will be formally launched.
10. At this point, all participant will be sent key documents and research papers, intended to inform their decision making during the Delphi process.
11. It is expected that the study will over-recruit by 20-30%, reaching a total of circa 50-60 individual participants. This allows for 2-3 experts in each field of practice.

12. Participants will be provided with a background document summarising the Rehabilitation Datasets being utilised internationally and key research papers relevant to the study.

    Phase 1:

    Exploring what information (data items) must be collected as part of the NRC's minimum dataset

    Round 1:

13. An initial list of proposed Core Data Items (outcome domains) agreed upon by the NRC dataset steering committee will be provided to the expert panel

14. These core data items (outcome domains) will include:

    i. Demographic information ii. Health indicators and associated outcome measures iii. Socio-economic data (healthcare costs, salary etc.) iv. Service delivery/care pathway variables (admission, transfer and discharge dates/locations etc.) v. Information associated with previous social function, employment and education and return to these activities following their injury/illness.

    vi. Injury/condition-specific data

15. Participants will priority rank each of the proposed data items based on the criteria outlined below.
16. Experts will also return a list of additional 'core' data items (outcome domains) they feel should be included in the National Rehabilitation Dataset by answering the question 'What additional data should be routinely collected for all rehabilitation patients in order to describe care and outcomes?'

17. At this time panel members will not be asked for matched outcome measures, but are encouraged to take note of these for later rounds and include them in comments. No limit will be placed on the number of data items proposed and participants will be encouraged to leave comments to express their opinion on any given item.

    Each data item will be ranked on a 9-point scale based on COMET initiative and OMERACT guidance as follows:

    (1-3) = 'Critical / Mandatory domains' (data that is considered essential for documentation and reporting the effectiveness and cost efficiency of the rehabilitation being provided)

    (4-6) = 'Important but not critical / important but optional domains' (data that improves the quality of data collected but is not critical to evaluating rehabilitation effectiveness and cost efficiency)

    (7-9) = 'Not that important / useful for research purposes' (optional data items that may supplement the core dataset for research purposes or specialist populations)

    Rounds 2-4:

18. All ranked data items will be anonymised and re-distributed to the expert panel. Each expert will evaluate their ranked responses before returning a final list to the Delphi coordinator. No limit will be placed on the required number of data items or health indicators that could be proposed.
19. The newly ranked list of data items will be re-distributed to the expert panel. Clearly outlining any uncertainties and disagreements to the expert panel for further consideration.
20. Any data item that has reached consensus of >70% after round 2 will be included in the final dataset. These core domains (data items) will then be excluded from subsequent rounds of voting.
21. If consensus is not met in round 2 then this round will be repeated until such a time that consensus is achieved prior to moving onto phase 2
22. If a consensus of >70% is not achieved in each of the core outcome domains after 4 round of voting, the highest-ranked responses with a consensus >60% will be accepted.
23. A written report summarising the proposed 'core' data items (outcome domains) will be distributed to the expert panel.
24. A video will be recorded by the study coordinator to present the consensus findings. Outlining the proposed 'core' data items and health indicators to be included in the NRC dataset, before outlining the next steps in the Delphi study.

25. Any ranked responses from phase 1 with 50-70% consensus as critical/mandatory that were not included in the final dataset will be included in the supplemental dataset alongside data items marked as 'important but not critical/optional domains' with a stable response rate.

    Phase 2: What tools should be used to evaluate the data items outlined in phase 1

    Round 1:

26. The expert panel will propose a list of outcome measure instruments to evaluate each of the proposed 'core' data items (inc. health indicators) identified in phase 1.
27. This list should be priority ranked based on the OMERACT and COSMIN checklists. Principally basing ranked choices on, (1) The appropriateness of the proposed outcome measure for the given population and (2) The quality of the outcome measure based on its reliability, validity, responsiveness to change and interpretability of the outcome data.

28. Where an outcome measure instrument is not required to measure fixed domains such as biological sex at birth or occupational status, the CI will include wording from the NHS data model/dictionary. Participants will be asked to comment on the domain wording.

    Ranked choices for each core data item will be ranks from 1-8 as follows:

    1-4 = Most appropriate outcome measure instrument / Domain 5-8 = Optional / supplemental outcome measure instrument / Domain that could be used for specific patient populations or for research purposes.

    Round 2-4:

29. The proposed outcome measures and their matched health indicators or data items will be anonymised and re-distributed to the expert panel, alongside a table outlining each outcome measure instrument's validity, reliability and responsiveness.
30. In round 2 participants will evaluate their ranked responses and return their final list.
31. A written report summarising the proposed 'core' data items and their associated outcome measure instruments, with any uncertainties / disagreements will be distributed to the expert panel.
32. Any outcome measure that has reached consensus of >70% will be included in the final dataset. The core domains (data items) these outcome measures assess will then be excluded from subsequent rounds of voting.
33. If consensus is not met in round 2 then this round will be repeated until such a time that consensus is achieved or 4 rounds are completed in phase 2. At this point the highest-ranked responses with a consensus >60% from each of the pre-identified categories will be accepted.
34. The finalised COS will be created and re-distributed to the panel for final approval alongside a short video presenting the consensus findings from phases 1 and 2 linking the study conclusions to International Classification of Disability and Health (ICF) core outcome sets and other relevant research.

Phase 3 (Optional):

If consensus was not achieved in phase 1 or 2 a nominal group technique will be used to reach consensus over the course of 1 working day.

During phase 3 the expert panel will discuss the supporting literature for each domain where uncertainty remains before voting on a final dataset.

If a consensus cannot be achieved, the meeting chair will have a deciding vote.

Data Analysis This study will use a mixed method approach of Quantitative and Qualitative analysis.

Quantitative analysis will include calculating the arithmetic mean and mode values before exploring standard deviations and interquartile ranges of the participant responses to represent the percentage agreement between participants' ratings (consensus). An outlier analysis will be completed to identify and exclude any results that could skew the interquartile range.

Qualitative analysis will include a thematic and content analysis of the statements each participant makes about a given topic or question using excel to aggregate and count similarly themed comments.

Thematic analysis will be coded as beliefs/opinions, differentiations, cause-effect relationships, examples, experiences, trends, no information, or misunderstandings.

Content analysis of participant's comments will be used to identify outcome measure instruments and other data items proposed by greater than one participant after each round. These will then be included into subsequent rounds of voting.