Routine health outcomes measurement

Routine health outcomes measurement is the process of examining whether or not interventions are associated with change (for better or worse) in the patient's health status. This change can be directly measured (e.g. by rating scales used by the clinician or patient) or assumed by the use of proxy measurement (e.g. a blood test result). Interventions can be direct (e.g. medication) or indirect (e.g. change in the process of health care like integration care by different specialists). Some definitions of health outcomes measurement stipulate that the population or group has to be defined (different outcomes are expected for different people & conditions). A strong example is that of Australia's New South Wales Health Department: health outcome is

"change in the health of an individual, group of people or population which is attributable to an intervention or series of interventions"[1]

In its purest form, measurement of health outcomes implies identifying the context (diagnosis, demographics etc.), measuring health status before an intervention is carried out, measuring the intervention, measuring health status again and then plausibly relating the change to the intervention.

Evidence-based practice describes a healthcare system in which evidence from published studies, often mediated by systematic reviews or processed into medical guidelines is incorporated into clinical practice. The flow of information is one way; from research to practice. However many interventions by health systems and treatments by their staff have never been, or cannot easily be, subject to research study. Of the rest, quite a lot is from research that is graded as low quality.[2] All health staff intervene in their patients on the basis of both information from research evidence and from their own experience. The latter is personal, subjective and strongly influenced by stark instances which may not be representative.[3] However, when information on these interventions and their outcomes are collected systematically it becomes "practice-based evidence"[4] and can complement that from academic research. To date, such initiatives have been largely confined to primary care[5] and rheumatology.[6] An example of practice-based evidence is found in the evaluation of a simple intervention like a medication. Efficacy is the degree with which it can improve patients in randomised controlled trials– the epitome of evidence-based practice. Effectiveness is the degree with which the same drug improves patients in the uncontrolled hurly-burly of everyday practice; data which are much more difficult to come by. Routine health outcomes measurement has the potential to provide such evidence.

The information required for practice-based evidence is of three sorts: context (e.g. case mix), intervention (treatment) and outcomes (change).[7] Some mental health services are developing a practice-based evidence culture with the routine measurement of clinical outcomes[8][9] and creating behavioral health outcomes management programs.

Perhaps because of instances of scandalously poor care (for example at the Bristol Royal Infirmary 1984-1995[17]) mortality data have become more and more openly available as a proxy for other health outcomes in hospitals,[18] and even for individual surgeons.[19] For many people, quality of life is a greater consideration so factors such as physical symptoms, psychological, emotional and spiritual, and information and support needs may take greater precedence. Therefore, as an indicator of the quality and safety of health care institutions, mortality remains important, but for individuals, it may not be the key goal.[20]

1. All three dimensions (context, intervention as well as outcomes) must be measured. It is not possible to understand outcomes data without all three of these.
2. Different perspectives on outcomes need to be acknowledged. For instance, patients, carers and clinical staff may have different views of what outcomes are important, how you would measure them, and even which were desirable[21]
3. Prospective and repeated measurement of health status is superior to retrospective measurement of change such as Clinical Global Impressions.[22] The latter relies on memory and may not be possible if the rater changes.
4. The reliability (statistics) and validity (statistics) of any measure of health status must be known so that their impact on the assessment of health outcomes can be taken into account. In mental health services these values may be quite low, especially when carried out routinely by staff rather than by trained researchers, and when using short measures that are feasible in everyday practice.
5. Data collected must be fed back to them to maximize data quality, reliability and validity.[23] Feedback should be of content (e.g. relationship of outcomes to context and interventions) and of process (data quality of all three dimensions)

Why is routine health outcomes measurement so rare? One can find reports of routine health outcomes measurement in many medical specialties and in many countries. However, the vast majority of these reports are by or about enthusiasts who have set up essentially local systems, with little connection with other similar systems elsewhere, even down the street. In order to realise the full benefits of an outcomes measurement system we need large-scale implementation using standardised methods with data from high proportions of suitable healthcare episodes being trapped. In order to analyse change in health status (health outcomes) we also need data on context, as recommended by Donabedian[15] and others, and data on the interventions being used, all in a standardised manner. Such large-scale systems are only at present evident in the field of mental health services, and only well developed in two locations: Ohio[8] and Australia,[9] even though in both of these data on context and interventions are much less prominent than data on outcomes. The major challenge for health outcomes measurement is now the development of usable and discriminatory categories of interventions and treatments, especially in the field of mental health.

Aspirations include the following benefits

• Aggregated data
  • Can form the basis of effectiveness data that complement efficacy data. This could show the actual benefits in everyday clinical practice of interventions previously tested by randomised clinical trials, or the benefits of interventions that have not been or cannot be tested in Randomized Controlled Trials and systematic reviews
  • Can identify hazardous interventions that are only apparent in large datasets
  • Can be used to show differences between clinical services with similar case mix and thus stimulate search for testable hypotheses that might explain these differences and lead to improvements in treatment or management
  • Can be used to compare the outcomes of treatment and care from different perspectives– e.g. clinical staff and patient
• Data about individual patients
  • Can be used to track changes during treatment over periods of time too long to be amenable to memory by an individual patient or clinician, and especially when more than one clinician or team is involved[24]
  • Can, especially when different perspectives are available, be used in discussions between patients, clinicians and carers about progress[25]
  • Can be used to speed up and crispen clinical meetings[26]

1. If attempts are made to purchase or commission health services using outcomes data, bias may be introduced that will negate the benefits, especially in the service provider produces the outcomes measurement. See Goodhart's Law
2. Inadequate attention may be paid to the analysis of context data, such as case mix, leading to dubious conclusions.[27]
3. If data are not fed back to clinicians participating then data quality (and quantity) is likely to fall below the thresholds necessary for reasonable interpretation.[28]
4. If only a small proportion of episodes of health care have completed outcomes data, then these data may not be representative of all episodes, although the threshold for this effect will vary from service to service, measure to measure.
5. Some risks of bias, widely foretold,[29] are proving to be insubstantial but need guarding against

Experience suggests that the following factors are necessary for routine health outcomes measurement

1. an electronic patient record system with easy extraction from data warehouse. Entry of outcomes data can then become part of the everyday entry of clinical data. Without this, aggregate data analysis and feedback is very difficult indeed.
2. resources and staff time set aside for training and receiving feedback
3. resources and personnel to extract, analyse and proactively present outcomes, casemix and, where available, intervention data to clinical teams
4. regular reports on data quality as part of a performance management process by senior managers can supplement, but not replace, feedback

Outcome measurement is therefore an important but neglected tool in improving quality of healthcare provision. It has been argued that it is vital that the patient has been meaningfully involved in decisions about whether or not to embark on an intervention (e.g. a test, an operation, a medicine). This is especially so if the decision is fateful (i.e. cannot be reversed).[31] Although a process rather than an outcome measure, the degree with which patients have been involved in shared decision making is clearly important.[32]

• Effectiveness
• Efficacy
• Evidence-based practice
• Healthcare system
• Primary care
• Research