Systematic reviews—Great gains and significant pitfalls
-
Katri Hamunen
1 Systematic reviews are important
Systematic reviews have become very popular and productive form of publishing within medical science. They are considered the top of hierarchy of medical evidence. Systematic reviews provide clinicians with tools for evidence-based practice. They are utilized in producing national and international clinical guidelines, decision-making of reimbursement policies and other health care decisions. This places a severe responsibility to all involved in production and publication of systematic reviews since they do not come without problems and they do not provide us with the final truth – for several reasons.
2 Many problems with systematic reviews
There may be significant problems with the primary studies e.g. they may be small, few in number, clinically heterogeneous, of poor quality, or lacking in internal and external validity. In addition, the review process carries its possible problems. Search strategy, study selection, assessment of study quality and bias, selection of outcomes and decisions on data pooling are all points where the authors will have to make choices between alternatives and therefore there is always the possibility of bias. Rarely is a result from a clinical study or systematic review so clear that it does not need any interpretation. The results need to be evaluated in light of study population, design and limitations, and previous information on the topic. This goes for both clinical trials and systematic reviews. We all are human and therefore more or less susceptible to confirmation bias, or in other words, we like studies and conclusions that support our own views, we do not like those papers that go against our own convictions.
3 Performing a systematic review is labor-intensive
A systematic review is an exhaustive summary of literature relevant to the topic in question. Performing a good quality systematic review is demanding and time consuming. A systematic review should always be based on a rigorous protocol that is followed and the progress of the process documented with equivalent rigorousness and transparency. It is important that authors conducting a systematic review have in-depth knowledge on the clinical problem and methodology of systematic reviews. Several guidelines and checklists have been developed in order to improve the quality of performing and reporting of systematic reviews and meta-analyses, e.g. the PRISMA statement and the Cochrane Handbook for Systematic Reviews of Interventions [1,2,3].
4 Systematic reviews of adverse effects are unreliable due to poor reporting
Adverse effects are the other side of the coin of any treatment. In most analgesic studies, analgesic efficacy is the main outcome and it is also used for power calculation. Adverse effects typically occur less frequently than analgesia and their true incidence is difficult to calculate reliably. Systematic reviews can be used to pool large amounts of data to partly overcome this problem. However, the incidence and severity of adverse effects are often poorly studied and reported in clinical trials [4,5], and even systematic reviews cannot overcome the problem in absence of reliable original data.
5 Systematic reviews provide ideas for research projects
In addition to generating evidence for clinical and administrative purposes, systematic reviews have their place also in providing ideas and material for future clinical trials and developing trial methodology. Sometimes a systematic review concludes that more randomized trials are needed in order to make firm conclusions about efficacy of a particular treatment. As such, this conclusion does not take medicine much further. However, if we look at the systematic review as an overview what has been tried and what is known about a particular condition or treatment modality, we can use this information to develop a research question and a protocol for new clinical trial. In fact, it has been suggested that a systematic review should form the basis or starting point of new Ph.D. projects and large grant applications. It is a suggestion that I warmly support. Performing a systematic review is also excellent training in planning and reporting own clinical trials.
6 Systematic reviews of methodology, on average data, or on individual patient data
Systematic reviews that do not aim in directly evaluating clinical efficacy of a given treatment but concentrate in exploring methodological aspects of studies are important for evolution of trial methodology and reporting [6,7,8]. One significant problem with existing systematic reviews is that they are often based on available average data retrieved from original clinical trials. However, clinical responses in pain are often skewed and using individual patient data instead of mean data in systematic reviews will give more reliable and useful results [9,10].
7 Evaluation of qualities of systematic reviews
There are also checklists for structured evaluation of published systematic reviews [11]. These may be useful especially for those readers who are not yet familiar with the methodology of systematic reviews. A lazy reader may skip the paragraphs describing the methodology of the review at hand and thereby may be in grave danger of making false conclusions on the results and generalization of the findings. Therefore, there are responsibilities also for users of systematic reviews.
8 How to read a systematic review
Important aspects to consider while reading a systematic review include selection of the databases and search strategy, inclusion and exclusion criteria, estimation of bias, evaluation of study quality, pooling of data. The results and conclusions of a systematic review must always be interpreted with respect of the data they were produced from. One of the challenges in reporting systematic reviews is to provide the readers enough transparency and information on these important steps in a concise and informative way.
9 Systematic reviews can be used as one of several tools to find the truth
Systematic reviews offer one tool alongside other research tools to increase our knowledge in medicine. They do not give us the final truth – for several reasons listed above – nor are they an automat for easy answers but they can be valuable tools in developing clinical management of patients and also by moving trial methodology forward. In optimal situation, clinical trials and systematic reviews support each other by giving ideas and knowledge to the next step of research.
10 A well-done systematic review in this issue: treatment of chronic pelvic pain
In this issue of the Scandinavian Journal of Pain Loving et al. [12] report their systematic review on efficacy of various physiotherapeutic treatments of female chronic pelvic pain. They found ten studies with highly variable outcomes. Their review provides an overview of the literature available on the topic. The conclusions from this review are limited. Based on the review there is no really proven intervention for this condition but clinicians have the available data presented and can make their own judgment what treatments they might want to consider to try for their patients. In addition, interventions investigated so far are nicely outlined and provide a good starting point for planning next clinical trial (s) on this problematic pain condition. I sincerely hope that clinicians and researchers will take up the challenge from this review and start planning future studies on this still very little investigated problem.
DOI of refers to article: 10.1016/j.sjpain.2011.12.002.
References
[1] Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, Clarke M, Devereaux JP, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med 2009;151:65–94.Search in Google Scholar
[2] Cochrane handbook for systematic reviews of interventions. http://www.cochrane-handbook.org.Search in Google Scholar
[3] Authoring or assessing a Cochrane Protocol, Review, or Review Update for the PaPaS Review Group. Cochrane Pain, Palliative and Supportive Care Group. http://papas.cochrane.org/papas-documents.Search in Google Scholar
[4] Edwards JE, McQuay HJ, Moore RA, Collins SL. Reporting of adverse effects in clinical trials should be improved: lessons from acute postoperative pain. J Pain Symptom Manage 1999;18:427–37.Search in Google Scholar
[5] Kalso E, Edwards J, McQuay HJ, Moore RA. Five easy pieces on evidence based medicine (5). Trading benefit against harm – pain relief vs. adverse effects. Eur J Pain 2002;6:409–12.Search in Google Scholar
[6] Moore RA, Straube S, Eccleston C, Derry S, Aldington D, Wiffen P, Bell RF, Hamunen K, Phillips C, McQuay H. Estimate at your peril: imputation methods for patient withdrawal can bias efficacy outcomes in chronic pain trials using responder analyses. Pain 2012;153:265–8.Search in Google Scholar
[7] Furlan A, Chaparro LE, Irvin E, Mailis-Gagnon A. A comparison between enriched and nonenriched enrollment randomized withdrawal trials of opioids for chronic noncancer pain. Pain Res Manage 2011;16:337–51.Search in Google Scholar
[8] Barden J, Edwards JE, Mason L, McQuay HJ, Moore RA. Outcomes in acute pain trials: systematic review of what was reported? Pain 2004;109:351–6.Search in Google Scholar
[9] Moore RA, Mhuircheartaigh RJ, Derry S, McQuay HJ. Mean analgesic consumption is inappropriate for testing analgesic efficacy in post-operative pain: analysis and alternative suggestion. Eur J Anaesthesiol 2011;28:427–32.Search in Google Scholar
[10] Moore RA, Edwards JE, McQuay HJ. Acute pain: individual patient meta-analysis shows the impact of different ways of analysing and presenting results. Pain 2005;116:322–31.Search in Google Scholar
[11] Systematic review critical appraisal sheet. Centre for Evidence Based medicine, University of Oxford. http://www.cebm.net/.Search in Google Scholar
[12] Loving S, Nordling J, Jaszczak P, Tomsen T. Does evidence support physiotherapy management of adult female chronic pelvis pain? A systematic review. Scand J Pain 2012;3:70–81.Search in Google Scholar
© 2012 Scandinavian Association for the Study of Pain
Articles in the same Issue
- Editorial comment
- HIGH risk of chronic pain in women with LOW education, LOW socio-economic status, and LOW mood
- Observational studies
- Pain among women: Associations with socio-economic factors over time and the mediating role of depressive symptoms
- Editorial comment
- Systematic reviews—Great gains and significant pitfalls
- Review
- Does evidence support physiotherapy management of adult female chronic pelvic pain? A systematic review
- Editorial comment
- Effective treatment of osteoarthritic pain, tackling the challenge with pets
- Review
- Translational animal models using veterinary patients – An example of canine osteoarthritis (OA)
- Editorial comment
- Pre-procedure anxiety aggravates pain—A problem also in adult patients
- Clinical pain research
- Anxiety and pain during bone marrow aspiration and biopsy
- Editorial comment
- What is required from studies evaluating multidisciplinary treatment in pain clinics?
- Clinical pain research
- Long-term outcome of multidisciplinary intervention of chronic non-cancer pain patients in a private setting
- Editorial comment
- Optimal thoracic epidural analgesia–Again
- Clinical pain research
- Epidural pain management after open lateral thoracotomy: Female patients have better pain relief and need smaller amounts of analgesics than males
Articles in the same Issue
- Editorial comment
- HIGH risk of chronic pain in women with LOW education, LOW socio-economic status, and LOW mood
- Observational studies
- Pain among women: Associations with socio-economic factors over time and the mediating role of depressive symptoms
- Editorial comment
- Systematic reviews—Great gains and significant pitfalls
- Review
- Does evidence support physiotherapy management of adult female chronic pelvic pain? A systematic review
- Editorial comment
- Effective treatment of osteoarthritic pain, tackling the challenge with pets
- Review
- Translational animal models using veterinary patients – An example of canine osteoarthritis (OA)
- Editorial comment
- Pre-procedure anxiety aggravates pain—A problem also in adult patients
- Clinical pain research
- Anxiety and pain during bone marrow aspiration and biopsy
- Editorial comment
- What is required from studies evaluating multidisciplinary treatment in pain clinics?
- Clinical pain research
- Long-term outcome of multidisciplinary intervention of chronic non-cancer pain patients in a private setting
- Editorial comment
- Optimal thoracic epidural analgesia–Again
- Clinical pain research
- Epidural pain management after open lateral thoracotomy: Female patients have better pain relief and need smaller amounts of analgesics than males
