A critical appraisal of the WHO 2024 systematic review of the effects of RF-EMF exposure on tinnitus, migraine/headache, and non-specific symptoms

Internal validity

1. What was the SR’s research question (PECO: Population; Exposure; Controls; Outcomes)? Was it appropriate and adhered to?

Röösli et al. [3] state that the aim of their SR was “to assess the effects of continuous or repeated local and whole human body RF-EMF exposure per unit-increase of one week or longer (E) on the occurrence of tinnitus, migraine and nonspecific symptoms (O), in the general population or workers (P) and to assess whether there is an exposure–response relationship between these outcomes and RF-EMF exposure levels (C).” The research question was therefore clearly posed, in the recommended format for SRs. As for the degree to which the review adhered to this research question, we are not concerned that this SR veered from its stated aims per se. What we focus on in this critique is the methods utilized by Röösli et al. in executing the SR.

1. Is it unlikely that important, relevant studies were missed? What are the implications?

The literature search protocol is well described and appears to follow the published protocol for this SR [15] providing confidence that the search is reproducible. It is unlikely that any important, relevant studies were missed, as witnessed by the fact that the initial count of papers found was 4,458. However, we are concerned by the omission of both interrupted time-series studies and human experimental (provocation) studies of RF-EMF exposures, as discussed in the next section. We are also concerned that only one study included in the final meta-analysed lists of Röösli et al. [3], Schoeni et al. [19], was focused on children and adolescents. There are published accounts of significant burdens of illness in these age-groups, such as headaches, attributed by study subjects to cellphone use [20], 21].

1. Were the criteria used to select articles for inclusion/exclusion appropriate? If not, why not?

This is a difficult and contentious question, because the 4,458 citations initially identified in the literature search were reduced by the authors, according to the paper’s flow-chart, to only 13 relevant, unique publications, concerning only eight independent studies of a quality deemed adequate for detailed analysis. While this level of exclusion is not unusual in literatures with many sub-standard primary studies, as is common in environmental epidemiology [22], 23] we would question some of the exclusion criteria, as potentially omitting useful evidence for this SR. For example, human experimental studies, typically involving deliberate provocation of such symptoms by laboratory-controlled exposures to RF-EMFs, have been made the subject of a separate SR, still to appear [24], and so are excluded from consideration by Röösli et al. [3] Although that literature is controversial [25] we would argue that omitting it entirely, a priori, from the SR by Röösli et al. [3] is contrary to best epidemiological practice, in that experimental studies – when of high quality – virtually always contribute stronger evidence of causation than observational studies [18].

Röösli et al. [3], 15] also excluded from detailed consideration any cross-sectional studies, on the grounds that they are not able to demonstrate clear temporality (i.e. that the exposure preceded the outcome in time), a strong Bradford Hill criterion for causation [18]. This meant that the only epidemiological study designs included in their final list of relevant and reasonable-quality studies were traditional cohort or case-control studies. However, we would point out that the symptoms focused on by Röösli et al. [3] (tinnitus, migraine/headache, sleep disturbance, as well as more non-specific complaints, such as fatigue, exhaustion and nervousness) are typically recurrent over long periods of time – often years. These symptoms’ potential causes/precipitants are therefore not ideally studied by standard cohort or case-control designs used in observational epidemiology, which tend to assume that the outcome is more or less irreversible, as is the case, for example, in completed stroke/myocardial infarction or confirmed cancer. For recurrent symptomatic symptoms over prolonged periods, alternative etiological study designs are often preferable, such as interrupted time-series analyses of cases, assessing the association between changes in exposure levels and onset or exacerbation of symptoms. In fact, Röösli himself has recently published, as senior author, a case time-series analysis of acute psychiatric symptom exacerbations in admitted patients, and peaks of noise from a local military airbase [26].

The omission of any such “non-standard” epidemiological study designs in this SR – which ended up with only 12 cohort and one case-control publications, based on only eight separate studies, worthy of detailed analysis, may well have introduced bias. That is because the traditional cohort study design aiming to illuminate disease causation, requires the exclusion of subjects with a history of the health outcome at baseline. In this case, persons with a history of tinnitus, migraine/headache, etc. in response to RF-EMF exposure (including persons regarding themselves as having Electromagnetic Hypersensitivity Syndrome – EHS, referred to as “IEI-EMF” by Röösli et al. [3]) appear to have been excluded at baseline from the eight separate cohort studies finally determined by Röösli et al. [3] to be of sufficient relevance and quality to be assessed in detail in their SR. This exclusion effectively limits the SR’s results to the causation of new-onset symptoms only – arguably a very small proportion of all relevant cases in the population at a given time, given that these conditions are typically recurrent.^[1] Furthermore, as discussed in detail below, related to the questionable statistical power of almost all the meta-analyses conducted by Röösli et al. [3], failure to include even a handful of such “non-standard” observational epidemiological studies would have contributed substantially to the very small number of studies finally analysed – which varied from just three (for tinnitus and sleep disturbance) to four (for migraine/headache and composite symptoms), across all the broad categories of RF-EMF exposure examined.

In sum, we submit that Röösli et al. [3] over-excluded potentially valuable studies with non-standard epidemiological designs such as time-series analysis, and also did not adequately justify their a priori exclusion of all human experimental “provocation” studies.

1. Were the included studies sufficiently valid for the type of question asked? If not, why not?

As described in great detail in the OHAT Risk of Bias tool, the full assessment of the validity of each of the primary studies, both those included and excluded in such an SR, requires intensive analysis of potential biases of several distinct types. Röösli et al. [3] cite eight such “domains” of bias related to: subject selection; attrition (losses to follow-up in cohort studies); information bias (errors) in both exposure (RF-EMF) and outcome measurement; selective reporting; appropriate statistical methods; and reverse causality.

We have examined the Risk of Bias analyses of Röösli et al. [3], for example, as tabulated in their Figure 2. We find their summary assessment to have been, if anything, more lenient than is warranted, in stating “About half of the assessed outcome-exposure-population combinations had low probability for risk of bias.” We reviewed a number of the primary studies included in the final meta-analysis of Röösli et al. [3] and find major methodological weaknesses that would normally exclude them from the final list of core studies for an SR.

An example of a primary study selected for meta-analysis by Röösli et al., despite its major methodological weaknesses, is the COSMOS study of headache and tinnitus as related to cellphone use in Finland and Sweden [27]. The COSMOS study contributes a substantial proportion of the data that go into two of the six meta-analyses of various exposure/outcome combinations which are depicted in Röösli et al.’s Figures 3–7 [3], so its influence on the pooled estimates of effect in those Forest Plots is substantial. However, the four-year long COSMOS cohort study suffered from: exclusion at baseline of subjects with a history of tinnitus or weekly headaches (see subheading above for why this is problematic); over-adjustment for potential confounding by inclusion in the main multivariable analysis of the covariate “daily painkiller use” (which is likely to reduce observed effect-sizes); and the choice of the reference level of cellphone use as “the bottom 50 % of the cellphone use distribution at baseline,” rather than a more extreme percentile cutoff. In addition, there is no real rationale for a four-year study duration of symptoms which are usually induced, in persons who report them related to RF-EMF exposure, after a much briefer latency (typically hours to days). Ascertaining such outcomes so long after baseline exposures were measured risks increasing exposure misclassification substantially, during an era when both cellphone technology and user habits were changing rapidly. In particular, ongoing major changes in cellphone signal technology throughout the long period of the COSMOS study’s estimation of exposure (including a stratum of subjects with over 15 years of exposure in total) could only increase the inherent exposure misclassification arising from the study’s use of rather crude proxy measures of actual RE-EMF exposure among study subjects (cumulative number and duration of calls) despite the fact that operator information was used to cross-validated self-reported phone use.

Virtually all of the methodological weaknesses listed above would create biases in the direction of reducing the observed strengths of association towards the null (e.g. Relative Risk of 1). For example, the overall Relative Risks (RRs) reported in the COSMOS study were not statistically significantly different from 1 for both tinnitus and headache, despite the study’s huge statistical power. In our view [28] these COSMOS findings were so likely to have been biased that this study should not have been selected for the final meta-analysis by Röösli et al. [3] Notably, removing it from the two meta-analyses in which it was included by Röösli et al. [3] would have substantially increased the pooled RR across the remaining two or three primary studies, since the large sample-size of the COSMOS study led to its receiving heavy weighting, in the meta-analyses by Röösli et al. [3], accounting for 35–40 % of the overall weight of all studies pooled.

Finally, some of the primary studies selected for meta-analyses clearly had lower than required statistical power to show a clinically significant association (e.g. RR>, say, 1.5) as statistically significant. This is illustrated by the three primary studies analysed for the outcome “tinnitus” – two of which [29], 30] have 95 % confidence intervals on their central estimates of Relative Risk which are very wide. In fact, the equivalent RR intervals of uncertainty run the effect-size equivalent from 1 (the null) to more than 2 (a moderate strength of association, almost always of clinical significance if valid, especially in environmental exposures with high rates of exposure in the general population, as is the case here.) There are equally low-powered studies in the other meta-analyses of Röösli et al. [3] – for example, in their Figure 7, three of the four primary studies of non-specific symptoms which were meta-analysed for “modelled exposure,” and two of the four primary studies of those same outcomes meta-analysed for “self-perceived exposure” have remarkably wide 95 % confidence intervals for their estimate of effect-size, which in all cases span the null hypothesis.

Low power in primary studies is normally not a reason to exclude them from meta-analysis, the main purpose of which is to overcome power problems by pooling results across studies. However, when the total number of primary studies is as small (three to four) as in all six Forest Plots depicted by Röösli et al.’s [3] Figures 3–7, the inclusion of so many small studies substantially reduces the power of the meta-analyses. The meta-analyses of Röösli et al. [3] therefore suffer from two separate but synergistic power problems: 1) the very small number of studies included for each exposure/outcome combination examined; 2) inadequate sample-size for a substantial proportion of those studies.

1. Were the results similar from study to study? If not, why not?

The inadequate power of the meta-analyses conducted by Röösli et al. [3] is made further problematic because of the very high levels of across-study heterogeneity [31]. This is explicitly clear from the I-squared values cited below the six Forest Plots (Figures 3–7) in their paper. These I-squared values are above 85 % for all the meta-analyses summarizing more than two primary studies – below which such statistics are pretty much meaningless. Any I-squared value above 75 % is regarded as indicating “substantial heterogeneity” across primary studies’ results [32], 33]. Such heterogeneity across primary studies’ results renders meta-analyses extremely unreliable and risks producing biased pooled results, due to the excessive influence of just one or two larger studies (as exemplified by the COSMOS study’s strong influence on two of the meta-analyses by Röösli et al. [3], discussed above). Widely cited methodological guidance for meta-analyses strongly recommends that identifying as both relevant and of adequate quality less than ten– and certainly five–primary studies of a given exposure/outcome combination, is a contraindication to performing meta-analysis at all, especially in the presence of substantial heterogeneity [34], [35], [36].

It is here that we encounter perhaps our major concern with the approach of Röösli et al. [3], in calculating meta-analytic summaries of the RRs (or standardized mean difference in the case of continuous outcomes, such as symptom scores) and their 95 % confidence intervals for each of five health outcomes, pooled across the finally selected primary studies (which never number more than four for any Forest Plot of a given exposure/outcome combination). Pooling results in this way is not best epidemiological practice, when – as Röösli et al. [3] freely admit – the number of studies available for each meta-analysis is small and the heterogeneity of effect-sizes found across those primary studies is very high. The appropriate scientific conclusion here is that the available and selected studies, for each exposure-outcome-population combination assessed, are so inconsistent in their results that they must not be estimating the same actual strength of association, so that pooling the results is contra-indicated [36].

A particular consequence of pooling studies with heterogeneous results is that it inevitably “dilutes” the influence of studies finding higher effect-sizes by adding in studies with smaller ones, which (as noted above) typically tend to suffer from biases that underestimate association strength, especially in environmental cohort studies, due to widespread misclassification/measurement-error for the exposure of interest. This effect is evident for the meta-analytic results depicted in Forest Plots, for several of the outcomes examined by Röösli et al. [3] Of even greater concern, three of the six depicted Forest Plots (those for “headaches,” “sleep disturbance” and “non-specific symptoms”) show some primary studies with statistically significant observed associations in the opposite direction to that expected. Such a counterintuitive finding can be confidently attributed to various sorts of study biases, simply because there is no known mechanism by which increased EMF exposure would protect from such symptoms. Rather than pool such conflicting and likely biased results, thereby reducing the overall pooled effect to near or below the null in these very heterogeneous sets of primary studies, it would have been better to treat studies finding such “opposite to expected” directions of association as “inadmissible due to bias,” and exclude them from further consideration (typically, when heterogeneity is encountered in a meta-analysis, supplemental analyses are conducted to identify effect modifiers; however, this was not practical for Röösli et al. [3] due to the small number of studies available after the extensive elimination of studies included in this SR.).

As a result of these concerns, we are of the view that calculating and depicting – in the six Forest Plots of this SR – pooled estimates, of the strengths of associations across primary studies, borders on pseudo-science. It implies that such pooled estimates are valid, when they are not. In short, they should not have been calculated for such sparse and heterogeneous sets of studies, many with profound methodological weaknesses likely to have led to bias.

1. What were the results? Are they credible?

We credit Röösli et al. [3] with acknowledging – e.g. in the very last line of their abstract – the “substantial uncertainty” arising from the limitations of the 13 included papers from eight independent studies, spanning several exposure-categories and the four primary outcomes tabulated in their Figure 2. We are concerned, however, that the phrasing used by Röösli et al. [3] to summarise their conclusions does not accurately reflect the implications for meta-analysis of this uncertainty. For example, the main conclusion of their abstract, “There is no indication that RF-EMF below guideline values causes symptoms …” is simply not scientifically accurate; indeed, it is misleading. It should read: “Overall, the quantity and quality of evidence available from the primary studies reviewed is insufficient to draw any valid conclusions about whether or not RF-EMF exposures below guideline values cause the symptoms studied.” This sort of scientific misstatement has long been acknowledged as one of the most frequent errors in meta-analyses, often referred to as “mistaking absence of evidence for evidence of absence (of an effect)” [36], 37].

One further consideration, in assessing the potential of the SR by Röösli et al. [3] for bias, is the affiliations and funding sources of the authors, which are only partly declared in their paper, compared to those documented in publicly available sources. This worrisome under-declaration of potential conflicts of interest is demonstrated by contrasting the statement of such potential conflicts at the end of this SR (first quotation below) with the statement of funding from the institution with which Röösli et al. [3] are associated (second quotation below):

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Martin Röösli’s research is entirely funded by public or not-for-profit foundations. He has served as advisor to a number of national and international public advisory and research steering groups concerning the potential health effects of exposure to nonionizing radiation, including the World Health Organization, the International Agency for Research on Cancer, the International Commission on Non-Ionizing Radiation Protection, the Swiss Government (member of the working group “mobile phone and radiation” and chair of the expert group BERENIS), the German Radiation Protection Commission (member of the committee Non-ionizing Radiation (A6) and member of the working group 5G (A630)) and the Independent Expert Group of the Swedish Radiation Safety Authority.

This statement does not reflect the true extent of the relationships with telecommunications firms of Röösli et al. [3] and their home institution (Swiss Research Foundation for Electricity and Mobile Communication – FSM), which are described in that Foundation’s 2022 Annual Report as follows [38]:

Organization and financing: the Research Foundation is sponsored by the ETH Zurich, and the companies Cellnex, Ericsson, Sunrise, Swisscom, and Swissgrid. Institutionally, the FSM is supported by the Swiss Federal Offices of Public Health (FOPH), Communications (OFCOM), Environment (FOEN), and Energy (SFOE), as well as by the Federal Inspectorate for Heavy Current Installations (ESTI). In addition, the following NGOs support the Foundation: the Swiss Academy of Engineering Sciences (SATW), Swiss Consumer Forum (KF), the Swiss Heritage Society (SHS), the Swiss Cancer League, Ingenieur Hospital Schweiz, the Swiss Electricity Industry Association (VSE), the Swiss Telecommunications Association (ASUT), Suissedigital, Electrosuisse, Swico, the Swiss Conferences of Cantonal Ministers for Construction, Planning and the Environment (BPUK), and for Energy (EnDK).

These potential conflicts of interest are of great concern. There are many aspects of both systematic reviews and meta-analysis which – despite the use of widely recommended tools such as the OHAT Risk of Bias scale and GRADE scheme for assessing strength of evidence – involve inherently subjective decisions. Such subjectivity inevitably leads to significant variation across such reviews, even when precisely the same primary studies are being assessed [17], 39]. This potential for subjectivity requires clearcut independence of such reviews’ co-authors from any and all influences which might lead to bias related to conflicts of interest.

External validity

1. Were the results suitable for generating scientifically robust exposure limits for “real-world” RF-EMFs?

We have a specific concern regarding the extrapolation of this SR’s results to the current challenges of protecting the public from the rapidly evolving RF-EMF exposures now occurring in many countries, such as those related to the rollout of 5G telecommunications technologies. That concern is that all of the specific exposures studied in the literature reviewed by Röösli et al. [3] are not representative of the very diverse emerging RF-EMF exposures now affecting human populations (and the environment more generally) in the real world [8], 9]. To be fair, this is a general problem with many health protection topics, not just regulatory policies for RF-EMFs – although it is barely discussed by Röösli et al. [3] Overcoming this major lag, between real-world technological changes and the available research literature on their safety for the public, will require a substantial reorganization of, and massively increased funding for, international studies by top, impartial researchers, of newer RF-EMFs’ adverse effects, both in humans and other species [4], [5], [6], [7], [8], [9].