Detecting treatment interference under K-nearest-neighbors interference

1.1 Motivating example: An anti-conflict program in New Jersey schools

To motivate our approach, we refer to a recent randomized field experiment assessing the efficacy of an anti-conflict intervention aimed to reduce conflict among middle school students in 56 schools in New Jersey [17]. In particular, the experiment was explicitly designed to determine whether benefits of the program can be propagated through social interactions between students.

The intervention was administered through “seed” students – those that are selected to actively participate and advocate for the anti-conflict program. These students attended meetings with the program staff every 2 weeks to address conflict behaviors in their schools and to talk about strategies to mitigate peer conflict. Additionally, seed students were encouraged to publicly reflect their opposition to conflict in their school – identifying a common conflict in their school and creating a hashtag about it – and were also asked to distribute orange wristbands with the intervention logo to students that demonstrate anti-conflict attitudes.

Seed students were randomly assigned as follows. First, within each of the 56 schools, between 40 and 64 students were identified as being eligible to be seed students. Then, from the 56 schools in the study, 28 schools were randomly assigned to receive the anti-conflict program. Finally, within each of these assigned schools, half of the eligible students were selected to be seed students. Analysis was performed only on students that were eligible to be seeds (N = 2,451).

Of particular note, to assess potential pathways for treatment interference, students were asked to identify, in order, the ten other students that they spent the most time with during the previous few weeks. These students include both seed and non-seed students. Specifically, the survey asks the following question: “In the last few weeks, I decided to spend time with these students at my school: (in school, out of school, or online) – Number 1 is for the person you spent most time with, then number 2, then number 3… You don’t have to fill in all the lines! To make it easier, you can write down their initials here, then find their number. It can be boys and girls!” [17]. Students’ responses to this question may include both seed and non-seed students. This yields a unique dataset in which the strength of the interaction between two individuals under study is explicitly recorded. Hence, statistical analyses may benefit from an interference model, such as KNNIM, that allows for direct incorporation of the relative strengths of the interactions. For this dataset, KNNIM models with K up to 10 may be applicable.

An analysis performed by Aronow and Samii [9] estimated the indirect effect of being a seed student on wearing an orange wristband to be about 0.15 with a 95% confidence interval between about 8 and 23 percentage points, i.e., students exposed to treated peers were about 15% more likely to report wearing an orange wristband in comparison with students in control schools.

3.1 Interaction measure

We begin formally introducing KNNIM by introducing an interaction measure d ( i , j ) that measures how strongly unit i associates with unit j . This measure does not necessarily need to be computed across every pair of units ( i , j ) ; however, we assume that at least K values of d ( i , j ) can be computed for each unit i , j ≠ i . Here, d ( i , j ) may be measured explicitly. For example Section 1.1 describes an example where respondents assign numbers to ten students, from 1 to 10, where 1 denotes the closest connection, 2 denotes the second closest connection, etc. [17]. Alternatively, d ( i , j ) may combine several interaction measures to form a proxy for overall interaction. For example, an experiment on a social network may define d ( i , j ) to be an index variable aggregating the number of comments, likes, and other forms of engagement performed by user i and directed towards user j . Smaller values of d ( i , j ) may correspond to stronger or weaker interactions from i toward j depending on researcher preference. In this article, we assume that smaller values correspond to stronger interactions.

Of particular note, the dissimilarity measure is allowed to be asymmetric, i.e., d ( i , j ) and d ( j , i ) may differ. Such a property may be necessary if one user strongly influences another user, but not vice versa. A common instance of this involves social media moguls; a mogul i may induce strong engagement from millions of followers j , but may interact sparingly with the vast majority of these followers. This would suggest that followers of the mogul may be strongly impacted by an intervention given to the mogul – indicated by a small value of d ( j , i ) – but the mogul’s behavior may not be altered by their followers – indicated by a large value of d ( i , j ) .

Additionally, it may also be the case that the same absolute value of d ( i , j ) may be interpreted differently across users. For example, suppose that d ( i , j ) is an index variable for engagement on a social media platform. If two users i and i ′ interact with the same user j in identical ways, we may have d ( i , j ) = d ( i ′ , j ) . However, if i engages with the platform often and i ′ does so sparingly, then d ( i , j ) may be relatively large for user i (i.e., i may interact even more with close users j * , leading to smaller values of d ( i , j * ) ), but d ( i ′ , j ) may be relatively small for user i ′ .

3.2 K-neighborhood interference assumption (K-NIA)

Let d ( i , ( j ) ) denote the j th smallest value of { d ( i , j * ) , j * ≠ i } , i.e., d ( i , ( 1 ) ) < d ( i , ( 2 ) ) < … . For ease of exposition, we assume that all values of d ( i , j ) are unique (in practice, ties may be broken arbitrarily). The K-neighborhood of unit i , denoted N i K , is the set of the K “closest” units to unit i :

Define N − i K = { 1 , … , N } \ ( i ∪ N i K ) as the set of units that are outside of i ’s K -neighborhood. Note that the sets { i , N i K , N − i K } form a partition of the N units.

Recall that W i is a treatment indicator for unit i , and let W = ( W 1 , W 2 , … , W N ) = { W i , W N i K , W N − i K } denote the vector of treatment assignments given to all units N . Additionally, recall that y i ( W ) denotes the potential outcome for unit i under treatment allocation W ∈ { 0 , 1 } N . Now, we give the following assumption that defines KNNIM:

Assumption 1 states that the potential outcome of unit i is only affected by its treatment and by the treatments assigned to its KNN. Changing treatments for other units outside the K -neighborhood will not affect the potential outcome of unit i . This is a special case of the neighborhood interference assumption (NIA) described in Sussman and Airoldi [12]. In its most general form, KNNIM assumes only that the treatment interference structure satisfies Assumption 1. For convenience, we will suppress the treatment statuses in W N − i K when referring to the potential outcomes y i .

For ease of exposition, it is often convenient to view units under study as an adjacency graph G = ( V , E ) , where vertices represent the units under study and edges denote the potential paths for treatment interference. For KNNIM, let G KNN = ( V , E KNN ) denote a directed graph on ‖ V ‖ = N vertices; each vertex i ∈ V corresponds to a unit under study. An edge i j → ∈ E KNN if and only if j is one of i ’s K -closest neighbors: j ∈ N i K . Note, by definition, i i → ∉ E KNN . Each edge i j → ∈ E KNN has weight equal to the interaction measure d ( i , j ) . In this article, we may refer to G KNN as the weighted adjacency graph. Throughout this article, the terms vertex, unit, and individual will be used interchangeably.

Let A denote the N × N adjacency matrix of G KNN , which indicates the presence or absence of an edge i j → in the graph G KNN , i.e., A i j = 1 if i j → ∈ E KNN and A i j = 0 otherwise. Note that the diagonal elements of the adjacency matrix are zero, i.e., A i i = 0 for all i .

3.3 Choosing the neighborhood size K

The choice of K for a given study may vary depending on the studies’ field, the purpose of the study, and the availability of data. The experimenter may also use prior knowledge from previous studies to help choose K – if previous studies have indicated that a person’s behavior is influenced by their two closest friends, setting K = 2 may be appropriate. When possible, the K should be selected in early phases of the study to help construct the adjacency matrix A when collecting data.

However, another factor that should be addressed when choosing the size of K is the sample size needed to accurately quantify, estimate, and draw inference on the K -nearest neighbors indirect effects. As mentioned previously, the number of possible exposures to treatments under KNNIM is 2 K + 1 . Hence, to ensure sufficient power, many methods that incorporate KNNIM will require a sufficient number of units assigned to each of these exposure levels. From our experience, a good heuristic is to require roughly 30 observations for each treatment exposure. Under this heuristic, most studies may find models with K = 2 or 3 to be most useful.

Issues may arise if responses are used to inform the value of K . For example, a post hoc selection of K could lead to inaccurate detection of treatment interference due to inherent multiple testing issues (inferences must account for testing both the appropriateness of K and the presence of interference in the model) and/or bias in indirect effect estimates. It may be possible to incorporate additional structure into KNNIM to allow for a rigorous treatment of this problem, but such work is outside of the scope of this article. See Alzubaidi and Higgins [31] for additional information about the estimation of indirect effects under KNNIM.

6.1 Test statistics for randomization tests

Aronow [14] introduced the randomization inference approach for testing for interference between units, where units are affected by their own treatment and by the treatment assigned to their immediate neighbors. In this test, the treatment status for a subset of focal units remains fixed; the rest of the units are the variant subset. The randomization inference is conditional on the observed treatment status of the fixed subset, i.e., this test is on indirect effects resulting from the treatment allocation on the variant subset of units. A variety of test statistics may be used under this framework.

Pearson’s correlation coefficient ρ between the outcomes of the fixed units ( Y F ) and the “distance” to the nearest unit of a particular treatment status in the variant subset ( D nearest ) may be used as the test statistic:

A common choice of distance is the Euclidean distance between pretreatment covariates. This distance can be incorporated into the KNNIM framework through the interaction measure d . Aronow [14] advocates for computing Pearson’s correlation coefficient on the ranks of these quantities; however, preliminary simulations suggest that the statistic ρ tends to be more powerful for the models considered in Section 8.

Athey et al. [15] extended this work and developed tests for more general realizations of interference (e.g., no higher-order interference). As part of this work, they suggested additional test statistics for detecting interference. The edge-level contrast statistic T elc – a modification of a test statistic proposed by Bond et al. [34] – is the difference between the average outcomes of the focal units with treated neighbors and the focal units with control neighbors. Here, T elc averages over edges i j , where i is a focal unit and j is not a focal unit:

where F i is an indicator variable satisfying F i = 1 if and only if i ∈ F .

A second test statistic is the score test statistic T score [15]. This statistic is motivated by a model of treatment interference in which the indirect effect is proportional to the fraction of treated neighbors [11,30]. The score test begins by computing

for each focal unit i ∈ F , where Y ¯ F , 1 obs and Y ¯ F , 0 obs are the average outcome for the treated and control focal units, respectively. Then, T score is the covariance between these r i terms and

which is the fraction of treated neighbors for unit i . This statistic is computed across only focal units that have at least one treated neighbor:

Finally, Athey et al. [15] considered the has-treated-neighbor test statistic T htn , a modification of Pearson’s correlation coefficient (5). Instead of using the distance to the nearest treated neighbor, this statistic uses an indicator variable E i for whether any of a unit’s neighbors in the variant subset are treated, i.e., E i = 1 if and only if ∑ j A i j W j ( 1 − F j ) > 0 . Then, T htn is the correlation between this indicator and the outcomes for the focal units F :

where Y ¯ F obs and S Y F obs are the sample mean and standard deviation of the outcomes for focal units, respectively, and S E is the sample standard deviation of the E i variables.

6.2 Experimental design approach

Saveski et al. [5] and Pouget-Abadie et al. [4] presented a two-stage experimental design to test for the presence of interference. In this design, the units under study are divided into two groups and two experiments are performed simultaneously: for one group, treatment is assigned completely at random, and for another group, units are clustered and treatment is assigned across clusters rather than units. Then, estimates of the average direct effect are computed under the assumption of no interference for both the completely randomized and cluster randomized designs. Finally, a standardized difference T exp is computed between these estimates:

where τ ˆ c r and τ ˆ c b r are the estimates of the direct effect under the completely randomized and cluster randomized designs, respectively, and σ ˆ p is a pooled standard deviation of responses from both the completely randomized and cluster randomized designs [5]. Large values of T exp imply the presence of indirect effects.

A conservative test of the null hypothesis of no treatment interference can be performed at the α significance level by rejecting the null hypothesis if and only if T exp ≥ α − 1 ∕ 2 . Additionally, as the number of units n → ∞ , it can be shown that T exp converges to a standard normal distribution (provided that cluster sizes remain fixed). Thus, an approximate size α test can be conducted by rejecting the null hypothesis of no interference if T exp ≥ z 1 − α ∕ 2 , where z 1 − α ∕ 2 is the 1 − α ∕ 2 quantile of the standard normal distribution.

8.1 Data generation procedure

We generate the responses under the following model that satisfies KNNIM with K = 3 :

In this model, we assume that the closest three neighbors affect the response Y i ; we use W i ℓ to denote the treatment status of the ℓ th-nearest neighbor of unit i . The covariates X i p , p = 1 , 2 , 3 , are independent and identically distributed N o r m a l ( 0 , 1 ) random variables; let X i = ( X i 1 , X i 2 , X i 3 ) . We use the Euclidean distance between the covariates X i and X j as the interaction measure d ( i , j ) – units with more similar values of covariates are more likely to interact with each other. We additionally assume that the interference adjacency graph G is exactly G KNN . Note that this assumption impacts T elc , T score , and T htn ; however, T knn will not be affected, provided that G KNN is a subgraph of G .

Note that Model (8) defines the set of the potential outcomes for each unit i . Additionally, conditional on the covariates, the only source of randomness is in the treatment assignment; there is no additional random error term. Simulated data are generated by randomizing treatment across units for a given potential outcome model. Different models are obtained through varying the β = ( β 1 , β 2 , β 3 , β d ) coefficients and the sample size N . We consider sample sizes of N = 256 and N = 1,024 .

For each choice of sample size, we consider 16 different models of interference. We describe these models in Table 1 in terms of the coefficient vector β . The first three elements of β represent the indirect effect contributed by first, second, and third nearest neighbor, respectively. The last element β d is the unit’s direct effect. In all models considered, the closer the relationship to unit i , the greater the indirect effect: ∣ β 1 ∣ ≥ ∣ β 2 ∣ ≥ ∣ β 3 ∣ . The indirect effects in every set of three models represent the degree of interference starting from no interference in the first three models, followed by very weak interference in the second three models, weak interference in the next three models, moderate interference in the next three models, and finally strong interference in the last four models.

For datasets with N = 256 observations, 1,000 realizations of potential outcomes following each model are generated. Tests of indirect effects are then applied to each of the 1,000 realizations. Results for N = 256 are given in Section 8.4. Due to computational limitations, only 100 realizations are generated for models containing N = 1,024 units. Results for N = 1,024 are given in Supplementary Material.

8.2 Simulation for randomization tests

We compare the performance of both conditional randomization tests and experimental design approaches for detecting interference. For the conditional randomization tests, for each set of generated potential outcomes, treatment is initially assigned completely at random to units, with half of the units receiving treatment and the other half receiving control. Then, focal units are selected according to Algorithm 1. We then proceed with randomization tests as described in Sections 4 and 6.1. We evaluate the performance of the following test statistics: Pearson’s correlation coefficient (Pearson) [14], the edge-level contrast statistic (ELC), the score statistic (Score), the has-treated-neighbor statistic (HTN) [15], and the K -nearest neighbors indirect effect test statistic (KNN).

Test statistics are computed across 1,000 randomizations for each realization of the potential outcomes; for each randomization, treatment statuses are fixed for focal units and are completely randomized across variant units. For each set of potential outcomes and for each choice of test statistic, we obtain a p -value for the null hypothesis of no treatment interference. Thus, for N = 256 , we obtain a distribution of 1,000 p -values for each test statistic under each model. The power of the tests can also be estimated by computing the fraction of p -values that fall beneath a pre-specified significance level α .

8.3 Simulation for experimental design approach

In addition, we follow the experimental design in Saveski et al. [5] (described in Section 6.2) to determine its efficacy for testing whether SUTVA holds under KNNIM. For each set of generated potential outcomes, we divide the units into clusters of four units using a heuristic algorithm for the clique partitioning problem with minimum clique size requirement from Ji [35] (Algorithm 4). This clustering is performed once per set of potential outcomes.

We then randomly select half of the clusters to be cluster randomized; for this group, treatment is assigned at the cluster level, with half of the clusters receiving treatment and the other half receiving control. For units belonging to the remaining clusters, each unit’s cluster assignment is ignored, and treatment is completely randomized across all of these remaining units. Again, half of these units receive treatment and the other half receive control. For each set of potential outcomes, the random selection of clusters and the treatment randomization are performed 1,000 times.

For each randomization, the statistic T exp in (7) is computed. We then perform a test of the null hypothesis of no treatment interaction at the α = 0.05 significance level. A conservative test rejects this null hypothesis if T exp ≥ α − 1 ∕ 2 and an asymptotic test rejects the null if T exp ≥ z 1 − α ∕ 2 . Thus, for N = 256 , we perform a total of 1,000,000 tests: 1,000 tests for each of the 1,000 generated potential outcomes. By computing the fraction of rejected null hypotheses, we are able to assess the type I error (Models 1–3) and the power (Models 4–16) of the experimental design approach.

8.4 Discussion

Figure 1 provides a visual comparison of the distribution of p -values for the randomization tests to detect interference under KNNIM. Table 2 provides the estimated type I error and power of these tests (conducted at significance level α = 0.05 ) across the 16 considered models. As is expected by design [36], the p -values of all randomization tests under models without treatment interference (Models 1–3) are approximately distributed uniformly between 0 and 1. All tests lack power under very weak interference (Models 4–6) where the highest power is 0.110 for KNN test followed by 0.108 for Score test. Under weak interference (Models 7–9), the ELC, Score, and KNN tests seem to outperform the Pearson and the HTN tests; the p -values are smaller overall for these three tests. Similar trends hold under moderate interference (Models 10–12) and strong interference (Models 13–16). In particular, under strong interference, Score, KNN, and ELC tests have near 100% power to detect treatment interference (Figure 2).

Figure 1

Boxplots of p -values for Pearson, HTN, ELC, Score, and KNN under various KNNIMs. We use N = 256 units and K = 3 nearest neighbors. Plots also contain the estimated type I error (Models 1–3) and power (Models 4–16) for the Score, KNN, ELC, HTN, and Pearson. The p -values are estimated using 1,000 randomizations for each of the 1,000 generated potential outcome realizations.

Figure 2

Boxplots of p -values for the Score, KNN, ELC, HTN, and Pearson under Models 3, 7, 9 10, and 12. We use N = 256 units and K = 3 nearest neighbors. The p -values are estimated using 1,000 randomizations for each of the 1,000 generated potential outcome realizations.

However, the ELC and HTN tests seem to have some difficulty with detecting indirect effects when direct effects become large. For example, the p -values for these three tests under Models 9 and 12 – models that have comparatively larger direct effects – are substantially larger than under Models 7 and 8 and Models 10 and 11, respectively. The Score and KNN tests do not suffer from this loss of power as direct effects increase. For example, for Model 9, the Score and KNN tests have an estimated power of 0.844 and 0.839, respectively, where the ELC and HTN tests have an estimated power of 0.553 and 0.249, respectively. Thus, for the considered tests, the Score and KNN tests seem to have the best combination of power in detecting treatment effects and isolating indirect effects in the presence of direct effects. Similar comparisons between the methods hold for datasets with N = 1,024 and/or when focal units are selected from only one treatment condition (see the Supplementary Material for details).

We note that these tests may perform differently under different models of response. On the one hand, our model assumptions are favorable to competing tests; some preliminary work has suggested that if the KNNIM model holds but the observed adjacency graph G contains edges outside of G KNN , then test statistics that rely on the fraction of treated neighbors (e.g., T elc , T score , and T htn ) lead to less-powerful tests, whereas the performance of the KNN tests will be unchanged. On the other hand, provided that the adjacency graph is correctly specified, preliminary work also seems to suggest that Score tests perform similarly to, if not better than, KNN tests under more complex model specifications – when including terms for the interaction of treatment indicators. Finally, other interference models (e.g., the model considered in Leung [29]) may favor one or more of these tests; however, investigation of these interference models is outside of the scope of this article.

Figure 3 gives the box plots of the estimated rejection rate across all 1,000 generated potential outcomes for both the conservative and asymptotic tests using the experimental design method [4,5] with N = 256 and significance level α = 0.05 . This plot also shows the estimated power of the considered randomization tests under these 16 models. Table 2 includes the median values of the rejection rates across the 1,000 generated potential outcomes for these tests. The conservative experimental approach appears to lead to a very conservative test; the true type I error is much smaller than α = 0.05 , and the test appears to have weak power under very weak, weak, and moderate interference. Even under Models 13–16, which exhibit strong interference, the conservative test only has a median power of approximately 0.6965.

Figure 3

Boxplots of the estimated rejection rates under the experimental design approach for both the conservative and asymptotic tests of the null hypothesis of no treatment interference under various KNNIMs. Plots also contain the estimated type I error (Models 1–3) and power (Models 4–16) for the Score, KNN, ELC, HTN, and Pearson. We use N = 256 units and K = 3 nearest neighbors. The rejection rates are estimated using 1,000 treatment assignments for each of the 1,000 generated potential outcomes. Tests are performed at significance level α = 0.05 .

The asymptotic test yields much more desirable results for our simulated data. Overall, the type I error seems quite close to the nominal α = 0.05 . The asymptotic test outperforms the Pearson and HTN randomization tests for almost all models of interference and has a power close to 1 of detecting interference under Models 13–16. However, the power of the asymptotic test still is behind that of the Score, KNN, and ELC tests across all models.

When we increase the sample size to N = 1,024 , the conservative approach seems to be powerful for moderate and strong interference, while the asymptotic approach is powerful for all interference models except the very weak interference models. However, both approaches remain comparatively less powerful than the Score, KNN, and ELC randomization tests (see Supplementary Materials for details).

9.1 Selecting K

Recall that the K = 10 closest connections were identified for each student. However, implementing a KNNIM model with K = 10 is impractical for this example. For a study of this size (N = 2,451), such a model would result in too many potential exposures for each unit (2,048 in total) to allow for meaningful inference to be performed on the indirect effect. Moreover, seed-eligible students often identify connections with ineligible students which are not included in A – in fact, most seed-eligible students have fewer than three connections with other seed-eligible students. This complicates the implementation of KNNIM with K = 10 , which (from Section 3) is only well identified when each observation has at least ten connections.

To determine whether a choice of K is appropriate for this application, we first subset all seed-eligible students that have at least K connections with other seed-eligible students. We then calculate how many of these students are exposed to each of the 2 K + 1 treatment exposures. Finally, we choose the largest K that yields sufficient sample sizes (at least 30 students) for each exposure for our KNNIM model.

To make this explicit, suppose we consider a KNNIM model with K = 2 . This sample contains N = 348 units – there are 348 seed-eligible students that interact with at least two other seed-eligible students. Moreover, there are eight treatment exposures possible for each student in this sample; in Table 3, we see that each possible exposure has at least 34 students assigned to that exposure. Hence, K = 2 seems to be an acceptable choice.

Now, suppose we restrict our analysis further to only eligible students in treated schools who have at least K = 3 seed-eligible nearest neighbors. In this case, the sample size is reduced to only 100 students. Additionally, from Table 4, we see that there are an insufficient number of units assigned to each exposure – in fact, there is only one student in the sample that for which that student and all its three seed-eligible nearest neighbors are all treated. We conclude that K = 3 yields an inappropriate model and continue our analysis using a KNNIM model with K = 2 .

9.2 Assessing indirect effects using randomization tests

We evaluate the performance of the randomization tests for the following statistics: Pearson, ELC, Score, HTN, and KNN. We choose focal units according to Algorithm 1, and treatment is re-randomized across non-focal units 1,000 times. The p -value is the proportion of the replications where the absolute value of the simulated test statistic is greater than the absolute value of the observed test statistic. Results are given in Table 5.

For this modified experiment, all randomization tests fail to detect an indirect effect. The p -value is smallest for the ELC test ( p = 0.14 ), followed by the Score test ( p = 0.22 ) and the KNN test ( p = 0.34 ).

For context, an analysis of this experiment by Aronow and Samii [9] estimated the indirect effect to be 0.154 – the probability that a non-seed student wears a wristband increases by about 15% if they have a connection with a seed student. Failure of these permutation tests to detect an indirect effect does not negate the findings of the original study. For example, from Section 8, we find that permutation tests struggle to detect indirect effects of similar sizes consistently. Additionally, this modified demonstration dramatically reduces the sample size of the original study, further decreasing the power of these tests.