Measuring Latent Constructs in Nonprofit Surveys with Item Response Theory: The Example of Political Ideology

A New Way to Measure Latent Constructs in Nonprofit Surveys: The Example of Political Ideology

Latent constructs are the unobservable characteristics of individuals, groups or organizations, such as public opinion, ideology, preferences and values. Although researchers use many methods to measure latent constructs, including scaled-items and factor analysis techniques, this study offers nonprofit scholars one approach to measure these characteristics – Item Response Theory (IRT) – which provides a powerful methodological tool for understanding latent constructs and the resulting effects of these characteristics on individual or organizational behavior. This is particularly crucial in studies of the nonprofit sector, as a significant amount of the data generated for study relies upon survey and interview instruments (Lin and Van Ryzin 2012). Item Response Theory has been used successfully to measure latent constructs in other fields as disparate as psychology, education, political science and public administration, and can easily be applied to studies of the nonprofit sector and nonprofit organizations.

This study demonstrates a way to measure one latent construct which may be of interest to nonprofit and public sector scholars – political ideology – and provides a demonstration of the method. Political ideology has been found to have broad-reaching implications for public policy formation, advocacy, strategic decision making and organizational behavior. Using answers to a statewide survey of nonprofit executives in California, the revealed political ideology of leaders is estimated using a technique developed by Clinton, Jackman, and Rivers (2004) and Martin and Quinn (2002). To increase confidence in the measure, several consistency checks are used. This article then follows with a discussion of other applications of IRT and how the technique may be applied to other studies. Importantly, IRT is effective in measuring constructs at both the individual and aggregate (organization) levels, opening up new lines of inquiry.

Political Ideology

Studying the role of political ideology in driving political and policy outcomes has an established tradition in political science (Cox and McCubbins 1986; McCubbins, Noll, and Weingast 1987; Krehbiel 1996; Downs 1957; Epstein and O’Halloran 1999; Ferejohn and Shipan 1990; Gailmard 2009; Truman 1951; Moe 1988, 1991, 1987). It is also of interest to those who study “interest groups” (often nonprofit organizations) (Poole and Rosenthal 2007; Bonica 2013; Mason 2015; Ainsworth 1993; Ainsworth and Sened 1993; McKay 2010), as well as in public management scholarship (Clinton et al. 2012; Clinton and Lewis 2008; Bertelli and Grose 2009; Clinton, Lewis, and Selin 2014; Chen and Johnson 2015). Scholars have long been interested in the interplay between politics and political actors, institutions and ideology. However, political ideology – outside of a self-reported party affiliation – is a complex latent construct that is difficult to measure. Political ideology “helps to interpret the social world … Specific ideologies crystallize and communicate the widely (but not unanimously) shared beliefs, opinions, and values of an identifiable group, class constituency or society” (Jost, Federico, and Napier 2009, 309). Most scholars and public opinion pollsters tend to measure political ideology with a question related to party affiliation, such as Democrat, Republican or Independent, or asking where respondents fall on a unidimensional left-right political spectrum using a three, five or seven point scale (i. e. from extremely conservative to extremely liberal).

This conceptualization of ideology is consistent with the spatial theory of voting developed by Hotelling (1929) and Downs (1957). Downs, specifically, described the ways in which voters were aligned along a left-right continuum, describing how voters cast their votes for the candidate that is closest to them on that spectrum. The position of the voter on that continuum is called their ideal point, as is the position of the candidate. The ideal point, therefore, represents utility for the voter. The closer the candidate is to the voter, the more utility they receive in casting their vote for that candidate. In the competition for votes, candidates can be seen as converging on the median voter in order to secure a majority and win the election (Bertelli 2012; Poole 2005; Downs 1957) This intuition has also been used to explain voting in legislative bodies as well, where individual members of legislatures tend to converge on the median voter in crafting, and passing legislation (Downs 1957; Shepsle 1986; Poole and Rosenthal 1985; Krehbiel 2010, 1996; Cox and McCubbins 2007).

Although it has been shown that political ideology can be measured on multiple dimensions (Feldman and Johnston 2014; Layman and Carsey 2002; Zumbrunnen and Gangl 2008), this study makes use of the most common conceptualization, which is a single dimension along a left-right continuum. The left end of the spectrum is most often defined as the liberal space and is often associated with advocating social change and rejecting inequality, and the right side is associated with conservatism, characterized with rejecting social change and accepting inequality (Jost et al. 2003b, 2003a). Alford, Funk, and Hibbing (2005) developed a separate spectrum bookended by two individual-level “phenotypes”. On one end are the “contextualists” who are more aligned with liberal politics, are associated with a higher tolerance of out-groups, an optimistic view of human nature generally, an opposition to hierarchy and authority, show higher levels of empathy and a reduced demand for “punitiveness”. On the other end are “absolutists” more aligned with the normative view of conservatism, shows support for rigid moral rules, higher levels of acceptance of inequality, in-group unity and higher expectations for punitiveness.

Jennings (1992) finds that the ideology of political elites was more stable with a constrained and unidimensional set of ideological beliefs compared to the general public, and Layman and Carsey (2002) also argue that political elites tend to align themselves along a single dimension. In a study of voters from 1972–2004, Jost (2006) finds that self-reported ideology aligned along a single left-right spectrum was a strong predictor of voting. Indeed, it has been recognized that political elites, including elected and government officials, activist groups, the media and academia, frequently use the single left-right dimension of ideology in framing and debating issues (Jost, Federico, and Napier 2009; McCarty, Poole, and Rosenthal 2005).

Measuring Political Ideology

In order to measure the revealed political ideology of political actors, a growing number of scholars have turned to item response theory (IRT) (Van Der Linden and Hambleton 1997). Initially developed as a collection of tools to measure latent constructs in both educational testing and psychology (Van Der Linden and Hambleton 1997; Baker 2001; Lord and Novick 1968; Guttman 1945), IRT offers some benefits compared to factor analysis models. First, IRT modeling is based on the relationship between the individual (item) response and the individual values of the latent traits. In other words, it measures the probability that a respondent will answer a particular item in a certain way. As Treier and Jackman (2002, 2) write, “IRT models provide the answers to the questions that most social scientists are asking when they engage in measurement: what do the data say about both scores on the latent constructs and the properties of [each of] the items?”. IRT also, unlike factor analysis, can handle a mix of data and mixed data models – including binary, ordinal and continuous, allowing for more flexibility in research design and analysis (Quinn 2004).

IRT models thus provide a distinct value of an individual/item on a continuous latent trait, or their ideal point. In other words, in measuring a latent trait, in this case revealed political ideology, IRT can distinguish very slight differences between individuals, groups or organizations. Poole and Rosenthal (1997, 2007) were among the first to demonstrate measures of political ideology using IRT that went beyond party affiliation, when they generated estimates of ideology for members of Congress using a large number of floor votes. The ideology of bureaucratic agencies and federal secretaries have also been measured (Clinton et al. 2012; Bertelli and Grose 2009; Chen and Johnson 2015), while others have used similar techniques to measure the political ideology of state legislatures (Shor and McCarty 2011), local elected officials (Connolly and Mason 2016), candidates for public office (Bonica 2013), and voters (Bafumi and Herron 2010). In the public sector, scholars have also sought to measure the political ideology of large public sector agencies (Clinton et al. 2012; Bertelli and Grose 2009; Clinton and Lewis 2008; Chen and Johnson 2015), and others have attempted to measure the political ideology of “interest groups” (often nonprofit organizations) (Mason 2015; McKay 2008, 2010; Bonica 2013; Barber 2015; Poole and Rosenthal 2007).

Data and Methods

Using a Bayesian method developed by Clinton, Jackman, and Rivers (2004) and an estimation technique developed by Martin and Quinn (2002) and Martin, Quinn, and Park (2011), the political ideology of leaders in nonprofit advocacy organizations is estimated in such a way as to compare them directly with other actors – in this case, California state legislators, although members of other legislatures, including state legislatures, city Councils or Congress can be used. This study uses the MCMCpack for R statistical software to estimate leader ideology, although other methods are available. One such alternative to is to use the technique used by Poole and Rosenthal (2007, 1997) and used by others such as McKay (2008, 2010), which uses DW-Nominate a stand-alone statistical program. Stata 14 now also includes IRT models for use by scholars.

To assemble the data for analysis, first the roll call votes of eleven votes in the California legislature were selected. These votes, taken on the floor of the 2012 California legislature, comprised of six Senate bills and five Assembly bills. The full list of the bills selected can be seen in Table 1. Each bill was voted on in both houses of the legislature, and the third (substantive) reading and vote was used for analysis. These measures were selected because they had a clear partisan split in both houses of the legislature. In other words, they were largely party-line votes with Democrats on one side and Republicans on the other. They also were seen as having high political salience outside the legislature, where other political elites (nonprofit leaders) and even average California voters, are assumed to have a position on each of the issues. For example, one bill chosen was a measure on extending welfare payments to unwed teenage mothers. Another was whether or not to fund California’s controversial high-speed rail project.

Each item (or survey question) has both difficulty and discrimination parameters on an item characteristic curve (Baker 2001). Originating in educational testing, a “difficult” question will function well among those more able. Here, it can also be conceptualized as the item’s “location on the latent trait” (Rusch et al. 2016). Discrimination, on the other hand, is how well the each item differentiates between conservative and liberal ideologies, and the steeper the curve, the better the item is able to discriminate between a liberal voter and a conservative voter (Baker 2001). The parameters estimated are the ideal points themselves, or their revealed political ideology, including a difficulty (the intercept) α_j and the discrimination (slope) where the legislator i votes on the roll-call j. Roll call votes were coded 1 for “Yea” and 0 for “Nay”. The discrimination parameter β_j captures the weight of change in the probability of voting yea on a roll call as the member’s ideal point θ_i moves from liberal to conservative. Using the two-parameter, one dimensional estimation technique developed by Martin, Quinn, and Park (2011) and Clinton, Jackman, and Rivers (2004), ideology is modeled as the unobserved utility of voting “yea” on any particular roll call vote:

To help identify the model, Republicans and Democrats that consistently voted along party lines were selected in each chamber. These members’ ideal points were then constrained to −1 (=very liberal) and 1 (=very conservative). This is a common strategy in the literature (see Poole and Rosenthal 1997; Clinton, Jackman, and Rivers 2004; Martin and Quinn 2002). In the Assembly, Jim Silva (R- Huntington Beach) and Tom Ammiano (D-San Francisco) were fixed at 1 and −1, respectively. The Senators selected were Joel Anderson (R-San Diego) and Juan Vargas (D-Chula Vista), outside San Diego. The parameter estimates for the roll call votes can be seen in Table 2. The large negative values indicate that the votes show discrimination between liberal and conservative members of the legislature, and show the expected negative value because each house is made up of a majority of Democrats where a “yea” vote was the liberal choice. Estimates in each chamber are based on a thinned chain of every 20th observation from 500,000 iterations of the Gibbs sampler, with 5000 draws excluded as burnin – a total of 25,000 draws for each chamber. The models displayed substantial evidence of convergence based on trace, density, and auto-correlation plots. In both chambers, less than 10 percent of Geweke (1992) and Heidelberg and Welch (1983) diagnostic statistics failed to suggest convergence.

The difficulty parameter identifies the location of the proposal on the left-right scale among the legislators. As proposals move farther away from the legislator’s ideology, they represent less utility for the legislator if he or she was to vote “yea”. Therefore, a “nay” vote can also be considered a vote for the status quo. The mean value for the California Assembly was −0.94 (Democrats −0.95, Republicans 1.37). For the California Senate, the mean value was −1.04 (Democrats it was −1.05, Republicans 1.37). Overall, the full legislator mean value was −0.15 (Democrats −0.91, Republicans 1.19). With means to the left of zero, these values demonstrate the left (or liberal) leanings of the California legislature as well as its polarized, bimodal distribution.

Findings and Consistency Checks

The process described above provided ideal point estimates for the legislature and then the legislature combined with nonprofit leaders. While the polarization of the legislators was apparent, showing the large difference between Democrats and Republicans in both houses, the nonprofits leaders are considerably more moderate, with a mean value of respondents following this procedure is 0.121, and the median is 0.11 (σ=0.60), indicating that they were slightly to the right of zero, and moderate compared to the mean and median voter in both houses of the California legislature (Senate mean −0.259, σ=1.23 of the merged sample; Assembly mean −0.295 σ=1.24). This is shown graphically in Figure 1, where the bimodal distribution of the legislature can be compared to the more normally distributed ideal points of the nonprofit leaders.

Figure 1:

Comparison of the ideology of senate, assembly and nonprofit leaders. Political ideology is measured on a left-right continuum. As an individual moves further to the right, they can be considered more conservative (N=217 nonprofit leaders, 80 California Assembly, and 40 California Senate).

Considering these were leaders of 501(c)(4) organizations, this might be considered surprising. There has long been a normative expectation that “interest groups” are extremists compared to the legislature, and apply pressure to legislators to vote on issues that are outside their own personal preferences. This was supported by Poole and Rosenthal’s (1997, 2007) early work with Item Response Theory, comparing the voting records of elected officials and the “votes” of interest groups in Washington, D.C. assembled with the use of organization scorecards (documents prepared by groups showing the voting records of legislators that support and oppose the organization’s official positions, providing a proxy for the organization’s positions on a range of issues). Their findings suggest that interest group ideology is more extreme than that of the members of Congress. However, more recent research has started to cast doubt on this assumption. Bonica (2013) demonstrated that Political Action Committees (PACs), those that provide funding for candidates running for office, are more ideologically moderate than Congress. Bafumi and Herron (2010) argued that voters are more ideologically moderate than those they’ve voted to send to Congress. Barber (2015) finds mixed results showing that PACs tend to be more moderate with their giving, while individual donors tend to support more extremist candidates for office. He speculates that PACs are more concerned about working both sides of the aisle to gain access to those that are ultimately elected. Since many 501(c)(4) groups are engaged in advocacy, it is likely that these nonprofit organizations are, like PACs, interested in access over partisan ideology. In fact, the data also show that nonprofit leaders are statistically similar to the majority of the state legislators, where 61 % of the nonprofit leaders were statistically similar to the median member of both the Senate and Assembly. In other words, despite the polarized distribution of California state legislators, nonprofit leaders tend to straddle both extremes in the center of the distribution.

Additional data on the leader’s organization, provided by the National Center for Charitable Statistics (NCCS) also helps to differentiate groups by their mission statement. For example, it was easy to compare the ideology of arts and culture organizations with animal rights groups. Figure 2 shows the mean ideology of the leaders by their organization’s mission, and does show some variance among leaders of different types of organizations. While this variance is interesting, and does provide questions for future research, this variable was primarily used for control and descriptive purposes.

Figure 2:

Leader ideology by organizational issue field (N=208).

Since the wording in the survey was a simpler description of the complex legislation considered by the state legislature (to reveal the “real meaning” of the legislation, as well as reduce the noise from complex legislative language), it’s possible to argue that the survey responses and roll call votes are not comparable. In an effort to check for the consistency of the IRT-generated estimates of revealed political ideology, four tests were conducted. First, an exploratory factor analysis was conducted to see whether the items hold together as one “factor”, although Ip (2010) demonstrates that unidimensional IRT models are largely robust to modestly multidimensional constructs. Factor loadings demonstrate all eleven items hold well together as a single factor (ideology) with a resulting Cronbach Alpha of 0.90. While it’s possible more than one underlying latent factor could be present, all these items hold together as one single dimension of ideology as used in this illustration. This was confirmed by an additional multidimensionality check in MCMCPack in R. Second, the survey items by nonprofit leaders were analyzed separately, independent to the roll call votes, constraining the scores based on one respondent who was reliably “liberal” (-1) and one that was reliably “conservative” (1). The mean ideal point for all respondents – independent of legislators – was 0.121. This test indicated that all questions continued to discriminate between liberal and conservative respondents (all items discriminated in the expected “liberal” direction), where a “Yes” on any given item indicated a liberal orientation.

Finally, two additional tests were conducted similar to those provided by Bafumi and Herron (2010) in their study comparing survey questions administered online to voters to roll call votes in the U.S. Congress. In the first test, the IRT-generated estimates of political ideology were correlated to a question on self-reported political ideology. The question, “When it comes to politics do you usually think of yourself as liberal or conservative,” offered eight choices, seven of which were “extremely liberal” to “extremely conservative” on a seven-point sale with an “I don’t know” option (with zero responses). The IRT estimate for ideology correlated with this self-reported liberal to conservative model at 0.63. In addition, a series of bivariate logistic regressions were conducted, with the survey response only on each survey item as the dependent variable and the IRT-generated ideology score as the independent variable. Each item was statistically significant at p<0.01. Table 4 provides the intercepts (constants) and slopes (coefficients) for each item along with the observations for each question. The number of observations changes slightly for each item due to skipped questions. The steep slope for nearly all items indicates that there is strong discrimination between liberal and conservative respondents. The negative values indicates that, for each question where the choice was a “yes”, the more conservative the respondent was, the less likely they were to select “yes”.

The survey also asked, “Do you usually think of yourself as a Republican, a Democrat, an Independent, or other”. As Figure 3 demonstrates graphically, Democrats are left of Independents, which are left of Republicans. Ultimately, these tests provide evidence to indicate that the method demonstrated here does seem to tap into an underlying latent construct.

Figure 3:

Political ideology of nonprofit leaders by self-reported party. As respondents move right, they can be considered more conservative (N=217).

Using Item Response Theory for Future Research in the Nonprofit Sector

While this paper has demonstrated one way to measure political ideology among social welfare organizations, there are many other applications of IRT that can be applied to research on nonprofit organizations and the policy arena. For one, while this specific application measured a latent characteristic at the individual level, it can also be used to measure characteristics at the group level by aggregating survey or observational data. For example, the political ideology of federal employees has been measured at the agency level (Clinton et al. 2012), and voters have been aggregated by district (Bafumi and Herron 2010) and compared with their members of Congress. These studies therefore provide a measure of a group’s or population’s overall ideology – a much more powerful tool than relying simply on public statements or voter scorecards (the method most often used for interest groups), election results or exit polling. It would be possible, for example, to measure the political ideology of a nonprofit organization in this fashion, by surveying employees or the board (or comparing the ideology of the two groups), or among organizational volunteers or donors. Are donors or volunteers more liberal than board members? What are the implications of that as the organization manages its stakeholder demands? For those organizations that are engaged in advocacy activities – it is also possible to envision a host of research questions of interest to nonprofit scholars. How does the ideology of the organization relate to its activities or mission statement? Commitment to diversity? How is the ideology of an organization’s leaders affect organizational outcomes or effectiveness? To innovation? How does the ideology of the organization, or leadership, affect strategic decisions regarding policy advocacy and implementation? These questions, too, can have far reaching implications for a nonprofit’s role in the wider political environment. One early study by Mason (2015) found that there were differences in advocacy tactics based on the political ideology of the organization’s leaders, but much more can be done.

Item Response Theory has also been used to measure other latent characteristics that are expected to have important relationships with both individual and organizational behavior. Bertelli et al. (2015), for example, measured job satisfaction, intrinsic motivation and autonomy among employees of federal agencies over time through the use of the federal surveys. Bertelli also separately studied bureaucratic turnover intention (2007) and motivation (2006). While most of these previous studies used unidimensional estimates of ideology or other characteristics, IRT can also be used to explore data in other ways, including assessing the item-related parameters themselves, the multidimensionality of various constructs and their relationship to other variables of interest. For example, multiple dimensions of volunteer motivations have already been explored (Penner 2002; Clary and Snyder 1999), and it might be possible to fine tune these measures with item response theory. Pinard’s (2011) study on the theories of social movements and social movement activists also provides an opportunity to measure the drivers of activism and advocacy. In additional exploration of political ideology, some scholars have explored ideology as a multi-dimensional construct (Feldman and Johnston 2014), particularly among the voting public (Layman and Carsey 2002). In a study of conservative voters, Zumbrunnen and Gangl (2008) found that conservatism was defined by two separate dimensions, cultural and economic. Cultural conservativism is identified as identifying strongly with “traditional values” and religious beliefs, while economic conservativism is related to both a free-market orientation and limited government. They found that cultural conservativism was the primary driver in the voter’s self-identified political ideology. Recently, Feldman and Johnston (2014) also showed how political ideology is best understood along two different dimensions, economic and social ideology.

Conclusion

Item Response Theory can be used to measure latent constructs of interest to nonprofit and public policy scholars. In this study, one technique was demonstrated to measure the revealed political ideology of nonprofit leaders, which may be of interest to those who are interested in understanding how the ideology of leaders, elected officials or organizations may affect policy formation, policy implementation or other behaviors. However, this paper also has some limitations. First, the response rate for the survey was lower than desired, and selection bias cannot be ruled out based on the response rate and the fact that some respondents seemed uncomfortable with the political nature of the survey questions. Readers should take this into consideration. Second, it’s important to note that this study compares the political ideology of elected officials with nonprofit leaders only on the eleven selected issues that were voted on in the 2012 California legislature, similar to a model used by Connolly and Mason (2016) and not inconsistent with that used by McKay (2008, 2010). While the approach presented here is simple to operationalize (which might be welcomed by scholars), there are other approaches to estimating the revealed ideology of legislators, such as including all roll call votes cast in a single year or over multiple years. Finally, the makeup of the California legislature changes every two years, issues gain and lose political salience and turnover in nonprofit organizations is not insignificant. It is important to replicate this design among other types of organizations, engaged in different types of activities, in other locations and at other points in time.

While this study demonstrated one way to measure for political ideology, and provided three tests for consistency, survey or observation instruments should continue to be designed, and tested, in ways that limit systematic bias and support both internal and external validity. Ultimately, IRT can be added to the toolbox of nonprofit scholars who are seeking to understand the way that unobservable characteristics of different populations of interest impact organizational behavior.