Bot invasion: protecting the integrity of online surveys against spamming

Possible motivations for bot spamming in online surveys

Few researchers have reported on the impact of bot spamming [5], 6], 10], despite the significant and catastrophic effect it has on survey methods and research integrity. Moreover, there is limited evidence establishing why bots are created and deployed to spam online surveys. It is likely that bots are used to collect incentives or to purposefully sabotage the data collected [5]. A speculation in one recent report on bot activity is that bots may use artificial intelligence to target keywords or symbols, such as ‘$’ to seek incentives [8]. One hypothesis is that controversial topics may be a reason for bot sabotage attention. It is likely that survey links are identified by spammers or bots when shared publicly on social media sites [5]. Bots can be programmed to automatically fill out survey forms, often utilising scripts or specialised software [2], 5]. In certain instances, bots may be deployed to manipulate survey outcomes to favour a particular agenda or viewpoint [2], 5]. This may involve providing biased or deceitful responses to influence the survey’s outcome [5], 6].

Recognising bots in survey data

We first suspected fraudulent responses due to a considerable increase in the recruitment rate within 1 h. This alerted us to an abnormality, and we then utilised additional steps to determine if they were likely bots. We detected fraudulent survey responses using various sources including email verification, invalid postal codes for the country selected time taken to complete surveys, and rare group selections (Table 1).

Criteria for detecting bots in this study: Respondents had to meet at least one of the following: (1) an unusual/unexpected increase in recruitment rates or (2) a completion time that was under 1 min or over 10 min, and two of the following: (1) an invalid email address as evidenced by undeliverable email receipt/no response to follow up survey, (2) an invalid postcode, or (3) chose a rare group option (i.e., someone who has received a vaccine patch). A summary of responses that met the criteria is displayed in Table 2.

Recruitment rates: Daily recruitment rate was 10 or less. Any one day in which recruitment was over 10, triggered further investigation. For example, in approximately 1 h we had 350 ‘participants’ recruited.

Time taken to complete survey: Any survey that was completed in 1 min or less or over 10 minutes was flagged for further investigation. Invalid email addresses: Follow up surveys were automatically sent in REDCap to participants two weeks after their first response. Invalid email addresses would bounce back and thus could be excluded from further involvement in the study, and were investigated as potential bots at this point.

Invalid postal codes for the country selected: Given the diversity of countries and the format of postcodes, this was a free-format field. The validity of postcodes was checked relative to the country of residence selected. For example, almost all Australian postcodes consist of four numbers (one territory has three numbers), All suspected bot responses contained postcodes with five numbers.

Rare group selection: Two fields were considered for this criteria: Exposure to Vaccine patches, and Identifying as First Nations. If the response chosen was ‘someone who has received a vaccine patch’, which is currently rare as this technology is in early clinical trials, they were considered suspected bots. Likewise if selecting ‘yes’ to identifying as First Nations (e.g., Aboriginal and Torres Strait Islander, Inuit, Métis, or Māori), which in Australia for example, Aboriginal and Torres Strait Islander people represent 3.8 % of the population [15], they were considered bots.

Bot prevention using reCAPTCHA

CAPTCHA is perhaps one of the most feasible options to prevent bots in online surveys and can be embedded in surveys (i.e., REDCap and Qualtrics) [16]. Implementing CAPTCHA or other human verification mechanisms can help differentiate between human respondents and bots [17]. These systems require users to perform tasks that are easy for humans but challenging for automated bots to complete [10]. reCAPTCHA, a Google technology, confirms a user’s human identity and shields websites from automated spam and abuse. Initially conceived by von Ahn and colleagues [17], it was later acquired by Google in 2009. reCAPTCHA’s primary role is to differentiate between human users and bots through challenges designed to be easy for humans but challenging for automated scripts [17]. These challenges commonly involve deciphering distorted text, selecting specific images from a set, or completing tasks requiring human cognitive abilities [18].

As time has progressed, reCAPTCHA has advanced to enhance user-friendliness and security [16], 17]. Notably, reCAPTCHA introduced machine learning algorithms to assess user behaviour and assign a risk score, eliminating the need for traditional CAPTCHA challenges [17]. reCAPTCHA is deeply integrated into various online platforms, including websites, online forms, and mobile applications [17]. It provides a robust defence against spam, abuse, and unauthorised access attempts, thus preserving the integrity and security of online surveys [16], 17]. Despite its effectiveness, reCAPTCHA has encountered criticism and controversy, particularly regarding its accessibility for users with disabilities and concerns over user privacy from public groups [19], 20]. It may also produce false negatives (failing to detect bots) and false positives (incorrectly identifying humans as bots) [21]. Google has addressed these concerns by improving accessibility features and minimising data collection practices using a score based on suspicion to avoid challenging the user. reCAPTCHA v3 or ‘invisible’ reCAPTCHA is designed to remove the need to interrupt users by running analyses in the background to detect patterns of bot activity on the web [22]. This could help prevent the exclusion of people with accessibility issues or disabilities that make reCAPTCHA a barrier to participation, thus reducing selection bias and increasing generalisability and external validity. However, this is a delicate balance between user privacy, anonymity, and accessibility. Thus, while reCAPTCHA v3 enhances usability and inclusivity (e.g., people with visual impairment), it does so by collecting more granular user data, which may compromise privacy; in contrast, traditional CAPTCHA methods, though more intrusive, limit data collection and better preserve user anonymity.

Alternative bot prevention strategies

Data validation techniques such as IP address tracking (determining a user’s location), browser fingerprinting (a method that captures a user’s data through web browsing activity), and response consistency checks can help identify suspicious patterns indicative of bot activity [1], 23]. Employing monitoring and filtering tools capable of detecting and blocking bot-generated responses in real-time can help mitigate the impact of spam on survey data quality [5]. Limiting anonymity in surveys by requiring users to sign in with verified accounts or providing incentives tied to legitimate user profiles can deter bots seeking to exploit the anonymity of online surveys [23], but may also reduce the recruitment rate and deter potential participants, especially those from culturally and linguistically diverse populations or with low digital and data literacy.

Implications for epidemiologic research

Bots have the potential to introduce selection, confounding and measurement bias into research as well as impacting upon the generalisability of our findings. As demonstrated in our data, had we failed to exclude the bot generated responses from our data, we would have had an inflated proportion of indigenous participants, all of whom were allocated to one treatment arm. Thus, introducing selection bias and confounding. Additionally, the main results were biased showing greater harms for the treatment when bot responses were included in the analysis. Including bots would have inflated the sample size, resulting in underestimated variance and overly narrow confidence intervals.

Use of reCAPTCHA or human verification

It is highly recommended that researchers use reCAPTCHA to reduce the additional labour and the potential risk of inaccurate survey data, especially for surveys that only require a single response. Many reputable survey administration software, such as REDCap or Qualtrics, have reCAPTCHA functions that can be activated easily.

Identifying bots in collected data

Even if attempts have been made to reduce bots from completing data collection, the following strategies may help in identifying and excluding potential bot data from a dataset. First Nation’s status as suspected bots may unintentionally exclude genuine responses and contribute to the underrepresentation of Indigenous participants. However, it was important that this was considered with other bot-identifying criteria to best avoid excluding genuine responses.

Collect email addresses for all participants

Even if a cross-sectional survey is being conducted and no further follow-up is required, sending a ‘thank you for participating’ email may assist in identifying potential bots. However, some sophisticated bots may use disposable email addresses [5].

Open-ended questions

By including open-ended questions, such as “Can you explain your preference for MAP or injection” may allow researchers to determine genuine responses. These questions should be concise to avoid burdening and discouraging participants. Analysing this may require additional resources, but natural language processing tools could assist [24].

Allowing ‘free-format’ fields

Allowing ‘free-format’ fields may assist researchers in identifying potential bots that have entered invalid options. This could include having participants write in their postal code instead of restricting this field to a specific format (e.g., Australian post codes consist of four digits). This may increase time for analysis but could be managed using natural language processing.

Other strategies previously discussed may be more taxing for participants and could impact participant engagement. Artificial intelligence (AI) and machine learning advances could further sophisticate bots and possibly bypass reCAPTCHA authentication methods [23]. Researchers can leverage machine learning algorithms to analyse user behavioural patterns (i.e., response times, answer consistency, and click patterns) to identify anomalies [25]. Metadata analysis, including IP address tracking, device fingerprinting, and geolocation checks, can further help distinguish human participants from bots [26]. Natural language processing techniques can be employed to evaluate open-ended survey responses for coherence, context relevance, and linguistic complexity [27]. Future authentication may need to adapt and use AI to fight AI, as AI-based detection systems have demonstrated effectiveness in dynamically adapting to evolving bot behaviour [28].