Investigating the acoustic fidelity of vowels across remote recording methods

2.5.1 TextGrid creation for original recordings

For each reference token, a TextGrid was produced with hand-alignments at the phoneme level. Vowels were marked from the onset of regular voicing; when preceded by a liquid or approximant, the midpoint of the two phones was treated as the vowel onset.

2.5.2 Alignment of test recordings

Input for all analyses were in WAV format, either as generated by recording or, if necessary, converted to WAV using FFmpeg. To extract the individual words from the test recordings and match them with the phoneme alignment, a cross-correlation was computed using Parselmouth (Jadoul et al. 2018), a Python library interfacing with Praat (Boersma and Weenink 2021), for each pair of original and test files, using an amplitude scaling with peak of 0.99 and zero signal outside of the time domain. The time at maximum of this cross-correlation was computed using Whittaker–Shannon interpolation with 70 points, and that value was used to shift the test audio into alignment with the original.

Certain configurations, particularly ones recorded over internet connections, experienced variable lag, resulting in misalignment. To correct for these lags, after computing the preliminary phase shift, the cross-correlation method was repeatedly applied to individual words. Each re-aligned token file was verified by one researcher, and 109 tokens were removed across 34 recording conditions due to persisting poor time alignment (55 % from male recordings, 45 % from female), corresponding to 0.5 % of recorded tokens. Eleven of the tokens removed due to lag showed evidence of temporal stretching within a word. Recordings on the S8 device were removed most often, comprising 61 of the removed tokens. Thirty-two tokens were removed from the set recorded on Cleanfeed using the Edge browser on the S8. This vastly dwarfs all other conditions; in no non-S8 recording condition were more than four tokens removed. The validated tokens were matched to the corresponding token TextGrid objects, yielding phoneme alignments for all tokens.

2.5.3 Measure extraction

For each vowel, we collected the measurements described below at 10 % increments between 20 % and 80 % of the vowel duration (7 time-points). Vowels shorter than 80 ms were excluded. From the original and test recordings every 10 ms, we computed:

1. f0 (in Hz): Using Praat via the Parselmouth package. Pitch maximum was set to 300 Hz for the male speaker, and 400 Hz for the female speaker. Pitch minimum was set to the default.

2. Formants (in Hz): Also using Parselmouth, values for F1–F3 were extracted. The maximum frequency was 5,000 Hz for the male speaker, and 5,500 Hz for the female speaker, with the number of formants set to 5 for both speakers.

3. Loudness (in sones): Using OpenSMILE (Eyben et al. 2010), computed based on auditory weighting of the spectrum.

4. MFCCs (Mel-frequency cepstral coefficients): Popularized by research in automatic speech recognition, MFCCs provide an alternative view of the distribution of energy at different frequencies and can avoid the tracking errors which affect formant extraction. MFCC1 can be interpreted as capturing the difference in energy at high and low frequencies, though higher order MFCCs lack such natural interpretations. We used OpenSMILE to extract MFCCs over 25 ms windows with 10 ms steps. MFCCs are unitless.

2.5.4 Statistical analysis

Differences in acoustic measurements as compared to the original recording^[2] vis-à-vis devices, software, web browsers, and microphones were analyzed using linear mixed-effects models fit with the lmerTest package (Kuznetsova et al. 2017) for R (R Core Team 2022). R formulas are specified under each table or figure for which models were fit.

Loudness and fundamental frequency (f0) modeling were conducted using measures for all vowels, and models for the first two formants were fit separately for each vowel. Separate models were fit for each of the two speakers. All measures were averaged across each vowel instance.^[3] All models except for those comparing hardware also included a random effect for device. When reporting significance, * refers to p < 0.05, ** to p < 0.01, and *** to p < 0.001.

3.1.1 F1 and F2 heatmaps

We present comparisons for F1 and F2 obtained through the linear mixed-effects models across phonemes and contrastive recording conditions as heatmaps, using the Seaborn Python package (Waskom 2021). The color saturation of each cell corresponds to the estimated effect size relative to the original recording. The divergent series colors reflect positive or negative estimated effect sizes. A short vertical dash in the cell indicates that the difference depicted did not reach statistical significance (p < 0.05).

3.1.2 MFCC heatmaps

Results for MFCC1 and MFCC2 are displayed in heatmaps (as described for F1 and F2) and discussed for the hardware comparison. MFCC plots for the remaining comparisons are available in the Supplementary Material.

3.1.3 Vowel plots

F1 and F2 measures were computed as the average value for the respective measure between the 20 % and 80 % points. Entries with missing data were removed. The data was trimmed at the upper and lower quartiles (for the purposes of plotting and Pillai scores below only). Plots were created using the dplyr (Wickham et al. 2022) and phonR (McCloy 2012) packages for R. Two types of plots were created for each speaker, for each comparison: (1) distribution plots of F1 × F2 of each instance of each vowel by condition; and (2) vowel space plots of the average F1 × F2 for each vowel by condition. For readability, /æ/ was excluded from the distribution plots. For all plots, the IPA symbol of the vowel is plotted at the mean F1 × F2 for that vowel.

3.1.4 Pillai statistic

The Pillai score is a test statistic produced by a MANOVA. Pillai scores and statistical significance based on p values are reported for F1 × F2 of each condition compared to the F1 × F2 of the original recording, with one table for each comparison from the same trimmed data used to generate F1 × F2 plots. The p value associated with the Pillai–Bartlett trace from the MANOVA indicates whether or not each vowel distribution is statistically significantly differentiated from the original recording by F1 and/or F2 values (Hay et al. 2006). The Pillai–Bartlett trace also produces a statistic ranging from 0, indicating complete separation of the vowel in each condition, to 1, indicating complete overlap.

3.2.1 Loudness

All hardware configurations resulted in lower loudness from the original, and all differences were statistically significant (Table 3). Furthermore, similar patterns of reduced loudness are found for all software, browser, and microphone comparisons, and will not appear in subsequent sections but are available in the Supplementary Material.

3.2.2 f0

For the female speaker, across all devices, there were no statistically significant differences from the original (Table 4). For the male speaker, there were statistically significant decreases in the mean f0 from the original for two devices, the S8 by 2.36 Hz and the LenP50 by 1.48 Hz. For all other devices, differences did not reach significance.

3.2.3 Differences in F1 and F2 and MFCCs from original

As depicted in the estimated effect sizes presented in the heatmaps in Figure 2, we can observe significant effects of hardware configuration for both speakers across all vowels for F1 and all vowels except /ɪ/ for the male speaker for F2. Both increases and decreases relative to the original recordings are seen. Effect sizes are larger for F2 than for F1 and for the female speaker than for the male speaker, with extreme negative values rather than positive ones for the female for F1. However, few systematic patterns are readily discernible for specific hardware configurations.

Figure 2:

Heatmap of difference in F1 and F2 from the original (in Hz), across devices. Short vertical dashes indicate that the differences are not significant for that phoneme. Calls: lmer (F1∼device + (1|word), data = device.speaker.vowel, REML = FALSE).

If we consider the heatmaps for the MFCC1 and MFCC2 estimated effect sizes (Figure 3), we again see near universal significant differences across the different hardware configurations for all vowels and both speakers. In contrast to the varied pattern observed for F1 and F2 differences, we observe relatively consistent positive effects for MFCC1 and consistent negative effects for MFCC2. We also note more reliable influences of specific hardware configurations across vowels and speakers, with smaller, though still significant, magnitudes for the S8 and LenT490. Interestingly, these relatively salient patterns capturing shifts in spectral energy distribution based on hardware configuration are obscured in observed formant values, likely due to interactions between these spectral differences and formant tracking processes.

Figure 3:

Heatmap of difference in MFCC1 and MFCC2 from the original. Short vertical dashes indicate that the differences are not significant for that phoneme. Calls: lmer (mfcc1∼device + (1|word), data = device.speaker.vowel, REML = FALSE).

3.2.4 Distribution of F1 × F2 and vowel space

For the male speaker (Figure 4, top left), distribution of individual instances of each vowel largely overlapped between each of the hardware settings and the original, although visual inspection indicates greater deviations for the LenT490 for /u/, /ɑ/, and /ɛ/, as well as /i/ and /ɛ/ for the iPhone, /ɛ/ for the MBA13, and /ɑ/ for the MBA19. The female speaker’s vowels (Figure 4, top right) had wider and more varied individual distributions across hardware. Distributions varied by vowel, with the largest differences in both F1 and F2 observed for /i/, /ɪ/, and /ɛ/ and in F1 for /u/ and /ɑ/.

Figure 4:

Distribution plots of each vowel instance for male (top left) and female speakers (top right) and vowel space plots for male (bottom left) and female speakers (bottom right).

For the male speaker, the vowel spaces for the different hardware varied (Figure 4, bottom left). Overall the S8 has the closest vowel space to the original, and the iPhone and LenT490 had greatest differences in F1 × F2 from the original (as illustrated by Pillai scores in Table 5).

The female speaker’s vowel spaces varied between hardware with larger differences in F1 × F2 than the male’s (Figure 4, bottom right). Overall, for the female speaker, the iPhone, MBA13, and MBA19 were most different in F1 × F2 from the original (Table 5).

3.3.1 f0

For the female speaker, both microphones yield differences in f0 of less than 7 Hz from the original 229 Hz; neither is significant. For the male speaker, neither the internal microphone average f0 nor the external microphone average f0 differs from the original more than 2 Hz. Neither of these differences is statistically significant (Table 6). However, manual inspection of data points indicates pitch tracking failures for a number of recordings for this speaker due to low-frequency noise when using the external microphone with Mac laptops.

3.3.2 Differences in F1 and F2 from original

Overall, the magnitude of differences in F1 and F2 from the original were similar for external and internal microphones (Figure 5). For F1, for both speakers, differences between the external microphone and the original are largely not significant. For F2, for both microphone setups, the female speaker generally has larger differences from the original than the male speaker.

Figure 5:

Heatmap of difference in F1 and F2 from the original (in Hz), across microphones. Short vertical dashes indicate that a comparison is not statistically significant. Measures for male /u/ are excluded due to frequent pitch tracking failures impacting formant tracking with the external microphone and Mac laptops for this vowel. Calls: lmer (F1∼mic + (1|device) + (1|word), data = mic.speaker.vowel, REML = FALSE).

3.3.3 Distribution of F1 × F2 and vowel space

Comparisons of vowel distributions and vowel space for the internal and external microphones appear in Figure 6 with Pillai scores in Table 7. For the male speaker, the distribution of individual vowel instances for the external microphone for /u/ differs greatly from both the internal microphone and original setups, particularly at F2, such that it overlaps with the distribution of /i/. The external microphone setup also results in slightly wider distributions for /i/, /ɛ/, and /ɑ/. The distribution with the internal microphone setup more closely matches the overall distribution of the original. For the male speaker, the vowel space with the internal microphone setup more closely matches that of the original overall, with largest differences in F1 × F2 for /ɛ/ and /u/ (Figure 6 and Table 7). While mitigated here by quartile filtering, the measures for /u/ may be affected by the presence of low-frequency noise in the case of external microphones on Mac laptops that led to the pitch tracking issues noted above for the male speaker.

Figure 6:

Distribution plots of each vowel instance for male (top left) and female speakers (top right) and vowel space plots for male (bottom left) and female speakers (bottom right).

For the female speaker, vowel distributions for the external and internal microphones pattern together except for smaller and shifted distributions for the internal microphone for /i/ and /u/ (Figure 6). Vowel spaces for this speaker are similar between the two microphone setups, with both differing from the original (Figure 6). Average F1 × F2 differs most from the original with the internal microphone for /ɪ/ and /æ/ and with the external microphone for /i/ and /u/ (Table 7).

4.2.1 Loudness

A statistically significant decrease in loudness was observed for all hardware, software, microphone, and browser configurations. These differences varied with configuration. Loudness can be adjusted post-recording and would not raise issues for most analyses; however, conducting a study in which loudness was a measure of interest, for example for stress or prominence, would benefit from relative measures of loudness or other forms of normalization to support comparisons across speakers.

4.2.2 f0

Overall, with suitable pitch range constraints and where pitch tracking was successful, both speakers’ f0 measurements remained faithful to the original regardless of setup, with differences well below the just-noticeable difference for pitch discrimination for falling pitch contours in speech (Turner et al. 2019). However, we do observe some systematic pitch tracking failures, associated with the external microphone on Mac laptop hardware. These recordings are marred by strong low-frequency noise, possibly due to hardware compatibility issues, which interferes with pitch tracking for the lower-pitched speaker, but not for the higher-pitched voice, highlighting differential impacts of the same acoustic distortion.

4.2.3 F1 × F2 vowel space and distribution

Vowel space distributions showed clear inconsistencies in formant tracking. This is particularly evident with the external microphone for the male speaker. Plotting F1 × F2 with both the distributions and the vowel spaces allowed for a clearer picture of how these measures were affected.

4.2.4 Differences between speakers

Different outcomes by configuration were observed for the male and the female speaker for all measurement types. Due to the small sample size of speakers, conclusions cannot be drawn as to whether differences were correlated with speaker sex or other aspects of their speech. Based on these findings, however, there is a strong argument for documenting recording conditions in as much detail as possible and employing speaker normalization methods when performing online studies with cross-speaker comparisons. Additionally, future study with multiple male and female speakers would be helpful to better determine the cause of differences and aid in normalization approaches.