Sound event detection by intermittency ratio criterium and source classification by deep learning techniques

2.1 Proposed methodology

The proposed methodology is outlined in Figure 1. The IR criterium used for sound event detection has been applied on the time history of the SPL data. Thus, the first step requires converting the audio recording file into a time series of SPL values with a preset time resolution. Then, the criterium of sound event detection selects all the SPL values exceeding the preset threshold (Equation (2)). Afterwards, a deep CNN would provide the probability of a source type associated to the event, considering that multiple events can overlap in an urban acoustic environment.

Figure 1

Outline of the methodology proposed to detect and classify the sound events.

2.2 Dataset used

The dataset used in this article is the one described by Vidaña-Vila [15], composed of audio recordings taken in the city center of Barcelona during two campaigns. The first one took place in Autumn 2020 at noon, when there were some mobility restrictions due to COVID-19 pandemic; the second one was performed in Spring 2021 during the afternoon, when the mobility restrictions were softened. The dataset contains a rich variety of acoustic data, as the soundscape was slightly different due to the effect of: i) the COVID-19 restrictions, ii) the season of the year when data was recorded, and iii) the hour of recording.

Four sensors were deployed at four different spots of a street intersection, the crossroad between Villarroel Street and Diputació Street (Figure 2) located in the Eixample area of Barcelona. The rationale behind the decision of locating the four sensors on a crossroad was that the four sensors would capture similar loud events (an acoustic event that is loud enough should be recorded in the four spots) while the background sounds would be different in the four spots, enabling us to study the benefit of placing more than one sensor on a street in terms of increasing the accuracy at the time of performing acoustic event classification or not [15]. For this work, this physical redundancy of sensors (i.e., more than one sensor capturing the same acoustic event from a different perspective) is not relevant, as it is not the topic under study. Therefore, using data from the four sensors will just enable us to have a larger and richer variety of data, as different events were occurring in the background of the four locations.

Figure 2

Crossroad in the Eixample area of Barcelona and sound recording spots selected for data collection (edited from OpenStreetMaps).

The hardware used to collect the audio data was a Zoom H5 recorder mounted on a tripod. The device was powered with two Alkaline batteries, enabling to have an autonomy of a few hours, which was enough for the duration of the recording campaign. The microphone used was the one provided by the same Zoom recorder: the XYH-5 microphone capsule with a windscreen [16]. This microphone capsule is composed of two condenser and unidirectional microphones paired at 90° angle, enabling stereo recordings. Therefore, the recordings obtained in the campaign are stereo. The sampling rate of the recordings was 44,100 Hz bit 16-bit depth.

As shown in Figure 3, the recorder was placed at an angle of 45° from the floor and pointing to the street.

Figure 3

Example of a recorder used mounted over a tripod.

Putting together the acoustic data from both campaigns in all the four recording spots, the dataset has a duration of 5 h in each sensing node, resulting in an aggregated amount of data of 20 h. More details on measurement and recording setup are reported by Vidaña-Vila [15].

In this work, data coming from the different sensors were processed individually, differently to the usage of the dataset in the study by Vidaña-Vila [15]. For the experimental evaluation, 4 h from each sensor (together with data coming from external sources) were used to train the DL model, whereas the remaining hour was used for testing. The detection of sound events was carried out only for the testing hour. Even though the recordings are in stereo mode, only the left channel of them was used, turning them into mono audio files. The decision of conserving just one channel was for simplicity purposes, making it easier to label the events using only one channel. The sounds contained in this dataset are the ones shown in Table 1.

Two complimentary datasets were used for training the automatic classifier CNN, namely the BCNDataset [17] and the UrbanSound8K dataset [18]. On the one hand, the BCNDataset was selected as it contains acoustic data from the same city like the evaluation dataset (the Eixample dataset), and on the other hand, UrbanSound8K dataset was used because all the sounds that are contained there can be heard very clearly, and can be considered foreground sounds.

The events present in the BCNDataset are similar to the ones present in the Eixample dataset. Actually, some labels are the same, such as rtn, sire, horn, brak, troll, peop or musi. Then, there are categories that are slightly different in both datasets but have a similar meaning in terms of sound. For those cases, the labels were unified to match the taxonomy of the Eixample dataset. These specific labels can be checked on Table 2. Those categories that did not contain acoustic events that could match the ones on the Eixample dataset, specifically the busd category, were discarded and tagged with a ‘–’.

For the UrbanSound8k dataset, the labels were unified as shown in Table 3. Those categories that did not contain acoustic events that could match the ones on the Eixample dataset were discarded and tagged with a ‘–’.

As it can be observed, adding the UrbanSound8K dataset and BCNDataset to the training set helps mitigating the class imbalance of data in some categories (such as car horns, sirens or street music). However, there are some categories that still have a few amount of samples such as bell sounds (with only 51 samples), beep sounds (with 31 samples), bike sounds (with 75 samples) or wind sounds (with 31 samples). The lack of data in this categories can be therefore mitigated by applying data augmentation techniques such as mix-up augmentation.

2.3 Sound event detection

The digital audio recordings in the “.wav” format have been processed to get the A-weighted SPL time history with Fast (F) weighting according to the IEC 61672-1 standard (exponential integration time τ = 125 ms, that is 8 SPL values per second, 28,800 SPL values in 1 h of recording). An example of such processing is shown in Figure 4, where the blue, red and green dotted lines report the equivalent continuous level L _Aeq and the percentile levels L _A10 and L _A95 hourly values, respectively. This plot is among the output of a script developed in the R environment [19] to process the L _AF time history in order to detect the sound events and to determine the acoustic descriptors given in Table 4.

Figure 4

Example of the L _AF hourly time history obtained from digital audio recording processing (spot 4). Blue, red and green dotted lines correspond to the equivalent continuous level (L _Aeq) and the percentile levels L _A10 and L _A95 hourly values, respectively.

For the sound event detection task, the criterium used in the metric termed IR has been applied [4]. In particular, the IR value [20], in percentage, is calculated as the ratio of the event-based sound energy L _Aeq,T,events to the overall sound energy L _Aeq,T , both referred to the time T:

The time T can be 1 h, the day and/or night period, 24 h and so forth. A sound event is detected whenever its A-weighted SPLs exceed the preset threshold K:

where the constant C is to be set. On the basis of practical experience on transportation noise situations, the Authors proposing IR suggest that C might not be smaller than 0 and not larger than about 10 dB [4]. For low values of C, almost any situation produces a large IR value, whereas high values of C almost always produce low IR, because only in extraordinarily intermittent situations the level rises above the high threshold. To make IR able to distinguish between situations with different degrees of intermittency, the criterium for setting C would be a preferably uniform spread of IR across the range of exposure situations as they occur in the real world. The balance between these extreme cases was investigated by numerical simulations of various traffic situations and resulted in C = 3 dB [4]. This has been the value applied in the present study regardless of the identified sound sources and because mostly of the events were from mixed road traffic.

An IR > 50% means that more than half of the sound dose is caused by “distinct” sound events. In situations with only events that clearly emerge from background noise (e.g., a receiver point close by a railway track), IR yields values close to 100%.The IR metric provides information on the noise temporal structure and can be fruitfully added to the noise energy content L _Aeq,T to describe the potential harmful effects on the exposed population. For instance, in the SiRENE study, the IR values were included in the façade noise maps computed for all dwellings in Switzerland [4]. Moreover, the IR provide interesting results to classify urban roads on the basis of traffic noise features [21].

An example of sound event detection is given in Figure 5, where the parameters of sound event(s), reported with red lines, are given as follows:

• τ _s and τ _e are the start and end time of the event, corresponding to the instants immediately before and just after the exceedances of the threshold K (Equation (2)) above which an event (red line) is detected accordingly to the IR criterium [4];

• τ = τ _e − τ _s is the event duration;

• τ _g is the time gap between two consecutive events;

Figure 5

Parameters of the sound event detection based on IR algorithm (adapted from the study by Alsina-Pagès [7]).

For any detected event, the following parameters were also determined (Figure 5):

• τ _max, the instant at which the maximum SPL of the event occurs (SPL_max);

• the onset O of the SPL O = SPL_max − SPL_s, where SPL_s is the SPL corresponding to the start time τ _s of the event;

• the onset rate OR of the SPL OR = O ( τ max − τ s ) ;

• the sound exposure level (SEL), corresponding to all the acoustic energy of the sound event as if this had occurred within a 1-s time period:

• the equivalent continuous level (L _Aeq):

Figure 6 shows an example of the plot obtained by the developed R script, where:

• the blue dotted line reports L _β , that is the threshold applied for the sound event detection according to the IR criterium;

• blue crosses correspond to the maximum sound level of each event with duration τ ≥ 1 s and onset O ≥ 5 dB(A), usually considered as “notice-event” in terms of perception;

• red circles correspond to the maximum sound level of each event with duration τ ≥ 1 s, onset O ≥ 5 dB(A) and onset rate OR ≥ 10 dB/s, respectively; some standards, like the NORDTEST Method [21], classify a sound as impulsive when the onset rate OR ≥ 10 dB(A)/s.

Figure 6

Example of the L _AF hourly time history (spot 3) with L _β the threshold applied for the sound event detection according to the IR criterium at K threshold (blue dotted line), maximum sound level of each event with duration τ ≥1 s and onset O ≥ 5 dB(A) (blue cross) and onset rate OR ≥ 10 dB/s (red circle).

2.4 Sound source classification

The CNN performs a polyphonic classification after the pre-processing stage of the raw audio files and the training process of the neural network.

2.4.1 Audio pre-processing

As inputs of the CNN, the audio files have been converted to spectrograms. Spectrograms are graphic representations of sound with x-axis representing time and y-axis the frequency and colored scale indicating the SPL. The spectrograms were calculated using the LibROSA Python library, version 0.8.1 [22]. The spectrograms of the raw audio files were determined without applying any pre-processing technique, such as noise reduction, to enable the CNN to classify not only the events in the foreground, but also the events in the background. The recording was windowed in fragments of 4 s each and, then, the short-time Fourier transform was calculated, which computes the discrete Fourier transforms over short overlapping windows [22]. Once the spectrograms were calculated, they are resized to a resolution of 224 × 224, aiming to maintain compatibility with the chosen CNN.

The 4 s windowing was necessary because one of the datasets used for training (the Eixample dataset) is labelled with weak labels of a 4 s resolution. Differently from strong labels, where each acoustic event is defined by the start and end time, weak labels only indicate the presence or absence of a source category in each fragment. In the Eixample dataset, this weak labelling process was carried out by fragments of 4 s, meaning that every 4 s there is a tag with all the categories that are present on it, regardless of their saliency.

A sample spectrogram is shown on Figure 7. Specifically, the spectrogram on the left shows a 4-s fragment containing the sounds of road traffic noise, a car brake, a car horn, and an impulsional sound. Therefore, this spectrogram contains 4 categories as labels, but the position of them inside of the spectrogram is not known. Similarly, the spectrogram on the right contains three sounds: road traffic noise, car brakes and people talking.

Figure 7

Example of two spectrograms. The left one contains noises from road traffic noise, a car brake, a car horn and an impulsional noise. The right one contains road traffic noise, car brakes and people talking.

The other two datasets used for training the neural network, namely the BCNDataset and the UrbanSound8k dataset, are labelled with strong labels.

The sounds of the UrbanSound8k dataset are not masked by other categories and do not enable to perform polyphonic classification directly, as each fragment contains an event of only one category. The fact that UrbanSound8k dataset contains fragments of durations of 4 s (or, in some cases, shorter), makes the selected window length a convenient size to work with. Those fragments that had a duration shorter than the selected size were synthetically modified and repeated until the 4-s duration was achieved.

For the BCNDataset, as the audio fragments contain strong labels, the labels were converted to weak labels by simply splitting the audio files into fragments of 4 s and aggregating all the labels contained in that amount of time.

The spectrograms of all the datasets were obtained using the same parameters. Then, each spectrogram was normalized according to its maximum and minimum value, in such a way that each spectrogram contained only values ranging between 0 and 1.

3.1 Sound event detection

For each 1-h time history of A-weighted SPL, the overall value of the acoustic descriptors calculated by the script developed in the R environment (Table 4) is given in Table 5. The difference of values among the recording spots are small, for instance 1 dB(A) for L _Aeq and 0.6 dB(A) for L _A90, usually representing the background sound level. The IR values range from 43.7 to 49.0%, indicating that the sound energy of the events roughly contributes to half of the overall energy. The noise climate, determined by the difference L _A10 – L _A90, ranges from 12.7 to 13.7 dB(A), indicating large SPL time variability, as also reported by sL _A.

Based on the IR criterium for sound event detection, namely a sound event is detected whenever its A-weighted SPLs exceed the preset threshold K (Equation (2)), the number of events detected in each 1h-recording are reported in Table 6, considering only the sound events with duration τ ≥ 1 s and onset O ≥ 5 dB(A), usually considered as “notice-event” (NE) in terms of perception, and those classified as impulsive (IE) because onset rate OR ≥ 10 dB(A)/s. The average values of some descriptors of the events are also given.

The probability density function of the SEL values is given in Figure 8 for each of the four recording spots. Spots 3 and 4 show a very similar distribution, whereas they are different for spots 1 and 2, the latter showing a bimodal distribution. The dotted lines correspond to the mode value for each distribution.

Figure 8

Density plot of SEL values of “notice-event” (NE) observed at the four sites. The legend numbers refer to the locations highlighted on Figure 2.

The R script provides also the A-weighted SPL time history of each detected event. Figure 9 reports such an example where the horizontal red dotted line corresponds to the threshold for the event detection L _β and the inclined blue line connects the SPL at the start time with the maximum SPL of the event.

Figure 9

Example of the A-weighted SPL time history of a detected event. The horizontal red dotted line corresponds to the threshold for the event detection L _β and the inclined blue line connects the SPL at the start time with the maximum SPL of the event.

Referring to the sound recording fragmentation into 4 s windows, the detected sound events were often non-completely included in one 4 s window, because their length was longer. Table 7 reports the percentage of detected notice-events NE spanning over adjacent 4 s windows for each site. Moreover, Figure 10 shows the distribution of the event durations τ observed at each site.

Figure 10

Density plot of event durations τ of “notice-event” (NE) observed at the four sites. The legend numbers refer to the locations highlighted on Figure 2.

The further step was to listen to the recordings of the detected sound events to recognize the corresponding source(s), according to the classification reported in Table 1. The labelling outcome selected by the listener was taken as reference for evaluating the performance of the sound source classification by DL technique.

3.2 Sound source classification

Once the events were detected by the IR, the CNN classified them into different categories. For model evaluation, each event was manually listened to in order to verify whether the neural network correctly classified the category causing the event. This process was carried out by two experts that listened all the events at different moments. After the listening process, the results from both experts were checked and discussed until consensus was reached.

The neural network had 21 output neurons (one for each class described in Table 1), each outputting a probability (between 0 and 1) that the corresponding event was present, using a sigmoid activation function. Then, this output was binarized using a custom threshold for each class aiming to maximize the true positive events and minimize the false negative events for that category. The thresholds were selected from a validation set, which was a subsample of the training dataset (10% of data from BCNDataset and 10% of data from the Eixample dataset) that was not used for training nor testing and in which we did not apply any data augmentation techniques. Therefore, the process for obtaining the thresholds was:

1. First, we passed the validation data through the network, obtaining an output probability values for each class.

2. Then, given that more than one category might be active on each input fragment due to the polyphonic nature of the training data, we calculated the ROC curve for every class, which gave us the true positive rate (TPR) and false negative rate (FNR) in different thresholds.

3. Next, we calculated the geometric mean of the TPR and FNR for every threshold as follows:

   (5) Geometric _ mean = TRP × ( 1 − FPR ) .

4. Finally, for each class, we selected the threshold that enabled us to have the highest geometric mean.

The results of this procedure resulted in 21 different thresholds, most of them having values of around 0.5. The specific value of threshold per class is: rtn = 0.5, peop = 0.66, brak = 0.56, bird = 0.52, motorc = 0.42, eng = 0.5, cdoor = 0.59, impls = 0.47, cmplx = 0.70, troll = 0.56, wind = 0.5, horn = 0.14, sire = 0.54, musi = 0.48, bike = 0.43, hdoor = 0.60, bell = 0.59, glass = 0.55, beep = 0.5, dog = 0.54, drill = 0.48.

In the testing phase, once the events detected by the IR were passed through the network, the expert listeners would review each fragment and check whether it contained the detected noise sources.

Given the data were collected in a very noisy environment with common overlapping sounds, the manual evaluation categorized each output result of every neuron and for every event as follows:

• TP:FG: meaning True Positive: Foreground. This category represents the sources that have been detected both by the CNN and the expert listener and for the noise sources that are occurring in the foreground;

• TP: BG: meaning True Positive: Background. This category represents the sources that have been detected both by the CNN and the expert listener but that the event was occurring in the background or was masked by more salient event;

• FP: meaning False Positive. This category represents the sources that have been detected by the CNN but that the expert listener could not hear them. Therefore, this category represents a mistake of the CNN;

• FN: meaning False Negative. This category reflects those events that were heard by the expert listener but that the CNN could not detect. Therefore, this category represents a mistake of the CNN too;

• TN: meaning True Negative. This category represents all the outputs that both the CNN and the manual listener did not detect. Therefore, this category represents a correct prediction.

As it can be observed, the standard metrics of TP, FP, FN and TN were used, but the True Positive events were divided into two categories: one for the most salient events and one for the events in the background. This way, it can be analyzed whether the positive events are only detected in the foreground or if there are events that are detected but are masked with other sounds.

The aggregated results from the four sensors, as shown in Table 8, provide insights into the performance of the CNN-based classification system, particularly in identifying foreground events, with a sensitivity equal to 0.829 and a specificity of 0.985.

The system demonstrated a commendable ability to detect the most salient sounds, with 534 events classified as TP: FG. These foreground events, which are the prominent parts of the audio, were accurately identified by the CNN. This indicates that the model is effective in capturing and classifying events that strongly influence the sonic environment.

However, the system also displayed limitations, primarily in the detection of background sources. Despite successfully classifying 74 TP: BG events, the CNN missed 110 events during manual listening analysis, mainly in the background. While this discrepancy exists, it is important to note that the system’s ability to correctly identify background sources is still notable. The fact that 74 background events were accurately identified indicates that the model is capable of capturing and classifying sources that are less prominent in the audio.

Actually, considering the system as a whole, as the events are detected through the IR, it is coherent to think that the classifier should focus on detecting the most prominent sound, which is the one that has made the IR trigger the event.

Also, the number of false negatives exceeds the count of false positives, indicating a higher tendency for the system to fail in recognizing background sources compared to the identification of sources not present in the audio file.

For a more detailed analysis of the classification results, Table 9 shows the same categories (True Positives in the Foreground, Tue Positives in the Background, False Positives, False Negatives and True Negatives) segregated by type of event and per class. This way, it is possible to analyze which of the categories are better classified.

Table 9

Results per class and per sensor

As it can be observed, there are some sources that were not present in the testing data, and therefore it was not possible to evaluate if the algorithm is able to classify them when they are present. These are: musi, glass, beep and dog. However, it is still relevant to include them in the table to check that the system does not produce False Positive events in that category. The lack of those events in the testing hour occurred as the hour selected for testing the system contains real-world data, and during that time of the data gathering campaign, those sources were not present.

As it can be observed, all the fragments of events contained the rtn source, with some overlapping sources as well. The model was able to predict successfully all those events. Moreover, the system was capable of detecting also most of the motorc events, showing a good capability of detecting motorized sources. Same happens with the bus door category or even the house door source, where most of the events are correctly classified.

Also, in general, the system tends to have less False Positive detections than False Negatives. With the data used for evaluation, it can be seen that none of the wind, drill or sire sounds present in the data were actually detected by the CNN. When listening to the events, it can be concluded that it is because these few samples were actually masked by other sounds. Therefore, this reveals that the system struggles to identify all of the polyphonic sounds of a sample when they are highly masked.

As of consistency between sensors, it seems that the data gathered in the four different sensors have the same behavior in the CNN, showing that the system is robust against using data coming from different physical sound devices.

If the TP:FG and the TP:BG are added together, we can obtain the general True Positive values of the system, enabling the calculation of standard metrics such as the Precision, Recall or F1-score of the system per class. Actually, for the sake of better understanding the behavior of the classifier, Table 10 shows the classification metrics of those categories which contained samples (at least, 5) in the testing set and aggregating the classification data of the four sensors.

Table 10 reveals key insights into the classifier’s performance: certain categories, such as rtn, brak, and bird, achieved exceptionally high F1-scores and the maximum Precision score, with rtn even attaining perfect scores across all metrics. Following this categories, motorc also demonstrates a strong overall performance, with an F1-score bigger than 90%. The performance of the classes rtn, brak, and motorc suggest that the classification model excels at classifying road traffic-related sounds, which is one of the categories with more interest in the END [15].

Conversely, categories such as troll and horn display lower performance, indicating more difficulties in accurately detecting these types of events. This reflects challenges in accurately detecting these types of events, possibly due to their more subtle or masked characteristics within the audio data used for testing and the fact that there are only few events of these categories in the testing set (only 14 troll events and 5 horns).

We must remember that the limited number of events in certain categories, such as troll and horn, can be attributed to several factors tied to the data collection and selection process: First, the testing data were derived from real-world, continuous recordings. Unlike synthetic or artificially balanced datasets, real-world data inherently reflects the natural distribution of sounds, which may not include all categories equally. Then, the testing data were chosen as a continuous time segment, representing a specific period of the day in which the IR criteria can be applied to detect the acoustic events, which is the target of this work.

In summary, while there is room for refinement, particularly in the detection of background sources, the CNN-based classification system demonstrates promising capabilities in identifying sources in the acoustic environment, achieving an overall aggregated metric surpassing the 84% of F1-score (both micro and macro averaged) among those categories that were more present in the testing data. Addressing these limitations – probably by using more training data in those categories that are less present in the dataset such as musi, glass, beep or dog – will lead to a more robust and reliable tool for assessing and managing environmental noise.