Unheard melodies and emotional peaks in Let It Go and Show Yourself: a multimodal sentiment analysis

4.1 Textual analysis

Sentiment analysis is crucial for understanding the polarity and intensity of human emotions expressed in text (Eyu et al. 2025). As defined by Batrinca and Treleaven (2015), it involves “the application of natural language processing, computational linguistics and text analytics to identify and extract subjective information in source materials” (p. 90). With this in mind, our textual analysis deploys two distinct sentiment analysis models, BERT and VADER, chosen for their complementary strengths and proven efficacy across diverse narrative genres (Barik and Misra 2024), further enhancing the robustness of analysis. BERT’s deep learning architecture allows for context-aware sentiment classification, making it particularly well-suited for detecting subtle shifts in emotion across the lyrical content. VADER, a lexicon-based tool, offers a complementary approach, particularly effective for handling more general sentiments often found in media contexts. A significant methodological challenge was the extraction of lyrics directly from the Frozen soundtracks due to the prominent presence of musical instrumentation. To overcome this, Optical Character Recognition (OCR) techniques were applied to available subtitles, providing a reliable source for textual extraction and ensuring the accuracy of subsequent sentiment analysis. This methodological choice demonstrates a practical approach to data acquisition in multimodal analysis. To maintain coherence with the songs’ narrative structures, a four-step process was implemented:

1. Segmentation Alignment. The texts within each previously defined narrative segment were concatenated into a single string. This ensured that the sentiment analysis reflected the entire narrative context of each segment, capturing the overall emotional arc rather than analyzing fragmented pieces of text.

2. Dual-Model Application. Both a BERT-based deep learning model (Hoang et al. 2019) and VADER (Catelli et al. 2022) were applied to each segment. This dual-model approach, a key methodological contribution, provided a more robust evaluation by integrating the strengths of both advanced machine learning and traditional lexicon-based sentiment analysis.

3. Heatmap Visualization. A heatmap visualization (Samek et al. 2017) was used to represent the dominant emotions across the different segments. This visualization technique effectively highlighted variations in sentiment throughout the songs, providing a clear and interpretable representation of the emotional trajectories.

4. OCR for Lyric Extraction. Due to the difficulty of extracting lyrics directly from the soundtracks, OCR was applied to readily available subtitles. This practical solution ensured accurate lyric extraction despite audio complexities.

4.2 Visual analysis

Our visual analysis employed Facial Emotion Recognition (FER), focusing specifically on Elsa’s facial expressions at key emotional junctures in both songs. FER’s deep learning-based architecture (Bhagat et al. 2024) is adept at detecting nuanced emotional transitions, providing a data-driven perspective on the visual manifestation of character emotions. The following detailed steps outline the visual analysis process:

1. Segment Alignment. As with the textual analysis, the video was divided into distinct segments based on the previously defined narrative and emotional shifts. This ensured alignment between the visual, textual, and auditory analyses.

2. Face Detection. For each segment, frames were sampled, and face detection was performed using a pretrained Haar cascade classifier. This provided a reliable method for isolating Elsa’s face within each frame.

3. Facial Emotion Recognition. The extracted facial images were processed using the FER model, a deep Convolutional Neural Network (CNN) trained to recognize a range of human emotions (e.g., happiness, sadness, anger, surprise, or fear).

4. Dominant Emotion Identification. For each video segment, the dominant emotion was identified by aggregating the results from individual frames, determining the emotion with the highest confidence score across all frames within that segment.

5. Expert Validation. To ensure the accuracy and reliability of the FER model, the identified dominant emotions were compared against expert interpretations based on visual, audio, and semiotic cues. This validation step is crucial for establishing the validity of the computational analysis.

6. Heatmap Visualization. The results were visualized using heatmaps, displaying normalized confidence scores of detected emotions across different segments, providing a clear visualization of emotional transitions.

To ensure the accuracy and reliability of the FER model, the identified dominant emotions were compared against expert interpretations based on visual, audio, and semiotic cues. Three annotators with backgrounds in film studies, musicology, and psychology participated in this process. These annotators possess expertise in analyzing emotional expression in film, allowing for a nuanced understanding of the characters’ emotional states. The annotation process involved each annotator independently rating the dominant emotion in each segment based on visual cues, audio cues, and narrative context. The emotion categories were operationally defined to ensure consistency. Inter-rater reliability was assessed using Fleiss’ kappa to quantify the agreement between annotators, demonstrating a substantial level of agreement (kappa = 0.81). Any discrepancies between FER model outputs and expert judgments were resolved through discussion and consensus among the annotators. This process ensured that the final interpretation reflected a comprehensive understanding of the emotional content. It is important to acknowledge the unique challenges of applying FER to stylized animations, where characters’ expressions may differ from real human expressions. The expert validation process helped to mitigate these challenges by providing a human-centered interpretation of the model’s output.

4.3 Auditory analysis

To provide a more holistic understanding of how musical features contribute to emotional peaks, we extracted key musical features using Music Information Retrieval (MIR) tools. The extracted features – tempo (BPM), spectral centroid (brightness), and Root Mean Square (RMS) (loudness) – are widely used in MIR for mood and emotion classification (Casey et al. 2008; Lartillot and Toiviainen 2007). The six-step process for auditory analysis was as follows:

1. Audio Extraction and Separation. The audio track was extracted from the video and separated into vocal and accompaniment channels using tools like FFmpeg and Spleeter. This separation allowed for a more focused analysis of the musical elements.

2. Feature Extraction. Relevant musical features (tempo, spectral centroid, RMS) were extracted from each segment using audio analysis libraries like LibROSA.

3. Sentiment Mapping. The extracted features were mapped to specific sentiments using heuristic rules based on empirical studies in music psychology (Juslin and Laukka 2003).

4. Result Interpretation. The results were interpreted for each segment to understand how the music influences the emotional experience.

5. Visualization. The extracted features and corresponding sentiments were visualized using tools like Matplotlib.

6. Evaluation and Reflection. The outcomes were evaluated to determine the effectiveness of the heuristic rules and reflect on their limitations, paving the way for future refinements using more sophisticated methods.

By means of combining these three distinct analytical approaches, this study provides a more comprehensive and nuanced understanding of the emotional landscapes of Frozen and Frozen II, directly addressing the need for multimodal analysis in understanding artistic expression. Combining computational analysis with expert validation allows for a more objective and data-driven approach to the study of emotion in media.

5.1 Textual analysis

The analysis of “Show Yourself,” presented in Table 3 and visualized in Figure 2, using BERT and VADER, reveals interesting discrepancies and alignments between the models and expert interpretation. At the song’s outset, Elsa experiences curiosity and uncertainty, as confirmed by expert interpretation. BERT detected anger, while VADER showed neutrality. This discrepancy suggests an underlying inner conflict masked by a controlled exterior, a nuance that VADER’s more general approach missed. As Elsa reaches the first revelation, a mix of hopeful realization and apprehension emerges. Both models indicated a shift toward a more negative outlook from an initial detection of anger by BERT, reflecting Elsa’s cautious approach to the unfolding truths. During the intensified search for answers, Elsa’s determination becomes evident. BERT detected sadness, potentially reflecting frustration, while VADER remained neutral, indicating Elsa’s focused pursuit of self-discovery. The final confrontation brings a moment of truth, marked by joy and positivity, aligning with both models’ readings. This signifies Elsa’s growing understanding of her true identity. At the song’s emotional climax, Elsa’s realization is intense. BERT detected anger, while VADER remained neutral, possibly highlighting the overwhelming nature of self-discovery, which can be both exhilarating and daunting. The song concludes with peaceful acceptance as Elsa comes to terms with her identity. Both models reflected a calm and balanced state, indicating her attainment of serenity and peace. This progression, from internal conflict to acceptance, effectively encapsulates Elsa’s emotional evolution throughout the song, showcasing the potential of sentiment analysis to map character development.

Figure 2:

Emotional analysis heatmap of Show Yourself.

The analysis of “Let It Go,” presented in Table 4 and visualized in Figure 3, revealed both strengths and limitations of the BERT and VADER models in capturing the song’s emotional peaks. BERT frequently misinterpreted intense emotions, labeling segments like “Introduction/Resignation” and “Creation of Ice Palace” as anger, contrasting with expert interpretations of mild frustration and creative freedom, respectively. This suggests that BERT may associate strong emotional expressions with negativity, even in positive or neutral contexts. In contrast, VADER often classified segments as positive, aligning with moments of joy and transformation, such as “Decision to Let Go” and “Transformation,” reflecting Elsa’s embrace of her powers and independence. However, VADER’s simpler classification sometimes lacked nuance, failing to distinguish between different types of positive emotions like empowerment and liberation. For instance, in the “Final Declaration” segment, VADER’s positive sentiment aligns with liberation but does not capture the moment’s full complexity. BERT’s misclassification of fear highlights its difficulty distinguishing high-energy emotions associated with fear and liberation. Overall, both models effectively detected clear emotional peaks, particularly when emotions were unambiguously positive. However, they struggled with more complex or mixed emotional states. BERT’s detailed labels provided specificity but at the risk of misclassification, while VADER’s broader categories were consistent but lacked depth.

Figure 3:

Emotional analysis heatmap of Let It Go.

The analysis, as detailed in Tables 3 and 4 and Figures 2 and 3, reveals that both BERT and VADER have limitations in representing emotions that are not clearly positive or negative. BERT showed potential for recognizing distinct emotional peaks but requires refinement to handle nuanced emotions. VADER’s neutrality suggests its utility for general sentiment analysis, but its lack of specificity limits in-depth analysis. To improve sentiment analysis in multimedia narratives, both models would benefit from further training on specialized datasets that capture the nuanced emotional transitions characteristic of storytelling. This would enhance their ability to accurately interpret a broader range of emotions. Integrating techniques focusing on contextual and sequential emotional analysis, such as attention mechanisms, could further improve alignment with expert interpretations, making sentiment analysis more relevant and insightful for multimodal analysis. This directly supports the need for the proposed multimodal model, which aims to address these very limitations.

5.2 Visual analysis

In the analysis of “Show Yourself,” the FER model was employed to detect and quantify the emotional peaks within the song, aligning these findings with the narrative structure and Elsa’s evolving self-awareness (see Table 5 and Figure 4). In the Introduction (0:00 - 0:42), characterized by anticipation, the FER model detected a high confidence level associated with curiosity and uncertainty, which aligns with Elsa’s cautious approach as she embarks on her journey. During the First Revelation (0:43–1:21), where hopefulness emerges, the FER model recorded emotions that match Elsa’s growing sense of optimism about uncovering her past, supported by a high confidence score of 0.9500. As the search intensifies (1:22–1:58), the FER model noted a slight dip in confidence, at 0.9300, which still reflects Elsa’s rising determination driven by her intrinsic need for self-discovery. In the Final Confrontation (1:59–2:36), marked as the moment of truth, the FER model recorded a confidence score of 0.9200, capturing the emotional intensity as Elsa prepares to face her reality. The Emotional Climax (2:37–3:05) represents Elsa’s peak of self-realization, with the FER model detecting a return to higher confidence levels (0.9300), indicating her deep emotional resonance upon realizing her true identity. Finally, in the Resolution and Acceptance segment (3:06–3:45), the FER model recorded the highest confidence score of 0.9800, reflecting Elsa’s serene acceptance of her identity and the song’s peaceful conclusion. The strong alignment between the FER model’s findings and expert analysis demonstrates the model’s capability to accurately mirror the emotional transitions and depth portrayed in “Show Yourself.” By synchronizing audio-visual cues with expert semiotic analysis, this approach provides a comprehensive understanding of Elsa’s character evolution, emphasizing the effectiveness of deep learning models in uncovering nuanced emotional insights within narrative-driven content. The analysis illustrates how the FER model effectively complements expert interpretations, capturing both the subtle and significant emotional shifts as Elsa journeys through self-discovery. The model’s consistently high confidence levels across the song’s segments reflect its ability to mirror the character’s psychological landscape and narrative-driven emotional peaks.

Figure 4:

Emotional peak with normalized confidence of Show Yourself using FER.

In the analysis of “Let It Go,” also visualized in Figure 5 and detailed in Table 6, the FER model was utilized to identify and quantify the emotional peaks within the song, aligning these results with the narrative progression and Elsa’s character development. In the Introduction (0:00–0:40), the detected emotions indicated mild frustration and resignation, aligning with Elsa’s initial struggle and fears. As Elsa decides to let go (0:41–1:02), the model captured rising determination, a transition that marks her shedding of past constraints. This shift is further underscored in the Empowerment segment (1:03–1:27), where the model’s high confidence scores reflect Elsa’s full embrace of her powers, symbolizing a profound self-empowerment. The creation of the Ice Palace (1:28–2:05) was marked by emotions associated with creative freedom, as Elsa transforms her environment, embodying control and artistry. The Transformation segment (2:06–2:37) showed emotions of happiness, corresponding to Elsa’s self-realization and acceptance of her identity, while the Final Declaration (2:38–3:00) captured feelings of liberation, aligning with Elsa’s complete detachment from societal norms. Finally, in the Conclusion (3:01–3:44), the analysis revealed calm confidence, correlating with Elsa’s state of peace and mastery over her powers. The normalized confidence scores from the FER model consistently peaked during these key emotional shifts, affirming the model’s capability to accurately detect and mirror the emotional depth and progression outlined by human interpretation. This synchronization between audio-visual cues and semiotic analysis offers a comprehensive understanding of how Elsa’s character evolution is visually and emotionally depicted in “Let It Go,” demonstrating the efficacy of using deep learning models in conjunction with expert analysis for nuanced emotional insight.

Figure 5:

Emotional peak with normalized confidence of Let It Go using FER.

5.3 Auditory analysis

This section presents the results of the auditory analysis, aligning musical features and sentiment analysis with the predefined emotional peaks from the scene analyses of “Show Yourself” (Frozen II) and “Let It Go” (Frozen), as detailed in Tables 7 and 8 and visualized in Figures 6 and 7, respectively. This combined approach, integrating audio analysis with the previously discussed textual and visual analyses, provides a richer understanding of how musical elements correlate with the narrative’s emotional flow and contributes to the overall multimodal sentiment analysis.

Figure 6:

Normalized musical features per segment of Show Yourself.

Figure 7:

Normalized musical features per segment of Let It Go.

The analysis of “Show Yourself,” presented in Table 7 and visualized in Figure 6, aligns the extracted musical features with the emotional narrative described by expert interpretation. Each segment’s sentiment is derived from the normalized values of key musical features such as tempo, spectral centroid, and RMS (loudness), reflecting the emotional shifts within the song. In the “Intro/Anticipation” segment, the high tempo and spectral centroid suggest a feeling of tension and anticipation, fitting with Elsa’s curiosity and uncertainty as she begins her journey. The second “First Revelation” segment is marked by low tempo and spectral centroid, with a slight increase in loudness, leading to a “Calm/Sad” sentiment. This aligns with Elsa’s hopeful yet reflective state as she inches closer to understanding her inner voice. By the “Search Intensifies” segment, despite the high tempo, the low spectral centroid and moderate loudness lead to a neutral sentiment. This reflects Elsa’s rising determination and the growing intensity of her search, driven by a sense of destiny. In the “Final Confrontation” segment, with high values across all features, the segment conveys an “Energetic/Happy” sentiment, aligning with the emotional charge of Elsa facing her moment of truth. This segment represents a pivotal point where Elsa is ready to confront and embrace her true self. High tempo, spectral centroid, and RMS continue to drive an “Energetic/Happy” sentiment in the “Emotional Climax” segment, fitting with the emotional climax of the song where Elsa fully realizes her identity and powers. The “Resolution/Acceptance” segment maintains high values across all features, conveying a sentiment of “Energetic/Happy,” which correlates with Elsa’s peaceful acceptance of herself. The conclusion of the song mirrors a serene and fulfilled emotional state.

Table 8 and Figure 7 illustrate a strong alignment between the musical features, sentiment analysis, and the narrative’s emotional arc in “Let It Go.” In the “Introduction/Resignation” segment, the low tempo, low spectral centroid, and minimal loudness result in a neutral sentiment, matching the scene where Elsa is still conflicted, constrained by fear and responsibility. The emotional interpretation of “Mild Frustration/Resignation” reflects this internal struggle. With low tempo and spectral centroid, and a slightly higher loudness, the “Decision to Let Go” segment’s sentiment is “Calm/Sad,” aligning with Elsa’s tentative step toward embracing her powers, characterized as “Rising Determination.” In the “Empowerment” segment, a high tempo, high spectral centroid, and moderate loudness contribute to a “Tense/Anxious” sentiment. This correlates with Elsa’s empowering moment where she fully embraces her abilities, mirroring the emotional peak described as “Empowerment.” By the “Creation of Ice Palace” phase, high values across tempo, spectral centroid, and RMS indicate an “Energetic/Happy” sentiment, fitting the scene of Elsa’s artistic expression with her ice powers, encapsulated by “Creative Freedom.” The “Transformation” segment maintains high tempo, spectral centroid, and loudness, leading to an “Energetic/Happy” sentiment. It aligns with Elsa’s acceptance of her true self, which the experts interpret as “Self-Realization/Acceptance.” Although the tempo remains high in the “Final Declaration” segment, the spectral centroid drops, and loudness reaches its peak, resulting in a neutral sentiment. This phase of the song represents Elsa’s liberation, which is echoed in the emotional peak “Liberation.” By the “Conclusion” segment, consistent high values in tempo and spectral centroid, along with slightly reduced loudness, evoke an “Energetic/Happy” sentiment, aligning with the peace and confidence Elsa has found, as described in “Calm Confidence.”

This combined analysis demonstrates how musical features can effectively map to specific emotional peaks, offering valuable insights into the emotional impact of a song within a narrative context (see Figures 6 and 7). The heuristic rules applied to derive sentiments from musical features provide a reliable method for understanding the emotional resonance of the song, demonstrating the power of music in enhancing storytelling. The findings emphasize the alignment between the audio analysis and the emotional journey depicted in both “Show Yourself” and “Let It Go,” supporting the narrative through carefully crafted musical elements. The application of these heuristic rules, grounded in music psychology and information retrieval research, confirms their validity and relevance in sentiment analysis of musical compositions.