Dependence modeling in general insurance using local Gaussian correlations and hidden Markov models

2.1 LGC

Let X = ( X 1 , X 2 ) be a two-dimensional random variable with density f ( x ) = f ( x 1 , x 2 ) and ψ be a Guassian bivariate density defined as follows:

where ν = ( ν 1 , ν 2 ) T is the running variable, μ ( x ) = ( μ 1 ( x ) , μ 2 ( x ) ) is the local mean vector and Σ ( x ) = σ i j ( x ) is the local covariance matrix.

The density f ( x 1 , x 2 ) can be approximated locally in a neighborhood of each point x by the variables’ general density ψ . The approximation approach allows the characterization of local dependencies within the data, even if the overall data distribution is not Gaussian.

The variables themselves are not required to follow a Gaussian distribution, as the local Gaussian approach will approximate a specific region of the dataset to adhere to Gaussian density characteristics. Thus, LGC provides a localized Gaussian approximation that captures subtle, region-specific relationships in the data.

With σ i 2 ( x ) = σ i i ( x ) , the local correlation at the point x is defined as ρ ( x ) = σ 12 ( x ) σ 1 ( x ) σ 2 ( x ) and hence in terms of local correlation at x the approximated density ψ is written as follows:

This construction can be easily extended to multivariate cases. However, a direct multivariate implementation often faces dimensionality issues, which cause estimates to deteriorate rapidly. As a result, [38] introduced a simplification by estimating each local correlation as a bivariate problem by considering only the pair of observation vectors. Therefore, the p -variate problems of estimating the local parameters depending on all coordinates is reduced to a series of bivariate problems of estimating pairwise local correlations depending on their respective coordinates. This simplification significantly enhances computational efficiency and interpretability.

Extra conditions are needed to ensure that 2 is well represented. Tjøstheim and Hufthammer [45] demonstrated that for a given neighborhood characterized by a bandwidth b , the local population parameters

can be obtained by minimizing a likelihood related penalty function

that measures the difference between f and ψ . Intuitively, this penalty function quantifies how well the local Gaussian approximation aligns with the true underlying data distribution. The method for estimating LGC parameters is explained in detail in Appendix C. Specifically, by maximizing the local likelihood numerically, we obtain local likelihood estimates θ ˆ b ( x ) , which also include estimates ρ ˆ b ( x ) of the local correlation. By applying this process at multiple grid points, we can derive various estimates of the LGCs. Thus, LGC flexibly captures varying dependence structures, accommodating complex, nonlinear, or heterogeneous relationships more effectively than traditional global correlation measures.

In our analysis of insurance data, we define X = ( X 1 , X 2 ) ∈ R 2 as a stochastic variable representing claim payouts across two distinct lines of business. This setup allows us to effectively explore local dependencies between different insurance segments.

We used the LGC R package “lg” to fit the LGC model, facilitating the calculation and visualization of LGC coefficients, enabling a localized examination of dependencies. We have included sample code in the supplementary material for further reference on how to fit the LGC model. For further insights into the “lg” R package, we refer the reader to the article by Otneim [37].

2.2 LGC test for independence

The LGC theory, as outlined by Berentsen et al. [5], facilitates the construction of an independence test between pairs of variables. The LGC method provides a nuanced measure of dependence, focusing specifically on local regions of the data, and contrasts sharply with conventional global tests that can overlook subtle variations in dependence. The local measure of dependence evaluates disparities between the data and the assumption of independence. Thus, it offers a sensitive tool for identifying dependencies that manifest locally rather than globally. Such localized detection is crucial for accurately understanding and modeling risk factors, which often do not exhibit constant relationships across different claim scenarios.

The hypothesis tested is given as follows:

We use bootstrapping to test local independence, estimating the null distribution of LGC by resampling from empirical marginal distributions. This resampling approach provides a reliable empirical basis for comparing observed local correlations against the assumption of independence. It allows multiple tests across different data regions, revealing areas where variables deviate significantly from independence and enhancing our understanding of dependence patterns.

Let X 1 and X 2 be two random variables with a general estimated functional for testing independence given as follows:

where h is a non-negative measurable function and F n is the empirical distribution function of X 1 , X 2 . Here, the integral aggregates local correlation statistics across specified regions to quantify the overall extent of departure from independence.

By choosing a set S , we can focus the test on specific regions. Bootstrapping generates multiple bootstrap samples from observed data, and LGC test statistics are computed for each. The distribution of these statistics is then used to estimate confidence intervals or compute p-values enabling a more detailed and rigorous investigation of local dependence structures, enhancing the interpretability and statistical reliability of the results.

2.3 Integration of LGC and HMMs

HMMs are often used in indirect observation studies of discrete-valued processes [33]. HMMs are particularly effective in modeling systems that switch between unobservable (hidden) states, which can influence observed outcomes in complex ways. By using HMMs, we can model hidden regimes underlying observable data, allowing us to model temporal, dynamic, and time-varying dependence [31]. The LGC-HMM framework is ideal for settings like insurance, where underlying risk regimes such as economic conditions or policy changes are not directly observed but impact claim behavior [31,36]. HMMs offer distinct advantages for analyzing insurance claim time series due to their ability to model unobserved state transitions and time-varying dependencies inherent in claims data [49]. While traditional autoregressive models or multivariate approaches assume static relationships, HMMs explicitly capture shifts in underlying risk regimes, which are critical in insurance contexts where external factors (e.g., regulatory changes, economic cycles, pandemics) alter claim dynamics over time [23]. Moreover, insurance claims often follow patterns influenced by hidden variables such as shifts in policyholder behavior or emerging fraud trends, which HMMs can probabilistically infer through latent states, enabling dynamic adjustments to dependence structures [11]. Furthermore, unlike ARMA models that assume fixed parameters, HMMs allow parameters to switch between states, accommodating abrupt changes in claim severities [24]. The claim datasets considered in this article contain structural breaks or change points, as shown in the figures in Appendix C, further justifying the application of HMMs in this context.

HMMs have diverse applications, including speech recognition, bioinformatics, and, more recently, financial risk management through analysis of stock prices and modeling market volatility [22,24]. Insurance data often exhibits varying patterns over time, showing high claim frequency and severity phases, followed by periods of reduced activity. Oflaz [36] employs HMMs to identify these changing patterns by permitting the hidden states to transition over time. This ability to segment time series into distinct, interpretable phases makes HMMs especially powerful for modeling regime-dependent characteristics in actuarial applications. This approach enables HMMs to effectively capture changes in dependence structures and adapt to different market conditions. However, traditional HMMs may fall short in detecting fine-grained, local dependencies that vary within regimes.

To address this, we propose an integrated framework that combines LGC with HMMs. While HMMs identify the broader hidden regimes, LGC captures the evolving, local dependence within each regime. This dual-layer modeling approach provides a more nuanced understanding of both global regime shifts and local variations in dependence structures.

In addition to using direct numerical maximization (DNM) as an estimation method for the HMMs, we utilize the template model builder (TMB) package in R. TMB is particularly suited for fitting complex latent variable models like HMMs due to its computational efficiency and automatic differentiation capabilities. For a detailed tutorial on employing TMB with HMM using DNM, we direct readers to Bacri et al. [4].

2.4 LGC across regimes

To assess time-varying and nonlinear dependence structures, we combine the regime-switching model with the LGC framework. This allows us to describe distinct regimes and examine whether the dependence structure between claim payouts differs across them. The motivation for this approach stems from the need to move beyond global dependence measures like Pearson correlation, which may mask localized or regime-specific features in the data. By estimating LGC maps within each regime identified by the HMM, we can detect subtle but meaningful variations in dependency structures.

Let X t = { X 1 t , X 2 t } denote a bivariate sample observed over t = 1 , … , T . Each observation is classified into a regime c t ∈ { 1 , … , m } using a fitted HMM and the local decoding procedure. This classification step enables the separation of the entire sample into subsets corresponding to distinct latent regimes, allowing regime-specific dependence structures to be analyzed.

The subset of observations belonging to regime c is denoted by:

Each regime-specific sample c X is then used to estimate the corresponding LGC map, denoted ρ c ( x ) , for that regime. These LGC maps provide a detailed, localized view of the dependency between the two variables, conditional on location in the sample space and regime. Comparing the maps across regimes enables us to determine whether the structure of dependence is stable or state dependent.

For each regime c , we estimate a 2 × 2 local correlation matrix ρ c ( x ) at a given grid point ( x , y ) by maximizing the local likelihood function defined in equation (A2) in Appendix C. Repeating this across multiple grid points results in one LGC map per regime.

To formally assess differences across regimes, we perform pairwise comparisons between the LGC maps. For each pair of regimes ( c , d ) with c ≠ d , the null and alternative hypotheses are formulated as follows:

Unlike traditional correlation comparison tests, this method evaluates the entire dependence surface, allowing detection of complex and asymmetric variations. The use of the full grid ensures that nonlinear and local features are not overlooked.

Grid points are indexed by i and j such that x i j = ( x i , y j ) with i , j = 1 , … , n . The test statistic for a given pair ( c , d ) is computed as follows:

where w ( x i , y j ) is a weight function.

The weight function w ( x i , y j ) allows us to give more emphasis to regions of interest in the sample space, such as the tails or central area, without discarding any data. It ensures the grid points remain relevant by controlling the distance between observations and estimation locations. We apply the pairwise bootstrap testing procedure described in Appendix E to assess the significance of observed differences in LGC maps between regimes.

2.5 Data description

We used data from two leading insurance firms: a Kenyan dataset from 2008 to 2021, covering periods of political unrest and the COVID-19 pandemic, and a Norwegian dataset from 2012 to 2018, characterized by economic stability. This allows us to explore how different economic conditions impact dependencies in insurance claims.

Table 1 shows that the Kenyan dataset displays characteristics of a positively skewed distribution with heavy tails and high variability for certain LoBs. Also, some LoBs indicate a distribution with thinner tails and a flatter peak compared to the normal distribution.

In Table 2, the descriptive statistics from the weekly datasets from both Kenya and Norway are shown. The Automobile LoBs from Kenyan data have lower skewness values and standard errors compared to homeowners’ insurance LoBs.

The augmented Dickey-Fuller (ADF) test results indicate that all four claim series commercial, private, house, and content – are stationary at the 5% level or better. This suggests that their statistical properties remain stable over time, at least in the weak sense of unit-root stationarity, satisfying a key assumption for time series modeling. While classical HMMs assume a stationary hidden state process and conditional stationarity within regimes, they do not require global stationarity of the observed data. The ADF results therefore align well with the HMM framework. Furthermore, the rolling statistics and change point detection plots in Appendices B.1 in Figure A4(a)–(f) provide additional support for regime-switching modeling. These figures highlight clear shifts in mean levels and volatility across time, with visible structural breaks suggesting the presence of distinct latent states. Together, the statistical test results and visual diagnostics justify the application of HMMs to capture the dynamic, state-dependent nature of insurance claim behavior.

Figure 1

LGC maps claim payouts of different pairs of LoBS for Kenya data.

Our analysis is divided into three parts. First, we used the LGC to model dependency structures of seven LoBs using 174 monthly average claim payouts in millions of Kenyan shillings. These lines of business include motor private (MP), motor commercial (MC), workers’ compensation (WC), fire industrial (FI), liabilities (LI), engineering (EN), and personal accidents (PA). We calculate the local correlations and perform the independence test using Pearson correlation (linear) and LGC test (both linear and nonlinear)

Second, we utilized LGC and HMMs to analyze the time-varying temporary dependence in 699 weekly average claim payouts for auto insurance (motor commercial and motor private LoBs) and 315 claim payouts for homeowner’s insurance (house and content LoBs) from Kenya and Norway, respectively. Both datasets were adjusted for inflation.

Finally, we used the auto insurance LoB data to demonstrate a risk management application by simulating from the fitted LGC model and calculating VaRs. We fit copulas to the data and compute the VaRs using the Monte Carlo simulation approach and compare the VaR results of the LGC model with those of the corresponding copulas.

3.1 Dependence modeling using LGC for general Insurance LoBs

Figure 1 shows LGC maps revealing nonlinear relationships between LoB pairs: liabilities vs engineering ( ρ = 0.14 ), liabilities vs personal accidents ( ρ = 0.17 ), engineering vs personal accidents ( ρ = 0.20 ), and motor private vs motor commercial ( ρ = 0.78 ). Other pairs are in Table 3, with LGC plots in the supplementary material.

Figure 2

LGC maps of average weekly claim payouts of auto and homeowners insurance of LoBS.

The standard Pearson correlation test assessed linear dependencies, while the LGC test examined both linear and nonlinear dependencies based on the statistics in Section 2.2 and equation (4). All four graphs in Figure 1 indicate a positive correlation in the lower left tail (low payouts) and a weak or negative correlation in the upper right tail (high payouts). A moderate to weak correlation appears in the center. The nonparametric LGC independence test results in Table 3 reveal these nonlinear relationships, highlighting local correlations that traditional methods might overlook. Table 3 presents Pearson’s ρ correlation test results for linear dependencies and the LGC test for both linear and nonlinear dependencies. In most cases, both tests show significant dependence between LoBs, as observed in pairs like engineering vs fire-industrial and motor-commercial vs motor-private, which exhibit moderate to strong linear and nonlinear dependence (Pearson’s ρ > 0.4 and p-values < 0.001 ). Some pairs display weak and nonsignificant linear dependence. However, the LGC test still detects significant dependence in these cases, indicating the presence of nonlinear relationships that Pearson’s ρ could not capture. The LGC test’s ability to detect nonlinear dependencies as shown in LGC map on the top left and right of Figure 1 highlights its robustness in identifying more complex interactions between variables that traditional correlation measures may miss.

3.2 Dependence modeling using LGC-HMM framework for weekly claims of Motor and Home insurance LoBs

Figure 2 presents the LGC maps of weekly claim payouts for auto and homeowners insurance from Kenya and Norway before applying regime-switching models. The plots reveal nonlinear and heterogeneous dependencies across different claim amount ranges. For auto insurance, local correlations range from moderately negative ( − 0.52 ) to moderately positive (0.52), with positive dependence concentrated at lower payouts and negative dependence emerging when commercial claims are high and private claims are low. On the other hand, homeowners’ insurance shows strong positive dependence at moderate house and content payouts, shifting to negative dependence at extreme content claim levels. These localized patterns underscore the limitations of global dependence measures such as the Pearson correlation, which are − 0.003 for auto and 0.163 for homeowners insurance (Table 5). The LGC maps in Figure 2 and the LGC test with p-values < 0.00 reveal clear evidence of nonlinear dependence. This confirms that low Pearson correlations do not rule out meaningful dependence, particularly in nonlinear contexts [3,31,45]. These findings motivate the use of HMMs to capture time-varying and regime-specific dependencies, with results shown in Figures 3 and 4.

Figure 3

Two-regime time-varying HMM plot for automobile insurance LoBs.

Figure 4

Two-regime time-varying HMM plot of house and content insurance.

Figure 5

LGC plots of showing nature of time-varying dependence for each of the two regimes for Auto Insurance.

The Gaussian-HMM classified both datasets into two regimes as presented in Figures 3 and 4. In a supplementary material, we present results for the three-regime Gaussian HMM and the t-distribution HMM. By comparing the HMM classifications in Figure 3 with historical events in Kenya, it is found that the HMM can effectively identify crisis periods. Regime 1 is associated with the 2007–2008 Kenyan political crisis [12], the COVID-19 outbreak in 2021, and short periods in regime 1, which could be weeks with terrorist attacks [34] or strikes. Regime 2 appears to represent periods of economic stability or noncrisis periods and claim payout are higher in that period. For home insurance data in Figure 4, regime 1 indicates lower payouts compared to regime 2, without clear patterns except in the period around the end of 2015 to early 2016 where all observations were in regime 2. This outcome could be related to the fall of the oil price in 2015 that impacted Norway’s economy [25].

In Table 4, we compare regimes 1 and 2 by their average weekly claims payouts. Both show nonnormality, indicated by high skewness and kurtosis coefficients. Regime 2 has notably higher volatility, with larger variances and kurtosis for both LoBs. This is emphasized by higher standard errors in regime 2 compared to regime 1 in both LoBs and datasets.

In Table 5, the null hypothesis of independence was rejected in both regimes, with p-values < 0.05 . Notably, the Pearson correlation test indicates that the correlation is significantly different from zero, but negative for regime 2 and positive for regime 1. Interestingly, the nature of the dependence structure shown in Figures 5 suggests similar results.

Figure 6

LGC plots showing nature of time varying dependence for each of the two regimes in Home owners Insurance.

The results for testing whether the dependence structure between the claim payouts varies across the regimes using the test statistic described in Section 2.4 and detailed in equation (5) gives a p -value of 0.00 for both datasets, indicating a statistically significant difference in dependence between regimes 1 and 2 , leading to the rejection of the null hypothesis. Consequently, we conclude that the dependence structure of regime 1 differs from that of regime 2 aligning with expectations from distinct structures depicted in the LGC Maps (Figures 5 and 6).

Figure 7

VaR and TVaR at different confidence intervals for No Regime data.

The plots in Figure 5 show regime-specific LGC maps for auto data from Kenya. The results reveal that in regime 1, the local correlations are predominantly positive, particularly in the lower left tail, while they tend to be close to zero or negative in the lower right tail and upper left corner. This pattern indicates the existence of nonlinear tail dependence. In addition, Table 5 shows that in regime 2, the correlation is negative, with a Pearson correlation coefficient of − 0.237 , compared to the 0.168 in regime 1. Both correlation coefficients are statistically significant based on the standard Pearson correlation test. However, before implementing the LGC-HMM model, the global correlation coefficients were not statistically significant ( − 0.003 ) .

Furthermore, in regime 1, the LGC test for independence yields a p -value = 0.04, while in regime 2, it is 0.00, signifying dependence in both regimes. The findings suggest that automobile claims payouts exhibit nonlinear dependence during crisis periods in regime 1, with higher dependence for low payouts in the left lower tail, and that in regime 2, dependence is higher in the center, but the relationship is negative during economic stability periods. We presented a comparison to a three-regime HMM and presented the plots in the supplementary material and estimated AIC and BIC in Table A3 in Appendix A.3. The performance metrics indicate that we find that a two-regime HMM is preferred over 3 regime for the auto insurance data. Similar results are observed with the homeowners’ insurance data from Norway. The estimated parameters of the HMM model are presented in Table A1 in Appendix A.1.

The aforementioned analysis indicates that using HMMs and LGC to examine dependencies reveals a significant difference in the dependence patterns between the two regimes. These methodologies have enhanced our understanding of complex structures and temporal dynamics, helped us identify local correlations and transitions over time, and improved our comprehension of risk relationships.

3.3 Risk modeling using value-at-risk and tail value at risk

To quantify potential extreme losses in insurance claims, we employ two widely used risk measures. The VaR at confidence level α ∈ ( 0 , 1 ) for a loss random variable L is defined as follows:

The Tail Value-at-Risk (also called Expected Shortfall) at level α is defined as the expected loss given that the loss exceeds the VaR:

We analyzed weekly data from Kenya, combining claims from two business lines (motor private and motor commercial) into a single portfolio with equal weights. We modeled the joint distribution of claims using copulas and LGC densities, generating 50,000 simulated observations per model. These simulations were transformed using the inverse lognormal distribution with a log-mean of 0.5 and a log-standard deviation of 0.2. Based on the transformed data, we estimated VaR and TVaR at various confidence levels. Historical estimates of VaR and TVaR, for a model called “Empirical,” were included as benchmarks alongside the copula-based and LGC-based risk measures.

Table 6 reports 99% VaR and TVaR estimates in millions of Kenyan Shillings for several models under three scenarios: No Regime (full sample), Regime 1, and Regime 2. The regime structure is inferred from the fitted HMM in (Figure 3) and supported by LGC dependence maps (Figure 5). The LGC model delivers the highest VaR and TVaR estimates across the No regime and Regime 1 cases, confirming its sensitivity to changes in dependence structure and its ability to capture heightened tail risk, especially in Regime 1. In contrast, the historical VaRs, i.e., the “Empirical” model provide the lowest estimates, reflecting limited capacity to account for evolving dependence dynamics. The Independence model shows minimal variation across regimes, underscoring its inability to detect shifts in tail dependence. The Clayton and Gumbel copulas are excluded from Regime 2 due to their restriction to positive dependence, making them unsuitable for regimes with negative or weakening dependencies. The Frank copula performs moderately well, with lower estimates in Regime 2 reflecting a shift to less severe joint tail events. Figures 7, 8, and 9 visually demonstrate the LGC model’s ability to adapt to structural changes in dependence, with its curves consistently lying above other models in the tail regions, especially under regime-specific scenarios. The results in Figures 7–9 visually illustrate how risk estimates, both VaR and TVaR, increase with higher confidence intervals across all scenarios, as expected. In the No Regime and Regime 1 cases, the Independence copula yields consistently lower risk estimates due to its zero-dependence assumption, which fails to capture actual tail dependence and may thus underestimate extreme losses.

Figure 8

VaR and TVaR at different confidence intervals for Regime 1 data.

Figure 9

VaR and TVaR at different confidence intervals for Regime 2 data.

The empirical (historical) model also produces low estimates across all scenarios, as it does not incorporate structural dynamics or tail dependencies, limiting its responsiveness to shifts in joint behavior. In contrast, the LGC model demonstrates clear sensitivity to changing dependence structures, adapting its estimates to underlying regime characteristics and consistently producing higher, more cautious values. In Regime 2, where negative dependence dominates, the Independence model overestimates risk by ignoring diversification effects. The Frank copula adjusts more effectively, capturing the benefits of negative correlation and yielding significantly lower VaR and TVaR estimates. Notably, the LGC model remains robust across all regimes, flexibly responding to positive and negative dependence structures.