Upper bound for variance of finite mixtures of power exponential distributions

1 Introduction

Both variance and differential entropy are commonly used metrics to measure uncertainty of random variables. While variance measures how a random variable is spread out from its expected value, differential entropy takes a different approach. Differential entropy measures uncertainty from an information approach, it measures the average amount of information of a random variable. Although variance and entropy are generally not the same, there is one case when variance and entropy are equivalent measures. When normal distribution is considered, the maximum entropy given variance is attained, this suggests that variance and entropy are related to some extent. For comprehensive reviews on entropy, we refer to [2] and [7]. See also [4].

When the underlying probability distribution is multimodal, variance becomes inefficient in representing the uncertainty of random variables. For example, consider a finite mixture of two equally weighted univariate normal distributions with unit variance and different means μ₁ and μ₂. The variance of the overall distribution approaches ∞ as ∣μ₁ − μ₂∣ → ∞. On the other hand, differential entropy remains stable in this situation, approaching log(2πe)2+log(2) as ∣μ₁ − μ₂∣ → ∞. Therefore, differential entropy appears to be a more appropriate measure of uncertainty of multimodal random variables.

Let X denote a random variable with probability density function f. It is well known that given the value of differential entropy,

the lower bound of variance of X is achieved by normal distribution. That is, there exists a sharp lower bound on variance proportional to entropy power e^2h(f), as shown in (1):

This result can be extended (see, for example, [6]) for lower bounds of the kth absolute central moment such that the lower bound of the kth absolute central moment is proportional to e^{kh (f)} as shown in (2):

In this case, the lower bound is attained by power exponential distribution (or generalised normal distribution) with shape parameter β=k. The probability density function of power exponential distribution is

where β>0 is the shape parameter, α is the scale parameter, and −∞ < μ < ∞ is the location parameter.

In general, a similar form of upper bound is not known. There are many cases where the variance is not bounded given finite differential entropy. For instance, non-logconcave distributions, for example, Pareto distributions with shape parameter ⩽2, or finite mixture distributions, do not generally have an upper bound on variance given entropy. [1] considered a finite mixture of power exponential distributions with equal means; they showed that the variance has an upper bound proportional to entropy power, if the maximum ratio of variances between component distributions is finite.

Although multimodal distributions are generally not upper bounded by entropy power, we believe it would be useful to derive an upper bound under some specific conditions, giving further understanding between differential entropy and variance. In this note, we consider a special class of multimodal distributions, a multimodal finite mixture of power exponential distributions. Entropy of finite mixture distributions cannot be expressed in closed-form generally, and the evaluation relies on various approximation methods. For example, [3] proposed a method based on a Taylor-series expansion method to approximate entropy of Gaussian mixtures. We show that the variance has a upper bound proportional to e^2h(f), if ratios of variances, and ratios of squared means between all component distributions are finite.

The main result on the upper bound of variance is given in Section 2. A numerical study is presented in Section 3. An extension for kth absolute central moments is given in Section 4. Conclusions are given in Section 5.

2 Upper bound on variance for multimodal distributions

Consider f(x) specified as follows

where β_i > 0 are the shape parameters, α_i are the scale parameters, −∞ < μ_i < ∞ are the location parameters, n is the number of components and 0 < ω_i < 1 are weights with ω₁ + ⋅+ ω_n=1.

Let

3 Special cases

It is easy to see that ri2→0 when μ_i → 0, and (5) reduces to the original upper bounds shown in [1], i.e., (4). The variance of X becomes equal to (5) if all component distributions are identical with zero mean, i.e., σi2=σj2, β_i=β_j for all i, j=1, …, n and μi2=0 for i=1, …, n. In this case, the variance equals to 1A(2,β)e2h(f).

If only σi2=σj2 for all i, j=1, …, n then (5) reduces to

If only β_i=β_j for all i, j=1, ⋅, n then (5) reduces to

If only μ_i=μ_j for all i, j=1, …, n then (5) reduces to

If only μ_i=0 for all i=1, …, n then (5) reduces to

4 Upper bound on the kth absolute central moments for unimodal distributions

The variance upper bound for finite mixture of generalized normal distributions can be easily extended for kth absolute central moment. We consider a finite mixture of generalized normal distributions in (3).

A lower bound can also be obtained in the case of Theorem 4.1. Using (2), we obtain

If k=2, (7) reduces to

5 Conclusions

In this note, we have derived upper bounds on variance for a special class of multimodal distributions under some constraints. Closeness of the upper bound under different conditions have been presented. We have also derived upper bounds on the kth absolute central moment for a special class of unimodal distributions.