Rényi entropy

In information theory, the Rényi entropy is a quantity that generalizes various notions of entropy, including Hartley entropy, Shannon entropy, collision entropy, and min-entropy. The Rényi entropy is named after Alfréd Rényi, who looked for the most general way to quantify information while preserving additivity for independent events. In the context of fractal dimension estimation, the Rényi entropy forms the basis of the concept of generalized dimensions.

The Rényi entropy is important in ecology and statistics as index of diversity. The Rényi entropy is also important in quantum information, where it can be used as a measure of entanglement. In the Heisenberg XY spin chain model, the Rényi entropy as a function of $α$ can be calculated explicitly because it is an automorphic function with respect to a particular subgroup of the modular group. In theoretical computer science, the min-entropy is used in the context of randomness extractors.

Definition

Let $X$ be a discrete random variable with possible outcomes in the set $𝒜 = {x_{1}, x_{2}, ..., x_{n}}$ and corresponding probabilities $p_{i} ≐ Pr (X = x_{i})$ for ⁠ $i = 1, \dots, n$ ⁠. Then, the Rényi entropy of order ⁠ $α$ ⁠, where $0 < α < \infty$ and ⁠ $α \neq 1$ ⁠, is defined as $H_{α} (X) = \frac{1}{1 - α} \log (\sum_{i = 1}^{n} p_{i}^{α}) .$ It is further defined at $α = 0, 1, \infty$ as $H_{α} (X) = lim_{γ \to α} H_{γ} (X) .$

The resulting unit of information is determined by the base of the logarithm, e.g. shannon for base 2, or nat for base e. If the probabilities are $p_{i} = 1 / n$ for all ⁠ $i = 1, \dots, n$ ⁠, then all the Rényi entropies of the distribution are equal: ⁠ $H_{α} (X) = \log n$ ⁠. In general, for all discrete random variables ⁠ $X$ ⁠, $H_{α} (X)$ is a non-increasing function in ⁠ $α$ ⁠.

Applications often exploit the following relation between the Rényi entropy and the α-norm of the vector of probabilities: $H_{α} (X) = \frac{α}{1 - α} \log ({‖ P ‖}_{α}) .$ Here, the discrete probability distribution $P = (p_{1}, \dots, p_{n})$ is interpreted as a vector in $ℝ^{n}$ with $p_{i} \geq 0$ and $\sum_{i = 1}^{n} p_{i} = 1$ .

The Rényi entropy for any $α \geq 0$ is Schur concave, a property that can be proven by the Schur–Ostrowski criterion.

Special cases

As $α$ approaches zero, the Rényi entropy increasingly weighs all events with nonzero probability more equally, regardless of their probabilities. In the limit for ⁠ $α \to 0$ ⁠, the Rényi entropy is just the logarithm of the size of the support of $X$ . The limit for $α \to 1$ is the Shannon entropy. For $α = 2$ , the Rényi entropy is related to the collision probability, and the quantity $2^{H_{2}} = 1 / \sum p_{i}^{2}$ is known as the participation ratio, widely used in condensed matter physics and random matrix theory to measure the effective number of states contributing to a distribution. As $α$ approaches infinity, the Rényi entropy is increasingly determined by the events of highest probability.

Hartley or max-entropy

$H_{0} (X)$ is $\log n$ where $n$ is the number of non-zero probabilities. If the probabilities are all nonzero, it is simply the logarithm of the cardinality of the alphabet (⁠ $𝒜$ ⁠) of ⁠ $X$ ⁠, sometimes called the Hartley entropy of ⁠ $X$ ⁠, $H_{0} (X) = \log n = \log | 𝒜 |$

Shannon entropy

The limiting value of $H_{α}$ as $α \to 1$ is the Shannon entropy: $H_{1} (X) \equiv lim_{α \to 1} H_{α} (X) = - \sum_{i = 1}^{n} p_{i} \log p_{i}$

Collision entropy

Collision entropy, sometimes just called "Rényi entropy", refers to the case ⁠ $α = 2$ ⁠, $H_{2} (X) = - \log \sum_{i = 1}^{n} p_{i}^{2} = - \log P (X = Y),$ where $X$ and $Y$ are independent and identically distributed. The collision entropy is related to the index of coincidence. It is the negative logarithm of the Simpson diversity index. The quantity $2^{H_{2}} = 1 / \sum p_{i}^{2}$ is also known in random matrix theory as the participation ratio, where it measures the effective number of eigenstates contributing to a quantum state or, more generally, the effective number of eigenvalues contributing to a matrix spectrum.

Min-entropy

In the limit as ⁠ $α \to \infty$ ⁠, the Rényi entropy $H_{α}$ converges to the min-entropy ⁠ $H_{\infty}$ ⁠: $H_{\infty} (X) ≐ min_{i} (- \log p_{i}) = - (max_{i} \log p_{i}) = - \log max_{i} p_{i} .$

Equivalently, the min-entropy $H_{\infty} (X)$ is the largest real number $b$ such that all events occur with probability at most ⁠ $2^{- b}$ ⁠.

The name min-entropy stems from the fact that it is the smallest entropy measure in the family of Rényi entropies. In this sense, it is the strongest way to measure the information content of a discrete random variable. In particular, the min-entropy is never larger than the Shannon entropy.

The min-entropy has important applications for randomness extractors in theoretical computer science: Extractors are able to extract randomness from random sources that have a large min-entropy; merely having a large Shannon entropy does not suffice for this task.

Inequalities for different orders α

That $H_{α}$ is non-increasing in $α$ for any given distribution of probabilities ⁠ $p_{i}$ ⁠, which can be proven by differentiation, as $- \frac{d H_{α}}{d α} = \frac{1}{(1 - α)^{2}} \sum_{i = 1}^{n} z_{i} \log (z_{i} / p_{i}) = \frac{1}{(1 - α)^{2}} D_{KL} (z ‖ p)$ which is proportional to Kullback–Leibler divergence (which is always non-negative), where $z_{i} = p_{i}^{α} / \sum_{j = 1}^{n} p_{j}^{α}$ . In particular, it is strictly positive except when the distribution is uniform.

At the $α \to 1$ limit, we have $- \frac{d H_{α}}{d α} \to \frac{1}{2} \sum_{i} p_{i} {(\ln p_{i} + H (p))}^{2}$ .

In particular cases inequalities can be proven also by Jensen's inequality: $\log n = H_{0} \geq H_{1} \geq H_{2} \geq H_{\infty} .$

For values of ⁠ $α > 1$ ⁠, inequalities in the other direction also hold. In particular, we have $H_{2} \leq 2 H_{\infty} .$

On the other hand, the Shannon entropy $H_{1}$ can be arbitrarily high for a random variable $X$ that has a given min-entropy. An example of this is given by the sequence of random variables $X_{n} \sim {0, \dots, n}$ for $n \geq 1$ such that $P (X_{n} = 0) = 1 / 2$ and $P (X_{n} = x) = 1 / (2 n)$ since $H_{\infty} (X_{n}) = \log 2$ but ⁠ $H_{1} (X_{n}) = (\log 2 + \log 2 n) / 2$ ⁠.

Rényi divergence

As well as the absolute Rényi entropies, Rényi also defined a spectrum of divergence measures generalising the Kullback–Leibler divergence.

The Rényi divergence of order ⁠ $α$ ⁠ or alpha-divergence of a distribution $P$ from a distribution $Q$ is defined to be $\begin{matrix} D_{α} (P ‖ Q) & = \frac{1}{α - 1} \log (\sum_{i = 1}^{n} p_{i}^{α} q_{i}^{1 - α}) \\ = \frac{1}{α - 1} \log 𝔼_{i \sim p} [{(p_{i} / q_{i})}^{α - 1}] \end{matrix}$ when ⁠ $0 < α < \infty$ ⁠ and ⁠ $α \neq 1$ ⁠. We can define the Rényi divergence for the special values $α = 0, 1, \infty$ by taking a limit, and in particular the limit $α \to 1$ gives the Kullback–Leibler divergence.

Some special cases:

⁠ $D_{0} (P ‖ Q) = - \log Q ({i : p_{i} > 0})$ ⁠: minus the log probability under $Q$ that $p i > 0$ ;
⁠ $D_{1 / 2} (P ‖ Q) = - 2 \log \sum_{i = 1}^{n} \sqrt{p_{i} q_{i}}$ ⁠: minus twice the logarithm of the Bhattacharyya coefficient; (Nielsen & Boltz (2010))
⁠ $D_{1} (P ‖ Q) = \sum_{i = 1}^{n} p_{i} \log \frac{p_{i}}{q_{i}}$ ⁠: the Kullback–Leibler divergence;
⁠ $D_{2} (P ‖ Q) = \log ⟨ \frac{p_{i}}{q_{i}} ⟩$ ⁠: the log of the expected ratio of the probabilities;
⁠ $D_{\infty} (P ‖ Q) = \log sup_{i} \frac{p_{i}}{q_{i}}$ ⁠: the log of the maximum ratio of the probabilities.

The Rényi divergence is indeed a divergence, meaning simply that $D_{α} (P ‖ Q)$ is greater than or equal to zero, and zero only when $P = Q$ . For any fixed distributions $P$ and $Q$ , the Rényi divergence is nondecreasing as a function of its order $α$ , and it is continuous on the set of $α$ for which it is finite, or for the sake of brevity, the information of order $α$ obtained if the distribution $P$ is replaced by the distribution $Q$ .

Financial interpretation

A pair of probability distributions can be viewed as a game of chance in which one of the distributions defines official odds and the other contains the actual probabilities. Knowledge of the actual probabilities allows a player to profit from the game. The expected profit rate is connected to the Rényi divergence as follows $E x p e c t e d R a t e = \frac{1}{R} D_{1} (b ‖ m) + \frac{R - 1}{R} D_{1 / R} (b ‖ m),$ where $m$ is the distribution defining the official odds (i.e. the "market") for the game, $b$ is the investor-believed distribution and $R$ is the investor's risk aversion (the Arrow–Pratt relative risk aversion).

If the true distribution is $p$ (not necessarily coinciding with the investor's belief ⁠ $b$ ⁠), the long-term realized rate converges to the true expectation which has a similar mathematical structure $R e a l i z e d R a t e = \frac{1}{R} (D_{1} (p ‖ m) - D_{1} (p ‖ b)) + \frac{R - 1}{R} D_{1 / R} (b ‖ m) .$

Properties specific to α = 1

The value ⁠ $α = 1$ ⁠, which gives the Shannon entropy and the Kullback–Leibler divergence, is the only value at which the chain rule of conditional probability holds exactly: $H (A, X) = H (A) + 𝔼_{a \sim A} [H (X | A = a)]$ for the absolute entropies, and $D_{K L} (p (x | a) p (a) ‖ m (x, a)) = D_{K L} (p (a) ‖ m (a)) + 𝔼_{p (a)} {D_{K L} (p (x | a) ‖ m (x | a))},$ for the relative entropies.

The latter in particular means that if we seek a distribution $p (x, a)$ which minimizes the divergence from some underlying prior measure $m (x, a)$ , and we acquire new information which only affects the distribution of $a$ , then the distribution of $p (x | a)$ remains $m (x | a)$ , unchanged.

The other Rényi divergences satisfy the criteria of being positive and continuous, being invariant under 1-to-1 co-ordinate transformations, and of combining additively when $A$ and $X$ are independent, so that if $p (A, X) = p (A) p (X)$ , then $H_{α} (A, X) = H_{α} (A) + H_{α} (X)$ and $D_{α} (P (A) P (X) ‖ Q (A) Q (X)) = D_{α} (P (A) ‖ Q (A)) + D_{α} (P (X) ‖ Q (X)) .$

The stronger properties of the $α = 1$ quantities allow the definition of conditional information and mutual information from communication theory.

Exponential families

The Rényi entropies and divergences for an exponential family admit simple expressions $H_{α} (p_{F} (x; θ)) = \frac{1}{1 - α} (F (α θ) - α F (θ) + \log E_{p} [e^{(α - 1) k (x)}])$ and $D_{α} (p : q) = \frac{J_{F, α} (θ : θ^{'})}{1 - α}$ where $J_{F, α} (θ : θ^{'}) = α F (θ) + (1 - α) F (θ^{'}) - F (α θ + (1 - α) θ^{'})$ is a Jensen difference divergence.

Quantum information

The Rényi entropy can be generalized to the quantum case as

H_{α} (ρ) := \frac{1}{1 - α} \log tr (ρ^{α})

where $ρ$ is normalized density matrix. Its limit as $α \to 1$ is the von Neumann entropy.

The Rényi relative entropy (or divergence) can be generalized in two possible ways: the straightforward quantum Rényi relative entropy

D_{α} (ρ ‖ σ) = \frac{1}{α - 1} \log tr (ρ^{α} σ^{1 - α})

and the sandwiched Rényi relative entropy

{\tilde{D}}_{α} (ρ ‖ σ) = \frac{1}{α - 1} \log tr [{(σ^{\frac{1 - α}{2 α}} ρ σ^{\frac{1 - α}{2 α}})}^{α}]

Both converge to the quantum relative entropy in the limit $α \to 1$ . However, the sandwiched Rényi relative entropy has more convenient properties when used to defined conditional entropies, and has found applications in quantum key distribution.

Stabilizer Rényi entropy

One measure of quantum magic is the stabilizer Rényi entropy of order α:

M_{α} = \frac{1}{1 - α} \log [\frac{1}{2^{N}} \sum_{\hat{P} \in ℙ_{N}} ⟨ \hat{P} ⟩^{2 α}]

with $\hat{P}$ being the multi-qubit Pauli group and $⟨ \hat{P} ⟩ = ⟨ ψ | \hat{P} | ψ ⟩$ is an expectation value (quantum mechanics) for an $N$ -qubit system.

Ecology

Hill (1973) showed that the exponential of the Rényi entropy of order α defines a one-parameter family of diversity indices, now known as Hill numbers: ^αD = exp(H_α). For α = 0, this gives the species richness (total count of species present); for α = 1, the exponential of the Shannon entropy; and for α = 2, the inverse Simpson index, which counts the effective number of equally common species. Hill's framework unifies several previously disparate diversity measures as members of a single parametric family, differing only in how much weight is given to rare versus common species.

Notes

1 2 3 Rényi (1961)
↑ Rioul (2021)
↑ Barros, Vanessa; Rousseau, Jérôme (2021-06-01). "Shortest Distance Between Multiple Orbits and Generalized Fractal Dimensions". Annales Henri Poincaré. 22 (6): 1853–1885. arXiv:1912.07516. Bibcode:2021AnHP...22.1853B. doi:10.1007/s00023-021-01039-y. ISSN 1424-0661. S2CID 209376774.
↑ Franchini, Its & Korepin (2008)
↑ Its & Korepin (2010)
↑ RFC 4086, page 6
↑ Bromiley, Thacker & Bouhova-Thacker (2004)
↑ Bell, R. J.; Dean, P. (1970). "Atomic vibrations in vitreous silica". Discussions of the Faraday Society. 50: 55–61. doi:10.1039/df9705000055.
↑ Beck & Schlögl (1993)
↑ $H_{1} \geq H_{2}$ holds because ⁠ $\sum_{i = 1}^{M} p_{i} \log p_{i} \leq \log \sum_{i = 1}^{M} p_{i}^{2}$ ⁠.
↑ $H_{\infty} \leq H_{2}$ holds because ⁠ $\log \sum_{i = 1}^{n} p_{i}^{2} \leq \log {sup}_{i} p_{i} (\sum_{i = 1}^{n} p_{i}) = \log {sup}_{i} p_{i}$ ⁠.
↑ $H_{2} \leq 2 H_{\infty}$ holds because $\log \sum_{i = 1}^{n} p_{i}^{2} \geq \log {sup}_{i} p_{i}^{2} = 2 \log {sup}_{i} p_{i}$
↑ Devroye, Luc; Györfi, Laszlo; Lugosi, Gabor (1996-04-04). A Probabilistic Theory of Pattern Recognition (Corrected ed.). New York, NY: Springer. ISBN 978-0-387-94618-4.
1 2 Van Erven, Tim; Harremoës, Peter (2014). "Rényi Divergence and Kullback–Leibler Divergence". IEEE Transactions on Information Theory. 60 (7): 3797–3820. arXiv:1206.2459. Bibcode:2014ITIT...60.3797V. doi:10.1109/TIT.2014.2320500. S2CID 17522805.
1 2 Soklakov (2018)
↑ Nielsen & Nock (2011)
1 2 Müller-Lennert, Martin; Dupuis, Frédéric; Szehr, Oleg; Fehr, Serge; Tomamichel, Marco (1 December 2013). "On quantum Rényi entropies: A new generalization and some properties". Journal of Mathematical Physics. 54 (12) 122203. arXiv:1306.3142. Bibcode:2013JMP....54l2203M. doi:10.1063/1.4838856.
↑ Arqand, Amir; A. Hahn, Thomas; Y. -Z. Tan, Ernest (2024). "Generalized Rényi entropy accumulation theorem and generalized quantum probability estimation". Physical Review X. 15 (4): 041013. arXiv:2405.05912. Bibcode:2025PhRvX..15d1013A. doi:10.1103/pgrn-mz9j.
↑ Chung, Rebecca R. B.; Ng, Nelly H. Y.; Cai, Yu (14 July 2025). "Generalized numerical framework for improved finite-sized key rates with Rényi entropy". Physical Review A. 112 (1) 012612. arXiv:2502.02319. Bibcode:2025PhRvA.112a2612C. doi:10.1103/tyts-8v8j.
↑ Leone, L.; Oliviero, S. F. E.; Hamma, A. (2022). "Stabilizer Rényi Entropy". Physical Review Letters. 128 (5) 050402. arXiv:2106.12587. Bibcode:2022PhRvL.128e0402L. doi:10.1103/PhysRevLett.128.050402. S2CID 235670417.
↑ Hill, M. O. (1973). "Diversity and evenness: a unifying notation and its consequences". Ecology. 54 (2): 427–432. doi:10.2307/1934352.

References

Beck, Christian; Schlögl, Friedrich (1993). Thermodynamics of chaotic systems: an introduction. Cambridge University Press. ISBN 0-521-43367-3.
Jizba, P.; Arimitsu, T. (2004). "The world according to Rényi: Thermodynamics of multifractal systems". Annals of Physics. 312 (1): 17–59. arXiv:cond-mat/0207707. Bibcode:2004AnPhy.312...17J. doi:10.1016/j.aop.2004.01.002. S2CID 119704502.
Jizba, P.; Arimitsu, T. (2004). "On observability of Rényi's entropy". Physical Review E. 69 (2) 026128. arXiv:cond-mat/0307698. Bibcode:2004PhRvE..69b6128J. doi:10.1103/PhysRevE.69.026128. PMID 14995541. S2CID 39231939.
Bromiley, P.A.; Thacker, N.A.; Bouhova-Thacker, E. (2004), Shannon Entropy, Rényi Entropy, and Information, CiteSeerX 10.1.1.330.9856 {{citation}}: Cite uses deprecated parameter |citeseerx= (help)
Franchini, F.; Its, A. R.; Korepin, V. E. (2008). "Rényi entropy as a measure of entanglement in quantum spin chain". Journal of Physics A: Mathematical and Theoretical. 41 (25302) 025302. arXiv:0707.2534. Bibcode:2008JPhA...41b5302F. doi:10.1088/1751-8113/41/2/025302. S2CID 119672750.
"Rényi test", Encyclopedia of Mathematics, EMS Press, 2001 [1994]
Hero, A. O.; Michael, O.; Gorman, J. (2002). Alpha-divergence for Classification, Indexing and Retrieval (PDF) (Technical Report CSPL-328). Communications and Signal Processing Laboratory, University of Michigan.
Its, A. R.; Korepin, V. E. (2010). "Generalized entropy of the Heisenberg spin chain". Theoretical and Mathematical Physics. 164 (3): 1136–1139. Bibcode:2010TMP...164.1136I. doi:10.1007/s11232-010-0091-6. S2CID 119525704.
Nielsen, F.; Boltz, S. (2010). "The Burbea-Rao and Bhattacharyya centroids". IEEE Transactions on Information Theory. 57 (8): 5455–5466. arXiv:1004.5049. doi:10.1109/TIT.2011.2159046. S2CID 14238708.
Nielsen, Frank; Nock, Richard (2012). "A closed-form expression for the Sharma–Mittal entropy of exponential families". Journal of Physics A. 45 (3) 032003. arXiv:1112.4221. Bibcode:2012JPhA...45c2003N. doi:10.1088/1751-8113/45/3/032003. S2CID 8653096.
Nielsen, Frank; Nock, Richard (2011). "On Rényi and Tsallis entropies and divergences for exponential families". Journal of Physics A. 45 (3) 032003. arXiv:1105.3259. Bibcode:2012JPhA...45c2003N. doi:10.1088/1751-8113/45/3/032003. S2CID 8653096.
Rényi, Alfréd (1961). "On measures of information and entropy" (PDF). Proceedings of the fourth Berkeley Symposium on Mathematics, Statistics and Probability 1960. pp. 547–561.
Rosso, O. A. (2006). "EEG analysis using wavelet-based information tools". Journal of Neuroscience Methods. 153 (2): 163–182. doi:10.1016/j.jneumeth.2005.10.009. PMID 16675027. S2CID 7134638.
Zachos, C. K. (2007). "A classical bound on quantum entropy". Journal of Physics A. 40 (21): F407–F412. arXiv:hep-th/0609148. Bibcode:2007JPhA...40..407Z. doi:10.1088/1751-8113/40/21/F02. S2CID 1619604.
Nazarov, Y. (2011). "Flows of Rényi entropies". Physical Review B. 84 (10): 205437. arXiv:1108.3537. Bibcode:2015PhRvB..91j4303A. doi:10.1103/PhysRevB.91.104303. S2CID 40312624.
Ansari, Mohammad H.; Nazarov, Yuli V. (2015). "Rényi entropy flows from quantum heat engines". Physical Review B. 91 (10) 104303. arXiv:1408.3910. Bibcode:2015PhRvB..91j4303A. doi:10.1103/PhysRevB.91.104303. S2CID 40312624.
Ansari, Mohammad H.; Nazarov, Yuli V. (2015). "Exact correspondence between Rényi entropy flows and physical flows". Physical Review B. 91 (17) 174307. arXiv:1502.08020. Bibcode:2015PhRvB..91q4307A. doi:10.1103/PhysRevB.91.174307. S2CID 36847902.
Soklakov, A. N. (2020). "Economics of Disagreement—Financial Intuition for the Rényi Divergence". Entropy. 22 (8): 860. arXiv:1811.08308. Bibcode:2020Entrp..22..860S. doi:10.3390/e22080860. PMC 7517462. PMID 33286632.
Ansari, Mohammad H.; van Steensel, Alwin; Nazarov, Yuli V. (2019). "Entropy Production in Quantum is Different". Entropy. 21 (9): 854. arXiv:1907.09241. Bibcode:2019Entrp..21..854A. doi:10.3390/e21090854. S2CID 198148019.
Leone, L.; Oliviero, S. F. E.; Hamma, A. (2022). "Stabilizer Rényi Entropy". Physical Review Letters. 128 (5) 050402. arXiv:2106.12587. Bibcode:2022PhRvL.128e0402L. doi:10.1103/PhysRevLett.128.050402. S2CID 235670417.
Rioul, Olivier (2021). "This is IT: A Primer on Shannon's Entropy and Information" (PDF). Information Theory. Progress in Mathematical Physics. Vol. 78. Birkhäuser. pp. 49–86. doi:10.1007/978-3-030-81480-9_2. ISBN 978-3-030-81479-3. S2CID 204783328.

[Renyi61-1] 1 2 3 Rényi (1961)

[2] Rioul (2021)

[3] Barros, Vanessa; Rousseau, Jérôme (2021-06-01). "Shortest Distance Between Multiple Orbits and Generalized Fractal Dimensions". Annales Henri Poincaré. 22 (6): 1853–1885. arXiv:1912.07516. Bibcode:2021AnHP...22.1853B. doi:10.1007/s00023-021-01039-y. ISSN 1424-0661. S2CID 209376774.

[4] Franchini, Its & Korepin (2008)

[5] Its & Korepin (2010)

[6] RFC 4086, page 6

[7] Bromiley, Thacker & Bouhova-Thacker (2004)

[8] Bell, R. J.; Dean, P. (1970). "Atomic vibrations in vitreous silica". Discussions of the Faraday Society. 50: 55–61. doi:10.1039/df9705000055.

[Beck1993-9] Beck & Schlögl (1993)

[10] $H_{1} \geq H_{2}$ holds because ⁠ $\sum_{i = 1}^{M} p_{i} \log p_{i} \leq \log \sum_{i = 1}^{M} p_{i}^{2}$ ⁠.

[11] $H_{\infty} \leq H_{2}$ holds because ⁠ $\log \sum_{i = 1}^{n} p_{i}^{2} \leq \log {sup}_{i} p_{i} (\sum_{i = 1}^{n} p_{i}) = \log {sup}_{i} p_{i}$ ⁠.

[12] $H_{2} \leq 2 H_{\infty}$ holds because $\log \sum_{i = 1}^{n} p_{i}^{2} \geq \log {sup}_{i} p_{i}^{2} = 2 \log {sup}_{i} p_{i}$

[13] Devroye, Luc; Györfi, Laszlo; Lugosi, Gabor (1996-04-04). A Probabilistic Theory of Pattern Recognition (Corrected ed.). New York, NY: Springer. ISBN 978-0-387-94618-4.

[VanErvenHarremoes2014-14] 1 2 Van Erven, Tim; Harremoës, Peter (2014). "Rényi Divergence and Kullback–Leibler Divergence". IEEE Transactions on Information Theory. 60 (7): 3797–3820. arXiv:1206.2459. Bibcode:2014ITIT...60.3797V. doi:10.1109/TIT.2014.2320500. S2CID 17522805.

[Soklakov_2018-15] 1 2 Soklakov (2018)

[16] Nielsen & Nock (2011)

[tomamichel-17] 1 2 Müller-Lennert, Martin; Dupuis, Frédéric; Szehr, Oleg; Fehr, Serge; Tomamichel, Marco (1 December 2013). "On quantum Rényi entropies: A new generalization and some properties". Journal of Mathematical Physics. 54 (12) 122203. arXiv:1306.3142. Bibcode:2013JMP....54l2203M. doi:10.1063/1.4838856.

[18] Arqand, Amir; A. Hahn, Thomas; Y. -Z. Tan, Ernest (2024). "Generalized Rényi entropy accumulation theorem and generalized quantum probability estimation". Physical Review X. 15 (4): 041013. arXiv:2405.05912. Bibcode:2025PhRvX..15d1013A. doi:10.1103/pgrn-mz9j.

[19] Chung, Rebecca R. B.; Ng, Nelly H. Y.; Cai, Yu (14 July 2025). "Generalized numerical framework for improved finite-sized key rates with Rényi entropy". Physical Review A. 112 (1) 012612. arXiv:2502.02319. Bibcode:2025PhRvA.112a2612C. doi:10.1103/tyts-8v8j.

[20] Leone, L.; Oliviero, S. F. E.; Hamma, A. (2022). "Stabilizer Rényi Entropy". Physical Review Letters. 128 (5) 050402. arXiv:2106.12587. Bibcode:2022PhRvL.128e0402L. doi:10.1103/PhysRevLett.128.050402. S2CID 235670417.

[21] Hill, M. O. (1973). "Diversity and evenness: a unifying notation and its consequences". Ecology. 54 (2): 427–432. doi:10.2307/1934352.

v t e Statistical mechanics
Theory	Principle of maximum entropy ergodic theory
Statistical thermodynamics	Ensembles partition functions equations of state thermodynamic potential: U H F G Maxwell relations
Models	Ferromagnetism models Ising Potts Heisenberg percolation Particles with force field depletion force Lennard-Jones potential
Mathematical approaches	Boltzmann equation H-theorem Vlasov equation BBGKY hierarchy stochastic process mean-field theory and conformal field theory
Critical phenomena	Phase transition Critical exponents correlation length size scaling
Entropy	Boltzmann Gibbs Shannon Tsallis Rényi von Neumann
Applications	Statistical field theory elementary particle superfluidity Condensed matter physics Complex system chaos information theory Boltzmann machine