Skip to main content

rel_entr

Function rel_entr 

Source
pub fn rel_entr(x_i: f64, y_i: f64) -> f64
Expand description

Elementwise function for computing relative entropy

Translated from scipy.special.rel_entr into pure Rust.

$$ \mathrm{rel\_entr}(x_i,\, y_i) = \begin{cases} x_i \ln(x_i / y_i) & x_i > 0 \wedge y_i > 0 \\ 0 & x_i = 0 \wedge y_i \ge 0 \\ \infty & \text{otherwise} \end{cases} $$

§Notes

This function is jointly convex in $x$ and $y$.

The origin of this function is in convex programming 1. Given two discrete probability distributions $p_1,\dotsc,p_n$ and $q_1,\dotsc,q_n$, the definition of relative entropy in the context of information theory is 2

$$ H\left[ \{ p_i \} \, \rvert \, \{ q_i \} \right] = \sum_i \mathrm{rel\_entr}(p_i,\, q_i). $$

§See also


  1. S. Boyd and L. Vandenberghe. Convex optimization. Cambridge University Press, 2004. https://doi.org/10.1017/CBO9780511804441 ↩

  2. Kullback-Leibler divergence, https://wikipedia.org/wiki/KL-divergence ↩