pub fn rel_entr(x_i: f64, y_i: f64) -> f64Expand description
Elementwise function for computing relative entropy
Translated from scipy.special.rel_entr into pure Rust.
$$ \mathrm{rel\_entr}(x_i,\, y_i) = \begin{cases} x_i \ln(x_i / y_i) & x_i > 0 \wedge y_i > 0 \\ 0 & x_i = 0 \wedge y_i \ge 0 \\ \infty & \text{otherwise} \end{cases} $$
§Notes
This function is jointly convex in $x$ and $y$.
The origin of this function is in convex programming 1. Given two discrete probability distributions $p_1,\dotsc,p_n$ and $q_1,\dotsc,q_n$, the definition of relative entropy in the context of information theory is 2
$$ H\left[ \{ p_i \} \, \rvert \, \{ q_i \} \right] = \sum_i \mathrm{rel\_entr}(p_i,\, q_i). $$
§See also
S. Boyd and L. Vandenberghe. Convex optimization. Cambridge University Press, 2004. https://doi.org/10.1017/CBO9780511804441 ↩
Kullback-Leibler divergence, https://wikipedia.org/wiki/KL-divergence ↩