Cross entropy
From Wikipedia, the free encyclopedia
In information theory, the cross entropy between two probability distributions measures the average number of bits needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution q, rather than the "true" distribution p.
The cross entropy for two distributions p and q over the same probability space is thus defined as follows:
![\mathrm{H}(p, q) = \mathrm{E}_p[-\log q] = \mathrm{H}(p) + D_{\mathrm{KL}}(p \| q)\!](../../../math/0/a/b/0ab52f3e2d2aa97a20317c6a92d976c7.png)
,
where H(p) is the entropy of p, and DKL(p | | q) is the Kullback-Leibler divergence of q from p (also known as the relative entropy).
For discrete p and q this means
The situation for continuous distributions is analogous:
NB: The notation H(p,q) is sometimes used for both the cross entropy as well as the joint entropy of p and q.
When comparing a distribution q against a fixed reference distribution p, cross entropy and KL divergence are essentially the same concept. In fact, they are identical up to an additive constant (since p is fixed): both take on their minimal values when p = q, which is 0 for KL divergence, and H(p) for cross entropy. In the engineering literature, the principle of minimising KL Divergence (Kullback's "Principle of Minimum Discrimination Information") is therefore often alternatively called the Principle of Minimum Cross-Entropy (MCE), or Minxent for short.
However, as discussed in the article Kullback-Leibler divergence, it is very often the distribution q which is the fixed prior reference distribution; and the distribution p which it is desired to optimise to be as close to q as possible, subject to some constraint. In this case the two minimisations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be DKL(p||q) , rather than H(p,q). Readers should beware.
Cross-entropy minimization is frequently used in optimization and rare-event probability estimation; see the cross-entropy method.
