Log probability

In computer science, the use of log probabilities is a way to represent probabilities in a way which has several practical computational advantages over the standard use of approximated real numbers in the $[0, 1]$ interval.

Since the log of a number in $[0, 1]$ is negative, negative log probabilities are more commonly used. The most common log probability representation is therefore to encode a probability $x \in [0, 1]$ as $x' = - \log(x) \in \mathbb{R}$ . The product of probabilities $x \cdot y$ can then be replaced with the more efficient calculation $x' + y'$ . The sum of probabilities $x + y$ is more complex to express and is written as $- \log(e^{- x'} + e^{- y'})$ . However, in many applications a multiplication of probabilities (giving the probability of all independent events occurring) is used more often than their addition (giving the probability of at least one of them occurring). Additionally, the cost of computing the addition can be avoided in some situations by simply using the highest probability as an approximation. Since probabilities are nonnegative this gives a lower bound.

Representing probabilities in this way has two main advantages:

Speed. Since multiplication is more expensive than addition, taking the product of a high number of probabilities is faster if they are represented in log form. (The conversion to log form is expensive, but is only incurred once.)
Accuracy. The use of log probabilities improves numerical stability.

The use of log probabilities is widespread in several fields of computer science such as information theory and natural language processing as it represents the surprisal, the minimum length of the message that specifies the outcome in an optimally efficient code.