This paper
A Mathematical Theory of Communication
Entropy as the fundamental limit of compression. Mutual information. The channel capacity theorem.
← A Mathematical Theory of Communication
Idea lineage.
From Hartley's logarithmic measure to the entropy function that underpins every compression algorithm, cryptographic system, and neural network loss function.
Ancestors
- 1928
Transmission of Information
Information as the logarithm of the number of possible sequences.
- 1936
On Computable Numbers
The universal machine: any computation can be encoded as data.
- 1945
First Draft of a Report on the EDVAC
Programs and data share the same store — the stored-program concept.
Built on this
- 1949
Communication Theory of Secrecy Systems
Perfect secrecy: a cipher is unbreakable if the key has at least as much entropy as the message.
- 1959
Coding Theorems for a Discrete Source
The source coding theorem: optimal compression approaches entropy.
- 2006
A Fast and Frugal Elicitation of Probabilities
Shannon entropy as a measure of uncertainty in human decision-making.
- 2017
Attention Is All You Need
Cross-entropy loss. The model minimises prediction uncertainty, measured in bits.