Personal tools
You are here: Home → Electrical and Computer Engineering → Information Theory → Definitions and Basic Facts

Definitions and Basic Facts

Document Actions
  • Send this
  • Print this
  • Content View
  • Bookmarks

Entropy Function   ::   Joint Entropy   ::   Relative Entropy   ::   Multivariable   ::   Convexity

Binary Entropy Function

We saw last time that the entropy of a random variable X is

\begin{displaymath}H(X) = -\sum_x p(x)\log p(x) \end{displaymath}

Suppose X is a binary random variable,

\begin{displaymath}X = \begin{cases}1 & \text{with probability } p \\ 0 & \text{with probability } 1-p \end{cases}\end{displaymath}

Then the entropy of X is

\begin{displaymath}H(X) = -p\log p -(1-p)\log(1-p) \end{displaymath}

Since this depends on p, this is also written sometimes as H(p). Plot. Observe: concave function of p. (What does this mean?) H(0) = 0, H(1) = 0. Why? Where is the max? More generally, the entropy of a binary discrete random variable with probability p is written as either H(X) or H(p).

Copyright 2008, Todd Moon. Cite/attribute Resource. admin. (2006, May 15). Definitions and Basic Facts. Retrieved August 28, 2008, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture2_1.htm. This work is licensed under a Creative Commons License. Creative Commons License
Reuse Course
Download this course