Data Compression

Document Actions

Introduction :: Kraft :: Optimal Codes :: Bounds :: Huffman :: Coding

Optimal codes

We deem a code to be optimal if it has the shortest average codeword length. The goal, after all, is to use the smallest number of bits to send the information. This may be regarded as an optimization problem. In designing the code, we must select the codeword lengths $l_1,l_2,\ldots, l_m$ so that that average length

$\begin{displaymath}L = \sum_i p_i l_i \end{displaymath}$

is as short as possible (less than any other prefix code), subject to the constraint that the lengths satisfy the Kraft inequality (so it will be a prefix code). That is, minimize

$\begin{displaymath}L = \sum_i p_i l_i \end{displaymath}$

subject

$\begin{displaymath}\sum D^{-l_i} \leq 1. \end{displaymath}$

We will make two simplifying assumptions to get started: (1) we will neglect integer constraints on the codelengths; and (2) we will assume Kraft holds with equality. Then we can write a Lagrange-multiplier problem

$\begin{displaymath}J = \sum_i p_i l_i + \lambda\sum_i D^{-l_i}. \end{displaymath}$

Taking derivative with respect to l_j and equating to zero

$\begin{displaymath}\frac{\partial J}{\partial l_j} = p_j - \lambda D^{-l_j} \log D = 0 \end{displaymath}$

leads to

$\begin{displaymath}D^{-l_j} = \frac{p_j}{\lambda\log D} \end{displaymath}$

Substituting into the constraint,

$\begin{displaymath}\sum_i \frac{p_i}{\lambda \log D} = 1 \end{displaymath}$

so $\lambda = 1/\log D$ , and

p_i = D^-l_i

and the optimal codelengths are $l_i^* = -\log_D p_i$ . (The ^* denotes the optimal value.) Under this solution, the minimal average codeword length is

$\begin{displaymath}L^* = \sum_i p_i l_i^* = H_D(X). \end{displaymath}$

(The subscript _D denotes the log with respect to D.)

Of course, in practice the codeword lengths must be integer values, so the result just obtained is a lower bound on the average codeword length. We will validate this lower bound in the following theorem:
$\begin{theorem} The expected length $L$\ of any instantaneous $D$-ary code for ... ...X) \end{displaymath}with equality if and only if $D^{-l_i} = p_i$. \end{theorem}$

$\begin{proof} Write \begin{displaymath} \begin{aligned} L-H_D(X) &= \sum_i p_i... ...e relative entropy is nonnegative and $c\leq 1$\ (Kraft inequality). \end{proof}$

Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). Data Compression. Retrieved August 28, 2008, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture6_3.htm. This work is licensed under a Creative Commons License.

Course Contents Information Theory Home About the Professor Syllabus Schedule Homework Assignments Programming Assignments Download This Course	Personal tools Log in You are here: Home → Electrical and Computer Engineering → Information Theory → Data Compression
	Info Data Compression Document Actions Introduction :: Kraft :: Optimal Codes :: Bounds :: Huffman :: Coding Optimal codes We deem a code to be optimal if it has the shortest average codeword length. The goal, after all, is to use the smallest number of bits to send the information. This may be regarded as an optimization problem. In designing the code, we must select the codeword lengths $l_1,l_2,\ldots, l_m$ so that that average length $\begin{displaymath}L = \sum_i p_i l_i \end{displaymath}$ is as short as possible (less than any other prefix code), subject to the constraint that the lengths satisfy the Kraft inequality (so it will be a prefix code). That is, minimize $\begin{displaymath}L = \sum_i p_i l_i \end{displaymath}$ subject $\begin{displaymath}\sum D^{-l_i} \leq 1. \end{displaymath}$ We will make two simplifying assumptions to get started: (1) we will neglect integer constraints on the codelengths; and (2) we will assume Kraft holds with equality. Then we can write a Lagrange-multiplier problem $\begin{displaymath}J = \sum_i p_i l_i + \lambda\sum_i D^{-l_i}. \end{displaymath}$ Taking derivative with respect to l_j and equating to zero $\begin{displaymath}\frac{\partial J}{\partial l_j} = p_j - \lambda D^{-l_j} \log D = 0 \end{displaymath}$ leads to $\begin{displaymath}D^{-l_j} = \frac{p_j}{\lambda\log D} \end{displaymath}$ Substituting into the constraint, $\begin{displaymath}\sum_i \frac{p_i}{\lambda \log D} = 1 \end{displaymath}$ so $\lambda = 1/\log D$ , and p_i = D^-l_i and the optimal codelengths are $l_i^* = -\log_D p_i$ . (The ^* denotes the optimal value.) Under this solution, the minimal average codeword length is $\begin{displaymath}L^* = \sum_i p_i l_i^* = H_D(X). \end{displaymath}$ (The subscript _D denotes the log with respect to D.) Of course, in practice the codeword lengths must be integer values, so the result just obtained is a lower bound on the average codeword length. We will validate this lower bound in the following theorem: $\begin{theorem} The expected length $L$\ of any instantaneous $D$-ary code for ... ...X) \end{displaymath}with equality if and only if $D^{-l_i} = p_i$. \end{theorem}$ $\begin{proof} Write \begin{displaymath} \begin{aligned} L-H_D(X) &= \sum_i p_i... ...e relative entropy is nonnegative and $c\leq 1$\ (Kraft inequality). \end{proof}$ Copyright 2008, by the Contributing Authors. Cite/attribute Resource. admin. (2006, May 17). Data Compression. Retrieved August 28, 2008, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture6_3.htm. This work is licensed under a Creative Commons License.	Reuse Course Download this course

Sections

Personal tools

Data Compression

Document Actions

Optimal codes