##### Personal tools
•
You are here: Home Data Compression

# Data Compression

##### Document Actions

Introduction   ::   Kraft   ::   Optimal Codes   ::   Bounds   ::   Huffman   ::   Coding

## Optimal codes

We deem a code to be optimal if it has the shortest average codeword length. The goal, after all, is to use the smallest number of bits to send the information. This may be regarded as an optimization problem. In designing the code, we must select the codeword lengths so that that average length

is as short as possible (less than any other prefix code), subject to the constraint that the lengths satisfy the Kraft inequality (so it will be a prefix code). That is, minimize

subject

We will make two simplifying assumptions to get started: (1) we will neglect integer constraints on the codelengths; and (2) we will assume Kraft holds with equality. Then we can write a Lagrange-multiplier problem

Taking derivative with respect to l j and equating to zero

Substituting into the constraint,

so , and

p i = D - l i

and the optimal codelengths are . (The * denotes the optimal value.) Under this solution, the minimal average codeword length is

(The subscript D denotes the log with respect to D .)

Of course, in practice the codeword lengths must be integer values, so the result just obtained is a lower bound on the average codeword length. We will validate this lower bound in the following theorem:

Copyright 2008, by the Contributing Authors. Cite/attribute Resource . admin. (2006, May 17). Data Compression. Retrieved January 07, 2011, from Free Online Course Materials — USU OpenCourseWare Web site: http://ocw.usu.edu/Electrical_and_Computer_Engineering/Information_Theory/lecture6_3.htm. This work is licensed under a Creative Commons License