# Data Compression

Introduction :: Kraft :: Optimal Codes :: Bounds :: Huffman :: Coding

## Optimal codes

We deem a code to be optimal if it has the shortest average codeword length. The goal, after all, is to use the smallest number of bits to send the information. This may be regarded as an optimization problem. In designing the code, we must select the codeword lengths
so that that average length

is as short as possible (less than any other prefix code), subject to the constraint that the lengths satisfy the Kraft inequality (so it will be a prefix code). That is, minimize

subject

We will make two simplifying assumptions to get started: (1) we will neglect integer constraints on the codelengths; and (2) we will assume Kraft holds with equality. Then we can write a Lagrange-multiplier problem

Taking derivative with respect to

*l*

_{ j }and equating to zero

leads to

Substituting into the constraint,

so , and

*p*

_{ i }=

*D*

^{ - l i }

and the optimal codelengths are . (The

^{ * }denotes the optimal value.) Under this solution, the minimal average codeword length is

(The subscript

_{ D }denotes the log with respect to

*D*.)

Of course, in practice the codeword lengths must be integer values, so the result just obtained is a
*
lower bound
*
on the average codeword length. We will validate this lower bound in the following theorem: