VSPACE50PT NOINDENT HUGEBFSERIES PREFACE VSKIP 20PTCHAPTERPREFACESECTIONWHY THIS BOOKTHE PURPOSE OF THIS BOOK IS TO BRIDGE THE GAP BETWEENINTRODUCTORYLEVEL SIGNAL PROCESSING CLASSES AND THE MATHEMATICSPREVALENT IN CURRENT SIGNAL PROCESSING RESEARCH AND PRACTICE  THE GAPIS BRIDGED BY PROVIDING A UNIFIED EM APPLIED TREATMENT OFFUNDAMENTAL MATHEMATICS SEASONED WITH DEMONSTRATIONS USING SC  MATLAB  THIS BOOK INTENDED NOT ONLY FOR STUDENTS OF SIGNALPROCESSING STILL PURSUING THEIR FORMAL EDUCATION BUT ALSO FORPRACTICING ENGINEERS WHO NEED TO BE ABLE TO ACCESS THE SIGNALPROCESSING RESEARCH LITERATURE AND FOR RESEARCHERS LOOKING FOR APARTICULAR RESULT THAT THEY WANT TO APPLY  IT IS THUS INTENDED BOTHAS A BF TEXTBOOK AND AS A BF REFERENCE  THE THEORY AND PRACTICE OF SIGNAL PROCESSING CONTRIBUTE TO AND DRAWFROM A VARIETY OF DISCIPLINES AMONG THEM CONTROLS COMMUNICATIONSSYSTEM IDENTIFICATION INFORMATION THEORY ARTIFICIAL INTELLIGENCESPECTROSCOPY PATTERN RECOGNITION TOMOGRAPHY IMAGE ANALYSIS ANDDATA ACQUISITION  TO FULFILL ITS ROLE IN THESE DIVERSE AREAS SIGNALPROCESSING EMPLOYS A VARIETY OF MATHEMATICAL TOOLS INCLUDINGTRANSFORM THEORY PROBABILITY OPTIMIZATION DETECTION THEORYESTIMATION THEORY NUMERICAL ANALYSIS LINEAR ALGEBRA FUNCTIONALANALYSIS AND MANY OTHERS  THE PRACTITIONER OF SIGNAL PROCESSING THE SIGNAL PROCESSOR  MAY USE SEVERAL OF THESE TOOLS IN THESOLUTION OF A PROBLEM FOR EXAMPLE BY SETTING UP A SIGNALRECONSTRUCTION ALGORITHM AND THEN OPTIMIZING THE PARAMETERS OF THEALGORITHM FOR OPTIMUM PERFORMANCE  MOST PRACTICING SIGNAL PROCESSORSMUST HAVE KNOWLEDGE OF BOTH THE BF THEORY AND THE BF  IMPLEMENTATION OF THE MATHEMATICS HOW AND WHY IT WORKS AND HOW TOMAKE THE COMPUTER DO IT  THE BREADTH OF MATHEMATICS EMPLOYED INSIGNAL PROCESSING COUPLED WITH THE OPPORTUNITY TO APPLY THE MATH TOPROBLEMS OF ENGINEERING INTEREST MAKES THE FIELD BOTH INTERESTING ANDREWARDINGTHE MATHEMATICAL ASPECTS OF SIGNAL PROCESSING ALSO INTRODUCE SOME OFITS MAJOR CHALLENGES HOW IS A STUDENT OR ENGINEERING PRACTITIONER TOBECOME VERSED IN THE VARIETY OF MATHEMATICAL TECHNIQUES WHILE STILLKEEPING AN EYE TOWARD THE APPLICATIONS  INTRODUCTORY TEXTS ON SIGNALPROCESSING TEND TO FOCUS HEAVILY ON TRANSFORM TECHNIQUES ANDFILTERBASED APPLICATIONS  WHILE AN ESSENTIAL PART OF THE TRAINING OFA SIGNAL PROCESSOR THIS FOCUS REVEALS ONLY THE TIP OF THE ICEBERG OFMATERIAL REQUIRED BY A LIFELONG PRACTICING ENGINEER  MORE ADVANCEDTEXTS USUALLY DEVELOP THE MATHEMATICAL TOOLS SPECIFIC TO A NARROWASPECT OF SIGNAL PROCESSING WHILE PERHAPS MISSING CONNECTIONS OFTHESE IDEAS TO RELATED AREAS OF RESEARCH  NEITHER OF THESE APPROACHESPROVIDES THE BACKGROUND NECESSARY TO READ AND UNDERSTAND BROADLY INTHE SIGNAL PROCESSING RESEARCH LITERATURE NOR TO EQUIP A PERSON WITHMANY SIGNAL PROCESSING TOOLSOVER THE YEARS THE SIGNAL PROCESSING LITERATURE HAS MOVED TOWARDINCREASING SOPHISTICATION  AS EXAMPLES APPLICATIONS OF THE SINGULARVALUE DECOMPOSITION SVD OR WAVELET TRANSFORMS ABOUND EVERYONE KNOWSSOMETHING ABOUT THESE BY NOW OR SHOULD  PART OF THIS MOVE TOWARDSOPHISTICATION IS FUELED BY COMPUTERS SINCE COMPUTATIONS FORMERLYREQUIRING CONSIDERABLE EFFORT AND UNDERSTANDING ARE NOW EMBODIED INCONVENIENT MATHEMATICAL PACKAGES  NAIVELY VIEWED THIS AUTOMATIONTHREATENS THE EXPERTISE OF THE ENGINEER WHY HIRE SOMEONE TO DO WHATCAN BE DONE IN TEN MINUTES WITH A SC MATLAB TOOLBOX  VIEWED MOREPOSITIVELY THE POWER OF THE COMPUTER PROVIDES A VARIETY OF NEWOPPORTUNITIES AS ENGINEERS ARE FREED FROM COMPUTATIONAL DRUDGERY TOPURSUE NEW APPLICATIONS  COMPUTER SOFTWARE AVAILABLE NOW PROVIDESPLATFORMS UPON WHICH INNOVATIVE IDEAS MAY BE DEVELOPED WITH GREATEREASE THAN EVER BEFORE  TAKING ADVANTAGE OF THE NEW FREEDOM TO DEVELOPUSEFUL CONCEPTS WILL REQUIRE A SOLID UNDERSTANDING OF MATHEMATICSBOTH TO APPRECIATE WHAT IS IN THE TOOLBOXES AND TO EXTEND BEYOND THEMTHIS BOOK IS INTENDED TO PROVIDE A FOUNDATION TO THE REQUISITEMATHEMATICSONE WAY FOR ASPIRINGPRACTITIONERS OF SIGNAL PROCESSING TO GET THE MATHEMATICAL BACKGROUNDTHEY NEED IS SIMPLY TO TAKE MORE MATHEMATICS CLASSES  WHILERECOMMENDED AS AN IDEAL FOR MANY SUCH A PROGRAM IS IMPRACTICAL THEYMAY FIND A COURSE IN PURE MATH TOO FAR REMOVED FROM THEIR OR THEIREMPLOYERS NEED FOR PRACTICAL KNOWLEDGEBEGINQUOTESOURCEWENDELL BERRYEM RECOLLECTED ESSAYS 19651980    P 197WHAT IS IT GOOD FOR WE ASK  AND ONLY IF IT PROVESIMMEDIATELY TO BE GOOD EM FOR SOMETHING ARE WE READY TO RAISE THEQUESTION OF VALUE HOW MUCH IS IT WORTH  BUT WE MEAN HOW MUCH MONEYFOR IF IT CAN ONLY BE GOOD FOR SOMETHING ELSE THEN OBVIOUSLY IT CANONLY BE EM WORTH SOMETHING ELSE  EDUCATION BECOMES TRAINING ASSOON AS WE DEMAND IN THIS SPIRIT THAT IT SERVE SOME IMMEDIATEPURPOSE AND THAT IT BE WORTH A PREDETERMINED AMOUNT  ONCE WE ACCEPTSO SPECIFIC A NOTION OF UTILITY ALL LIFE BECOMES SUBSERVIENT TO ITSUSE ITS VALUE IS DRAINED INTO ITS USE  ENDQUOTESOURCEBEGINQUOTESOURCEPATRICK BILLINGSLEYPREFACE P V EMPROBABILITY AND MEASURE 1986EDWARD DAVENANT SAID HE WOULD HAVE A MAN KNOCKT IN THE HEAD THATSHOULD WRITE ANYTHING IN MATHEMATIQUES THAT HAD BEEN WRITTEN OFBEFORE  LDOTS WHAT IS NEW HERE THENENDQUOTESOURCEBEGINQUOTESOURCEHENRY DAVID THOREAU  FOR EVERY THOUSAND HACKING AT THE LEAVES OF EVIL THERE IS ONE  STRIKING AT THE ROOTENDQUOTESOURCETHE LEVEL OF THIS BOOK ASSUMES THAT STUDENTS HAVE HAD A COURSE INTRADITIONAL TRANSFORMBASED DSP AT THE SENIOR OR FIRSTYEAR GRADUATELEVEL AND ALSO A TRADITIONAL COURSE IN STOCHASTIC PROCESSES  WHILECONCEPTS IN THESE AREAS ARE REVIEWED THIS BOOK DOES NOT SUPPLANT THEMORE FOCUSED COVERAGE THAT THESE COURSES CAN PROVIDESECTIONFEATURES OF THE BOOKSOME HIGHLIGHTS OF THE BOOK INCLUDEBEGINITEMIZEITEM AN EMPHASIS ON VECTORSPACE GEOMETRY WHICH PUTS LEASTSQUARES  AND MINIMUM MEANSQUARES IN THE SAME FRAMEWORK  THE CONCEPT OF  SIGNALS AS VECTORS IN AN APPROPRIATE VECTOR SPACE IS EMPHASIZED  THE VECTOR SPACE APPROACH PROVIDES A NATURAL FRAMEWORK FOR TOPICS  SUCH AS WAVELET TRANSFORMS AND DIGITAL COMMUNICATIONS AS WELL AS  THE TRADITIONAL TOPICS SUCH AS OPTIMUM PREDICTION FILTERING AND  ESTIMATION  IN THIS CONTEXT THE MORE GENERAL NOTION OF METRIC  SPACES IS INTRODUCED WITH A DISCUSSION OF SIGNAL NORMSITEM A THOROUGH DESCRIPTION OF THE LINEAR ALGEBRA USED IN SIGNAL  PROCESSING BOTH IN CONCEPT AND IN NUMERICAL IMPLEMENTATION  WHILE  LIBRARIES ARE COMMONLY AVAILABLE TO DO LINEAR ALGEBRA COMPUTATIONS  WE FEEL THAT THE NUMERICAL TECHNIQUES PRESENTED EXERCISE INTUITION  ON THE GEOMETRY OF VECTOR SPACES AND BUILD UNDERSTANDING OF THE  ISSUES THAT MUST BE ADDRESSED IN PRACTICAL PROBLEMS  THE LINEAR ALGEBRA INCLUDES A THOROUGH DISCUSSION OF EIGENBASED  METHOD OF COMPUTATION INCLUDING EIGENFILTERS MUSIC AND ESPRIT  THERE IS ALSO A CHAPTER DEVOTED TO THE PROPERTIES AND APPLICATIONS  OF THE SVD  TOEPLITZ MATRICES WHICH APPEAR THROUGHOUT THE SIGNAL  PROCESSING LITERATURE ARE TREATED BOTH FROM A NUMERICAL POINT OF  VIEW  AS AN EXAMPLE OF RECURSIVE ALGORITHMS  AND ALSO IN  CONJUNCTION WITH THE LATTICEFILTERING INTERPRETATION      THE MATRICES IN LINEAR ALGEBRA ARE VIEWED AS OPERATORS AND THE  IMPORTANT CONCEPT OF AN OPERATOR IS INTRODUCED  ASSOCIATED NOTIONS  SUCH AS RANGE NULLSPACE AND NORM OF AN OPERATOR ARE PRESENTED  WHILE A FULL COVERAGE OF OPERATOR THEORY IS NOT PROVIDED THERE IS A  STRONG FOUNDATION HERE THAT SERVES TO BUILD INSIGHT FOR OTHER  OPERATORS  ITEM IN ADDITION TO THE LINEAR ALGEBRAIC CONCEPTS A DISCUSSION OF  EM COMPUTATION IS ALSO PRESENTED  ALGORITHMS FOR COMPUTING THE  COMMON FACTORIZATIONS EIGENVALUES EIGENVECTORS SVDS AND MANY  OTHERS ARE PRESENTED WITH SOME NUMERICAL CONSIDERATION FOR  IMPLEMENTATION  WHILE NOT ALL OF THIS MATERIAL IS NECESSARILY  INTENDED FOR CLASSROOM USE IN CONVENTIONAL SIGNAL PROCESSING CLASSES   THERE IS NOT TIME FOR ALL OF THIS IN MOST CLASSES  THE  MATERIAL PROVIDES AN IMPORTANT PERSPECTIVE TO PERSPECTIVE  PRACTITIONERS AND A STARTING POINT FOR IMPLEMENTATIONS ON OTHER  PLATFORMS  INSTRUCTORS MAY CHOOSE TO EMPHASIZE CERTAIN NUMERIC  CONCEPTS BECAUSE THEY HIGHLIGHT THE GEOMETRY OF VECTOR SPACES  ITEM THE CAUCHYSCHWARTZ INEQUALITY IS USED IN A VARIETY OF PLACES AS  AN OPTIMIZING PRINCIPLEITEM RLS AND LMS ADAPTIVE FILTERS ARE PRESENTED AS NATURAL OUTGROWTHS  OF MORE FUNDAMENTAL CONCEPTS MATRIX INVERSE UPDATES AND STEEPEST  DESCENT  NEURAL NETWORKS AND BLIND SOURCE SEPARATION ARE ALSO  PRESENTED AS AN APPLICATION OF STEEPEST DESCENTITEM SEVERAL CHAPTERS ARE DEVOTED TO ITERATIVE AND RECURSIVE METHODS  EMPLOYED IN SIGNAL PROCESSING  WHILE ITERATIVE METHODS ARE OF GREAT  THEORETICAL AND PRACTICAL SIGNIFICANCE NO OTHER SIGNAL PROCESSING  TEXTBOOK PROVIDES THIS BREADTH OF COVERAGE  METHODS PRESENTED  INCLUDE PROJECTION ON CONVEX SETS COMPOSITE MAPPING THE EM  ALGORITHM CONJUGATE GRADIENT AND METHODS OF MATRIX INVERSE  COMPUTATION USING ITERATIVE METHODSITEM DETECTION AND ESTIMATION ARE PRESENTED WITH SEVERAL  APPLICATIONS INCLUDING SPECTRUM ESTIMATION PHASE ESTIMATION AND  MULTIDIMENSIONAL DIGITAL COMMUNICATIONSITEM OPTIMIZATION IS A KEY CONCEPT ON SIGNAL PROCESSING AND EXAMPLES  OF OPTIMIZATION BOTH UNCONSTRAINED AND CONSTRAINED APPEAR  THROUGHOUT THE TEXT  A THEORETICAL JUSTIFICATION FOR LAGRANGE  MULTIPLIER METHODS AS WELL AS THEIR PHYSICAL INTERPRETATION ARE  EXPLICITLY SPELLED OUT IN A CHAPTER ON OPTIMIZATION  A SEPARATE  CHAPTER DISCUSSES LINEAR PROGRAMMING AND ITS APPLICATIONSITEM IN ADDITION OPTIMIZATION ON GRAPHS SHORTEST PATH PROBLEMS ARE  ALSO EXAMINED ALONG WITH A VARIETY OF APPLICATIONS IN  COMMUNICATIONS AND SIGNAL PROCESSINGITEM THE EM ALGORITHM IS PRESENTED HERE THE ONLY TREATMENT KNOWN IN  A SIGNAL PROCESSING TEXTBOOK  THIS POWERFUL ALGORITHM IS USED FOR  MANY OTHERWISE INTRACTABLE ESTIMATION AND LEARNING PROBLEMSENDITEMIZETHE PRESENTATION IS AT A MORE FORMAL LEVEL THAN HAS BECOME TRADITIONALIN MANY RECENT DSP BOOKS FOLLOWING A THEOREMPROOF FORMATTHROUGHOUT THE TEXT  AT THE SAME TIME IT IS LESS FORMAL THAN MANYMATH BOOKS COVERING THIS MATERIAL  IN THIS WE HAVE ATTEMPTED TO HELPTHE STUDENTS FEEL COMFORTABLE WITH RIGOROUS THINKING WITHOUTOVERWHELMING THE STUDENT WITH TECHNICALITIES  A BRIEF REVIEW OFMETHODS OF PROOFS IS ALSO PROVIDED TO HELP STUDENTS DEVELOP A SENSE OFHOW TO APPROACH PROOFS  ULTIMATELY THE AIM OF THE BOOK IS TOEDUCATE ITS READER IN HOW TO THINK ABOUT PROBLEMS  TO THIS END INSOME PLACES MATERIAL IS COVERED MORE THAN ONCE FROM DIFFERENTPERSPECTIVES EG MORE THAN ONE PROOF FOR SOME RESULTS TODEMONSTRATE THAT THERE IS USUALLY MORE THAN ONE WAY TO APPROACH APROBLEMTHROUGHOUT THE TEXT THE INTENT HAS BEEN TO EXPLAIN THE WHAT ANDWHY OF THE MATHEMATICS BUT WITHOUT BECOMING OVERWROUGHT WITH SOMEOF THE MORE TECHNICAL MATHEMATICAL OCCUPATIONS  IN THIS REGARD THEBOOK DOES NOT NECESSARILY THOROUGHLY TREAT QUESTIONS OF HOW WELLFOR EXAMPLE IN OUR COVERAGE OF LINEAR NUMERICAL ANALYSIS THEPERTURBATION ANALYSIS THAT CHARACTERIZES MUCH OF THE RESEARCHLITERATURE HAS BEEN LARGELY IGNORED  NOR DO ISSUES OF COMPUTATIONALCOMPLEXITY FORM A MAJOR CONSIDERATION  CONSIDER THIS AUTOMOTIVEANALOGY  OUR INTENT IS TO GET UNDER THE HOOD OF THE CAR TO ASUFFICIENT LEVEL THAT IT IS CLEAR WHY THE ENGINE RUNS AND WHAT IT CANDO BUT WITHOUT PROVIDING A MOLECULARLEVEL DESCRIPTION OF THEMETALLURGICAL STRUCTURE OF THE PISTON RINGS  SUCH FINEGRAINEDINVESTIGATIONS ARE A NECESSARY PART OF THE RESEARCH INTO FINETUNINGTHE PERFORMANCE OF THE ENGINE  OR THE ALGORITHM  BUT ARE NOTAPPROPRIATE FOR A READER LEARNING THE MECHANICSTHROUGHOUT THE BOOK AND IN THE APPENDICES THERE IS ALSO GREAT DEAL OFMATERIAL THAT WILL BE OF REFERENCE VALUE TO PRACTICING ENGINEERS  FOREXAMPLE THERE ARE FACTS REGARDING MATRIX RANK THE INVERTIBILITY OFMATRICES PROPERTIES OF HERMITIAN MATRICES PROPERTIES OF STRUCTUREDMATRICES PRESERVED UNDER MULTIPLICATION AND AN EXTENSIVE TABLE OFGRADIENTS  NOT ALL OF THIS MATERIAL IS NECESSARILY INTENDED FORCLASSROOM USE IN CONVENTIONAL SIGNAL PROCESSING CLASSES BEINGPROVIDED TO ENHANCE THE VALUE OF THE BOOK AS A REFERENCENEVERTHELESS WHERE SUCH REFERENCE MATERIAL IS PROVIDED IT IS USUALLYACCOMPANIED BY AN EXPLANATION OF THE DERIVATION SO THAT RELATED FACTSNOT LISTED MAY OFTEN BE DERIVED BY THE READER  THE INTENT ALWAYS ISTO EDUCATE AND EMPOWER THE READER NOT SIMPLY PROVIDE THE ANSWERBEGINQUOTESOURCEWENDELL BERRYRECOLLECTED ESSAYS 19651980    P X FINALLY I WOULD LIKE TO ALERT THE READER TO MY CONVICTION  THAT THIS IS A PIECE OF UNFINISHED BUSINESS AND THAT MORE TIME AND  WORK WILL REVEAL FURTHER NEED OF CORRECTIONENDQUOTESOURCE NEWLENGTHKNUTHLENGTH SETTOWIDTHKNUTHLENGTH EM VOLUME I FUNDAMENTAL ALGORITHMS PP VIIVIII BEGINQUOTESOURCEDONALD KNUTHPARBOXTKNUTHLENGTHEM THE ART       OF COMPUTER PROGRAMMING PAR     EM VOLUME I FUNDAMENTAL ALGORITHMS PP VIIVIII MY ORIGINAL GOAL WAS TO BRING READERS TO FRONTIERS OF KNOWLEDGE IN EVERY SUBJECT THAT WAS TREATED  BUT IT IS EXTREMELY DIFFICULT TO KEEP UP WITH A FIELD THAT IS ECONOMICALLY PROFITABLE DOTS THE SUBJECT HAS BECOME A VAST TAPESTRY OF TENS OF THOUSANDS OF SUBTLE RESULTS CONTRIBUTED BY TENS OF THOUSANDS OF PEOPLE ALL OVER THE WORLD THEREFORE MY NEW GOAL HAS BEEN TO CONCENTRATE ON CLASSIC TECHNIQUES LIKELY TO REMAIN IMPORTANT FOR MANY MORE DECADES AND TO DESCRIBE THEM AS WELL AS I CAN ENDQUOTESOURCEAS WITH KNUTHS BOOK WHILE THIS BOOK WILL NOT PROVIDE THE FINAL WORD IN ANY RESEARCH AREAWE HOPE THAT FOR MANY RESEARCH PATHS IT WILL AT LEAST PROVIDE A GOODFIRST STEP  THE CONTENTS OF THE BOOK HAVE BEEN SELECTED ACCORDING TOA VARIETY OF CRITERIA  THE PRIMARY SELECTION CRITERION IS WHETHERMATERIAL HAS BEEN OF USE OR INTEREST TO US IN OUR RESEARCH  QUESTIONSFROM STUDENTS AND THE NEED TO FIND A CLEAR EXPLANATION FOR THEM HAVELEAD TO INCLUSION OF OTHER MATERIAL  THE EXCEPTIONAL WRITINGS FOUNDIN OTHER TEXTBOOKS AND PAPERS HAS BEEN A FACTOR  SOME OF THE MATERIALHAS BEEN INCLUDED FOR ITS PRACTICALITY AND SOME FOR ITS OUTSTANDINGBEAUTYTHERE IS ONGOING DEBATE REGARDING THE TEACHING OF MATHEMATICS TOENGINEERS  RECENT PROPOSALS SUGGEST USING JUST IN TIMEMATHEMATICS PROVIDING THE MATHEMATICAL CONCEPT ONLY WHEN THE NEED FORIT ARISES IN THE SOLUTION OF ENGINEERING PROBLEMS  THIS APPROACH HASARISEN AS A RESPONSE TO THE CHARGE THAT MATHEMATICAL PEDAGOGY HAS BEENPRESENTED USING A JUST IN CASE APPROACH WELL TEACH YOU ALL THISSTUFF JUST IN CASE YOU EVER HAPPEN TO NEED IT  IN REALITY NEITHER OFTHESE APPROACHES ARE EITHER FULLY DESIRABLE OR ACHIEVABLE POTENTIALLYLACKING RIGOR AND DEPTH ON THE ONE HAND AND LACKING MOTIVATION ANDINSIGHT ON THE OTHER  AS AN ALTERNATIVE WE HOPE THAT THEPRESENTATION IN THIS BOOK IS JUSTIFIED SO THAT THE LEVEL OFMATHEMATICS IS SUITED TO ITS APPLICATION AND THE APPLICATIONS ARESEEN IN CONJUNCTION WITH THE CONCEPTSIN ADDITION TO ATTEMPTING TO PROVIDE THOROUGH EXPLANATIONS OF MANYCORE TOPICS WE ALSO ATTEMPT TO PLANT SOME SEEDS OF IDEAS  THESEINCLUDE SUCH TOPICS AS COMMUTATIVE DIAGRAMS INFINITE PRODUCTSINCIDENCE MATRICES INFORMATION THEORY AND GRAPH THEORY  WE HAVEALSO ATTEMPTED IN THE TO EXPLAIN SOME OF THE LIMITATIONS OF THEMETHODS AND TO PROVIDE REFERENCES TO ALTERNATIVE TECHNIQUES  ALSOSINCE A MATERIAL IS LEARNED BEST BY APPRECIATING ITS CREATOR WE HAVEPROVIDED A FEW HISTORICAL VIGNETTES  THESE HAVE BEEN DRAWN MOSTLYFROM CITEBOYER CITEOTHERMATHHIST AND CITEMATHUNIVTHE GOAL OF PROVIDING A THOROUGH COVERAGE OF THE CONCEPTS IS FAR FROMACHIEVED IN THIS VOLUME  ALONG THE WAY WE WERE FORCED TO JETTISONENTIRE PARTS OF THE BOOK IN THE INTEREST IN OBTAINING AN OSTENSIBLYPORTABLE BOOK  FOR THOSE WITH AN INTEREST IN NUMBER THEORYPOLYNOMIAL THEORY INTERPOLATION AND APPROXIMATION INTEGRALEQUATIONS OR A VARIETY OF OTHER TOPICS WE EXPRESS OUR REGRETSSECTIONTHE PROGRAMSTHROUGHOUT THE TEXT THERE ARE MANY ALGORITHMS WRITTEN IN SC MATLABTHESE EXAMPLES WILL ALLOW THE READER TO SEE HOW THE CONCEPTS DEVELOPEDIN THE TEXT MIGHT BE IMPLEMENTED ALLOW EASY EXPLORATION OF THECONCEPTS AND SOMETIMES THE LIMITATIONS OF THE THEORY AND SHOULDPROVIDE A USEFUL LIBRARY OF CORE FUNCTIONALITY FOR A VARIETY OF SIGNALPROCESSING RESEARCH  WITH THE THOROUGH THEORETICAL AND APPLIEDDISCUSSION SURROUNDING AN ALGORITHM THIS BOOK IS NOT SIMPLY A RECIPEBOOK BUT THE INGREDIENTS ARE PROVIDED TO STIR UP SOME INTERESTINGSTEWSIN MOST CASES THE ALGORITHMS ARE NOT TYPESET IN THE BOOK  INSTEADTHE ICON PARNOINDENT INCLUDEGRAPHICSPICON NOINDENTIS USED TO INDICATE THAT AN ALGORITHM IS TO BE FOUND ON THE INCLUDEDCDROM  IN SOME INSTANCES THE ALGORITHM CONSISTS OF SEVERAL RELATEDFILESIN THE INTEREST OF BREVITY TYPE CHECKING OF ARGUMENTS HAS NOT BEENINCORPORATED INTO THE FUNCTIONS  OTHERWISE THE CODE IS BELIEVED TOWORK AT LEAST TO PRODUCE THE EXAMPLES DESCRIBED IN THE BOOK BUTBUGFIXES AND IMPROVEMENTS ARE ALWAYS WELCOMEWE MAKE THE STANDARD DISCLAIMER OF WARRANTY BF WE MAKE NO  WARRANTY EXPRESS OR IMPLIED THAT THE PROGRAMS OR ALGORITHMS  PRESENTED IN THIS BOOK OR ITS ACCOMPANYING MEDIA ARE FREE OF  ERROR OR THAT THEY WILL MEET YOUR REQUIREMENTS FOR ANY PARTICULAR  APPLICATIONS  THEY SHOULD NOT BE RELIED UPON FOR SOLVING A PROBLEM  WHOSE INCORRECT SOLUTION COULD RESULT IN INJURY TO A PERSON OR LOSS  OF PROPERTY  ANY AND ALL USE OF THE PROGRAMS AND ALGORITHMS  ASSOCIATED WITH THIS BOOK IS AT YOUR OWN RISK  THE AUTHORS AND  PUBLISHER DISCLAIM ALL LIABILITY FOR DIRECT OR CONSEQUENTIAL DAMAGES  RESULTING FROM YOUR USE OF THE PROGRAMSYOU ARE FREE TO USE THE PROGRAMS OR ANY DERIVATIVE OF THEM FOR ANYSCIENTIFIC PURPOSE BUT PLEASE REFERENCE THIS BOOK  UPDATED VERSIONSOF THE PROGRAMS AND OTHER INFORMATION CAN BE FOUND AT THE WEBSITE VERBSOME WEBSITE ADDRESS PROBABLY WWWPRENHALLCOMMOON  ANNA SHOULD  BE LOOKING INTO THISSECTIONEXERCISESEXERCISES ARE FOUND AT THE END OF EACH CHAPTER  THESE EXERCISES ARELOOSELY DIVIDED INTO SECTIONS BUT IT MAY BE NECESSARY TO DRAW FROMMATERIAL IN OTHER SECTIONS OR EVEN OTHER CHAPTERS IN ORDER TO SOLVESOME OF THE PROBLEMS  THERE ARE RELATIVELY FEW MERELY NUMERICAL EXERCISES  WITH THECOMPUTER DOING AUTOMATED COMPUTATIONS IN MANY CASES SIMPLY RUNNINGTHE NUMBERS DOESNT SEEM TO PROVIDE INFORMATIVE EXERCISES  READERSARE ENCOURAGED OF COURSE TO PLAY AROUND WITH THE ALGORITHMS PROVIDEDTO GET A SENSE OF HOW THEY WORK  INSIGHT CAN FREQUENTLY GAINED ONSOME DIFFICULT PROBLEMS BY TRYING SEVERAL RELATED NUMERICAL EXAMPLESTHE INTENT OF THE EXERCISES IS TO ENGAGE TO READER IN THE DEVELOPMENTOF THE THEORY IN THE BOOK  MANY OF THE EXERCISES ARE TO PROVIDEDERIVATIONS FOR RESULTS PRESENTED IN THE CHAPTERS OR TO PROVE SOME OFTHE LEMMAS AND THEOREMS  OTHER EXERCISES REQUIRE PROGRAMMING ANEXTENSION OR MODIFICATION OF A SC MATLAB ALGORITHM PRESENTED IN THECHAPTER  STILL OTHER EXERCISES LEAD THE STUDENT THROUGH ASTEPBYSTEP PROCESS LEADING TO SOME SIGNIFICANT RESULTS FOR EXAMPLEA DERIVATION OF GAUSSIAN QUADRATURE A DERIVATION OF LINEAR PREDICTIONTHEORY EXTENSION OF INVERSES OF TOEPLITZ MATRICES OR ANOTHERDERIVATION OF THE KALMAN FILTER  WE HOPE THAT AS STUDENTS WORKTHROUGH THESE EXERCISES THEY WILL DEVELOP SKILL IN ORGANIZING THEIRTHINKING TO APPROACH OTHER PROBLEMS AS WELL AS ACQUIRE BACKGROUND INA VARIETY OF IMPORTANT TOPICSMOST OF THE EXERCISES REQUIRE A FAIR DEGREE OF INSIGHT AND EFFORT TOSOLVE  STUDENTS SHOULD PLAN ON BEING CHALLENGED  WHEREVER POSSIBLESTUDENTS SHOULD BE ENCOURAGED TO INTERACT WITH THE COMPUTER FORCOMPUTATIONAL ASSISTANCE INSIGHT AND FEEDBACKA SOLUTIONS MANUAL IS AVAILABLE TO INSTRUCTORS TO INSTRUCTORS WHO HAVEADOPTED THE BOOK FOR CLASSROOM USE  NOT ONLY ARE SOLUTIONS PROVIDEDBUT IN MANY CASES SC MATLAB AND SC MATHEMATICA CODE IS ALSOPROVIDED INDICATING HOW A PROBLEM MIGHT BE APPROACHED USING THECOMPUTER  BY PROVIDING GUIDANCE INTO HOW TO APPROACH THE PROBLEM THESOLUTIONS MANUAL CAN ALSO BE A VALUABLE RESOURCE FOR STUDENTS OFSIGNAL PROCESSING  SOLUTIONS TO SOME OF THE EXERCISES CAN BE FOUND ONTHE CDROM IE ANSWERS AT THE BACK OF THE BOOK THERE ARE SEVERAL DIFFERENT TYPES OF EXERCISES  SOME ARE MERELY COMPUTATIONAL  OTHERS INTRODUCE EXTENSIONS OF THE METHODS OF THE SECTION TO NEW PROBLEMS  IN SOME CASES THE EXERCISES ARE USED TO PRESENT NEW MATERIAL  OTHER EXERCISES REQUIRE THE STUDENT TO PROVE RESULTS USED IN THE TEXT WHILE OTHERS REQUIRE PROGRAM IMPLEMENTATION AND EVALUATION OF A CONCEPTSELECTION OF EXERCISES BY AN INSTRUCTOR CAN BE MADE ON THE BASIS OFTHE LEVEL OF PREPARATION OF THE STUDENTS AND THE AMOUNT OF TIME ASTUDENT IS EXPECTED TO SPEND WORKING PROBLEMSSECTIONPOSSIBLE COURSES OF STUDYTHERE IS SUFFICIENT MATERIAL HERE THAT A VARIETY OF USEFUL COURSESCOULD BE PUT TOGETHER USING THIS BOOK  THERE IS CLEARLY MOREINFORMATION IN THIS BOOK THAN CAN BE COVERED IN A SINGLE SEMESTER OREVEN A FULL YEAR  SEVERAL DIFFERENT COURSES OF STUDY COULD BE DEVISEDBASED ON THIS BOOK AND INSTRUCTORS ARE PROVIDED THE OPPORTUNITY TOCHOOSE THE MATERIAL SUITABLE FOR THE NEEDS AND DEVELOPMENT OF THEIRSTUDENTS  FOR EXAMPLE DEPENDING ON THE FOCUS OF THE CLASSINSTRUCTORS MAY CHOOSE TO SKIP COMPLETELY THE NUMERICAL ASPECTS OFALGORITHMS OR THEY MAY CHOOSE THEM AS A FOCUS OF THE COURSEHERE ARE SOME POSSIBLE COURSE OPTIONSBEGINENUMERATEITEM THE MATERIAL IN THE FIRST TWO PARTS IS REGARDED AS FOUNDATIONAL  UPON WHICH THE MAJOR CONCEPTS OF SIGNAL PROCESSING ARE BUILT  THE  FIRST PART PROVIDES A REVIEW OF SIGNAL MODELS AND REPRESENTATIONS  EG DIFFERENCE EQUATIONS TRANSFER FUNCTIONS STATE SPACE FORM  AND INTRODUCES SEVERAL IMPORTANT SIGNAL PROCESSING PROBLEMS SUCH AS  SPECTRUM ESTIMATION AND SYSTEM IDENTIFICATION  THE SECOND PART  PROVIDES A THOROUGH FOUNDATION IN LINEAR ALGEBRA WORKING FROM AN  UNDERGRADUATE LEVEL UP THROUGH SEVERAL APPLICATIONS  SELECTIONS  FROM THESE FIRST TWO PARTS WITH POSSIBLE ADDITIONS FROM THE FIRST  APPENDIX ON MATHEMATICAL FUNDAMENTALS WOULD MAKE A SOLID  SINGLESEMESTER COURSE FOR A COURSE TITLED SOMETHING LIKE  MATHEMATICAL METHODS FOR SIGNALS AND SYSTEMS A POSSIBLE COURSE  SEQUENCE FOR SUCH A COURSE MIGHT BE AS FOLLOWS  BEGINITEMIZE  ITEM MOVE FAIRLY QUICKLY THROUGH CHAPTER 1 12 WEEKS  SOME MAY    WISH TO ENTIRELY SKIP SECTIONS 18 AND 110 DEPENDING ON INTEREST  ITEM IN CHAPTER 2 MOVE QUICKLY TO THE VECTOR SPACE CONCEPTS THEN    FOCUS ON THE CONCEPT OF ORTHOGONALITY  FOR MANY CLASSES IT MAY    BE USEFUL TO SKIP THE MORE TECHNICAL SECTIONS ASSOCIATED WITH    INFINITEDIMENSIONAL VECTOR SPACES FOR EXAMPLE SECTIONS 212    213 AND 216  APPROX 2 WEEKS  ITEM SPEND TIME IN CHAPTER 3 ON LEASTSQUARES AND MINIMUM    MEANSQUARE FILTERING AND ESTIMATION CONCEPTS AND THE DUAL    APPROXIMATION PROBLEM SECTIONS 31314  23 WEEKS  THEN    DEPENDING ON INTEREST EXAMINE EITHER WAVELET TRANSFORMS OR    DIGITAL COMMUNICATIONS FROM THIS GEOMETRIC VIEWPOINT 1 WEEK  ITEM IN CHAPTER 4 FOCUS ON SECTIONS 41 THROUGH 45 TO GET THE    GEOMETRY OF THE OPERATORS THEN 49 FOR A RETURN TO THE    LEASTSQUARES IDEA AND 410 FOR PRACTICAL COMPUTATION ISSUES    THEN INTRODUCE THE RLS FILTER IN SECTION 411 AND VISIT    PARTITIONED MATRIX INVERSES IN SECTION 412  23 WEEKS  ITEM IN CHAPTER 5 FOCUS ON SECTIONS 52 AND 53  THE QR    FACTORIZATION IN PARTICULAR IS A FOUNDATION FOR MANY SIGNAL    PROCESSING ALGORITHMS  IF A NUMERIC IMPLEMENTATION VIEWPOINT IS    NOT OF INTEREST THEN MATERIAL AFTER SECTION 535 MAY BE OMITTED    23 WEEKS  ITEM SECTIONS 61  65 CONSTITUTE THE PRINCIPAL THEORY OF THE    CHAPTER  AFTER THESE SECTIONS HAVE BEEN COVERED APPLICATIONS    DRAWN FROM SECTIONS 67 THROUGH 612 WITH 68 AND 69 ARE    PROBABLY OF THE MOST INTEREST  IF A NUMERIC FOCUS IS DESIRED    SECTION 614 MAY BE COVERED 23 WEEKS  ITEM THE THEORY OF THE SVD IN SECTIONS 71 THROUGH 75 SHOULD BE    COVERED FOLLOWED BY A SUBSET OF APPLICATIONS FROM SECTIONS 76    THROUGH 79  23 WEEKS  ITEM TOPICS RELATED TO SPECIAL MATRICES WITH SPECIAL EMPHASIS ON    TOEPLITZ MATRICES CAN FILL THE REMAINING TIME   ENDITEMIZEITEM THE MATERIAL FROM CHAPTERS 10 THROUGH 14 WOULD FIT WELL INTO A  FIRST COURSE ON DETECTION AND ESTIMATION ESPECIALLY WHEN  SUPPLEMENTED BY SOME OF THE MATERIAL ON LINEAR ALGEBRA SUCH AS  EIGENDECOMPOSITIONS AND THE SINGULAR VALUE DECOMPOSITION  ITEM AN ALTERNATE WAY OF USING THE BOOK IS IN A ONESEMESTER TOOLS  COURSE WHICH SELECTS TOPICS FROM PARTS I II AND III  ASSUMING  FAMILIARITY WITH CONTINUOUSTIME AND DISCRETETIME SYSTEMS TOPICS  IN THIS COURSE COULD INCLUDE  BEGINENUMERATE  ITEM THE MULTIVARIATE GAUSSIAN DENSITY SECTION 17  1 WEEK  ITEM ESSENTIAL VECTOR SPACE NOTIONS SECTIONS 21 THROUGH 26    210 213 214 AND 215 2 WEEKS  ITEM APPLICATIONS OF VECTOR SPACE CONCEPTS EG LEASTSQUARES AND    MINIMUM MEANSQUARES FILTERING SECTIONS 31 32 34 38    THROUGH 312 3 WEEKS  ITEM MATRIX FACTORIZATIONS SECTIONS 52 AND 53 NO NUMERIC    DISCUSSION  1 WEEK  ITEM SINGULAR VALUE DECOMPOSITIONS SECTIONS 71 72 73 75    WITH SOME APPLICATIONS SUCH AS SECTION 76 2 WEEKS  ITEM INTRODUCTION TO DETECTION AND ESTIMATION SECTIONS 101 102    103 105 106 1 WEEK  ITEM DETECTION THEORY SECTIONS 111 THROUGH 116    3 WEEKS  ITEM ESTIMATION THEORY SECTIONS 121 122 124 125 126 2    WEEKS  ITEM KALMAN FILTERING SECTIONS 131 132 OR 133 1 WEEK  ENDENUMERATEITEM ANOTHER COURSE COULD BE ITERATIVE METHODS FOR SIGNAL  PROCESSING WHICH WOULD FOCUS ON CHAPTERS IN PART IV  THE COURSE  MATERIAL COULD WELL BE ACCOMPANIED BY A STUDENT RESEARCH PROJECTITEM ANOTHER COURSE COULD BE METHODS OF OPTIMIZATION FOR SIGNAL  PROCESSING WHICH WOULD FOCUS ON CHAPTERS IN PART VITEM YET ANOTHER ALTERNATIVE IS A WRAPUP COURSE FOR STUDENTS IN THE  SIGNAL AND SYSTEM AREA WHO ARE FAMILIAR WITH THEIR TOPIC AREAS AND  WISH TO SHARPEN THEIR ANALYTICAL SKILLS SOMEWHAT  THIS COURSE COULD  BE SIMILAR TO THE FIRST ONE MENTIONED WITH LESS TIME SPENT IN  CHAPTER 1 AND MORE TIME SPENT EXAMINING NUMERICAL IMPLEMENTATIONS  TOPICS FROM THE LAST PARTS OF THE BOOK COULD ALSO BE SELECTEDENDENUMERATESECTIONACKNOWLEDGEMENTSBEGINQUOTESOURCEISAAC NEWTONIF I HAVE SEEN FURTHER IT IS BY STANDING ON YE SHOULDERS OFGIANTSI DO NOT KNOW WHAT I MAY APPEAR TO THE WORLD BUT TO MYSELF I SEEM TOHAVE BEEN ONLY LIKE A BOY PLAYING ON THE SEASHORE AND DIVERTINGMYSELF IN NOW AND THEN FINDING A SMOOTHER PEBBLE OR A PRETTIER SHELLTHAN ORDINARY WHILST THE GREAT OCEAN OF TRUTH LAY UNDISCOVERED BEFORE MEENDQUOTESOURCEFOR PROVIDING A CHALLENGING AND STIMULATING ENVIRONMENT IN WHICH THEDEVELOPMENT OF THIS BOOK COULD OCCUR I OFFER MY APPRECIATION TO THELATE DR RICHARD HARRIS CHAIRMAN OF THE ELECTRICAL AND COMPUTERENGINEERING DEPARTMENT AT UTAH STATE UNIVERSITY WHO PASSED AWAYSUDDENLY AS THE BOOK WAS NEARING COMPLETION  FOR SUGGESTIONSCOMMENTS AND MUCHNEEDED CRITICISM THE COMMENTS OF MANY REVIEWERSARE APPRECIATED  PAUL BECKER AND HIS ERSTWHILE GROUP ATADDISONWESLEYLONGMAN HAS PROVIDED FRIENDLY ENCOURAGEMENT AND IT HASBEEN A PLEASURE WORKING WITH HIM  THE PRODUCTION STAFF AT INTERACTIVECOMPOSITION CORPORATION HAVE BEEN MONUMENTALLY PRODUCTIVE AND I THANKTHEM MAKING THIS ALL COME TOGETHERFOR STIMULATING AND BAFFLING CONVERSATIONS AND QUESTIONS I THANK MYSTUDENTS  I AM GRATEFUL FOR COMMENTS SUGGESTIONS ENCOURAGEMENTSAND ADVICE FROM FRIENDS AND COLLEAGUES WHO HAVE READ PORTIONS OF THISAND PROVIDED INPUT  THE PART ON DETECTION AND ESTIMATION THEORY COMES FROM WYNN STIRLINGAND I AM GRATEFUL AND HONORED THAT HIS NOTES CAN BE INCORPORATED INTOTHIS BOOK AND FOR THE OPPORTUNITY TO COLLABORATE WITH HIMDESPITE THE ASSISTANCE REVIEWS OVERSIGHT AND EDITING OF MANYPEOPLE I AM SURE THERE STILL LURK UNDETECTED ERRORS  THESE ARE MINEAND I DEEPLY REGRET THEM  IF YOU FIND ANY PLEASE LET ME KNOW SOTHEY CAN BE STAMPED OUTTO THOSE WHO HAVE PLAYED ON THE SHORES OF KNOWLEDGE AND FOUND SO MANYBRILLIANT SHELLS I EXTEND ENTHUSIASTIC APPRECIATION  I ALSO THANKTHOSE WHO BY THEIR WRITING AND INTERPRETATIONS BY THEIR TEACHING ANDDEDICATION HAVE EXTENDED MY VIEWS BY HELPING ME CLIMB UP TOWARD THESHOULDERS OF THE GIANTS  MY PARENTS HAVE INSTILLED IN ME THECURIOSITY AND WONDER ABOUT THE WORLD AROUND ME  TO MY MOTHER THANKSFOR AN INSATIABLE CURIOSITY ABOUT LIFE  TO MY FATHER THANKS FORPROVIDING THE PATTERNMY MOST HEARTFELT THANKS GO TO BARBARA WHO MORE THAN ANYONE HASSHOULDERED WITH ME THE BURDEN OF SEEING THIS THROUGH AND HAS SHARED MEWITH THIS BOOK  SHE ALSO APPRECIATES THE NEED TO KNOW  THANKS ALSOTO OUR CHILDREN  LESLIE KYRA KAYLIE JENNIE KIANA AND SPENCER WHO PROVIDE MORE THAN SUFFICIENT REASON FOR JOY IN MY LIFE 1EMHFILL  TKM LOCAL VARIABLES TEXMASTER TEST END INTRODUCTORY CHAPTERBEGINTABBING MMQUAD  MMQUAD  MMQUAD  MMQUAD  KILL WHY THIS BOOK    MATHEMATICAL AREAS ENCOMPASSED AND EXAMPLE APPLICATIONS  TO BUILD INSIGHT AND MATURITY  TO PROVIDE A WINDOW ON SIGNAL PROCESSING LITERATURE MATHEMATICAL TOPICS ENCOMPASSED BY DSP OUTLINE AND STRUCTURE OF THE BOOK SOME CANONICAL PROBLEMS AND MODELS  THE MULTIVARIATE NORMAL MODEL   SYSTEM MODELING AND IDENTIFICATION  PREDICTION FILTERING AND SPECTRAL ESTIMATION  TRANSFER FUNCTION AND STATESPACE FORMS  STOCHASTIC MODELING HIDDEN MARKOV MODELS  SIGNAL DETECTION AND ESTIMATION MODAL ANALYSIS ARRAY PROCESSING  ESTIMATION KALMAN FILTERING  TIMEFREQUENCY ANALYSIS WAVELETSCHAPTERINTRODUCTIONLABELCHAPINTROBEGINQUOTESOURCEHUGH NIBLEYEM APPROACHING ZIONTHERE IS FULLTIME EMPLOYMENT FOR ALL SIMPLY IN EXPLORING THE WORLDWITHOUT DESTROYING IT AND BY THE TIME WE BEGIN TO UNDERSTANDSOMETHING OF ITS MARVELOUS RICHNESS AND COMPLEXITY WELL ALSO BEGINTO SEE THAT IT DOES HAVE USES WE NEVER SUSPECTEDLDOTSENDQUOTESOURCEBEGINQUOTESOURCEMICHAEL SPIVAKEM A COMPREHENSIVE INTRODUCTION TO    DIFFERENTIAL GEOMETRY  TODAY A DILEMMA CONFRONTS ANY ONE INTENT ON PENETRATING THE  MYSTERIES OF DIFFERENTIAL GEOMETRY  ON THE ONE HAND ONE CAN CONSULT  NUMEROUS CLASSICAL TREATMENTS OF THE SUBJECT IN AN ATTEMPT TO FORM  SOME IDEA HOW THE CONCEPTS WITHIN IT DEVELOPED  UNFORTUNATELY A  MODERN MATHEMATICAL EDUCATION TENDS TO MAKE CLASSICAL MATHEMATICAL  WORKS INACCESSIBLE LDOTS ON THE OTHER HAND ONE CAN NOW FIND TEXTS  AS MODERN IN SPIRIT AND CLEAN IN EXPOSITION AS BOURBAKIS  ALGEBRA  BUT A THOROUGH STUDY OF THESE BOOKS USUALLY LEAVES ONE  UNPREPARED TO CONSULT CLASSICAL WORKS AND ENTIRELY IGNORANT OF THE  RELATIONSHIP BETWEEN ELEGANT MODERN CONSTRUCTIONS AND THEIR CLASSICAL  COUNTERPARTS  MOST STUDENTS EVENTUALLY FIND THAT THIS IGNORANCE OF  THE ROOTS OF THE SUBJECT HAS ITS PRICE  NO ONE DENIES THAT MODERN  DEFINITIONS ARE CLEAR ELEGANT AND PRECISE  ITS JUST THAT ITS  IMPOSSIBLE TO COMPREHEND HOW ANY ONE EVER THOUGHT OF THEM  AND EVEN  AFTER ONE DOES MASTER A MODERN TREATMENT OF DIFFERENTIAL GEOMETRY  OTHER MODERN TREATMENTS OFTEN APPEAR SIMPLY TO BE ABOUT TO TOTALLY  DIFFERENT SUBJECTS LDOTS  AT THIS POINT I AM REMINDED OF A PAPER DESCRIBED IN LITTLEWOODS  EM MATHEMATICIANS MISCELLANY  THE PAPER BEGAN THE AIM OF THIS  PAPER IS TO PROVE LDOTS AND IT TRANSPIRED ONLY MUCH LATER THAT  THIS AIM WAS NOT ACHIEVED THE AUTHOR HADNT CLAIMED THAT IT WAS  WHAT I HAVE OUTLINED ABOVE IS THE CONTENT OF A BOOK THE REALIZATION  OF WHOSE PLAN AND THE INCORPORATION OF WHOSE DETAILS WOULD PERHAPS  BE IMPOSSIBLE WHAT I HAVE WRITTEN IS A SECOND OR THIRD DRAFT OF A  PRELIMINARY VERSION OF THIS BOOKENDQUOTESOURCESECTIONWHAT IS SIGNAL PROCESSINGTHE SCOPE OF SIGNAL PROCESSING FAR EXCEEDS THE CAPABILITY OF ANYSINGLE BOOK TO CONTAIN IT  THOUGH THE SUBJECT HAS GROWN SO BROAD ASTO OBVIATE A PERFECT AND PRECISE DEFINITION OF WHAT IS ENTAILED IN ITCERTAIN CONCEPTS MUST BE CONSIDERED AS INDISPENSABLE FOR RUDIMENTARYUNDERSTANDING  CERTAINLY SIGNAL PROCESSING INCLUDES THE MATERIALTAUGHT IN TRADITIONAL DIGITAL SIGNAL PROCESSING DSP COURSES SEEEG CITEPROAKIS1OPPENHEIMSCHAFER SUCH AS TRANSFORMS OF MANYVARIETIES Z LAPLACE FOURIER ETC AND THE CONCEPTS OF FREQUENCYRESPONSE IMPULSE RESPONSE AND CONVOLUTION FOR BOTH DETERMINISTICAND RANDOM SIGNALS  IT ALSO INCLUDES THE BASIC CONCEPTS OF FILTERINGAND FILTER DESIGN  THESE CONCEPTS ARE ASSUMED AS A BACKGROUND TO THISTEXT AND ARE USED AS NECESSARY THROUGHOUT THE TEXT  TRADITIONAL AREASIN SIGNAL PROCESSING INCLUDE AS TAKEN FROM THE IEEE EM TRANSACTIONS  ON SIGNAL PROCESSING CLASSIFICATIONS FILTER DESIGN FASTFILTERING ALGORITHMS TIMEFREQUENCY ANALYSIS MULTIRATE FILTERINGSIGNAL RECONSTRUCTION ADAPTIVE FILTERS NONLINEAR SIGNALS ANDSYSTEMS SPECTRAL ANALYSIS AND EXTENSIONS OF THESE CONCEPTS TOMULTIDIMENSIONAL SYSTEMS  THESE TOPICS ARE EMPLOYED IN A VARIETY OFAPPLICATION AREAS  IMPLEMENTATION IN HARDWARE OR SOFTWARE IS ALSOAN IMPORTANT FACET OF SIGNAL PROCESSING  PROVIDING A THOROUGHCOVERAGE OF THESE TOPICS ALONE REQUIRES MULTIPLE VOLUMESBUT IN THE VIEW OF THIS BOOK SIGNAL PROCESSING HAS AN EVEN GREATERREACH BECAUSE OF ITS INFLUENCE BY AND ON RELATED DISCIPLINES  SIGNALPROCESSING OVERLAPS WITH THE STUDY TRADITIONALLY KNOWN AS EM  CONTROLS SINCE CONTROL ULTIMATELY INVOLVES PRODUCING A SIGNALBASED UPON MEASURED OUTPUT OF A PLANT BY MEANS OF SOME PROCESSING UPONTHAT SIGNAL  BEFORE A SYSTEM CAN BE CONTROLLED THE PARTICULARPARAMETERS OF THAT SYSTEM USUALLY MUST BE DETERMINED SO EM SYSTEM  IDENTIFICATION IS AN ASPECT OF SIGNAL PROCESSING  THIS IN TURNRELATES TO EM SPECTRUM ESTIMATION AND ALL OF ITS APPLICATIONSSIGNAL PROCESSING HAS STRONG TIES TO EM COMMUNICATIONS THEORY ANDRECENTLY ESPECIALLY TO DIGITAL COMMUNICATION SINCE THE CAPABILITIESOF MODERN COMMUNICATION SYSTEMS ARE THE RESULT OF THE SIGNALPROCESSING PERFORMED WITHIN THEM  RELATED TO DIGITAL COMMUNICATIONARE QUESTIONS OF EM DETECTION AND EM ESTIMATION THEORY HOW TOGET THE BEST INFORMATION OUT OF SIGNALS MEASURED IN THE PRESENCE OFRANDOM NOISE  DETECTION AND ESTIMATION THEORY IN TURN RELATE TO EM  PATTERN RECOGNITION  DIGITAL COMMUNICATION ALSO SPILLS OVER INTOTHE AREAS OF EM INFORMATION THEORY AND EM CODING THEORY  SYSTEMIDENTIFICATION AND ESTIMATION THEORY TREAT QUESTIONS OF SOLVINGOVERDETERMINED SYSTEMS OF EQUATIONS THAT IN TURN HAVE APPLICATION INEM TOMOGRAPHY  THESE IN TURN HAVE SOME BEARING ON QUESTIONS OFAPPROXIMATION AND SMOOTHING OF SIGNALS  IF A TREATMENT OF FUNDAMENTALSIGNAL PROCESSING TOPICS REQUIRES SEVERAL VOLUMES THEN INCLUSION OFTHESE LATTER TOPICS REQUIRES A LIBRARYSIGNAL PROCESSING COVERS A LARGE TERRITORY  HOWEVER THERE IS ACOMMON THREAD AMONG ALL THE AREAS MENTIONED  THEY ALL INVOLVE AFAIR DEGREE OF MATHEMATICAL SOPHISTICATION AND IN BOTH THEORY ANDPRACTICE ASSUME AN ANALYTICAL AND A COMPUTATIONAL COMPONENT  MOST OFTHESE AREAS SHARE A LARGE OVERLAP IN CONCEPTUAL CONTENT  WEPROPOSE THE FOLLOWING AS A TENTATIVE DEFINITION OF SIGNAL PROCESSINGAT LEAST FOR THE PURPOSES OF THIS BOOKBEGINDEFINITION INDEXSIGNAL PROCESSING DEFINITION  BF SIGNAL PROCESSING IS THAT AREA OF APPLIED MATHEMATICS THAT DEALS  WITH OPERATIONS ON OR ANALYSIS OF SIGNALS IN EITHER DISCRETE OR  CONTINUOUS TIME TO PERFORM USEFUL OPERATIONS ON THOSE SIGNALSENDDEFINITIONWITH ITS FOCUS ON APPLIED MATHEMATICS THIS BOOK NEGLECTS SEVERALIMPORTANT ASPECTS OF SIGNAL PROCESSING INCLUDING HARDWARE DESIGN ANDIMPLEMENTATION ON SIGNAL PROCESSING CHIPS  USEFUL OPERATIONIS DELIBERATELY LEFT AMBIGUOUS  DEPENDING UPON THE APPLICATION AUSEFUL OPERATION COULD BE CONTROL DATA COMPRESSION DATATRANSMISSION DENOISING PREDICTION FILTERING SMOOTHING DEBLURRINGTOMOGRAPHIC RECONSTRUCTION IDENTIFICATION CLASSIFICATION OR AVARIETY OF OTHER OPERATIONS THE PRIMARY INTENT OF THIS BOOK IS BF TO PRESENT A TREATMENT OF  RELEVANT MATHEMATICS SO THAT STUDENTS AND PRACTITIONERS OF SIGNAL  PROCESSING AND RELATED FIELDS ARE ABLE TO READ APPLY AND  ULTIMATELY CONTRIBUTE TO A VARIETY OF AREAS OF SIGNAL PROCESSING  RESEARCH AND PRACTICE  THE INTENT IS NOT TO EXPLORE PUREMATHEMATICS HOWEVER BUT RATHER TO PROVIDE A MATHEMATICAL MODICUMSUFFICIENT TO EXPLAIN AND EXPLORE THE MORE IMPORTANT MATHEMATICALPARADIGMS USED IN SIGNAL PROCESSING EM ALGORITHMS  A STUDENT WITHA BACKGROUND FROM THIS BOOK SHOULD BE ABLE TO MOVE EXPEDITIOUSLY TO APARTICULAR AREA OF INTEREST AND BEGIN MAKING EFFECTIVE PROGRESS IN THESPECIALIZED LITERATURE OF THAT AREA  WE HAVE ENDEAVORED TO MAINTAIN APRECARIOUS BALANCE PURISTS IN MATHEMATICS WILL FIND SOME OF THEANALYTICAL METHODS DEFICIENT WHILE PRAGMATISTS WILL ARGUE THAT THEREARE FAR TOO MANY EQUATIONS  TO USE A GARAGE ANALOGY WE HAVE PROVIDEDENOUGH INFORMATION TO GET UNDER THE HOOD OF THE CAR TAKING APART FOREXAMINATION MANY OF THE ENGINE COMPONENTS BUT WITHOUT GETTING INTODETAIL AT THE LEVEL OF METALLURGICAL PHENOMENA  SUCH MINUTEINVESTIGATIONS ARE BEST CONDUCTED AFTER THE STUDENT UNDERSTANDS HOWTHE CAR OPERATES  IN ADDITION TO THEORY THE BOOK CONTAINS AVARIETY OF MATERIAL COMPARABLE TO WHAT IS FOUND IN OTHER ADVANCEDSIGNAL PROCESSING TEXTS  IN ADDITION TO THE PRIMARY GOAL OF THIS BOOK THERE ARE TWO OTHERSFIRST TO DEVELOP WITHIN THE STUDENT A DEGREE OF MATHEMATICALMATURITY  THE STUDENT WITH THIS MATURITY WILL IT IS HOPED BE ABLETO ORGANIZE EFFECTIVE APPROACHES OF HISHER OWN TO A VARIETY OFPROBLEMS  THIS MATURITY WILL BE DEVELOPED BY WORKING PROBLEMSFOLLOWING AND DOING PROOFS AND WRITING AND RUNNING PROGRAMSITEM IN ADDITION TO THE MATHEMATICAL CONTENT THE SC MATLAB  PROGRAMS PROVIDED IN THE BOOK SHOULD BE USEFUL BOTH AS STANDALONE  FUNCTIONS AND AS BUILDING BLOCKS TO FURTHER UNDERSTANDINGSECOND THE BOOK IS INTENDED AS A USEFUL REFERENCE WITH REFERENCEMATERIAL GATHERED ON SEVERAL AREAS IN SIGNAL PROCESSING SUCHAS DERIVATIVES LINEAR ALGEBRA OPTIMIZATION INEQUALITIES ETCTHIS STATEMENT OF INTENT SHOULD MAKE CLEAR WHAT THIS BOOK IS NOTTHERE ARE SEVERAL VERY GOOD BOOKS AVAILABLE ON APPLICATION AREAS INSIGNAL PROCESSING SUCH AS SPECTRUM ESTIMATION ADAPTIVE FILTERINGARRAY PROCESSING AND SO ON THIS BOOK DOES NOT CHOOSE ANY OF THOSEPARTICULAR AREAS AS ITS FOCUS  THUS WHILE MANY DIFFERENT TECHNIQUESOF SPECTRUM ESTIMATION WILL BE PRESENTED AS APPLICATIONS OF THETECHNIQUES DISCOVERED ISSUES CENTRAL TO THE STUDY OF SPECTRUMESTIMATION SUCH AS COMPARISONS OF THE DIFFERENT TECHNIQUES IN TERMSOF SPECTRAL RESOLUTION BIAS ETC ARE NOT PRESENTED HERESIMILARLY THE MAJOR PARADIGMS OF ADAPTIVE FILTERING ARE PRESENTED ASAPPLICATIONS OF OTHER IMPORTANT CONCEPTS EG LEASTSQUARES ANDMINIMUM MEANSQUARES AND RECURSIVE COMPUTATION OF MATRIX INVERSESBUT A THOROUGH TREATMENT OF THE CONVERGENCE OF THE FILTERS IS AVOIDEDRATHER THAN FOCUSING ON ONE PARTICULAR AREA OF RESEARCH INTEREST THISBOOK PRESENTS THE TOOLS THAT ARE USED IN THESE RESEARCH AREAS ENABLINGTHE INTERESTED STUDENT TO MOVE INTO A VARIETY OF DIFFERENT AREASSECTIONMATHEMATICAL TOPICS EMBRACED BY SIGNAL PROCESSINGSO WHAT DOES A SIGNAL PROCESSOR  THAT IS AN INDIVIDUAL WHO WANTSTO DESIGN SIGNAL PROCESSING ALGORITHMS NOT THE SPECIALIZEDMICROPROCESSOR THAT MIGHT BE USED TO IMPLEMENT THE ALGORITHMS  NEEDTO KNOW TO BE EFFECTIVE  DEPENDING ON THE PROBLEM SEVERALMATHEMATICAL TOOLS CAN BE EMPLOYEDBEGINDESCRIPTIONITEMLINEAR SIGNALS AND SYSTEMS AND TRANSFORM THEORY THESE TOPICS  CORE TO MANY UNDERGRADUATE AND INTRODUCTORY GRADUATE COURSES ARE  ASSUMED AS BACKGROUND TO THIS BOOK  FAMILIARITY WITH BOTH  CONTINUOUS AND DISCRETETIME SYSTEMS IS ASSUMED ALTHOUGH A REVIEW  OF SOME TOPICS IS PROVIDED IN SECTION REFSECLTI  ITEMPROBABILITY AND STOCHASTIC PROCESSES THIS IS A CRITICALLY  IMPORTANT AREA THAT IS ALSO ASSUMED AS BACKGROUND  STUDENTS SHOULD  BE ACQUAINTED WITH PROBABILITY AND HAVE HAD A COURSE IN STOCHASTIC  PROCESSES AS A PREREQUISITE TO THIS BOOK  PROBABILITY IS AN  IMPORTANT TOOL AND STUDENTS ARE ADVISED TO CONTINUE SHARPENING  THEIR SKILLS WITH IT  A BRIEF REVIEW OF IMPORTANT TOPICS IN  STOCHASTIC PROCESSES IS PROVIDED IN APPENDIX REFAPPDXRP  ITEMPROGRAMMING A SIGNAL PROCESSOR MUST KNOW HOW TO PROGRAM IN AT  LEAST ONE HIGHLEVEL LANGUAGE  IN MOST CASES SIGNAL PROCESSING  ULTIMATELY BOILS DOWN TO A SOFTWARE OR HARDWARE IMPLEMENTATION ON  SOME KIND OF COMPUTING PLATFORM  THIS REQUIRES DEPLOYMENT OF THE  CONCEPT SIMULATION AND TESTING ALL USUALLY SOFTWARERELATED  ACTIVITIES  AN UNDERSTANDING OF BASIC PROGRAMMING CONCEPTS SUCH AS  VARIABLES PROGRAM FLOW RECURSION DATA STRUCTURES AND PROGRAM  COMPLEXITY IS ASSUMEDITEMCALCULUS AND ANALYSIS THESE FOUNDATION CONCEPTS OCCUR  REPEATEDLY IN THE SIGNAL PROCESSING LITERATURE  A BROAD AND SHALLOW  COVERAGE OF ANALYSIS APPEARS IN APPENDIX REFAPPDXSETFUNCTITEMVECTOR SPACES AND LINEAR ALGEBRA WHILE EVERY UNDERGRADUATE  ENGINEER HAS SOME EXPOSURE TO LINEAR ALGEBRA THESE TOPICS ARE SO  IMPORTANT TO SIGNAL PROCESSING THAT ADDITIONAL EXPOSURE IS CRITICAL  MANY OF THE BASIC CONCEPTS ARE REVIEWED IN THIS BOOK WITH AN EYE  TOWARD APPLICATIONS IN SIGNAL PROCESSING  BECAUSE OF ITS  IMPORTANCE CHAPTERS REFCHAPVECTSP THROUGH REFCHAPKRONECKER  ARE DEVOTED LARGELY TO LINEAR ALGEBRA AND ITS APPLICATIONSITEMNUMERICAL METHODS WITH THE INCREASING PENETRATION OF COMPUTERS  INTO ENGINEERING CULTURE THERE IS PARADOXICALLY A DECREASE IN MANY  STUDENTS EXPOSURE TO NUMERICAL METHODS  AND YET A SIGNIFICANT  PORTION OF SIGNAL PROCESSING CONSISTS OF NOTHING MORE THAN NUMERICAL  METHODS APPLIED TO A PARTICULAR SET OF PROBLEMS INVOLVING SIGNALS  MANY OF THE TECHNIQUES DESCRIBED IN THIS BOOK ARE BORROWED FROM THE  NUMERICAL METHODS LITERATUREITEMFUNCTIONAL ANALYSIS IN SIGNAL PROCESSING A SIGNAL IS A  FUNCTION  THE TOOLS FROM FUNCTIONAL ANALYSIS PROVIDE A FRAMEWORK  FROM WHICH TO VIEW THE SIGNAL LEADING THE WAY TO POWERFUL SIGNAL  TRANSFORMS AND SIGNAL SPACES IN DIGITAL COMMUNICATIONS  IN THIS  BOOK WE PRESENT CONCEPTS FROM FUNCTIONAL ANALYSIS IN THE CONTEXT OF  VECTOR SPACES PARTICULARLY IN CHAPTERS REFCHAPVECTSP AND  REFCHAPVECTAPITEMSTATISTICAL DECISION THEORY STATISTICAL DECISION THEORY CAN BE  DESCRIBED AS THE SCIENCE OF MAKING DECISIONS IN THE FACE OF RANDOM  UNCERTAINTY  SUCH DECISIONMAKING ALSO DESCRIBES WHAT IS DONE IN  MANY SIGNAL PROCESSING APPLICATIONS  THE APPLICATION OF STATISTICS  TO SIGNAL PROCESSING CAN BE DIVIDED INTO TWO MAJOR OVERLAPPING  AREAS BF DETECTION THEORY AND BF ESTIMATION THEORY  DETECTION THEORY IS A FRAMEWORK FOR MAKING DECISIONS IN THE PRESENCE  OF NOISE  ESTIMATION THEORY PROVIDES A MEANS OF DETERMINING THE  VALUE OF A QUANTITY IN THE PRESENCE OF NOISE  DETECTION AND  ESTIMATION ARE COVERED IN CHAPTERS REFCHAPFORMALISM THROUGH  REFCHAPKALMANITEMOPTIMIZATION A COMMON THEME RUNNING THROUGH MANY SIGNAL  PROCESSING APPLICATIONS IS OPTIMIZATION WHATEVER IS BEING COMPUTED  WE WISH TO DO IT IN THE BEST POSSIBLE WAY  OR IF WE CANNOT GET TO  THE OPTIMAL OPERATION POINT IN ONE STEP WE WILL PROGRESS TOWARD IT  AS WE CONTINUE TO PROCESS DATA THAT IS WE WILL ADAPT  BECAUSE OF  ITS UBIQUITY IN APPLICATION IN PART REFPARTOPT WE PRESENT  FUNDAMENTAL CONCEPTS IN OPTIMIZATION INCLUDING AND CONSTRAINED  OPTIMIZATION LINEAR PROGRAMMING AND PATH SEARCH ALGORITHMS  IN  ADDITION OPTIMIZATION PROBLEMS PARTICULARLY FOR CONSTRAINED  OPTIMIZATION ARE PRESENTED THROUGHOUT THE TEXT AND IN THE  EXERCISESITEMMODERN ALGEBRA MODERN ALGEBRA PROVIDES A VOCABULARY OF  IMPORTANT CONCEPTS AND TOOLS USEFUL IN THE DEVELOPMENT OF SEVERAL  FAST ALGORITHMS  WE PRESENT THE BASIC DEFINITIONS AND SOME  USEFUL EXAMPLES IN SECTION REFSECALGEBRA WITH A FEW ADVANCED  EXAMPLES AND APPLICATIONS IN SECTION REFSECALG2ITEMCOMPLEX ANALYSIS ALL ENGINEERS KNOW ABOUT COMPLEX NUMBERS BUT  NOT ENOUGH KNOW ABOUT THE WONDERS OF COMPLEX ANALYSIS  SINCE  TRANSFORMS ALMOST INVARIABLY INVOLVE COMPLEX FUNCTIONS IT IS  IMPORTANT TO KNOW SOMETHING ABOUT COMPLEX ANALYSIS AND HOW IT  APPLIES TO TRANSFORM THEORY  THE ESSENTIALS ARE PRESENTED IN  CHAPTER REFCHAPCOMPLEXITEMPOLYNOMIAL THEORY POLYNOMIALS ARISE AS TRANSFER FUNCTIONS AND  CHARACTERISTIC EQUATIONS IN ANALYSIS OF LINEAR SYSTEMS  POLYNOMIALS  ARE ALSO DENSE IN THE SET OF CONTINUOUS FUNCTIONS WHICH MEANS THAT  THERE IS SOME POLYNOMIAL ARBITRARILY CLOSE TO ANY CONTINUOUS  FUNCTIONS  FOR MANY PRACTICAL PURPOSES WHATEVER WE WANT TO DO WITH  A CONTINUOUS FUNCTION WE CAN DO WITH A POLYNOMIAL  IN ADDITION  SEVERAL USEFUL ANALYTICAL HAVE BEEN DEVELOPED IN ASSOCIATION WITH  POLYNOMIALS SUCH AS THE ROUTHHURWITZ ALGORITHM AND THE JURY TEST  THESE AND OTHERS IMPORTANT CONCEPTS RELATED TO POLYNOMIALS ARE  PRESENTEDITEMNUMBER THEORY NUMBER THEORY THE STUDY OF INTEGERS AND THEIR  PROPERTIES ARISES IN SIGNAL PROCESSING BECAUSE NUMBERS REPRESENTED  IN A COMPUTER ARE ULTIMATELY INTEGERS  THE CONCEPTS OF NUMBER  THEORY PROVIDE A FRAMEWORK FOR SEVERAL FAST ALGORITHMS FOR  CONVOLUTION AND TRANSFORMS  SEE CHAPTER REFCHAPNUMTH FOR SOME  PRINCIPLES AND APPLICATIONSITEMAPPROXIMATION AND INTERPOLATION FILTER DESIGN IS FUNDAMENTALLY  AN EXERCISE IN APPROXIMATION CERTAIN FILTER REQUIREMENTS ARE KNOWN  AND IT IS DESIRED TO FIND A REALIZABLE FILTER THAT MEETS THE  REQUIREMENTS AS CLOSELY AS POSSIBLE  INTERPOLATION IS RELATED TO  UPSAMPLING HOW TO FIND OUT WHAT HAPPENS BETWEEN THE SAMPLES  CHAPTER REFCHAPINTERP DEVELOPS THESE IDEASITEMITERATIVE METHODS MANY SIGNAL PROCESSING METHODS CONVERGE TO  THEIR SOLUTION AFTER SEVERAL ITERATIONS  FOR EXAMPLE ADAPTIVE  FILTERS AND NEURAL NETWORKS  WE PRESENT SOME BASIC CONCEPTS AND  EXAMPLES OF ITERATIVE METHODS IN CHAPTERS REFCHAPITER1 THROUGH  REFCHAPEMENDDESCRIPTIONTO THESE MIGHT BE ADDED THE TOPICS OF MODERN ALGEBRA NUMBER THEORYCOMPLEX ANALYSIS INTERPOLATION AND APPROXIMATION THEORY AND OTHERTOPICS TOO NUMEROUS TO FIT WITHIN THE COVERS OF A SINGLE BOOKTHESE TOPICS COVER A VERY LARGE TERRITORY  IN EACH OF THESE TOPICAREAS NUMEROUS VOLUMES HAVE BEEN WRITTEN  OUR INTENT IS TO NOT TOPROVIDE AN EXHAUSTIVE TREATMENT IN EACH AREA BUT TO PRESENT ENOUGHINFORMATION TO PROVIDE A USEFUL SET OF TOOLS WITH BROAD APPLICATIONOUR APPROACH IS DIFFERENT FROM MANY OTHER BOOKS ON SIGNAL PROCESSINGIN THAT WE DO NOT EXHAUSTIVELY EXAMINE A PARTICULAR DISCIPLINE OFSIGNAL PROCESSING  FOR EXAMPLE SPECTRUM ESTIMATION  BRINGING INMATHEMATICAL TOOLS AS NECESSARY TO TREAT ISSUES THAT ARISE  INSTEADWE PRESENT THE MATHEMATICAL PERSPECTIVE FIRST INTRODUCING NEW SIGNALPROCESSING PROBLEMS AND ENHANCING UNDERSTANDING OF ALREADYINTRODUCEDPROBLEMS AS THE MATERIAL PERMITS  BY THIS MEANS PARALLELS MAY BEDRAWN BETWEEN AREAS THAT SHARE MATHEMATICAL TOOLS BUT THAT ARE NOTCOMMONLY PRESENTED TOGETHERSECTIONMATHEMATICAL MODELSTHROUGHOUT MOST OF THE REMAINDER OF THIS CHAPTER WE PRESENT EXAMPLESSEVERAL DIFFERENT MODELS THAT ARE COMMONLY USED IN SIGNAL PROCESSINGTHE MODELS ARE ROUGHLY CATEGORIZED AS FOLLOWSBEGINENUMERATEITEM LINEAR SIGNAL MODELS FOR DISCRETE AND CONTINUOUS TIME INCLUDING  TRANSFER FUNCTION AND STATE SPACE REPRESENTATIONS ALSO  APPLICATIONS OF THESE MODELS TO SIGNAL PROCESSING PROBLEMS SUCH AS  PREDICTION OR SPECTRUM ESTIMATIONITEM ADAPTIVE FILTERING MODELS AND APPLICATIONS TO PREDICTION  SYSTEM IDENTIFICATION ETCITEM THE GAUSSIAN RANDOM VARIABLE RV INCLUDING THE IMPORTANT  IDEA OF CONDITIONING UPON AN OBSERVATIONITEM HIDDEN MARKOV MODELSENDENUMERATETHESE EXAMPLES ILLUSTRATE SOME OF THE NOTATION USED THROUGHOUT THISBOOK AND PROVIDE A STARTING POINT FOR SEVERAL OF THE SIGNALPROCESSING APPLICATIONS THAT ARE EXAMINED  THUS THE MATERIAL MOSTLYSETS THE STAGE POSING QUESTIONS AND INTRODUCING ASSOCIATED WITH THEMODELS LEAVING THE QUESTIONS TO BE ANSWERED IN LATER CHAPTERS  THEMATERIAL HERE IS PRESENTED PARTLY BY WAY OF REVIEW AND PARTLY AS APARTIAL SURVEY AND MOTIVATOR OF CONCEPTS TO BE DEVELOPED THROUGHOUTTHE BOOK  SEVERAL NEW IDEAS ARE TOUCHED ON HERE THOUGH WITH THEINTENT THAT IT WILL MOTIVATE AND FORESHADOW THE TOPICS IN UPCOMINGCHAPTERSAFTER THIS INTRODUCTORY MATERIAL WE PRESENT A DISCUSSION OF PROOFSTHE CHAPTER ENDS WITH THE DEVELOPMENT OF A FAST ALGORITHM  FINALLYAN ALGORITHM  FOR SOLUTION OF A SYSTEM OF TOEPLITZ EQUATIONSTHIS ALGORITHM  MORE COMMONLY DISCUSSED IN THE ERROR CONTROLLITERATURE THAN THE SIGNAL PROCESSING LITERATURE  TIES TOGETHERSEVERAL THEMES OF THE CHAPTER LINEAR SYSTEMS NOTATION AUTOREGRESSIVEMODELS ALGORITHMS AND PROOFSSECTIONMODELS FOR LINEAR SYSTEMS AND SIGNALSLABELSECLTIMOST OF THE SYSTEMS TREATED IN SIGNAL PROCESSING ARE ASSUMED TO BELINEARINDEXLINEAR SYSTEM A CONCEPT THAT SHOULD BE FAMILIAR FROMINTRODUCTORY SIGNAL PROCESSING COURSES  WE WILL FOCUS PRINCIPALLY ONSYSTEMS THAT ARE ALSO TIME INVARIANT SUCH SYSTEMS ARE SAID TO BELINEAR TIMEINVARIANT LTI  SYSTEMS ARE DIVIDED ACCORDING TO WHETHERTHEY OPERATE IN CONTINUOUS TIME OR DISCRETE TIME  IN DISCRETE TIMETHE DATA ASSOCIATED WITH TIME T ARE INDICATED BY EITHER SQUAREBRACKETS INDEX  SQUARE BRACKETSDISCRETETIMESUCH AS XT OR BY SUBSCRIPTS SUCH AS XT WHERE TIS AN INTEGER  WE WILL ALSO EMPLOY OTHER VARIABLES AS A DISCRETETIMEINDEX SUCH AS N OR K  FOR CONTINUOUSTIME SIGNALS THE NOTATIONXT OR XT IS COMMONLY EMPLOYED WHERE T IS A REAL NUMBERINDEX CONTINUOUS TIMEWE WILL FIRSTPRESENT SOME CONCEPTS AND NOTATION FOR DISCRETE TIME SIGNALS ANDSYSTEMS THEN TRANSLATE THE NOTATION TO CONTINUOUSTIME  THE MATERIAL IN THIS SECTION IS INTENDED TO BE PRIMARILY AS A REVIEWTHIS SECTION IS FAIRLY LONG DUE TO THE IMPORTANCE OF THE MATERIAL AND THENUMBER OF INTERESTING PROBLEMS IT INTRODUCESSUBSECTIONLINEAR DISCRETETIME MODELSSUBSUBSECTIONDIFFERENCE EQUATIONSLET FT DENOTE THE SCALAR INPUT TO A DISCRETETIME LINEAR SYSTEMAND LET YT DENOTE THE SCALAR OUTPUT  IT IS COMMON TO ASSUME ANINPUTOUTPUT RELATION OF THE FORM OF THE INDEXDIFFERENCE EQUATIONDIFFERENCE EQUATIONBEGINMULTLINE YT  ABAR1 YT1  ABAR2 YT2  CDOTS  ABARP YTP    BBAR0 FT  BBAR1 FT1  CDOTS  BBARQ FTQLABELEQARMA0ENDMULTLINETHE EQUATION IS SHOWN UNDER GENERAL ASSUMPTION OF COMPLEX SIGNALS ANDTHE BAR OVER THE COEFFICIENTS DENOTES EM COMPLEX CONJUGATION  SEEBOX REFBOXCOMPLEXNOT  BY REDEFINING EACH COEFFICIENT ABARIAND BBARI IN TERMS OF ITS CONJUGATE REFEQARMA0 COULD ALSOBE WRITTEN WITHOUT THE CONJUGATES ASBEGINMULTLINE YT  A1 YT1  A2 YT2  CDOTS  AP YTP    B0 FT  B1 FT1  CDOTS  BQ FTQ NONUMBERENDMULTLINEWITH CONSISTENT AND CAREFUL USE OF THE NOTATION THE QUESTION OFWHETHER THE COEFFICIENTS ARE CONJUGATED IN THE DEFINITION OF THELINEAR MODEL IS OF NO ULTIMATE SIGNIFICANCE  THE ANSWERS OBTAINEDARE INVARIABLY THE SAME  HOWEVER THE BULK OF SIGNAL PROCESSINGLITERATURE SEEMS TO FAVOR THE CONJUGATED REPRESENTATION INREFEQARMA0  OF COURSE WHEN THE SIGNALS AND COEFFICIENTS ARESTRICTLY REAL THE CONJUGATION IS SUPERFLUOUS AND THE SYSTEM CAN ALSOBE WRITTEN IN THE FORMBEGINMULTLINE YT  A1 YT1  A2 YT2  CDOTS  AP YTP    B0 FT  B1 FT1  CDOTS  BQ FTQ NONUMBERENDMULTLINEWITHOUT THE CONJUGATES ON THE COEFFICIENTSBEGINTEXTBOX09TEXTWIDTHNOTATION FOR COMPLEX QUANTITIESLABELBOXCOMPLEXNOTINDEXCOMPLEX CONJUGATE INDEXBAROVERLINE SEECOMPLEX CONJUGATEWE USE THE ENGINEERS NOTATION JSQRT1 RATHER THAN THEMATHEMATICIANS I  HOWEVER IN SOME PLACES J WILL BE USED AS ANINDEX OF SUMMATION CONTEXT SHOULD MAKE CLEAR WHAT IS INTENDEDINDEXJJ INDEXIIA BAR OVER A QUANTITY DENOTES EM COMPLEX CONJUGATION  OTHERAUTHORS COMMONLY INDICATE COMPLEX CONJUGATION USING A SUPERSCRIPTASTERISK AS A  HOWEVER THE ABAR NOTATION IS USED IN THISBOOK TO INDICATE CONJUGATION SINCE A IS ALSO COMMONLY USED TODENOTE A PARTICULAR VALUE OF A SUCH AS A MINIMIZING VALUE OR TOINDICATE THE ADJOINT OF A LINEAR OPERATORENDTEXTBOXIN THE CASE OF A SYSTEM THAT IS NOT TIMEINVARIANT THE COEFFICIENTSMAY BE A FUNCTION OF THE TIME INDEX T  WE WILL ASSUME FOR THE MOSTPART CONSTANT COEFFICIENTS  THE RELATION REFEQARMA0 CAN BEWRITTEN ASBEGINEQUATION SUMK0P ABARK YTK  SUMK0Q BBARK FTKLABELEQARMAENDEQUATIONWITH A0  1  IN REFEQARMA WHEN P0BEGINEQUATIONYT  SUMK0Q BBARK FTKLABELEQFIR1ENDEQUATIONTHE SIGNAL YT IS CALLED IN THE STATISTICAL LITERATURE A EM  MOVING AVERAGE MA SIGNAL INDEXMOVING AVERAGE SINCE IT ISFORMED BY SIMPLY ADDING UP SCALED VERSIONS OF THE INPUT SIGNAL OVERA WINDOW OF Q1 VALUES  THE NUMBER Q IS THE EM ORDER OF THE MASIGNAL  THE SIGNAL IS DENOTED EITHER AS MA OR MAQINDEXMASEEMOVING AVERAGE WE CAN ALSO WRITE REFEQFIR1 USINGA CONVENIENT VECTOR NOTATION  LET FBFT  BEGINBMATRIX FT  FT1  VDOTS  FTQENDBMATRIX QQUAD TEXTAND QQUADBBF  BEGINBMATRIX B0  B1  VDOTS  BQ ENDBMATRIXTHEN  YT  BBFH FBFT  OVERLINEFBFTTBBFTHE VECTOR NOTATION USED IN THIS EXAMPLE IS SUMMARIZED IN BOXREFBOXVECTORNOTINDEXBOLD FONTSEEFONTSIN REFEQARMA WHEN Q0 SO THAT YT  BBAR0 UN SUMK1P ABARK YTKTHE SIGNAL Y IS SAID TO AN EM AUTOREGRESSIVE AR SIGNAL OF ORDER PINDEXARSEEAUTOREGRESSIVEINDEXAUTOREGRESSIVE  EM AUTO BECAUSE IT EXPRESSES THE SIGNAL IN TERMS OF ITSELF  EM REGRESSIVE IN THE SENSE THAT A FUNCTIONAL RELATIONSHIP EXISTS  BETWEEN TWO   OR MORE VARIABLES  AN AUTOREGRESSIVE MODEL IS DENOTED AS AR OR  ARP  WRITING YBFT  BEGINBMATRIX YT1  YT2  VDOTS  YTPENDBMATRIX QQUAD TEXTAND QQUADABF  BEGINBMATRIX A1  A2  VDOTS  AP ENDBMATRIXWE CAN WRITE THE AR SIGNAL AS YT  BBAR0 UT  ABFH YBFTBEGINTEXTBOX09TEXTWIDTHNOTATION FOR VECTORSLABELBOXVECTORNOTBEGINENUMERATEITEM VECTORS IN A FINITEDIMENSIONAL VECTOR SPACE ARE TYPICALLY  DENOTED IN BOLD FONT SUCH AS BBF  INDEXFONTSBOLDITEM ALL VECTORS IN THIS BOOK ARE ASSUMED TO BE COLUMN VECTORS  IN  SOME CASES A VECTOR WILL BE TYPESET IN HORIZONTAL FORMAT WITH T  TRANSPOSE TO INDICATE THAT IT SHOULD BE TRANSPOSED  THUS WE COULD  HAVE EQUIVALENTLY WRITTEN BBF  B0 B1 LDOTS BQTQQUAD TEXTORQQUADBBFT  B0B1 LDOTS BQINDEXTTINDEXHHINDEXTRANSPOSETITEM IN GENERAL THE ITH COMPONENT OF A VECTOR BBF WILL BE  DESIGNATED AS BI  WHETHER THE INDEX I STARTS WITH 0 OR 1 OR  SOME OTHER VALUE DEPENDS ON THE NEEDS OF THE PARTICULAR PROBLEMITEM THE NOTATION BBFH DENOTES THE EM HERMITIAN TRANSPOSE IN  WHICH BBF IS TRANSPOSED AND ITS ELEMENTS ARE CONJUGATEDINDEXTRANSPOSEH HERMITIAN BBFH  BBAR0 BBAR1 LDOTS BBARQENDENUMERATETHESE RULES NOTWITHSTANDING FOR NOTATIONAL CONVENIENCE WE WILLSOMETIMES DENOTE THE VECTOR WITH N ELEMENTS AS AN NTUPLE SO THAT XBF  BEGINBMATRIX X1  X2  LDOTS  XN ENDBMATRIXTQQUADTEXTANDQQUAD XBF  X1X2LDOTSXNARE OCCASIONALLY USED SYNONYMOUSLY  THIS NTUPLE NOTATION IS USEDPARTICULARLY WHEN XBF IS REGARDED AS A POINT IN RBBNFURTHERMORE SINCE WE WILL GENERALIZE THE CONCEPT OF VECTORS TOINCLUDE FUNCTIONS THE MATH ITALIC NOTATION X WILL BE USED IN THEMOST GENERAL CASE TO REPRESENT VECTORS EITHER IN RBBN OR ASFUNCTIONSINDEXFONTSMATH ITALICINDEXTTSEETRANSPOSEINDEXHHSEETRANSPOSEBOXINDENT MATRICES ARE REPRESENTED WITH CAPITAL LETTERS AS IN A OR X  THEMATRIX I IS AN IDENTITY MATRIXINDEXIIINDEXVECTOR NOTATIONINDEXMATRIX NOTATIONTHE NOTATION ZEROBF IS USED TO INDICATE A VECTOR OR MATRIX OFZEROS WITH THE SIZE DETERMINED BY CONTEXT INDEX0ZEROBFSIMILARLY THE NOTATION ONEBF IS USED TO INDICATE A VECTOR ORMATRIX OF ONES WITH THE SIZE DETERMINED BY CONTEXT INDEX1ONEBFENDTEXTBOXTHE GENERAL FORM IN REFEQARMA COMBINING BOTH THE AUTOREGRESSIVEAND THE MOVING AVERAGE COMPONENTS IS CALLED AN EM AUTOREGRESSIVE  MOVINGAVERAGE OR ARMA OR ARMAPQ  WHERE ALL THE SIGNALSARE DETERMINISTIC THE TERM DARMA DETERMINISTIC ARMA IS SOMETIMESEMPLOYED INDEXARMA INDEXAUTOREGRESSIVE MOVING AVERAGESUBSUBSECTIONSYSTEM FUNCTION AND IMPULSE RESPONSEIN THE INTEREST OF GETTING A SYSTEM FUNCTION THAT DOES NOT DEPEND UPONINITIAL CONDITIONS WE ASSUME THAT THE INITIAL CONDITIONS ARE ZERO ANDTAKE THE ZTRANSFORM TO OBTAIN YZ SUMK0P ABARK ZK  FZ SUMK0Q BBARK ZKWHICH WE WRITE AS YZ AZ  FZ BZWE WILL OCCASIONALLY WRITE THE TRANSFORM RELATIONSHIP AS YT LEFTRIGHTARROW YZWHERE THE PARTICULAR TRANSFORM INTENDED IS DETERMINED BYCONTEXT INDEXLEFTRIGHTARROW  WE WILL ALSO DENOTEZTRANSFORMS BY YZ  ZCYT INDEXZZCTHE EM SYSTEM FUNCTION INDEXSYSTEM FUNCTIONSEETRANSFER FUNCTION ISBEGINEQUATION HZ  FRACYZFZ  FRACSUMK0Q BBARK  ZKSUMK0P ABARK ZK   FRACSUMK0Q BBARK  ZK1 SUMK1P ABARK ZK  FRACBZAZLABELEQHZ1ENDEQUATIONTHIS IS ALSO CALLED USUALLY INTERCHANGEABLY THE EM TRANSFER  FUNCTION INDEXTRANSFER FUNCTION OF THE SYSTEM  WE WRITEBEGINEQUATION YZ  HZ FZLABELEQYHZ1ENDEQUATIONAND REPRESENT THIS AS SHOWN IN FIGURE REFFIGSYST1BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODEINPUTPICTUREDIRSYST1LATEXINPUTPICTUREDIRSYST1    CAPTIONINPUTOUTPUT RELATION FOR A TRANSFER FUNCTION    LABELFIGSYST1  ENDCENTERENDFIGUREIF THE SYSTEM IS AR THEN HZ  FRAC11 SUMK1P ABARK ZK  FRAC1AZAND HZ IS SAID TO BE AN EM ALLPOLESYSTEM INDEXALLPOLESEEAUTOREGRESSIVEIF THE SYSTEM ISMA THEN HZ  SUMK0Q BBARK  ZK  BZWHICH IS CALLED AN EM ALLZERO SYSTEM  INDEXALLZEROSEEMOVING    AVERAGE THE CORRESPONDING INDEXDIFFERENCE EQUATION DIFFERENCE    EQUATION REFEQFIR1 HAS ONLY A FINITE NUMBER OF NONZERO OUTPUTS WHEN THE INPUT IS A DELTAFUNCTION FT  DELTAT WHERE  INDEXDELTA FUNCTION  DELTAT  BEGINCASES  1  T  0   0  T NEQ 0ENDCASESWE WILL ALSO WRITE THE DELTA FUNCTION AS DELTAT  OCCASIONALLY THEFUNCTION DELTATTAU WILL BE WRITTEN AS DELTATTAUA SYSTEM THAT HAS ONLY A FINITE NUMBER OF NONZERO OUTPUTS INRESPONSE TO A DELTA FUNCTION IS REFERRED TO AS A INDEXFINITE IMPULSE RESPONSE FINITE IMPULSE  RESPONSE FIR SYSTEM  INDEXFIRSEEFINITE IMPULSE RESPONSE ASYSTEM WHICH IS NOT FIR IS INDEXINFINITE IMPULSE RESPONSE INFINITEIMPULSE RESPONSE IIRWE CAN VIEW SIGNAL YZ AS THE OUTPUT OF A SYSTEM WITH SYSTEM FUNCTIONHZ DRIVEN BY AN INPUT FZ  TAKING THE INVERSE ZTRANSFORM OFREFEQYHZ1 AND RECALLING THE CONVOLUTION PROPERTY INDEXCONVOLUTIONMULTIPLICATION IN THE TRANSFORM DOMAIN CORRESPONDS TO CONVOLUTION INTHE TIME DOMAIN WE OBTAIN YT  SUMKINFTYINFTY UK HTKWHERE HT THE IMPULSE RESPONSEINDEXIMPULSE RESPONSE IS THE INVERSE TRANSFORM OFHZ    TO COMPUTE THE INVERSE TRANSFORM OF HZ WE FIRST FACTOR HZINTO MONOMIAL FACTORS USING THE ROOTS OF THE NUMERATOR AND DENOMINATORPOLYNOMIALS HZ  FRACBBAR0 PRODK1Q 1ZI Z1PRODK1P 1PI Z1  FRACBZAZWHERE THE ZI ARE THE NONZERO ROOTS OF BZ CALLED THE EM  ZEROS OF THE SYSTEM FUNCTION AND THE PI ARE THENONZERO ROOTS OF AZ CALLED THE EM POLES INDEXPOLE OF THESYSTEM FUNCTION  IN THIS FORM WE OBSERVE THAT IF A POLE IS EQUAL TOA ZERO THE FACTORS CAN BE CANCELED OUT OF BOTH THE NUMERATOR ANDDENOMINATOR TO OBTAIN AN EQUIVALENT TRANSFER FUNCTION  A WORD OFCAUTION EVEN THOUGH TERMS MAY CANCEL FROM THE NUMERATOR ANDDENOMINATOR AS SEEN FROM THE TRANSFER FUNCTION THE PHYSICALCOMPONENTS THAT THESE TERMS MODEL MAY STILL EXIST AND COULD INTRODUCEDIFFICULTY  A SYSTEM WITH THE SMALLEST DEGREE NUMERATOR ANDDENOMINATOR IS SAID TO BE A EM MINIMAL SYSTEM INDEXMINIMAL  SYSTEMBEGINEXAMPLETHE SYSTEM FUNCTION HZ  FRAC1 7Z1  12 Z215Z1  06Z2CAN BE FACTORED AS HZ  FRAC13Z114Z112Z113Z1  FRAC14Z112Z1THUS THE HZ IS NOT A MINIMAL REALIZATIONENDEXAMPLESUBSUBSECTIONPARTIAL FRACTION EXPANSION PFEASSUMING FOR THE MOMENT THAT THE POLES ARE UNIQUE NO REPEATED POLESAND THAT QP THEN BY PARTIAL FRACTION EXPANSION PFEINDEXPARTIAL FRACTION EXPANSION PFE THE SYSTEM FUNCTION CAN BEEXPRESSED ASBEGINEQUATIONHZ  SUMK1P FRACNK1PK Z1LABELEQHZ2ENDEQUATIONWHERE NK  HZ1PK Z1BIGRZPKTAKING THE CAUSAL INVERSE ZTRANSFORM OF REFEQHZ2 WE OBTAIN HT  SUMK1P NK PKTQQUAD T GEQ 0THE FUNCTIONS PKN ARE THE NATURAL MODES INDEXMODE OF THE SYSTEMHZ  CLEARLY FOR THE CAUSAL MODES TO BE BOUNDED IN TIME WE MUSTHAVE PK LEQ 1 INDEXSTABILITYIN GENERAL THE OUTPUT OF A LINEAR TIMEINVARIANT SYSTEM IS THE SUM OFTHE NATURAL MODES OF THE SYSTEM PLUS THE INPUT MODES OF THE SYSTEMBEGINEXAMPLE  LET HZ  FRAC13Z1 1 11Z1  3Z2  FRAC13Z115Z116Z1THEN A PARTIAL FRACTION EXPANSION IS HZ  FRAC215Z1  FRAC316Z1THE IMPULSE RESPONSE IS HT  25T  36TUTWHERE UT IS THE UNITSTEP FUNCTION INDEXUNITSTEP FUNCTION UT  BEGINCASES  1  T GEQ 1   0  T  0ENDCASESENDEXAMPLETO COMPUTE THE PFE WHEN Q GEQ P THE RATIO OF POLYNOMIALS IS FIRSTDIVIDED OUT  WHEN THERE ARE REPEATED POLES INDEXREPEATED POLESSOMEWHAT MORE CARE IS REQUIRED  FOR EXAMPLE A ROOT REPEATED RTIMES AS IN HZ  FRACBZ1P Z1RGIVES RISE TO THE PARTIAL FRACTION EXPANSIONBEGINEQUATION HZ  FRACK01P Z1R  FRACK11P Z1R1   CDOTS  FRACKR11PZ1LABELEQHRRENDEQUATIONWHEREFOOTNOTETHE SYMBOL J HERE DOES NOT REPRESENT SQRT1  IN  INSTANCES WHERE CONFUSION IS UNLIKELY WE MAY USE J AS AN INDEX  VALUEBEGINEQUATIONKJ  FRAC1PJ J1J FRACDJDZ1J 1PZ1RHZLABELEQPFEZTENDEQUATIONTHE INVERSE ZTRANSFORM CORRESPONDING TO REFEQHRR IS OF THE FORM HT  C0PT  C1 T PT  CDOTS  CR1 TR PT UTWHERE THE COEFFICIENTS CI ARE LINEARLY RELATED TO THE PFECOEFFICIENTS KIUSING COMPUTER SOFTWARE TO COMPUTE PARTIAL FRACTION EXPANSIONS SUCHAS THE TT RESIDUE OR TT RESIDUEZ COMMAND IN SC MATLAB ISRECOMMENDEDBEGINEXAMPLELET HZ  FRAC3  24 Z1  6 Z217Z1  1Z2WE DESIRE TO FIND THE IMPULSE RESPONSE HT  SINCE THE DEGREE OFTHE NUMERATOR IS THE SAME AS THE DEGREE OF THE DENOMINATOR WE DIVIDETHEN FIND THE PARTIAL FRACTION EXPANSION BEGINALIGNEDHZ  60  FRAC444Z1  5712Z115Z1  60  FRAC11012Z1  FRAC5315Z1ENDALIGNEDTHEN HT  60 DELTAT  1102T  535T QQUAD T GEQ 0ENDEXAMPLEBEGINEXERCISESITEM SHOW THAT REFEQPFEZT FOR THE PARTIAL FRACTION EXPANSION OF  A ZTRANSFORM WITH REPEATED ROOTS IS CORRECTITEM DETERMINE THE PARTIAL FRACTION EXPANSION OF THE FOLLOWINGBEGINARRAYLHSPACE8EMLTEXTA  HZ  FRAC13Z1115Z1  56 Z2 TEXTB  HZ  FRAC15Z1  6Z2115Z1  56 Z2TEXTC  HZ  FRAC2  3Z113 Z12 TEXTD  HZ  FRAC56Z113 Z1214Z1ENDARRAYCHECK YOUR RESULTS USING TT RESIDUEZ IN SC MATLABITEM INVERSES OF HIGHERORDER MODES  BEGINENUMERATE  ITEM PROVE THE FOLLOWING PROPERTY FOR Z TRANSFORMS  IF XT LEFTRIGHTARROW XZTHEN TXT LEFTRIGHTARROW Z FRACD XZDZITEM USING THE FACT THAT PT UT LEFTRIGHTARROW 11PZ1  SHOW THAT T PT UT LEFTRIGHTARROW FRACPZ11PZ12ITEM DETERMINE THE Z TRANSFORM OF T2 PT UTITEM BY EXTRAPOLATION DETERMINE THE ORDER OF THE POLE OF A MODE OF  THE FORM TK PT UT  ENDENUMERATEENDEXERCISESSUBSECTIONSTOCHASTIC MA AND AR MODELSLABELSECARPROCESSIN STOCHASTIC INDEXSTOCHASTIC PROCESSES MA AND AR MODELS THE INPUTFT IS ASSUMED TO TO BE A WHITE DISCRETETIME RANDOM PROCESS THATIS USUALLY ZERO MEAN  THE READER IS ENCOURAGED TO REVIEW THECONCEPTS OF RANDOM PROCESSES SUMMARIZED IN APPENDIX REFAPPDXRPTHE INPUT COEFFICIENT B0 IS SET TO 1 WITH THE INPUT POWERDETERMINED BY THE VARIANCE OF THE SIGNAL  THUS EFT  0 QQUAD TEXTFOR ALL  TAND EFT FBARS  BEGINCASES  SIGMAF2  T  S 0  TEXTOTHERWISEENDCASESSUBSUBSECTIONAUTOCORRELATION FUNCTIONSIGNAL PROCESSING OFTEN INVOLVES COMPARING TWO SIGNALS ONE MEANS OFCOMPARISON IS BY MEANS OF CORRELATION  WHEN A SIGNAL IS COMPARED WITHITSELF THE INDEXCORRELATION CORRELATION IS CALLEDAUTOCORRELATIONINDEXAUTOCORRELATION  FOR STOCHASTIC SIGNALS WEDEFINE THE AUTOCORRELATION OF A ZEROMEAN WIDESENSE STATIONARYSIGNAL YT ASBEGINEQUATION RYYLK  EYTK YBARTLLABELEQAUTOCORRDEFENDEQUATIONOR EQUIVALENTLY RYYK  EYT YBARTK  THEAUTOCORRELATION FUNCTION HAS THE PROPERTY THATBEGINEQUATION RYYK  RBARYYKLABELEQRHERMENDEQUATIONFOR REAL RANDOM PROCESSES RYYK  RYYK AN EVEN FUNCTIONOF K INDEXEVEN  FUNCTIONFOR THE MA PROCESS YT  FT  BBAR1 FT1  CDOTS BBARQ FTQIT IS STRAIGHTFORWARD TO SHOW THAT THE AUTOCORRELATION FUNCTION ISBEGINEQUATION RYYK  SIGMAF2 SUML BLK BBARLLABELEQMAAUTOCORRENDEQUATIONWHERE THE SUM IS OVER ALL VALUES L SUCH THAT BL OR BLK ARENOT ZERO AND B0  1  FOR THE AR MODELBEGINEQUATION YT  ABAR1 YT1  CDOTS  ABARP YTP  FTLABELEQAR2ENDEQUATIONMULTIPLY BOTH SIDES BY YBARTL AND TAKE EXPECTATIONS TO OBTAINBEGINEQUATION  LABELEQYW1  ELEFTSUMK0P ABARK YTKYBARTLRIGHT  EFT  YBARTLENDEQUATIONWE RECOGNIZE THAT EYTK YBARTL  RYYLKAND THAT THE RIGHTHAND SIDE EFTYBARTL  0FOR L0 SINCE FT IS A WHITENOISE PROCESS  THEN USING THE FACTTHAT A01 WE CAN WRITEBEGINEQUATION RYYL  ABAR1 RYYL1  ABAR2 RYYL2  CDOTS ABARP RYYLPQQUADTEXT FOR  L  0LABELEQAR3ENDEQUATIONTHIS DIFFERENCE EQUATION FOR THE AUTOCORRELATION IS SIMILAR TO THE EQUATION FORTHE ORIGINAL DIFFERENCE EQUATION IN REFEQAR2  STACKINGREFEQAR3 FOR L12LDOTSP WE OBTAIN BEGINEQUATION  LABELEQYW2  BEGINBMATRIX RYY0 RYY1  CDOTS  RYYP1 RYY1  RYY0  CDOTS  RYYP2 VDOTS RYYP1  RYYP2  CDOTS  RYY0  ENDBMATRIXBEGINBMATRIX ABAR1  ABAR2  VDOTS  ABARPENDBMATRIX BEGINBMATRIX RYY1  RYY2  VDOTS  RYYP ENDBMATRIXENDEQUATIONCONJUGATING BOTH SIDES USING REFEQRHERM WE OBTAINBEGINEQUATION  LABELEQYW3  BEGINBMATRIX RYY0 RYY1  CDOTS  RYYP1 RBARYY1  RYY0  CDOTS  RYYP2 VDOTS RBARYYP1  RBARYYP2  CDOTS  RYY0  ENDBMATRIXBEGINBMATRIX A1  A2  VDOTS  APENDBMATRIX BEGINBMATRIX RBARYY1  RBARYY2  VDOTS   RBARYYP ENDBMATRIX ENDEQUATIONTHESE EQUATIONS ARE KNOWN AS THE EM YULEWALKER EQUATIONSINDEXYULEWALKER EQUATIONS  WE COMMONLY WRITE REFEQYW3 AS R WBF  RBFWHERE WBF  BEGINBMATRIX A1  A2 CDOTS  APENDBMATRIXTQQUAD RBF  BEGINBMATRIXRBARYY1  RBARYY2  CDOTS   RBARYYP ENDBMATRIXTHE MATRIX R IS SAID TO BE THE EM AUTOCORRELATION MATRIX OF YTHROUGHOUT THE BOOK WE WILL HAVE CONSIDERABLE TO SAY ABOUT THEPROPERTIES OF R AND ALGORITHMS THAT OPERATE ON IT  FOR NOW WE MAKETHE FOLLOWING OBSERVATIONSBEGINENUMERATEITEM R IS EM HERMITIAN SYMMETRIC INDEXHERMITIAN    SYMMETRICSEESYMMETRIC INDEXSYMMETRIC WHICH MEANS THAT R  RHWE WILL SEE THAT THIS MEANS THAT THE EIGENVALUES OF R ARE REAL ANDTHE EIGENVECTORS CORRESPONDING TO DISTINCT EIGENVALUES ARE ORTHOGONALIF R IS REAL THEN R IS EM SYMMETRIC RT  RITEM R IS A EM TOEPLITZ MATRIX INDEXTOEPLITZ MATRIX WHICH  MEANS THAT R IS CONSTANT ALONG THE DIAGONALS  IF RIJ DENOTES  THE IJTH  ELEMENT OF R THEN RJJ  RIJTHE ELEMENTS OF R DEPENDS ONLY ON THE DIFFERENCE BETWEEN THE INDEXVALUES  WE SHALL SEE THAT THE TOEPLITZ STRUCTURE OF R LEADS TOEFFICIENT ALGORITHMS FOR SOLVING EQUATIONS SIMILAR TO THE YULEWALKEREQUATIONSENDENUMERATEBEGINEXERCISESITEM SHOW THAT THE AUTOCORRELATION FUNCTION DEFINED IN  REFEQAUTOCORRDEF HAS THE PROPERTY THAT  RYYK  RBARYYKITEM SHOW THAT REFEQMAAUTOCORR IS CORRECTITEM FOR THE MA PROCESS YT  FT  2FT1  3FT2WHERE FT IS ZEROMEAN WHITE RANDOM PROCESS WITH SIGMAF2  1DETERMINE THE MATSIZE44 AUTOCORRELATION MATRIX RITEM FOR THE FIRSTORDER REAL AR PROCESS YT1  A1 YT   FT1WITH A11 WITH EFT  0 SHOW THATBEGINEQUATIONSIGMAY2  EY2T  FRACSIGMAF21A12LABELEQFIRSTARVARENDEQUATIONITEM FOR AN AR PROCESS REFEQAR2 DRIVEN BY A WHITENOISE  SEQUENCE FT WITH VARIANCE SIGMAF2 SHOW THATBEGINEQUATIONSIGMAF2  SUMI0P AI RYYI HAYKIN P 120LABELEQARINPUTVARENDEQUATIONITEM LET YT  7YT1  12 Y2T  FT WHERE FT IS A  ZEROMEAN WHITENOISE RANDOM PROCESS WITH SIGMAF2  2    BEGINENUMERATE  ITEM WRITE  THE YULEWALKER EQUATIONS FOR Y  ITEM DETERMINE RYY1 AND RYY2  ITEM FIND SIGMAY2  ENDENUMERATEITEM CONSIDER THE SECONDORDER REAL AR PROCESSBEGINEQUATION YT2  A1 YT1  A2 YT  FT2LABELEQYULEWALKER21ENDEQUATIONWHERE FT IS A ZEROMEAN WHITENOISE SEQUENCE  THE DIFFERENCEEQUATION IN REFEQAR3 HAS A CHARACTERISTIC EQUATION WITH ROOTS P1 P2  FRAC12A1 PM SQRTA12  4A2BEGINENUMERATEITEM USING THE YULEWALKER EQUATIONS SHOW THAT IF THE  AUTOCORRELATION VALUES  RYYLK  EYTKYBARTLARE KNOWN THEN THE MODEL PARAMETERS MAY  BE DETERMINED FROMBEGINEQUATIONBEGINSPLITA1  FRACR1R0  R2 R20   R21 A2  FRACR0R2  R21 R20   R21ENDSPLITLABELEQYW5ENDEQUATIONITEM ON THE OTHER HAND IF SIGMAY2  R0 AND A1 AND  A2 ARE KNOWN SHOW THAT THE AUTOCORRELATION VALUES CAN BE  EXPRESSED AS  BEGINEQUATION    LABELEQYW6BEGINSPLIT    RYY1  FRACA11A2 SIGMAY2RYY2  SIGMAY2LEFT FRACA121A2  A2RIGHTENDSPLITENDEQUATIONITEM USING REFEQARINPUTVAR AND THE RESULTS OF THIS PROBLEM  SHOW THAT  BEGINEQUATION    LABELEQYW7    RYY0  SIGMAY2  LEFTFRAC1A21A2RIGHT    FRACSIGMAF21A22  A12  ENDEQUATIONITEM USING RYY0  SIGMAY2 AND RYY1  A1  SIGMAY21A2 AS INITIAL CONDITIONS FIND AN EXPLICIT SOLUTION  TO THE YULEWALKER DIFFERENCE EQUATION RYYK  A1 RYYK1  A2 RYYK2  0IN TERMS OF P1 P2 AND SIGMAY2 HAYKIN P 121ENDENUMERATEITEM FOR THE SECONDORDER DIFFERENCE EQUATION YT2  7 YT1 12 YT  FT2WHERE FT IS A ZEROMEAN WHITE SEQUENCE WITH SIGMAF2  1DETERMINE SIGMAY2  RYY0 RYY1 AND RYY2ITEM A RANDOM PROCESS YT HAVING ZEROMEAN AND MATSIZEMM  AUTOCORRELATION MATRIX R IS APPLIED TO AN FIR FILTER WITH IMPULSE  RESPONSE VECTOR HBF  H0H1H2LDOTSHM1  DETERMINE  THE AVERAGE POWER OF THE FILTER OUTPUTENDEXERCISESSUBSECTIONREALIZATIONSA BLOCK DIAGRAM OR EM REALIZATION OF REFEQARMA CAN BEEASILY DERIVED  THE REALIZATION PRESENTED HERE ISKNOWN IN THE CONTROL LITERATURE AS THE EM CONTROLLER CANONICAL  FORM  INDEXCONTROLLER CANONICAL FORM WRITE THE SYSTEM FUNCTION ASBEGINEQUATION HZ  FRACYZWZ FRACWZFZ  LEFTSUMK0Q BBARK  ZK RIGHT LEFTFRAC11    SUMK1P ABARK ZKRIGHT  H1Z H2ZLABELEQHZ2AENDEQUATIONWHERE THE SIGNAL WZ HAS BEEN ARTIFICIALLY INTRODUCED  FROM THETRANSFER FUNCTION H2Z WE GET THE RELATIONSHIPBEGINEQUATIONWZ1SUMK1P ABARK ZK  FZLABELEQTRANSFER1ENDEQUATIONCORRESPONDING TO THE DIFFERENCE EQUATIONWT  ABAR1 WT1  CDOTS  ABARP WTP  FTOR WT  FT  ABAR1 WT1  ABAR2 WT2  LDOTS  ABARP WTPA BLOCK DIAGRAM OF A  REALIZATION OF REFEQTRANSFER1 IS SHOWN IN FIGUREREFFIGTRANSFER1BEGINFIGURETBP  BEGINCENTER INPUTPICTUREDIRTRANSFER1LATEX INPUTPICTUREDIRTRANSFER1  CAPTIONREALIZATION OF THE AR PART OF A TRANSFER FUNCTION  LABELFIGTRANSFER1ENDCENTERENDFIGUREFROM H1Z IN REFEQHZ2A WE HAVE YZ  WZBZWITH THE CORRESPONDING DIFFERENCE EQUATION YT  BBAR0 WT  BBAR1 WT1  CDOTS  BBARQ WTQTHIS REALIZATION DRAWN ASSUMING THAT PQ IS SHOWN IN FIGUREREFFIGTRANSFER2 BEGINFIGUREHTBP  BEGINCENTERINPUTPICTUREDIRTRANSFER2LATEXINPUTPICTUREDIRTRANSFER2  CAPTIONCONTROLLER CANONICAL REALIZATION OF A TRANSFER FUNCTION  LABELFIGTRANSFER2ENDCENTERENDFIGUREWE EXPLORE OTHER POSSIBLE REALIZATIONS IN THE EXERCISESSUBSUBSECTIONSTATESPACE FORMINDEXSTATESPACE FORM CONSIDER THE BLOCK DIAGRAM IN FIGUREREFFIGTRANSFER3 IN WHICH THE OUTPUTS OF THE DELAY BLOCKS ARELABELED X1 X2 LDOTS XP FROM RIGHT TO LEFTBEGINFIGUREHTBP  BEGINCENTERINPUTPICTUREDIRTRANSFER3LATEXINPUTPICTUREDIRTRANSFER3  CAPTIONREALIZATION OF A TRANSFER FUNCTION WITH STATE VARIABLE LABELS  LABELFIGTRANSFER3ENDCENTERENDFIGUREFROM THIS BLOCK DIAGRAM WE OBTAIN THE FOLLOWING EQUATIONSBEGINEQUATIONBEGINSPLITX1T1  X2T X2T1  X3T  EQSKIP VDOTS XP1T1  XPT XPT1  FT ABAR1 XPT ABAR2 XP1T  CDOTS ABARP1 X2T  ABARP X1T SMALLSKIP YT  BBARP X1T  BBARP1 X2T  CDOTS  BBAR2XP1T  BBAR1 XPT    QQUAD BBAR0FT  ABAR1 XPT  ABAR2 XP1T  CDOTS ABARP X1TENDSPLITLABELEQSTATE1ENDEQUATIONOBSERVE THAT THE DIRECT CONNECTION FROM INPUT F TO OUTPUT Y IS VIAB0  THE VARIABLES X1 X2 LDOTS XP ARE THE BF STATE  VARIABLES  LET XBFT BE THE BF STATE VECTOR XBFT  BEGINBMATRIX X1T  X2T  VDOTS  XPTENDBMATRIX WE ALSO INTRODUCE THE VECTORSBBF  UNDERBRACE00LDOTS 01P TEXT ELEMENTST CBF  BEGINBMATRIX BBARP  BBAR0 ABARP  BBARP1   BBAR0 ABARP1   VDOTS  BBAR1  BBAR0ABAR1  ENDBMATRIXQQUAD TEXTANDQQUADD  BBAR0 AND THE MATRIXBEGINEQUATION A  BEGINBMATRIX0100 CDOTS  00 0010 CDOTS  00 VDOTS 0000 CDOTS  01 ABARP  ABARP1  ABARP2  ABARP3  CDOTS ABAR2  ABAR1  ENDBMATRIXLABELEQASTATEMATENDEQUATIONIF B00 THEN CBF IS CBFT  BBARPBBARP1LDOTSBBAR1WHICH EXPLICITLY DISPLAYS THE NUMERATOR COEFFICIENTS OF HZTHE EQUATIONS IN REFEQSTATE1 CAN BE WRITTEN USING THESEDEFINITIONS ASBEGINEQUATIONBEGINSPLITXBFT1  A XBFT  BBF FT YT  CBFT XBFT  D FTENDSPLITLABELEQSTATE2ENDEQUATIONAN EQUATION OF THE FORM REFEQSTATE2 IS IN EM STATESPACEFORM  THE SYSTEM IS DENOTED AS ABBFCBFTD OR WHEN D0 ASABBFCBFT  ALTHOUGH THE TRANSFORMATION FROM THE TRANSFERFUNCTION TO REFEQSTATE2 WAS MADE BY A PARTICULAR STATEASSIGNMENT THE REFEQSTATE2 IS OF GENERAL APPLICABILITY AND THEMATRICES DOES NOT NECESSARILY HAVE THE STRUCTURE OFREFEQASTATEMAT WHEN THE STATESPACE SYSTEM IN REFEQSTATE2 DOES HAVE THE AMATRIX OF THE FORM REFEQASTATEMAT THE STATESPACE SYSTEM ISSAID TO BE IN EM CONTROLLER FORM INDEXCONTROLLER FORM  THEFORM OF THE MATRIX A WITH ONES ABOVE THE DIAGONAL AND COEFFICIENTS ONTHE LAST ROW IS CALLED A FIRST EM COMPANION MATRIXCOMPANION MATRICES AREDISCUSSED IN SECTION REFSECCOMPANMAT INDEXCOMPANION MATRIXSUBSUBSECTIONSYSTEM TRANSFORMATIONS SIMILAR MATRICESTHE STATEVARIABLE REPRESENTATION IS NOT UNIQUE  IN FACT AN INFINITENUMBER OF POSSIBLE REALIZATIONS EXIST WHICH ARE MATHEMATICALLYEQUIVALENT ALTHOUGH NOT NECESSARILY IDENTICAL IN PHYSICAL OPERATIONWE CAN CREATE A NEW STATEVARIABLE REPRESENTATION BY LETTING XBF TZBF FOR ANY INVERTIBLE MATSIZEPP MATRIX T  THENREFEQSTATE2 BECOMES BEGINALIGNEDTZBFT1  ATZBFT  BBF FT YT  CBFT TZBF  D FTENDALIGNEDWHICH CAN BE WRITTEN ASBEGINEQUATION  LABELEQSTATE3  BEGINSPLITZBFT1  ABAR ZBFT  BBFBAR FT YT  CBFBART ZBFT  DBAR FTENDSPLITENDEQUATIONWHERE ABAR  T1 A T QQUAD BBFBAR  T1BBFQQUAD CBFBAR  TTCBF QQUAD DBAR  DTHE BAR DOES NOT INDICATE CONJUGATION IN THIS INSTANCE  MATRICESA AND ABAR THAT ARE RELATED AS ABAR  T1 A T ARE SAID TOBE EM SIMILAR  INDEXSIMILAR MATRIX IT IS STRAIGHTFORWARD TOSHOW THAT THE SYSTEM ABARBBFBARCBFBARTDBAR HAS THE SAMEINPUTOUTPUT RELATIONSHIPS DYNAMICS AND TRANSFER FUNCTION AS DOESTHE SYSTEM ABBFCBFTD  WHICH MEANS AS WE SHALL SEE THATA AND ABAR HAVE THE SAME EIGENVALUESSUBSUBSECTIONTIMEVARYING STATESPACE MODELWHEN THE SYSTEM IS TIMEVARYING INDEXTIMEVARYING SYSTEM THESTATESPACE REPRESENTATION ISBEGINEQUATIONBEGINSPLITXBFT1  AT XBFT  BBFT FT YT  CBFTT XBFT  DT FTENDSPLITLABELEQSTATE4ENDEQUATIONIN WHICH THE EXPLICIT DEPENDENCE OF ATBBFTCBFTTDT ONTHE TIME INDEX T IS SHOWNSUBSUBSECTIONTRANSFER FUNCTION FROM THE STATESPACE MODELTHE TIMEINVARIANT STATESPACE FORM CAN BE REPRESENTED USING ASYSTEM FUNCTION  WE CAN TAKE THE ZTRANSFORM OF REFEQSTATE2THE ZTRANSFORM OF A VECTOR IS SIMPLY THE TRANSFORM OF EACHCOMPONENT  WE OBTAIN THE EQUATIONSBEGINALIGNZ XBFZ  A XBFZ  BBF FZ LABELEQSS1 YZ  CBFT XBFZ  D FZ LABELEQSS2ENDALIGNFROM REFEQSS1 WE OBTAIN ZI  AXBFZ  BBF FZTHE MATRIX I IS THE IDENTITY MATRIX  THEN XBFZ  ZIA1 BBF FZWHERE ZIA1 IS THE MATRIX INVERSE OF ZIA  MATRIX INVERSESARE DISCUSSED IN CHAPTER REFCHAPMATINV  SUBSTITUTINGXBFZ INTO REFEQSS2 WE OBTAIN YZ  CBFT ZIA1 BBF  DFZSINCE YZ AND FZ ARE SCALAR SIGNALS WE CAN FORM THEIR RATIO TOOBTAIN THE SYSTEM FUNCTIONBEGINEQUATION HZ  FRACYZFZ  CBFT ZIA1 BBF  DLABELEQHZ3ENDEQUATIONBEGINEXAMPLE  WE WILL GO FROM A SYSTEM FUNCTION TO STATESPACE FORM AND BACK  LET HZ  FRAC3  2Z1  4 Z21  3Z1  5 Z2IN SOME LITERATURE IT IS COMMON TO ELIMINATE NEGATIVE POWERS OFZ IN THE SYSTEM FUNCTIONS  THIS CAN BE DONE BY MULTIPLYING BYZ2Z2 HZ  FRAC3Z2  2Z  4Z2  3Z  5PLACING THE SYSTEM IN CONTROLLER FORM WE HAVE BBF  BEGINBMATRIX0 1 ENDBMATRIX QQUAD CBF BEGINBMATRIX435  233 ENDBMATRIX  BEGINBMATRIX  11  7  ENDBMATRIX A  BEGINBMATRIX01  53 ENDBMATRIX QQUAD D3TO RETURN TO A TRANSFER FUNCTION WE FIRST COMPUTE ZIA  BEGINBMATRIXZ  1  5  Z3 ENDBMATRIXAND ZIA1  FRAC1ZZ3  5BEGINBMATRIX Z3  1  5   ZENDBMATRIXINDEXMATRIX INVERSEMATSIZE22THE INVERSE OF A MATSIZE22 MATRIX IS BOXEDBEGINBMATRIXA  B  CD ENDBMATRIX  FRAC1AD  BCBEGINBMATRIX D  B  C  A ENDBMATRIX THEN USING REFEQHZ3 WE OBTAIN HZ  FRAC1Z23Z5117BEGINBMATRIXZ3  1  5  ZENDBMATRIX BEGINBMATRIX0  1 ENDBMATRIX  D  FRAC3Z2  2Z   4Z23Z5AS EXPECTEDTO EMPHASIZE THAT THE STATESPACE REPRESENTATION IS NOT UNIQUE LET ATILDE  BEGINBMATRIX 5  45  15  35ENDBMATRIXQQUADBBFTILDE  BEGINBMATRIX 1  1 ENDBMATRIX QQUAD CBFTILDE BEGINBMATRIX 2  9 ENDBMATRIX QQUAD DTILDE  3THIS SYSTEM IS NOT IN CONTROLLER FORM  WE  MAY VERIFY THAT HTILDEZ  CBFTILDET ZIATILDE1 BBFTILDE  DTILDE  HZENDEXAMPLESUBSUBSECTIONSOLUTION OF THE STATESPACE DIFFERENCE EQUATIONIT CAN BE SHOWN SEE EXERCISE REFEXSTATEOUT THAT STARTING FROM ANINITIAL STATE XBF0 THE STATESPACE SYSTEM REFEQSTATE2 HASTHE SOLUTIONBEGINEQUATION XBFT  AT XBF0  SUMK0T1 AK BBF FT1KLABELEQXNDT1ENDEQUATIONTHE SUM IS SIMPLY THE CONVOLUTION OF AT BBF WITH FT1  THEOUTPUT IS YT  CBFT AT XBF0  SUMK0T1 CBFT AK BBFFT1K  D FTTHE QUANTITIES CBFT AK BBF ARE CALLED THE EM MARKOV  PARAMETERS INDEXMARKOV PARAMETERS OF THE SYSTEM THEY CORRESPONDTO THE IMPULSE RESPONSE OF THE SYSTEM ABBFCBFTSUBSUBSECTIONMULTIPLE INPUTS AND OUTPUTSSTATESPACE REPRESENTATION CAN BE USED TO REPRESENT SIGNALS WITHMULTIPLE INPUTS AND OUTPUTS  FOR EXAMPLE A SYSTEM MIGHT BE DESCRIBEDBY BEGINALIGNEDXBFT1  BEGINBMATRIX X1T1  X2T1  X3T1  ENDBMATRIX BEGINBMATRIX 321  125  211 ENDBMATRIXXBFT  BEGINBMATRIX21  15  11 ENDBMATRIX BEGINBMATRIX F1T  F2T ENDBMATRIX YBFT  BEGINBMATRIX Y1T   Y2T ENDBMATRIX BEGINBMATRIX2  4 6  120 ENDBMATRIXXBFTENDALIGNEDTHIS SYSTEM HAS THREE STATE VARIABLES TWO INPUTS AND TWO OUTPUTSIN GENERAL A MULTIINPUT MULTIOUTPUT SYSTEM IS OF THE FORMBEGINEQUATIONBEGINSPLITXBFT1   A XBFT  B UBFT YBFT  C XBFT  D UBFTENDSPLITLABELEQSTATEGENENDEQUATIONIF THERE ARE P STATE VARIABLES AND L INPUTS AND M OUTPUTS THEN BEGINALIGNEDA  TEXT IS  MATSIZEPP B  TEXT IS  MATSIZEPL C  TEXT IS  MATSIZEMP D  TEXT IS  MATSIZEMLENDALIGNEDSUBSUBSECTIONSTATESPACE SYSTEMS IN NOISEA SIGNAL MODEL THAT ARISES FREQUENTLY IN PRACTICE ISBEGINEQUATIONBEGINSPLITXBFT1  A XBFT  B UBFT  WBFT YBFT  C XBFT  D UBFT  VBFTENDSPLITLABELEQSTATEGEN1ENDEQUATIONTHE SIGNALS WBFT AND VBFT REPRESENT NOISE PRESENT IN THESYSTEM  THE VECTOR WBFT IS AN INPUT TO THE SYSTEM THATREPRESENTS UNKNOWN RANDOM COMPONENTS  FOR EXAMPLE IN MODELINGAIRPLANE DYNAMICS WBFT MIGHT REPRESENT RANDOM GUSTS OF WINDTHE VECTOR VBFT REPRESENTS MEASUREMENT NOISE  MEASUREMENT NOISEIS A FACT OF LIFE IN MOST PRACTICAL CIRCUMSTANCES  GETTING USEFULRESULTS OUT OF NOISY MEASUREMENTS IS AN IMPORTANT ASPECT OF SIGNALPROCESSING  IT HAS BEEN SAID THAT NOISE IS THE SIGNAL PROCESSORSBREAD AND BUTTER WITHOUT THE NOISE MANY PROBLEMS WOULD BE TOO TRIVIALTO BE OF SIGNIFICANT INTERESTTHIS BOOK WILL TOUCH ON SOME ASPECTS OF SYSTEMS IN STATESPACE FORMBUT A THOROUGH STUDY OF LINEAR SYSTEMS INCLUDING STATESPACECONCEPTS IS BEYOND THE SCOPE OF THIS BOOK  FOR SUPPLEMENTARYTREATMENTS SEE THE REFERENCE SECTION AT THE END OF THIS CHAPTERBEGINEXERCISESITEM PLACE THE FOLLOWING INTO STATE VARIABLE FORM CONTROLLER  CANONICAL FORM AND DRAW A REALIZATIONBEGINARRAYLHSPACE8EMLTEXTA  HZ  FRAC13Z1115Z1  56 Z2 TEXTB  HZ  FRAC15Z1  6Z2115Z1  56 Z2ENDARRAYITEM DETERMINE THE FIRST FOUR NONZERO MARKOV PARAMETERS OF THE SYSTEMS IN THE  PREVIOUS EXERCISEITEM IN ADDITION TO THE BLOCK DIAGRAM SHOWN IN FIGURE  REFFIGTRANSFER2 THERE ARE MANY OTHER FORMS  THIS PROBLEM  INTRODUCES ONE OF THEM THE EM OBSERVER CANONICAL FORM INDEXOBSERVER CANONICAL FORM  BEGINENUMERATE  ITEM SHOW THAT THE ZTRANSFORM RELATION IMPLIED BY REFEQARMA    CAN BE WRITTEN ASBEGINEQUATIONBEGINSPLIT YZ   BBAR0 FZ  BBAR1 FZ  ABAR1 YZZ1  BBAR2 FZ  ABAR2 YZ Z2   CDOTS  BBARP FZ  ABARP YZZ1ENDSPLITLABELEQBLOCK2ENDEQUATIONITEM DRAW A BLOCK DIAGRAM REPRESENTING REFEQBLOCK2 CONTAINING  P DELAY ELEMENTS ITEM LABEL THE OUTPUTS OF THE DELAY ELEMENTS FROM RIGHT TO LEFT AS  X1 X2 LDOTS XP  SHOW THAT THE SYSTEM CAN BE PUT INTO STATE  SPACE FORM WITH A  BEGINBMATRIX ABAR1  1  0  CDOTS  0 ABAR2  0  1  CDOTS  0 VDOTS ABARP1  0  0 CDOTS  0 ABARP  0  0  CDOTS  1 ENDBMATRIXQQUAD BBF  BEGINBMATRIX BBAR1  ABAR1 BBAR0  BBAR2  ABAR2 BBAR0 CDOTS BBARP1  ABARP1 BBAR0 BBARP  ABARP BBAR0 ENDBMATRIXQQUAD CBF  BEGINBMATRIX 1  0  0  VDOTS  0 ENDBMATRIXQQUAD D  BBAR0A MATRIX A OF THIS FORM IS SAID TO BE IN EM SECOND COMPANION  FORM INDEXSECOND COMPANION FORMITEM DRAW THE BLOCK DIAGRAM IN OBSERVER CANONICAL FORM FOR HZ  FRAC 2  3Z1  4 Z21  Z1  6Z2  7 Z3AND DETERMINE THE SYSTEM MATRICES ABBF CBFT DENDENUMERATEITEM ANOTHER BLOCK DIAGRAM REPRESENTATION IS BASED UPON THE PARTIAL  FRACTION EXPANSION    BEGINENUMERATE  ITEM ASSUME INITIALLY THAT THERE ARE NO REPEATED ROOTS SO THAT HZ  SUMK1P FRACNK1PK Z1ITEM DRAW A BLOCK DIAGRAM REPRESENTING THE PARTIAL FRACTION EXPANSION  BY USING THE FACT THAT FRACYZFZ  FRAC11P Z1 HAS THE BLOCK DIAGRAM BEGINCENTERINPUTPICTUREDIRTRANSFER5LATEXENDCENTERITEM LET XI I12LDOTS P DENOTE THE OUTPUTS OF THE DELAY  ELEMENTS  SHOW THAT THE SYSTEM CAN BE PUT INTO STATESPACE FORM  WITH  A  BEGINBMATRIX P1  0  0  CDOTS 00P2  0  CDOTS  0  VDOTS 0  0  0  CDOTS  PP ENDBMATRIXQQUAD BBF  BEGINBMATRIX 1  1  VDOTS  1 ENDBMATRIXQQUAD CBF  BEGINBMATRIX  N1  N2  VDOTS  NP ENDBMATRIXQQUAD D  B0A MATRIX A IN THIS FORM IS SAID TO BE A EM DIAGONALMATRIX INDEXDIAGONAL MATRIXITEM DETERMINE THE PARTIAL FRACTION EXPANSION OF  HZ  FRAC 1  2Z1 1  5 Z1  6 Z2AND DRAW THE BLOCK DIAGRAM BASED UPON IT  DETERMINE ABBF CBFDITEM WHEN THERE ARE REPEATED ROOTS THINGS ARE SLIGHTLY MORE  COMPLICATED  CONSIDER FOR SIMPLICITY A ROOT APPEARING ONLY  TWICE  DETERMINE THE PARTIAL FRACTION EXPANSION OF HZ  FRAC1Z11 2 Z115Z12BE CAREFUL ABOUT THE REPEATED ROOT  ITEM DRAW THE BLOCK DIAGRAM CORRESPONDING TO HZ IN PARTIAL  FRACTION FORM USING ONLY THREE DELAY ELEMENTS  ITEM SHOW THAT THE STATE VARIABLES CAN BE CHOSEN SO THAT A  BEGINBMATRIX 5 00 15  0 0  0  2  ENDBMATRIXA MATRIX IN THIS FORM BLOCKS ALONG THE DIAGONAL EACH BLOCK BEINGEITHER DIAGONAL OR DIAGONAL WITH ONES IN IT AS SHOWN IS IN EM JORDANFORM INDEXJORDAN FORMENDENUMERATEITEM SHOW THAT THE SYSTEM IN REFEQSTATE3 HAS THE SAME TRANSFER  FUNCTION AND SOLUTION AS DOES THE SYSTEM IN REFEQSTATE2ITEM LABELEXSTATEOUT SHOW THAT REFEQXNDT1 IS CORRECTITEM FOR A SYSTEM IN STATESPACE REPRESENTATION  BEGINENUMERATE  ITEM SHOW THAT REFEQXNDT1 IS CORRECT  ITEM FOR A TIMEVARYING SYSTEM AS IN REFEQSTATE4 DETERMINE    A REPRESENTATION SIMILAR TO REFEQXNDT1  ENDENUMERATEITEM CITEKAILATH80 LET A1BBF1CBF1T AND  A2BBF2CBF2T BE TWO SYSTEMS  DETERMINE THE SYSTEM  ABBFCBFT OBTAINED BY CONNECTING THESE  BEGINENUMERATE  ITEM IN SERIES  ITEM IN PARALLEL  ITEM IN A FEEDBACK CONFIGURATION WITH A1BBF1CBF1T IN    THE FORWARD LOOP AND A2BBF2CBF2T IN THE FEEDBACK LOOP  ENDENUMERATEITEM SHOW THAT  BEGINBMATRIX A A1  0  A2 ENDBMATRIX QQUADBEGINBMATRIXBBF  ZEROBF ENDBMATRIX QQUAD CBFT QBFTAND BEGINBMATRIX A 0  A1  A2 ENDBMATRIX QQUADBEGINBMATRIXBBF  QBF ENDBMATRIX QQUAD CBFT ZEROBFAND ABBFCBFT ALL HAVE THE SAME TRANSFER FUNCTION FOR ALLVALUES OF A1 A2 AND QBF THAT LEAD TO VALID MATRIXOPERATIONS  CONCLUDE THAT REALIZATIONS CAN HAVE DIFFERENT NUMBERS OF STATESENDEXERCISESSUBSECTIONCONTINUOUSTIME NOTATIONLABELSECCONTSTATEFOR CONTINUOUSTIME SIGNALS AND SYSTEMS THE CONCEPTS FOR INPUTOUTPUTRELATIONS TRANSFER FUNCTIONS AND STATESPACE REPRESENTATIONSTRANSLATE DIRECTLY WITH Z1 UNIT DELAY REPLACED BY1S INTEGRATION  THE READER IS ENCOURAGED TO REVIEW THEDISCRETETIME NOTATIONS PRESENTED ABOVE AND REFORMULATE THEEXPRESSIONS GIVEN IN TERMS OF CONTINUOUSTIME SIGNALS  THE PRINCIPALDIFFERENCE BETWEEN DISCRETE TIME AND CONTINUOUS TIME ARISES IN THEEXPLICIT SOLUTION OF THE DIFFERENTIAL EQUATIONBEGINEQUATIONBEGINSPLITXBFDOTT  AT XBFT  BT FBFT YBFT  CT XBF  DT FBFTENDSPLIT  LABELEQXNCT1ENDEQUATIONFOR THE TIMEINVARIANT SYSTEM WHEN ABCD IS CONSTANT THESOLUTION ISBEGINEQUATION XBFT  EAT XBF0  INT0 T EATLAMBDA BFBFLAMBDA DLAMBDALABELEQXBFT2ENDEQUATIONWHERE EAT IS THE EM MATRIX EXPONENTIAL INDEXMATRIX EXPONENTIAL DEFINED IN TERMS OF ITS  TAYLOR SERIES INDEXTAYLOR SERIESBEGINEQUATION EAT  I  AT  A2 FRACT22  A3 FRACT33  CDOTS LABELEQEXPMAT1ENDEQUATIONWHERE I IS THE EM IDENTITY MATRIX  WE NOTE IN PARTICULAR THAT FRACDDT EAT  AEATSEE SECTION REFSECTAYLOR FOR A REVIEW OF TAYLOR SERIES ANDSECTION REFSECDIAGONAL FOR MORE ON THE MATRIX EXPONENTIAL  THEMATRIX EXPONENTIAL CAN ALSO BE EXPRESSED IN TERMS OF LAPLACETRANSFORMS INDEXMATRIX EXPONENTIALSEESTATE TRANSITION MATRIX EAT  LC1SIA1WHERE SIA IS KNOWN AS THE EM CHARACTERISTIC MATRIX OF A ANDLC CDOT DENOTES THE LAPLACE TRANSFORM OPERATOR INDEXLLCINDEXLAPLACE TRANSFORM LCFT  INT0INFTY FTESTDTAN INTERESTING AND FRUITFUL CONNECTION IS THE FOLLOWING  RECALL THEGEOMETRIC EXPANSIONBEGINEQUATION FRAC11X  1XX2  X3  CDOTSLABELEQGEOM1ENDEQUATIONWHICH CONVERGES FOR X  1  INDEXGEOMETRIC SERIES THIS ALSOAPPLIES TO GENERAL OPERATORS INCLUDING MATRICES SO THAT FOR ANOPERATOR FBEGINEQUATION  IF1  I  F  F2  F3  CDOTSLABELEQNEUMANN1ENDEQUATIONWHEN F1  THE NOTATION F SIGNIFIES THE OPERATOR NORM ITIS DISCUSSED IN SECTION REFSECMATNORM  THE EXPANSIONREFEQNEUMANN1 IS KNOWN AS THE NEUMANN EXPANSION SEE SECTIONREFSECNEUM  INDEXNEUMANN EXPANSION USING REFEQNEUMANN1THE EXPRESSION SIA1 IS FRAC1SI  AS  A2S2  CDOTSFROM WHICH THE TAYLOR SERIES FORMULA REFEQEXPMAT1 FOLLOWSIMMEDIATELY USING THE INVERSE LAPLACE TRANSFORMFOR THE TIMEINVARIANT SINGLEINPUT SINGLEOUTPUT SYSTEM BEGINALIGNEDXBFDOTT  A XBFT  BBF FT YT  CBFT XBFTENDALIGNEDTHE TRANSFER FUNCTION IS HS  CBFSIA1 BBFUSING REFEQNEUMANN1 WE WRITE HS  SUMI1INFTY HI SIWHERE HI  CBFT AI1 BBF ARE THE MARKOV PARAMETERS OF THECONTINUOUSTIME SYSTEM INDEXMARKOV PARAMETERSTHE FIRST TERM OF REFEQXBFT2 IS THE SOLUTION OF THE HOMOGENEOUSDIFFERENTIAL EQUATION XBFDOTT  A XBFTWHILE THE SECOND TERM OF REFEQXBFT2 IS THE PARTICULAR SOLUTIONOF  XBFDOT  AT XBFT  BT FBFTIT IS STRAIGHTFORWARD TO SHOW SEE EXERCISE REFEXUPDATEDEQ THATSTARTING FROM A STATE XBFTAU THE STATE AT TIME T CAN BEDETERMINED ASBEGINEQUATION XBFT  EATTAU XBFTAU  INTTAUT EATTAU BFBFLAMBDA DLAMBDALABELEQSTATEUPDATEENDEQUATIONSINCE EATTAU PROVIDES THE MECHANISM FOR MOVING FROM STATEXBFTAU TO STATE XBFT IT IS CALLED THE EM STATE  TRANSITION MATRIX LABELSTATE TRANSITION MATRIXFOR THE TIMEVARYING SYSTEM REFEQXNCT1 INDEXTIMEVARYING SYSTEM THE SOLUTION CAN BE WRITTEN ASBEGINEQUATIONXBFT  PHIT0 XBF0  INT0T PHITLAMBDA BLAMBDAFBFLAMBDA DLAMBDALABELEQXBFT3ENDEQUATIONWHERE PHITTAU IS THE STATETRANSITION MATRIX INDEXSTATE  TRANSITION MATRIX  NOT DETERMINED BYTHE MATRIX EXPONENTIAL IN THE TIMEVARYING CASE  THE FUNCTIONPHITTAU HAS THE FOLLOWING PROPERTIESBEGINENUMERATEITEM PHITT  IITEM PARTIALDPHITTAUT  AT PHITTAUITEM PHITTAU  PHITAUT1 THE MATRIX INVERSEENDENUMERATESUBSUBSECTIONCONTINUOUSTIME NOTATIONWE NOW SUMMARIZE BRIEFLY SOME SYSTEMS CONCEPTS FOR CONTINUOUS TIMETHE KEY DIFFERENCE IS THAT INSTEAD OF DIFFERENCE EQUATIONS ANDZTRANSFORMS WE DEAL WITH DIFFERENTIAL EQUATIONS AND LAPLACETRANSFORMS  WE WILL DEAL INPUTOUTPUT RELATIONSHIPS OF THE FORM YT  A1 FRACDDTYT  A2 FRACD2DT2 YT  CDOTS AP FRACDPDTP YT  B0 UT B1 FRACDDT UT  CDOTS  BQ FRACDQDTQ UTTAKING THE LAPLACE TRANSFORM AGAIN SETTING INITIAL CONDITIONS TOZERO YS SUMK0P AK SK  UZ SUMK0Q BK SKWHICH WE WRITE AS YS AS  US BSTHE SYSTEM FUNCTION ISBEGINEQUATION HS  FRACYSUS  FRACSUMK0Q BK SKSUMK0P AK SK   FRACSUMK0Q BK SK1 SUMK1PAK SK  FRACBSAS LABELEQHS1ENDEQUATIONTHE OUTPUT CAN BE EXPRESSED IN TERM OF THE INPUT AS YS  HSUSTAKING THE INVERSE  LAPLACE TRANSFORM AND RECALLING THECONVOLUTION PROPERTY MULTIPLICATION IN THE TRANSFORM DOMAINCORRESPONDS TO CONVOLUTION IN THE TIME DOMAIN WE OBTAIN YT  INTINFTYINFTY UTAU HTTAU DTAUWHERE THE IMPULSE RESPONSE HT IS THE INVERSE LAPLACE TRANSFORM OFHS  IN COMPUTING THE INVERSE TRANSFORM THE NUMERATOR ANDDENOMINATOR POLYNOMIALS OF THE SYSTEM FUNCTION HS OFREFEQHS1 ARE FACTORED HS  FRACB0 PRODK1Q SZIA0 PRODK1P SPIWHERE THE ZI ARE THE NONZERO ZEROS OF BS AND THE PI ARE THENONZERO ZEROS OF AS  IN THIS FORMWE OBSERVE THAT IF A POLE IS EQUAL TO A ZERO THE FACTORS CAN BECANCELED OUT OF BOTH THE NUMERATOR AND DENOMINATOR TO OBTAIN ANEQUIVALENT TRANSFER FUNCTION  ASSUMING FOR SIMPLICITY OF DISCUSSIONTHAT THE POLES ARE ALL UNIQUE NO REPEATED POLES AND THAT QP THENBY PARTIAL FRACTION EXPANSION THE SYSTEM FUNCTION CAN BE EXPRESSED ASBEGINEQUATIONHZ  SUMK1P FRACNKSPKLABELEQHS2ENDEQUATIONWHERE NK  HZSPKBIGLSPKTAKING THE CAUSAL INVERSE LAPLACE TRANSFORM OF REFEQHS2 WEOBTAIN HT  SUMK1P NK EPK TQQUAD T GEQ 0THE FUNCTIONS EPK T ARE THE NATURAL MODES OF THE SYSTEMFOR THE MODES TO BE BOUNDED IN TIME WE MUST HAVE REALPK LEQ 0IF THERE ARE REPEATED POLES IN HS A PARTIAL FRACTION CAN STILL BEOBTAINED BUT SOMEWHAT MORE CARE IS REQUIRED  IF HZ  FRACBSSPRTHEN THE PFE IS HZ  FRACK0SPR  FRACK1SPR1  CDOTS FRACKR1SPWHEREBEGINEQUATION  LABELPFES  KJ  FRAC1J FRACDJDSJ SPR HS BIGLSPENDEQUATIONIF Q GEQ P THEN THE RATIO OF POLYNOMIALS IS FIRST DIVIDED OUTBLOCK DIAGRAMS FOR CONTINUOUSTIME TRANSFER FUNCTIONS CAN BE DERIVEDJUST AS FOR DISCRETETIME TRANSFER FUNCTIONS  FIGUREREFFIGTRANSFER4 SHOWS THE CONTROLLER CANONICAL FORM OF A BLOCKDIAGRAM WITH STATEVARIABLE LABELS ON THE OUTPUTS OF THE INTEGRATORSBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODEINPUTPICTUREDIRTRANSFER4LATEX    CAPTIONCONTROLLER CANONICAL FORM FOR A CONTINUOUS TIME SYSTEM    LABELFIGTRANSFER4  ENDCENTERENDFIGUREFROM THE DIAGRAM WE CAN READ OFF THE STATEVARIABLE EQUATIONSBEGINEQUATIONBEGINSPLITXDOT1T  X2T XDOT2T  X3T  VDOTS XDOTP1T  XPT XDOTPT  UT A1 XPT A2 XP1T  CDOTS  AP1 X2T AP X1T SMALLSKIP YT  BP X1T  BP1 X2T  CDOTS  B2 XP1T  B1XPT    QQUAD B0UT  A1 XPT  A2 XP1T  CDOTS  AP X1TENDSPLITLABELEQSTATE3ENDEQUATIONTHE STATE VECTOR IS XBFT  BEGINBMATRIX X1T  X2T  VDOTS  XPTENDBMATRIX IN STATEVARIABLE FORM WE HAVEBEGINEQUATIONBEGINSPLITXBFDOTT  A XBFT  BBF UT YT  CBFT XBFT  D UTENDSPLITLABELEQSTATE4ENDEQUATIONWHERE XBFDOTT MEANS TO TAKE THE TIME DERIVATIVE OF EACH COMPONENTOF XBFT SEPARATELY AND A BBF CBF AND D ARE ASBEFORE  THE TRANSFER FUNCTION HS CAN BE EXPRESSED IN TERMS OF THE SYSTEMABBF CBF D ASBEGINEQUATION HS  FRACYSUS  CBFT SIA1 BBF  DLABELEQHS3ENDEQUATIONTHE OUTPUT CAN BE EXPRESSED AS YT  CBFT EATXBF0  INT0T CBFT EATTAU BUTAU  DTAUWHERE EAT IS THE MATRIX EXPONENTIAL DEFINED IN TERMS OF ITSTAYLOR SERIES EX  I  X  FRACX22  FRACX33  CDOTSFOR ANY SQUARE MATRIX X  TAYLOR SERIES ARE REVIEWED IN SECTIONREFSECTAYLOR  THE DYNAMICAL PROPERTIES OF THE MATRIX EXPONENTIALARE DISCUSSED IN SECTION REFSECMATEXPMORE GENERALLY WITH MULTIPLE INPUTS AND MULTIPLE OUTPUTS AND IN THEPRESENCE OF NOISE WE HAVE HAVE BEGINALIGNED XBFDOTT  A XBFT  B UBFT   WBFT YBFT  C XBFT  D UBFT  VBFTENDALIGNEDBEGINEXERCISESITEM FOR THE SYSTEM FUNCTION HZ  FRACS3  6S2  11 S  6 S3  9S2  23 S  15BEGINENUMERATEITEM DRAW THE CONTROLLER CANONICAL BLOCK DIAGRAMITEM DRAW THE BLOCK DIAGRAM IN JORDAN FORM DIAGONAL  FORM INDEXJORDAN FORMITEM HOW MANY MODES ARE REALLY PRESENT IN THE SYSTEM  THE PROBLEM  HERE IS THAT A EM MINIMAL REALIZATION OF A IS NOT OBTAINED  DIRECTLY FROM THE HZ AS GIVENENDENUMERATEITEM CITEKAILATH80 IF ABBFCBFTD WITH D NEQ 0  DESCRIBES A SYSTEM HS IN STATESPACE FORM SHOW THAT A  BBF CBFTD BBFD CBFTD 1DDESCRIBES A SYSTEM WITH SYSTEM FUNCTION 1HSITEM LABELEXUPDATEDEQ STATESPACE SOLUTIONS  BEGINENUMERATE  ITEM SHOW THAT REFEQXBFT2 IS A SOLUTION TO THE DIFFERENTIAL  EQUATION IN REFEQXNCT1 FOR CONSTANT ABCDITEM SHOW THAT AN UPDATE FROM XBFTAU TO XBFT IS AS GIVEN  IN REFEQSTATEUPDATEITEM SHOW THAT REFEQXBFT3 IS A SOLUTION TO THE DIFFERENTIAL  EQUATION IN REFEQXNCT1 FOR NONCONSTANT ABCD PROVIDED  THAT PHI SATISFIES THE PROPERTIES GIVEN  ENDENUMERATEITEM FIND A SOLUTION TO THE DIFFERENTIAL EQUATION DESCRIBED BY THE  STATESPACE EQUATIONSBEGINALIGNED XBFDOTT  BEGINBMATRIX 01  10 ENDBMATRIX XBFT YT  1 0 XBFTENDALIGNEDWITH XBF0  XBF0  THESE EQUATIONS DESCRIBE SIMPLE HARMONICMOTIONITEM FOR THE SYSTEM DESCRIBED BYBEGINALIGNED XBFDOTT  BEGINBMATRIX 2  0  1  1 ENDBMATRIX XBFT BEGINBMATRIX 2  1 ENDBMATRIX FT YT  0  2 XBFTENDALIGNEDBEGINENUMERATEITEM DETERMINE THE TRANSFER FUNCTION HSITEM FIND THE PARTIAL FRACTION EXPANSION OF HSENDENUMERATEITEM VERIFY REFEQGEOM1 BY LONG DIVISION LABELGEOMETRIC    SERIESITEM ENDEXERCISESSUBSECTIONISSUES AND APPLICATIONSTHE NOTATION INTRODUCED IN THE PREVIOUS SECTIONS ALLOWS US NOW TODISCUSS A VARIETY OF ISSUES OF BOTH PRACTICAL AND THEORETICALIMPORTANCE  HERE ARE A FEW EXAMPLESBEGINITEMIZEITEM GIVEN A DESIRED FREQUENCY RESPONSE SPECIFICATION   EITHER HEJOMEGAFOR DISCRETETIME SYSTEMS OR HJOMEGAFOR CONTINUOUSTIME SYSTEMS DETERMINE THE COEFFICIENTS AIAND BI TO MEET OR CLOSELY APPROXIMATE THE RESPONSESPECIFICATION  THIS IS THE EM FILTER DESIGN PROBLEM INDEXFILTER  DESIGNITEM GIVEN A SEQUENCE OF OUTPUT DATA FROM A SYSTEM HOW CAN THE  PARAMETERS OF THE SYSTEM BE DETERMINED IF THE INPUT SIGNAL IS KNOWN  IF THE INPUT SIGNAL IS NOT KNOWNITEM DETERMINE  A MINIMAL REPRESENTATION OF A SYSTEMITEM GIVEN A SIGNAL OUTPUT FROM A SYSTEM DETERMINE A PREDICTOR FOR  THE SIGNAL INDEXLINEAR PREDICTORITEM DETERMINE A MEANS OF EFFICIENTLY CODING REPRESENTING A SIGNAL  MODELED AS THE OUTPUT OF AN LTI SYSTEMITEM DETERMINE THE SPECTRUM OF THE OUTPUT OF AN LTI SYSTEMITEM DETERMINE THE MODES OF A SYSTEMITEM FOR ALGORITHMS OF THE SORT JUST PRESCRIBED DEVELOP  COMPUTATIONALLY EFFICIENT ALGORITHMSITEM SUPPOSE THE MODES OF A SIGNAL ARE NOT WHAT WE WANT THEM TO BE  DEVELOP A MEANS OF USING FEEDBACK TO BEND THEM TO SUIT OUR PURPOSESENDITEMIZEEXAMINATION OF MANY OF THESE ISSUES IS TAKEN UP AT APPROPRIATE PLACESTHROUGHOUT THIS BOOK WITH VARYING DEGREES OF COMPLETENESSSUBSUBSECTIONESTIMATION OF PARAMETERS LINEAR PREDICTIONINDEXLINEAR PREDICTORIT MAY OCCUR THAT A SIGNAL CAN BE MODELED AS THE OUTPUT OF ADISCRETETIME SYSTEM WITH SYSTEM FUNCTION HZ FOR WHICH THEPARAMETERS PQ B0LDOTS BQ A1 LDOTS AP ARE NOT KNOWN  GIVEN A SEQUENCEOF OBSERVATIONS Y0Y1LDOTS WE WANT TO DETERMINE IF POSSIBLETHE PARAMETERS OF THE SYSTEM  THIS BASIC PROBLEM HAS TWO MAJORVARIATIONSBEGINITEMIZEITEM THE INPUT FT IS DETERMINISTIC AND KNOWNITEM THE INPUT FT IS RANDOMENDITEMIZEOTHER COMPLICATIONS MAY ALSO BE MODELED IN PRACTICE  FOR EXAMPLE ITMAY BE THAT THE OUTPUT YT IS CORRUPTED BY NOISE SO THAT THE DATAAVAILABLE IS ZT  YT  WTWHERE WT IS A NOISE OR ERROR SIGNAL  THIS IS A SIGNAL PLUSNOISE MODEL THAT WE WILL EMPLOY FREQUENTLY INDEXSIGNAL PLUS  NOISEIN THE CASE WHERE THE INPUT IS KNOWN AND THERE IS NEGLIGIBLE OR NOMEASUREMENT NOISE IT IS STRAIGHTFORWARD TO SET UP A SYSTEM OF LINEAREQUATIONS TO DETERMINE THE SYSTEM PARAMETERS  FOR THE ARMAPQSYSTEM OF REFEQARMA IF THE ORDER PQ IS KNOWN A SYSTEM OFEQUATIONS TO FIND THE UNKNOWN PARAMETERS CAN BE SET UP ASBEGINEQUATIONLABELEQARMAIDAXBF  BBFENDEQUATIONIN WHICH A  BEGINBMATRIX YP1  YP2  CDOTS  Y0  FP  FP1   CDOTS  FPQ1 YP  YP1  CDOTS  Y1  FP1  FP1  CDOTS  FPQ  VDOTS  YN1  YN2  CDOTS  YNP  FN  FN1  CDOTS  FNQ1 ENDBMATRIX XBF  BEGINBMATRIX ABAR1  ABAR2  VDOTS  ABARP  BBAR0   BBAR1  VDOTS  BBARQENDBMATRIXQQUAD TEXTANDQQUADBBF   BEGINBMATRIX YP  YP1  VDOTS  YNENDBMATRIXWHERE N IS LARGE ENOUGH THAT THERE ARE AS MANY EQUATIONS ASUNKNOWNS  WHEN THERE IS MEASUREMENT NOISE IN THE SYSTEM N CAN BEINCREASED SO THAT THERE ARE MORE EQUATIONS THAN UNKNOWNS AND ALEASTSQUARES SOLUTION CAN BE COMPUTED AS DISCUSSED IN CHAPTERSREFCHAPVECTAP AND REFCHAPMATFACT  AN IMPORTANT SPECIAL CASE IN THIS PARAMETER ESTIMATION PROBLEM INWHICH THE INPUT IS ASSUMED TO BE NOISE AND WHEN HZ IS KNOWN TO BE OR ASSUMED TO BE  AN ARP SYSTEM WITH P KNOWN HZ  FRAC11SUMK1P AK ZKSUCH A MODEL IS COMMONLY ASSUMED IN SPEECH PROCESSING INDEXSPEECH  PROCESSING WHERE A SPEECH SIGNAL IS MODELED AS THE OUTPUT OF ANALLPOLE SYSTEM DRIVEN BY EITHER A ZEROMEAN UNCORRELATED SIGNAL INTHE CASE OF UNVOICED SPEECH SUCH AS THE LETTER S OR BY APERIODIC PULSE SEQUENCE IN THE CASE OF VOICED SPEECH SUCH AS THELETTER A  WE ASSUME THAT THE SIGNAL IS GENERATED ACCORDING TO YT  ABFT YBFT1  FTFURTHER ASSUMING HERE THE MODEL USES REAL DATA  OUR ESTIMATED MODELHAS OUTPUT YHATT WHERE YHATT  ABFHATT YBFTAND ABFHAT  BEGINBMATRIX AHAT1  AHAT2  VDOTS  AHATPENDBMATRIXTHE MARK HAT INDEXHAT ON A QUANTITY INDICATES ANESTIMATED OR APPROXIMATE VALUE  WE CAN INTERPRET THE ESTIMATED ARSYSTEM AS A EM LINEAR PREDICTOR THE VALUE YHATT IS THEPREDICTION OF YT GIVEN THE PAST DATA YT1 YT2LDOTSYTP  THE PREDICTION PROBLEM CAN BE STATED AS FOLLOWS DETERMINETHE PARAMETERS AHAT1LDOTSAHATP TO GET THE BESTPREDICTION  THERE IS AN ERROR BETWEEN WHAT IS ACTUALLY PRODUCED BYTHE SYSTEM AND THE PREDICTED VALUE ET  YT  YHATTTHIS IS ILLUSTRATED IN FIGURE REFFIGPREDICT1  A GOODPREDICTOR WILL MAKE THE ERROR AS SMALL AS POSSIBLE IN SOME SENSETHE SOLUTION TO THE PREDICTION PROBLEM IS DISCUSSED IN CHAPTERREFCHAPVECTAPBEGINFIGUREHTBP  CENTERLINEINPUTPICTUREDIRPREDICT1LATEX  CENTERLINEINPUTPICTUREDIRPREDICT1  CAPTIONPREDICTION ERROR   LABELFIGPREDICT1ENDFIGUREONE APPLICATION OF LINEAR PREDICTION IS TO DATA COMPRESSIONINDEXDATA COMPRESSION WE DESIRE TO REPRESENT A SEQUENCE OF DATAUSING THE SMALLEST NUMBER OF BITS POSSIBLE  IF THE SEQUENCE WERECOMPLETELY DETERMINISTIC SO THAT YT IS A DETERMINISTIC FUNCTIONOF PRIOR OUTPUTS WE WOULD NOT NEED TO SEND ANY BITS TO DETERMINEYT IF THE PRIOR OUTPUTS WERE KNOWN WE COULD SIMPLY USE A PERFECTPREDICTOR TO REPRODUCE THE SEQUENCE  IF YT IS NOT DETERMINISTICWE PREDICT YT THEN CODE QUANTIZE ONLY THE PREDICTION ERROR  IFTHE PREDICTION ERROR IS SMALL THEN ONLY A FEW BITS ARE REQUIRED TOACCURATELY REPRESENT IT  CODING IN THIS WAY IS CALLED DIFFERENTIALPULSE CODE MODULATION  WHEN PARTICULAR FOCUS IS GIVEN TO THE PROCESSOF DETERMINING THE PARAMETERS ABFHAT IT MAY BE CALLED LINEARPREDICTIVE CODING LPC  TO BE SUCCESSFUL IT MUST BE POSSIBLE TODETERMINE THE COEFFICIENTS INSIDE THE PREDICTORLINEAR PREDICTION ALSO HAS APPLICATIONS TO PATTERN RECOGNITIONSUPPOSE THERE ARE SEVERAL CLASSES OF SIGNALS TO BE DISTINGUISHED FOREXAMPLE SEVERAL SPEECH SOUNDS TO BE RECOGNIZED  EACH SIGNAL WILLHAVE ITS OWN SET OF PREDICTION COEFFICIENTS SIGNAL 1 HAS ABF1SIGNAL 2 HAS ABF2 AND SO FORTH  AN UNKNOWN INPUT SIGNAL CAN BEREDUCED BY ESTIMATING THE PREDICTION COEFFICIENTS THAT REPRESENT ITTO A VECTOR ABF  THEN ABF CAN BE COMPARED WITH ABF1ABF2 AND SO FORTH USING AN APPROPRIATE COMPARISON FUNCTION TODETERMINE WHICH SIGNAL THE UNKNOWN INPUT IS MOST SIMILAR TOWE CAN EXAMINE THE LINEAR PREDICTION PROBLEM FROM ANOTHER PERSPECTIVEIF YZ  HZFZTHEN FZ  YZ FRAC1HZTHAT IS FT  YT  ABFT YBFT1IF WE REGARD YT AS THE INPUT THEN FT IS THE OUTPUT OF ANINVERSE SYSTEM  IF WE HAVE AN ESTIMATED SYSTEMHHATZ  FRAC11  SUMK1P AHATK ZKTHEN THE OUTPUT FHATT  YT  ABFHATT YBFT1SHOULD BE CLOSE IN SOME SENSE TO FT  A BLOCK DIAGRAM IS SHOWNIN FIGURE REFFIGINVLP  IN THIS CASE WE WOULD WANT TO CHOOSE THEPARAMETERS ABFHAT TO MINIMIZE IN SOME SENSE THE ERROR FT FHATT  THAT IS WE WANT TO DETERMINE A GOOD INVERSE FILTER FORHZBEGINFIGUREHTBP  CENTERLINEINPUTPICTUREDIRPREDICT2LATEX  CENTERLINEINPUTPICTUREDIRPREDICT2  CAPTIONLINEAR PREDICTOR AS AN INVERSE SYSTEM  LABELFIGINVLPENDFIGUREINTERESTINGLY USING EITHER THE POINT OF VIEW OF FINDING A GOODPREDICTOR OR FINDING A GOOD INVERSE FILTER PRODUCES THE SAME ESTIMATEIT IS ALSO INTERESTING IS THAT COMPUTATIONALLY EFFICIENT ALGORITHMSEXIST FOR SOLVING THE EQUATIONS THAT ARISE IN THE LINEAR PREDICTIONPROBLEM THESE ARE DISCUSSED IN CHAPTER REFCHAPSPECIALMATSUBSUBSECTIONESTIMATION OF PARAMETERS SPECTRUM ANALYSISINDEXSPECTRUM ANALYSISIT IS COMMON IN SIGNAL ANALYSIS TO CONSIDER THAT A GENERAL SIGNAL ISCOMPOSED OF SINUSOIDAL SIGNALS ADDED TOGETHER  DETERMINING THESEFREQUENCY COMPONENTS BASED UPON MEASURED SIGNALS IS CALLED EMSPECTRUM ESTIMATION OR EM SPECTRAL ANALYSIS  THERE ARE TWOGENERAL APPROACHES TO SPECTRAL ANALYSIS  THE FIRST APPROACH IS BYMEANS OF FOURIER TRANSFORMS IN PARTICULAR THE DISCRETE FOURIERTRANSFORM  THIS APPROACH IS CALLED NONPARAMETRIC SPECTRUMESTIMATION  THE SECOND APPROACH IS A PARAMETRIC APPROACH IN WHICH AMODEL FOR THE SIGNAL IS PROPOSED SUCH AS THE ONE IN REFEQARMAAND THEN THE PARAMETERS ARE ESTIMATED FROM THE MEASURED DATA  ONCETHESE ARE KNOWN THE SPECTRUM OF THE SIGNAL CAN BE DETERMINEDPROVIDED THAT THE MODELING ASSUMPTIONS ARE ACCURATE IT IS POSSIBLE TOOBTAIN BETTER SPECTRAL RESOLUTION WITH FEWER PARAMETERS USINGPARAMETRIC METHODSDISCUSSION OF SPECTRUM ANALYSIS REQUIRES SOME FAMILIARITY WITH THECONCEPTS OF ENERGY AND POWER SPECTRAL DENSITIES  INDEXENERGY  SPECTRAL DENSITY INDEXPOWER SPECTRAL DENSITYINDEXDISCRETETIME FOURIER TRANSFORM FOR A DISCRETETIMEDETERMINISTIC SIGNAL YT THE DISCRETETIME FOURIER TRANSFORM DTFT IS BOXEDYOMEGA  SUMTINFTYINFTY YT EJOMEGA T WHERE JSQRT1  THE EM ENERGY SPECTRAL DENSITY ESD IS AMEASURE OF HOW MUCH ENERGY THERE IS AT EACH FREQUENCY  AN EM ENERGY  SIGNAL YT HAS FINITE ENERGY INDEXENERGY SIGNAL SUMTINFTYINFTY YT2  INFTYFOR A DETERMINISTIC ENERGY SIGNAL THE ESD IS DEFINED BY GYYOMEGA  YOMEGA2WHERE THE SUBSCRIPT Y ON GYY INDICATES THE SIGNAL WHOSE ESD ISREPRESENTED  THE EM AUTOCORRELATION FUNCTION OF A DETERMINISTICSEQUENCE IS RHOYYK  SUMTINFTYINFTY YT YBARTKINDEXAUTOCORRELATION FUNCTIONTHEN SEE EXERCISE REFEXESD1BEGINEQUATIONGYYOMEGA  SUMKINFTYINFTY RHOYYK EJOMEGA KLABELEQESD1ENDEQUATIONTHAT IS THE ENERGY SPECTRAL DENSITY IS THE DTFT OF THEAUTOCORRELATION FUNCTIONTHE POWER SPECTRAL DENSITY PSD IS EMPLOYED FOR SPECTRAL ANALYSIS OFSTOCHASTIC SIGNALS  IT PROVIDES AN INDICATION OF HOW MUCH AVERAGEPOWER THERE IS IN THE SIGNAL AS A FUNCTION OF FREQUENCY  WE ASSUMETHAT THE SIGNAL IS ZERO MEAN EYT  0  FOR THE SIGNAL YTWITH AUTOCORRELATION FUNCTION RYYK WE ALSO ASSUME THAT THEAUTOCORRELATION DROPS OFF SUFFICIENTLY FAST THATBEGINEQUATION LIMN RIGHTARROW INFTY FRAC1N SUMKNN K RYYK  0LABELEQPSDDECENDEQUATIONTHE PSD IS DEFINED AS SYYOMEGA  SUMKINFTYINFTY RYYK EJOMEGA KTHAT IS THE PSD IS THE DTFT OF THE AUTOCORRELATION SEQUENCE ONE OF THE IMPORTANT PROPERTIES OF THE PSD IS THAT SYYOMEGA GEQ 0 QQUAD TEXTFOR ALL  OMEGATHIS CORRESPONDS TO THE PHYSICAL FACT THAT REAL POWER CANNOT BENEGATIVEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODECENTERLINEINPUTPICTUREDIRSYST2LATEXCENTERLINEINPUTPICTUREDIRSYST2    CAPTIONPSD INPUT AND OUTPUT    LABELFIGSYST2  ENDCENTERENDFIGUREA SIGNAL FT WITH PSD SFOMEGA INPUT TO A SYSTEM WITH SYSTEMFUNCTION HZ PRODUCES THE SIGNAL YT AS SHOWN IN FIGUREREFFIGSYST2  LET US DEFINE HOMEGA  HEJOMEGA  HZBIGLZEJOMEGATHE FIRST EQUALITY IS BY DEFINITION AND IS ACTUALLY AN ABUSE OFNOTATION  HOWEVER IT AFFORDS SOME NOTATIONAL SIMPLICITY AND IS VERYCOMMON  THEN SEE APPENDIX REFAPPDXRP THE PSD OF THE OUTPUT IS SYYOMEGA  HOMEGA2 SFFOMEGATHE SPECTRUM ESTIMATION PROBLEM IS AS FOLLOWS GIVEN A SET OFOBSERVATIONS FROM A RANDOM SIGNAL Y0Y1LDOTSYN DETERMINEESTIMATE THE PSD  IN THE PARAMETRIC APPROACH TO SPECTRUMESTIMATION WE REGARD YT AS THE OUTPUT OF A SYSTEM HZ  ITIS COMMON TO ASSUME THAT THE INPUT SIGNAL IS A ZEROMEAN WHITE SIGNALSO THAT SFFOMEGA  TEXTCONSTANT  SIGMAF2THE PARAMETERS OF HZ AND THE INPUT POWER PROVIDE THE INFORMATIONNECESSARY TO ESTIMATE THE OUTPUT SPECTRUM SYOMEGASUBSECTIONIDENTIFICATION OF THE MODESLABELSECMODAL1INDEXMODAL ANALYSIS RELATED TO SPECTRUM ESTIMATION IS THEIDENTIFICATION OF THE MODES IN A SYSTEM  WE PRESENT THE FUNDAMENTALCONCEPT USING A SECONDORDER SYSTEM WITHOUT THE COMPLICATION OF NOISEIN THE SIGNAL  ASSUME THAT A SIGNAL YT IS THE OUTPUT OF ASECONDORDER HOMOGENEOUS SYSTEMBEGINEQUATION YT2  A1 YT1  A2 YT  0LABELEQMODE1ENDEQUATIONSUBJECT TO CERTAIN INITIAL CONDITIONS  THE CHARACTERISTIC EQUATION OFTHIS SYSTEM ISBEGINEQUATION Z2  A1 Z  A2  0LABELEQMODE2ENDEQUATIONTHE MODES OF THE SYSTEM ARE DETERMINED BY THE ROOTS OF THECHARACTERISTIC EQUATION  WRITING Z2  A1 Z  A2  ZP1ZP2AND ASSUMING THAT P1 NEQ P2 THEN YT  C1P1T  C2P2T QQUAD T GEQ 0WHERE THE MODE STRENGTHS AMPLITUDES C1 AND C2 ARE DETERMINEDBY THE INITIAL CONDITIONSBASED UPON THE NOISEFREE EQUATION REFEQMODE1 WE CAN WRITE ASET OF EQUATIONS TO DETERMINE THE SYSTEM PARAMETERS A1A2 BEGINBMATRIX Y1  Y0  Y2  Y1 VDOTSENDBMATRIX  BEGINBMATRIXA1  A2 ENDBMATRIX BEGINBMATRIX Y2    Y3   VDOTS ENDBMATRIXPROVIDED THAT THE MATRIX IN THIS EQUATION HAS FULL RANK THEPARAMETERS A1 AND A2 CAN BE FOUND BY SOLVING THIS SET OFEQUATIONS FROM WHICH THE MODES CAN BE IDENTIFIED BY FINDING THE ROOTSOF REFEQMODE2  USING THIS METHOD TWO MODES CAN BE IDENTIFIEDUSING AS FEW AS FOUR MEASUREMENTS  TWO REAL SINUSOIDS WITH TWOCOMPLEX EXPONENTIAL MODES IN EACH CAN BE IDENTIFIED WITH AS FEW ASEIGHT MEASUREMENTS AND THEY CAN IN PRINCIPLE AND IN THE ABSENCE OFNOISE BE DISTINGUISHED NO MATTER HOW CLOSE IN FREQUENCY THEY AREBEGINEXAMPLE  SUPPOSE THAT YT IS KNOWN TO CONSIST OF TWO REAL SINUSOIDAL  SIGNALS YT  A COSOMEGA1 T  THETA1  B COSOMEGA2 T THETA2EACH COSINE FUNCTION CONTRIBUTES TWO MODES COSOMEGA1 T  FRACEJOMEGA1 T  EJOMEGA1 T2SO WE WILL ASSUME THAT YT IS GOVERNED BY THE FOURTHORDERDIFFERENCE EQUATION YT  A1 YT1  A2 YT2  A3 YT3  0THEN ASSUMING THAT CLEAN NOISEFREE MEASUREMENTS ARE AVAILABLE WECAN SOLVE FOR THE COEFFICIENTS OF THE DIFFERENCE EQUATION BYBEGINEQUATION BEGINBMATRIXY3  Y2  Y1  Y0 Y4  Y3  Y2  Y1 Y5  Y4  Y3  Y2 Y6  Y5  Y4  Y3 ENDBMATRIXBEGINBMATRIXA1  A2  A3  A4 ENDBMATRIX BEGINBMATRIX Y4 Y5  Y6  Y7ENDBMATRIX LABELEQMODEEX1ENDEQUATIONIF THE MEASURED OUTPUT DATA SET ISBEGINALIGNED YBF   Y0 Y1 LDOTS Y7 255433  191774  115137  033427  0451325 11354  167244  20477ENDALIGNEDTHEN SUBSTITUTION IN REFEQMODEEX1 YIELDS A1A2A3A4   3715354404371531 Z4 37153 Z3 54404Z237153Z 1WHICH HAS ROOTS AT EPM J 05QQUAD TEXTANDQQUADEPM J 02SO THE FREQUENCIES OF THE MODES ARE OMEGA1  05 AND OMEGA2 02  ONCE THE FREQUENCIES ARE KNOWN THE AMPLITUDES AND PHASES CANALSO BE DETERMINED 3COS2T  PI4  2 COS5 T  PI6ENDEXAMPLEGENERALIZATION OF THESE CONCEPTS TO A SYSTEM OF ANY ORDER IS DISCUSSEDIN SECTION REFSECMODALMAT TREATMENT OF THE MEASUREMENT NOISE ISDISCUSSED IN SECTIONS REFSECMUSIC AND REFSECESPRITSUBSECTIONCONTROL OF THE MODESINDEXCONTROLSUPPOSE WE HAVE A SYSTEM DESCRIBED BY THE DYNAMICS BEGINBMATRIXX1T1  X2T1 ENDBMATRIX  BEGINBMATRIX 05  0  0  3 ENDBMATRIXBEGINBMATRIXX1T  X2T ENDBMATRIX   BEGINBMATRIX1  1 ENDBMATRIXFTBECAUSE THE A MATRIX IS A DIAGONAL MATRIX THE STATE VARIABLEEQUATIONS ARE SAID TO BE UNCOUPLED X1T1  05X1T  FTDOES NOT DEPEND ON X2 AND X2T1   3 X2T  FTDOES NOT DEPEND UPON X1  THE QUESTION OF HOW TO PUT A GENERALSYSTEM INTO DIAGONAL INDEXDIAGONAL MATRIX FORM IS ADDRESSED INSECTION REFSECDIAGONAL  THE HOMOGENEOUS RESPONSES ZEROINPUT OFTHE MODES SEPARATELY ARE X1T  05N X10 QQUAD X2T  3N X20THE STATE VARIABLE X1T DECAYS TO ZERO AS N RIGHTARROW INFTYWHILE THE STATE VARIABLE X2T BLOWS UP  IF THIS REPRESENTED THESTATE OF A MECHANICAL SYSTEM SUCH EXPONENTIAL GROWTH WOULD PROBABLYBE UNDESIRABLE  A NATURAL QUESTION ARISES IS IT POSSIBLE TODETERMINE AN INPUT SEQUENCE FT IN CONJUNCTION WITH FEEDBACK THATCONTROLS THE SYSTEM SO THAT BOTH STATE VARIABLES REMAIN STABLE  THEMEANS OF ACCOMPLISHING THIS FALLS VERY NATURALLY INTO PLACE USING SOMETECHNIQUES FROM LINEAR ALGEBRA   SEE SECTION REFSECMOVEEIGBEGINEXERCISESITEM SYSTEM IDENTIFICATION IN THIS EXERCISE YOU WILL DEVELOP A  TECHNIQUE FOR IDENTIFICATION OF THE PARAMETERS OF A CONTINUOUSTIME  SECONDORDER SYSTEM BASED UPON FREQUENCY RESPONSE MEASUREMENTS BODE  PLOTS  ASSUME THAT THE SYSTEM TO BE IDENTIFIED HAS AN OPENLOOP  TRANSFER FUNCTION HOS  FRACBSSA  BEGINENUMERATE  ITEM SHOW THAT WITH THE SYSTEM IN A FEEDBACK CONFIGURATION AS SHOWN    IN FIGURE REFFIGBODEID1 THE TRANSFER FUNCTION CAN BE WRITTEN    AS HCS  FRACYSFS  FRAC11ABS  1BS2ITEM SHOW THAT FRAC1HCJOMEGA  AJOMEGA ANGLE PHIJOMEGAWHERE AJOMEGA  FRAC1BSQRTBOMEGA22  AOMEGA2 QQUADTEXTANDQQUAD TAN PHIJOMEGA  FRACAOMEGAB  OMEGA2THE QUANTITIES AJOMEGA AND PHIJOMEGA CORRESPOND TO THERECIPROCAL AMPLITUDE AND THE PHASE DIFFERENCE BETWEEN INPUT ANDOUTPUTITEM SHOW THAT IF AMPLITUDEPHASE MEASUREMENTS ARE MADE AT N  DIFFERENT FREQUENCIES OMEGA1 OMEGA2 LDOTS OMEGAN THEN  THE UNKNOWN PARAMETERS A AND B CAN BE ESTIMATED BY SOLVING THE  OVERDETERMINED SET OF EQUATIONS BEGINBMATRIX AJOMEGA1  OMEGA1 SQRT1 1TAN2 PHIJOMEGA1 TAN PHIJOMEGA1  OMEGA1 AJOMEGA2  OMEGA2 SQRT1 1TAN2 PHIJOMEGA2 TAN PHIJOMEGA2  OMEGA2 VDOTS AJOMEGAN  OMEGAN SQRT1 1TAN2 PHIJOMEGAN TAN PHIJOMEGAN  OMEGAN ENDBMATRIXBEGINBMATRIX B  A ENDBMATRIX BEGINBMATRIX 0  OMEGA12 TANPHIJOMEGA1 0  OMEGA22 TANPHIJOMEGA2 VDOTS 0  OMEGAN2 TANPHIJOMEGAN ENDBMATRIX  ENDENUMERATE  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODE      INPUTPICTUREDIRFEEDBACK1      CAPTIONSIMPLE FEEDBACK CONFIGURATION      LABELFIGBODEID1    ENDCENTER  ENDFIGUREITEM VERIFY REFEQESD1  LABELEXESD1ITEM SHOW THAT  SUMNINFTYINFTY YN2  FRAC12PI INTPIPISOMEGA DOMEGAHINT RECALL THE INVERSE FOURIER TRANSFORM INDEXFOURIER TRANSFORM YT FRAC12PI INTPIPI YOMEGA EJ OMEGA TDOMEGAITEM SHOW THAT FOR A STOCHASTIC SIGNAL SOMEGA  LIMNRIGHTARROW INFTY ELEFT FRAC1N  LEFTSUMN1N YN EJOMEGA NRIGHT2 RIGHTHINT  SHOW AND USE THE FACT THAT SUMN1N SUMM1N FNM  SUMLN1N1 NLFLITEM MODAL ANALYSIS THE FOLLOWING DATA IS MEASURED FROM A  THIRDORDER SYSTEM Y   0320002500010000022200006000120000500001BEGINENUMERATEITEM DETERMINE THE MODES IN THE SYSTEM AND PLOT THEM IN THE COMPLEX PLANEITEM THE DATA CAN BE WRITTEN AS YT  C1P1T  C2P2T  C3P3T QQUAD T GEQ 0DETERMINE THE CONSTANTS C1 C2 AND C3ITEM TO EXPLORE THE EFFECT OF NOISE ON THE SYSTEM ADD RANDOM  GAUSSIAN NOISE TO EACH DATA POINT WITH VARIANCE SIGMA2  001  THEN FIND THE MODES OF THE NOISY DATA  REPEAT SEVERAL TIMES WITH  DIFFERENT NOISE AND COMMENT ON HOW THE MODAL ESTIMATES MOVEENDENUMERATEITEM MODAL ANALYSIS  IF YT HAS TWO REAL SINUSOIDS YT  A COSOMEGA1 T  THETA1  B COSOMEGA2 T THETA2AND THE FREQUENCIES ARE KNOWN DETERMINE A MEANS OF COMPUTING THEAMPLITUDES AND PHASESENDEXERCISESSECTIONADAPTIVE FILTERINGLABELSECADFILTINDEXADAPTIVE FILTERAN ADAPTIVE FILTER IS A FILTER USUALLY WITH AN FIR IMPULSERESPONSE IN WHICH THE COEFFICIENTS ARE OBTAINED BY ATTEMPTING TOFORCE THE OUTPUT OF THE FILTER YT TO MATCH SOME DESIRED INPUTSIGNAL DT    SCHEMATICALLY THE FILTER IS SHOWN IN FIGUREREFFIGADFILT1  THE ERROR SIGNAL  ET  DT  YTIS USED IN SPECIALIZED ALGORITHMS THE ADAPTATION RULE TO ADJUST THECOEFFICIENTS OF THE ADAPTIVE FILTER  A VARIETY OF ADAPTATION RULESARE EMPLOYED IN PARTICULAR WE WILL STUDY THE RECURSIVE LEASTSQUARESRLS ALGORITHM PRESENTED IN SECTION REFSECRLS AND THE LEAST MEANSQUARES LMS ALGORITHM PRESENTED IN SECTION REFSECLMSBEGINFIGUREHTBPBEGINCENTERINPUTPICTUREDIRADFILT1ENDCENTERCAPTIONREPRESENTATION OF AN ADAPTIVE FILTER  LABELFIGADFILT1ENDFIGUREADAPTIVE FILTERS ARE EMPLOYED IN A VARIETY OF CONFIGURATIONS SOME OFWHICH ARE HIGHLIGHTED IN THIS SECTIONSUBSECTIONSYSTEM IDENTIFICATIONINDEXSYSTEM IDENTIFICATIONAN ADAPTIVE FILTER CAN ESTIMATE THE THE TRANSFER FUNCTION OF ANUNKNOWN PLANT USING THE CONFIGURATION SHOWN IN FIGUREREFFIGADFILSYSID  THE ADAPTIVE FILTER AND THE PLANT ARE BOTHDRIVEN BY THE SAME INPUT SIGNAL AND THE DESIRED SIGNAL DT IS THEPLANT OUTPUT  THE ADAPTIVE FILTER WILL CONVERGE TO A BESTREPRESENTATION OF THE UNKNOWN SYSTEM  IF THE SYSTEM IS AN IIR SYSTEMAND THE ADAPTIVE FILTER IS AN FIR SYSTEM OR IF THE ORDER OF THEADAPTIVE FILTER IS LESS THAN THE ORDER OF THE SYSTEM THEN THEADAPTIVE FILTER CAN BE AT BEST AN APPROXIMATION OF THE TRUE SYSTEMRESPONSEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRADFILTSYSID    CAPTIONIDENTIFICATION OF AN UNKNOWN PLANT    LABELFIGADFILSYSID  ENDCENTERENDFIGURESUBSECTIONINVERSE SYSTEM IDENTIFICATIONINDEXINVERSE SYSTEM IDENTIFICATIONWHEN THE ADAPTIVE FILTER IS CONFIGURED AS SHOWN IN FIGUREREFFIGADFILTINVSYS THEN IT WILL CONVERGE WHEN THE OUTPUT OF THEADAPTIVE FILTER MATCHES THE DELAYED INPUT OF THE INVERSE SYSTEM ASCLOSELY AS POSSIBLE  IDEALLY THE ADAPTIVE FILTER WILL CONVERGE TOTHE INVERSE OF THE PLANT SO THAT THE CASCADE OF THE PLANT AND THEADAPTIVE FILTER IS SIMPLY A DELAY  THIS CONFIGURATION IS EMPLOYED INSOME MODEMS TO REDUCE THE EFFECT OF THE CHANNEL ON THE TRANSMITTEDSIGNAL  THE SIGNAL REPRESENTING A SEQUENCE OF INPUT BITS FTPASSES THROUGH A CHANNEL WITH AN UNKNOWN TRANSFER FUNCTION HZ  ATTHE RECEIVER THE SIGNAL IS PROCESSED BY AN ADAPTED INVERSE SYSTEMBEFORE DETECTING THE BITSAN EXAMPLE OF THE OPERATION IN THISCONFIGURATION IS PROVIDED IN SECTION REFSECRLSBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRADFILTINVSYS    CAPTIONADAPTING TO THE INVERSE OF AN UNKNOWN PLANT    LABELFIGADFILTINVSYS  ENDCENTERENDFIGURESUBSECTIONADAPTIVE PREDICTORSINDEXLINEAR PREDICTION IN THE CONFIGURATION SHOWN IN FIGUREREFFIGADFILPREDICTOR THE INPUT TO THE ADAPTIVE FILTER IS ADELAYED VERSION OF THE DESIRED SIGNAL  IN THIS CASE THE ADAPTIVEFILTER CONVERGES IN SUCH A WAY AS TO PROVIDE A PREDICTOR OF THE INPUTSIGNAL IF PREDICTION IS POSSIBLE  IN THIS MODE IT CAN BE USED FORALL THE APPLICATIONS MENTIONED PREVIOUSLY FOR LINEAR PREDICTORSINCLUDING DATA COMPRESSION PATTERN RECOGNITION OR SPECTRUMESTIMATIONBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRADFILTPRED    CAPTIONAN ADAPTIVE PREDICTOR    LABELFIGADFILPREDICTOR  ENDCENTERENDFIGURESUBSECTIONINTERFERENCE CANCELLATIONINDEXINTERFERENCE CANCELLATIONIN THE CONTEXT OF INTERFERENCE CANCELLATION THE SIGNAL DT ISCOMMONLY REFERRED TO AS THE PRIMARY SIGNAL WHILE THE FILTER INPUTIS REFERRED TO AS THE SECONDARY SIGNAL  THE PRIMARY DT ISMODELED AS THE SUM OF A SIGNAL OF INTEREST XT PLUS  NOISE DT  XT  WTTHE SECONDARY INPUT CONSISTS OF A NOISE SIGNAL FT  NTSEE FIGURE REFFIGADCANCELBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRADCANCEL    CAPTIONCONFIGURATION FOR INTERFERENCE CANCELLATION    LABELFIGADCANCEL  ENDCENTERENDFIGUREAS AN EXAMPLE SUPPOSE THAT A BACKGROUND ACOUSTIC NOISE SOURCE SAYTHE HUM OF A FAN WT SUPERIMPOSED ON A DESIRED AUDIO SIGNALXT WHICH IS RECORDED USING A MICROPHONE TO FORM THE PRIMARYINPUT  A SECOND MICROPHONE PLACED FAR FROM THE DESIRED SIGNAL RECORDSTHE NOISE NT BUT NOT THE DESIRED SIGNAL  THERE IS A DIFFERENTACOUSTIC TRANSFER FUNCTION BETWEEN THE SOURCE AND EACH OF THE TWOMICROPHONES HENCE NT IS NOT THE SAME AS WT  THE ADAPTIVEFILTER IS DRIVEN TO MINIMIZE THE ERROR WHICH ADAPTS TO ACCOMMODATETHIS DIFFERENCE IN TRANSFER FUNCTION FROM THE NOISE SOURCE  THUS THERESULTING DIFFERENCE SIGNAL ET WILL HAVE INSOFAR AS POSSIBLETHE NOISE FROM THE REFERENCE SIGNAL SUBTRACTED FROM THE NOISE FROM THEPRIMARY SIGNALTHE INTERFERENCE CANCELLATION CONFIGURATION HAS BEEN USED IN SEVERALAPPLICATIONS SUCH AS NOISE CANCELLATION ECHO CANCELLATION ANDADAPTIVE BEAMFORMING IN ARRAY PROCESSINGSECTIONGAUSSIAN RANDOM VARIABLES AND RANDOM PROCESSESLABELSECMULTGAUSSINDEXGAUSSIAN RANDOM VARIABLE INDEXRANDOM VARIABLEGAUSSIANINDEXNORMAL RANDOM VARIABLESEEGAUSSIAN RANDOM VARIABLE WE BEGINBY REVIEWING THE BASIC PROPERTIES OF SINGLE GAUSSIAN RANDOM VARIABLESSEE BOX REFBOXPROBNOT FOR NOTATIONAL CONVENTIONS  LET W BE AGAUSSIAN RANDOM VARIABLE WITH MEAN MU AND VARIANCE SIGMA2NOTATIONALLY WE  WRITE  W SIM NCMUSIGMA2THE SCALAR GAUSSIAN PROBABILITY DENSITY FUNCTION PDF SHOULD BEFAMILIARBOXED FWW  FRAC1SIGMASQRT2PI EW     MU22SIGMA2WHERE MU IS THE MEAN AND SIGMA2 IS THE VARIANCE OF THEDISTRIBUTION  THAT IS MU  EW  INTINFTYINFTY W FWW DW FRAC1SQRT2PI SIGMA INTINFTYINFTY W EW   MU22SIGMA2 DWAND SIGMA2  EWMU2  EW2  MU2  FRAC1SQRT2PI  SIGMA INTINFTYINFTY W2 EW  MU22SIGMA2 DW  MU2FIGURE REFFIGGAUSS1 ILLUSTRATES A GAUSSIAN PDF WITH MU0 ANDSIGMA2 1BEGINFIGURE PLOTGAUSSMCENTERLINEEPSFIGFILEPICTUREDIRPLOTGAUSSEPSCAPTIONTHE GAUSSIAN DENSITYLABELFIGGAUSS1ENDFIGUREBEGINTEXTBOX09TEXTWIDTHNOTATION FOR RANDOM VARIABLES AND VECTORSLABELBOXPROBNOTSCALAR RANDOM VARIABLES ARE REPRESENTED USING CAPITAL LETTERS WHILE APARTICULAR OUTCOME VALUE FOR A RANDOM VARIABLE IS INDICATED IN LOWERCASE USUALLY THE SAME LETTER THUS X IS A RANDOM VARIABLE AND XMAY BE AN OUTCOME OF THE RANDOM VARIABLE  INDEXFONTSCAPITALINDEXCAPITAL LETTERSSEEFONTS RANDOM VECTORS ARE USUALLYPRESENTED AS BOLD CAPITAL LETTERS  WHERE THE NOTATION OF THELITERATURE COMMONLY EMPLOYS LOWER CASE WE FOLLOW SUITINDEXFONTSBOLD CAPITALINDEXPROBABILITY DENSITY FUNCTION PDF INDEXPROBABILITY MASS  FUNCTION PMF BOXINDENT A PROBABILITY DENSITY FUNCTION PDF ORPROBABILITY MASS FUNCTION PMF FOR A RANDOM VARIABLE X IS WRITTENAS FXX  HOWEVER IT WILL BE COMMON THROUGHOUT THE TEXT TOSUPPRESS THE SUBSCRIPT NOTATION LETTING THE ARGUMENT OF THE FUNCTIONPROVIDE THE INDICATION OF THE RANDOM VARIABLE  THUS WE WILLFREQUENTLY WRITE FX TO MEAN FXXENDTEXTBOXASSOCIATED WITH THE GAUSSIAN PDF ARE THE FOLLOWING USEFUL INTEGRALSTRUE FOR ALL VALUES OF MU AND SIGMA NEQ 0BEGINEQUATIONBOXEDFRAC1SIGMASQRT2PI INTINFTYINFTY  EXMU22SIGMA2  DX  1LABELEQGAUSSINT1ENDEQUATIONBEGINEQUATIONBOXEDFRAC1SIGMA SQRT2PI INTINFTYINFTY X  EXMU22SIGMA2  DX  MULABELEQGAUSSINT2ENDEQUATIONBEGINEQUATIONBOXEDFRAC1SIGMA SQRT2PI INTINFTYINFTY X2  EXMU22SIGMA2  DX  SIGMA2  MU2LABELEQGAUSSINT3ENDEQUATIONMEASURED SIGNALS ARE COMMONLY CORRUPTED BY NOISE  IF YBFTREPRESENTS A VECTOR SYSTEM OUTPUT THE MEASURED VALUE IS OFTEN MODELEDAS ZBFT  YBFT  WBFTWHERE WBFT IS A VECTOR OF NOISE SAMPLES  WBFT  BEGINBMATRIX W1T  W2T  VDOTS  WKTENDBMATRIXTHIS IS THE SIGNAL PLUS NOISE MODEL INDEXSIGNAL PLUS NOISEIN THE ABSENCE OF SPECIFIC REASONS TO THE CONTRARY IT IS COMMON TOASSUME THAT ADDITIVE NOISE SIGNALS ARE DISTRIBUTED WITH A EM  GAUSSIAN OR NORMAL DISTRIBUTION  QUANTIZATION NOISE IS ANEXCEPTION TO THIS ASSUMPTION IT IS USUALLY MODELED AS A UNIFORMRANDOM VARIABLE  THERE ARE REASONS FOR ASSUMING THAT RANDOMVARIABLES AND RANDOM PROCESSES ARE GAUSSIAN  FIRST GAUSSIAN NOISEOCCURS PHYSICALLY  FOR EXAMPLE THE THERMAL NOISE AT THE FRONT END OFA RADIO RECEIVER IS OFTEN GAUSSIAN  SECOND GAUSSIAN NOISE SIGNALSHAVE A VARIETY OF USEFUL PROPERTIES WHICH SIMPLIFY SEVERAL THEORETICALDEVELOPMENTS  SOME OF THESE PROPERTIES ARE AS FOLLOWSINDEXGAUSSIAN RANDOM VARIABLEATTRIBUTESBEGINENUMERATEITEM BY THE CENTRAL LIMIT THEOREM THE DISTRIBUTION OF SUMS OF  SEVERAL RANDOM VARIABLES TENDS TOWARD A GAUSSIAN DISTRIBUTION  MORE  PRECISELY IF X1 X2 LDOTS XN ARE INDEPENDENT RANDOM  VARIABLES WITH MEANS MU1 MU2 LDOTS MUN AND VARIANCES  SIGMA12 SIGMA22 LDOTS SIGMAN2 RESPECTIVELY THEN Y  SUMI1N FRACXI  MUISIGMAIHAS A DISTRIBUTION WHICH APPROACHES A GAUSSIAN DISTRIBUTION WITH MEAN0 AND VARIANCE 1 AS N BECOMES LARGE ENOUGH  IN THE LIMIT AS NRIGHTARROW INFTY THEN Y SIM NC01  THE CENTRAL LIMIT THEOREMACCOUNTS IN LARGE MEASURE FOR THE OCCURRENCE OF GAUSSIAN NOISE INPRACTICE THE MEASURED NOISE IS ACTUALLY THE SUM OF MANY SMALLINDEPENDENT EFFECTSBEGINEXAMPLE  AN APPRECIATION OF THE CENTRAL LIMIT THEOREM CAN BE GAINED BY  LOOKING AT THE SUM OF ONLY THREE VARIABLES  LET X1 X2 AND  X3 BE INDEPENDENT RANDOM VARIABLES UNIFORMLY DISTRIBUTED FROM  12 TO 12  NOTATIONALLY WE WRITE XI SIM UC1212  THE PDF FOR THIS UNIFORM RANDOM VARIABLE IS SHOWN IN FIGURE  REFFIGPDF1A  LET Z  X1  X2  KEEP IN MIND THAT THE PDF  OF THE SUM OF INDEPENDENT RANDOM VARIABLES IS THE CONVOLUTION OF THE  PDFS INDEXCONVOLUTION THE PDF OF Z IS THUS THE HAT SHAPED  FUNCTION SHOWN IN FIGURE REFFIGPDF1B THE CONVOLUTION OF TWO  FLAT PULSES  LET Y  ZX3  X1  X2  X3  THE PDF OF Y  OBTAINED AGAIN BY CONVOLUTION IS SHOWN IN FIGURE REFFIGPDF1C  THIS IS A PIECEWISE QUADRATIC FUNCTION BUT OBSERVE HOW IT IS  ALREADY BEGINNING TO LOOK LIKE THE GAUSSIAN DENSITY IN FIGURE  REFFIGGAUSS1BEGINFIGUREHTBP  CENTERINGSUBFIGUREFXXEPSFIGFILEPICTUREDIRUNIF1EPSSUBFIGUREFZZEPSFIGFILEPICTUREDIRUNIF2EPSSUBFIGUREFYYEPSFIGFILEPICTUREDIRUNIF3EPS PLOTGAUSS2M  CAPTIONDEMONSTRATION OF THE CENTRAL LIMIT THEOREM  LABELFIGPDF1ENDFIGUREENDEXAMPLEITEM A GAUSSIAN RANDOM VARIABLE W IS ENTIRELY DETERMINED BY ITS MEAN  AND ITS VARIANCE  A GAUSSIAN RANDOM PROCESS WT IS DETERMINED BY ITS  MEAN MWT  EWTAND AUTOCORRELATIONBEGINEQUATION RWTS  EWTWBARSLABELEQGAUSSCORRCH1ENDEQUATIONA GAUSSIAN RANDOM PROCESS WITH CONSTANT MEAN SUCH THAT RWTS RWST THAT IS WITH THE AUTOCORRELATION DEPENDING UPON THE TIMEDIFFERENCE IN SAMPLE POINTS IS STATIONARYITEM LINEAR OPERATIONS ON GAUSSIAN RANDOM VARIABLES PRODUCE GAUSSIAN  RANDOM VARIABLES  THAT IS IF X AND Y ARE JOINTLY GAUSSIAN  THEN Z  AX  BYIS ALSO GAUSSIAN FOR ANY CONSTANTS A AND B  IN PARTICULAR THESUM OF GAUSSIANS IS GAUSSIAN  THIS FOLLOWS SINCE THE CONVOLUTION OFGAUSSIANS IS GAUSSIAN  FURTHERMORE IF A GAUSSIAN RANDOM PROCESS IS INPUT TO A LINEAR SYSTEMTHEN THE OUTPUT IS ALSO A GAUSSIAN RANDOM PROCESS  ALL THAT MUST BEDETERMINED IS THE MEAN AND AUTOCORRELATION OF THE OUTPUT SIGNAL ANDIT IS FULLY CHARACTERIZED ITEM MAXIMUM LIKELIHOOD DETECTION OR ESTIMATION INVOLVING GAUSSIAN  RANDOM VARIABLES CORRESPONDS TO A EUCLIDEAN DISTANCE METRIC  THIS  IS GENERALLY GEOMETRICALLY PALATABLE AND ANALYTICALLY TRACTABLEITEM WIDESENSE STATIONARY WSS GAUSSIAN RANDOM PROCESSES ARE ALSO  STRICTSENSE STATIONARY SSS SEE APPENDIX REFAPPDXRP  INDEXWIDESENSE STATIONARYITEM UNCORRELATED GAUSSIAN RANDOM VARIABLES ARE ALSO INDEPENDENTITEM A GAUSSIAN CONDITIONED UPON A GAUSSIAN IS GAUSSIAN  INDEXGAUSSIAN RANDOM VARIABLECONDITIONAL DENSITYENDENUMERATEJUSTIFICATIONS FOR MANY OF THESE PROPERTIES ARE PROVIDED THROUGHOUTTHIS BOOK AS THEY ARISEFOR A GAUSSIAN RANDOM VECTOR WBF OF DIMENSION K WITH MEAN MUBFAND COVARIANCE MATRIX R WE WRITE WBF SIM NCMUBF R  THEPDF ISBEGINEQUATIONBOXED FWBFWBF  FRAC12PIK2 R12 EXPFRAC12WBF  MUBFT R1WBF  MUBFLABELEQMULTGAUSSENDEQUATIONWHERE MUBF IS THE MEAN MUBF  EWBF  BEGINBMATRIX EW1  EW2  VDOTS   EWK ENDBMATRIXAND R IS THE MATSIZEKK COVARIANCE MATRIX R  EWBFMUBFWBF  MUBFT  EWBFWBFT  MUBF MUBFTINDEXBAR CDOT INDEX  CDOT THE NOTATION R IN REFEQMULTGAUSS INDICATES THE ABSOLUTEVALUE OF THE DETERMINANT OF THE MATRIX R SEE SECTIONREFSECDETERM  IN OTHER CONTEXTS THE NOTATION R WILLINDICATE THE DETERMINANT BUT THE ABSOLUTE VALUE IS NEEDED IN THISCASE SINCE A DENSITY FUNCTION IS ALWAYS NONNEGATIVEMANY OF THE SIGNIFICANT CONCEPTS ASSOCIATED WITH GAUSSIAN RANDOMVECTORS CAN BE OBTAINED BY EXAMINATION OF TWODIMENSIONAL VECTORSWHEN WBF  W1W2TBEGINEQUATION R  BEGINBMATRIX SIGMA12  SIGMA12  SIGMA12   SIGMA22 ENDBMATRIXLABELEQR22ENDEQUATIONWHERE  SIGMA12  EW12  MU12 QQUAD SIGMA22  EW22  MU22 AND SIGMA12  EW1 W2  MU1 MU2THE EM  CORRELATION COEFFICIENT IS DEFINED ASBEGINEQUATION RHO  FRACEW1 W2  MU1 MU2 SIGMA1 SIGMA2LABELEQCORRCOEFFENDEQUATIONUSING THE CAUCHYSCHWARZ INEQUALITY WHICH IS INTRODUCED IN SECTIONINDEXCAUCHYSCHWARZ INEQUALITY REFSECCS IT CAN BE SHOWN THAT 1 LEQ RHO LEQ 1THE CORRELATION COEFFICIENT PROVIDES INFORMATION ABOUT HOW W1VARIES WITH W2  IF RHO  1 THEN W1  W2 AND W1 TELLSEVERYTHING THERE IS TO KNOW ABOUT W2 AND VICE VERSA  IF RHO 1 THEN W1  W2  IF RHO  0 THEN THE VARIABLES ARE SAID TOBE EM UNCORRELATED INDEXUNCORRELATED W1 DOES NOT PROVIDE ANYINFORMATION ABOUT W2  MORE GENERALLY FOR A KDIMENSIONAL RANDOMVECTOR WBF IF THE CORRELATION MATRIX R IS DIAGONALINDEXDIAGONAL MATRIX THE COMPONENTS OF WBF ARE UNCORRELATEDWE CAN WRITE THE INVERSE OF THE COVARIANCE MATRIX REFEQR22 INTERMS OF THE CORRELATION COEFFICIENT AND VARIANCES ASBEGINEQUATIONR1  FRAC11RHO2 BEGINBMATRIX FRAC1SIGMA12   FRAC RHOSIGMA1 SIGMA2 EXMATSP  FRAC RHOSIGMA1 SIGMA2  FRAC1SIGMA22 ENDBMATRIXLABELEQINVCOVARENDEQUATIONTHE JOINT PDF OF W1 AND W2 CAN NOW BE WRITTEN ASBEGINEQUATIONBEGINSPLITFW1W2   FRAC12PI SIGMA1 SIGMA2 SQRT1RHO2   EXPLEFTFRAC121RHO2LEFT FRACW1  MU12      SIGMA12   RIGHT RIGHT  QQUAD LEFT LEFT FRACW2  MU22SIGMA22  FRAC2RHOW1       MU1W2  MU2SIGMA1 SIGMA2RIGHT RIGHTLABELEQ2GAUSSENDSPLITENDEQUATIONA SURFACECURVE PLOT OF THIS FUNCTION IS SHOWN IN FIGUREREFFIG2GAUSPLOT FOR MUX  MUY  0 SIGMAX2 SIGMAY21 FOR TWO VALUES OF RHOBEGINFIGUREHTBPCENTERINGSUBFIGURERHO09EPSFIGFILEPICTUREDIRGAUSS21EPSWIDTH045TEXTWIDTHSUBFIGURERHO0EPSFIGFILEPICTUREDIRGAUSS22EPSWIDTH045TEXTWIDTHPLOTGAUSS3MCAPTIONPLOT OF TWODIMENSIONAL GAUSSIAN DISTRIBUTIONLABELFIG2GAUSPLOTENDFIGUREIN REFEQ2GAUSS IF RHO0 THEN FW1W2  FRAC12PI SIGMA1 SIGMA2EXPLEFTFRAC12  LEFT FRACW1  MU12SIGMA12  FRACW2       MU22SIGMA22RIGHT RIGHT  FW1FW2SUBSTANTIATING THE CLAIM MADE PREVIOUSLY THAT UNCORRELATED GAUSSIANRANDOM VARIABLES ARE INDEPENDENTSUBSECTIONCONDITIONAL GAUSSIAN DENSITIESLABELSECCONDESTINDEXGAUSSIAN RANDOM VARIABLECONDITIONAL DENSITY CONDITIONALPROBABILITIES CONSTITUTE THE CORE OF MANY DETECTION AND ESTIMATIONALGORITHMS  IN THIS SECTION WE PRESENT A SIMPLE EXAMPLE OFCONDITIONING AS A FORERUNNER TO THE MORE COMPLETE DEVELOPMENT OFSTATISTICAL DECISION MAKING IN PART REFPARTDETESTSUPPOSE THAT X AND Y ARE JOINTLY GAUSSIAN RANDOM VARIABLES XSIM NCMUX SIGMAX2 Y SIM NCMUY SIGMAY2 WITHCORRELATION COEFFICIENT RHO  WE WANT TO EM ESTIMATE A VALUE FORX WHICH WE WILL DENOTE AS XHAT  IN THE ABSENCE OF ANY INDEXESTIMATIONMEASUREMENTS A REASONABLE VALUE FOR XHAT IS SIMPLY THE MEAN OFX SO XHAT  MUXINDEXCONDITIONAL PROBABILITY SUCH AN ESTIMATE  OBTAINABLEWITHOUT THE BENEFIT OF ANY MEASUREMENTS  IS A EM PRIOR OR EM A  PRIORI INDEXPRIOR ESTIMATE ESTIMATE AND THE DENSITY FXX ISKNOWN AS THE EM A PRIORI DENSITY FOR X  WHEN A MEASUREMENT OFY IS AVAILABLE SAY YY THEN THIS CAN BE USED TO MODIFY OUR PRIORESTIMATE OF X SINCE X AND Y ARE CORRELATED  ONE APPROACH TOTHIS IS TO FORM THE CONDITIONAL PDF FXYXY THE DENSITY OF XGIVEN THAT YY IS KNOWN AND DETERMINE OUR ESTIMATE XHAT BY THEMEAN OF THIS NEW DENSITY  THE CONDITIONAL DENSITY IS DEFINED ASINDEXCONDITIONAL PROBABILITY FXYXY  FXY  FRACFXYFYFROM REFEQ2GAUSS WITH X  W1 AND YW2 WE OBTAININDEXCONDITIONAL PROBABILITYGAUSSIANBEGINEQUATIONBEGINSPLITFXY  FRAC FRAC12PI SIGMAX SIGMAY SQRT1RHO2  EXPLEFT     FRAC121RHO2LEFTFRACXMUX2SIGMAX2       FRACYMUY2SIGMAY2  FRAC2RHOSIGMAX SIGMAY      XMUXYMUYRIGHTRIGHTFRAC1SQRT2PI SIGMAY  EXPFRAC12SIGMAY2YMUY2   FRAC1SQRT2PI1RHO2 SIGMAX EXPLEFTFRAC12 SIGMAX2  SQRT1RHO2 XMUX FRACSIGMAXSIGMAYRHOYMUY2RIGHTENDSPLITLABELEQFXYENDEQUATIONTHE ALGEBRA HERE REQUIRES COMPLETING THE SQUARE AS DESCRIBED ININDEXCOMPLETING THE SQUAREAPPENDIX REFAPPDXCTS  FROM THE FORM OF THE PDF WE RECOGNIZE THATFXY IS GAUSSIAN WITH MEANBEGINEQUATION EXY  MUX  FRACSIGMAXSIGMAYRHOYMUYLABELEQCONDMEANGAUSS1ENDEQUATIONAND VARIANCEBEGINEQUATION VARXY  SIGMAX2 SQRT1RHO2LABELEQCONDVARGAUSS1ENDEQUATIONIF X AND Y ARECORRELATED THAT IS RHO NEQ 0 THEN KNOWING Y SHOULD TELL USSOMETHING ABOUT X  BASED ON THE INFORMATION AVAILABLE ABOUT Y AREASONABLE ESTIMATE OF X IS THE CONDITIONAL MEANBEGINEQUATION XHAT  MUX  FRACSIGMAXSIGMAYRHOYMUYLABELEQCONDMEAN0ENDEQUATIONTHE VARIANCE OF THIS ESTIMATE IS THE CONDITIONAL VARIANCE OFREFEQCONDVARGAUSS1  WE CAN MAKE A MEANINGFUL INTERPRETATION OFTHE ESTIMATE REFEQCONDMEAN0  IF THERE IS NO CORRELATION THECONDITIONAL MEAN IS THE SAME AS THE PRIOR MEAN  IF RHO IS SMALLWE MAKE ONLY A SMALL MODIFICATION TO THE PRIOR MEAN  IF SIGMAY ISLARGE THEN THE CORRECTION TO THE PRIOR MEAN IS SMALL AS IT SHOULD BEIF WE HAVE LARGE UNCERTAINTY ABOUT THE OUTCOME Y  WE ALSO OBSERVETHAT INCORPORATING INFORMATION ABOUT Y REDUCES THE VARIANCE IN X SIGMAX2 SQRT1RHO2 LEQ SIGMAX2SINCE RHO LEQ 1THIS CONDITIONAL DENSITY WITH ONLY TWO VARIABLES IS EXTENDED INSECTION REFSECINVPART TO GENERAL GAUSSIAN VECTORS CONDITIONED ONGAUSSIAN VECTORSTHIS EXAMPLE INTRODUCES AN IMPORTANT PART OF ESTIMATION THEORY  ANOBSERVED OR MEASURED VARIABLE SUCH AS Y IN THE FOREGOING CAN BEUSED TO MODIFY OUR UNDERSTANDING OF VARIABLES THAT WE HAVE NOTMEASURED OR CANNOT MEASURE  A POWERFUL EXTENSION OF THIS SIMPLEEXAMPLE IS THE KALMAN FILTER IN WHICH THE STATE OF A SYSTEM IN RANDOMNOISE SUCH AS IN REFEQSTATEGEN1 IS ESTIMATED BASED UPONOBSERVATIONS THAT ARE ALSO IN NOISE  IN THE KALMAN FILTER THEDENSITY OF THE STATE VARIABLE FXBFT IS MODIFIED BY THEOBSERVATION YBFT TAKING INTO ACCOUNT THE DYNAMICS OF THE SYSTEMAND THE MECHANISM FOR OBSERVATION  THE KALMAN FILTER IS DISCUSSED INCHAPTER REFCHAPKALMAN INDEXKALMAN FILTERSEVERAL OTHER EXTENSIONS AND ISSUES NOW ARISE AMONG THEMBEGINITEMIZEITEM GIVEN A SEQUENCE OF DATA FROM SOME SOURCE WHICH IS ASSUMED TO BE  DRAWN ACCORDING TO A GAUSSIAN DISTRIBUTION HOW CAN THE PARAMETERS  OF THE GAUSSIAN DISTRIBUTION BE ESTIMATED  HOW CAN THE QUALITY OF  THE ESTIMATES BE ASSESSED  THESE QUESTIONS ARE ANSWERED IN PART BY  EM ESTIMATION THEORY  AN EARLY ANSWER IS EXPLORED IN EXERCISE  REFEXGAUSSESTITEM IF A SIGNAL IS CHOSEN AT RANDOM FROM AMONG A DISCRETE SET OF  SIGNALS AND THEN OBSERVED IN ADDITIVE NOISE HOW CAN THE CHOSEN  SIGNAL BE DISCRIMINATED  THIS IS THE EM DETECTION PROBLEM WHICH  LIES AT THE HEART OF DIGITAL COMMUNICATIONITEM GIVEN CORRELATED RANDOM VECTORS XBF AND YBF HOW CAN THE  CONDITIONAL DENSITY FXBFYBF BE COMPUTED HOW MAY THIS BE  APPLIEDITEM HOW CAN GAUSSIAN RANDOM VARIABLES OF GIVEN PARAMETERS BE  GENERATED AND USED IN SIMULATION FOR TESTING OF SIGNAL PROCESSING  ALGORITHMS  AN ANSWER FOR SCALAR GAUSSIAN RVS IS FOUND IN  EXERCISE REFEXGENGAUSSENDITEMIZEBEGINEXERCISES  ITEM SHOW THAT R1 FROM REFEQINVCOVAR IS CORRECTITEM SHOW THAT REFEQ2GAUSS FOLLOWS FROM REFEQMULTGAUSS AND  REFEQINVCOVARITEM SUPPOSE THAT XSIM NCMUXSIGMAX2 AND N SIM  NC0SIGMAN2 ARE INDEPENDENTLY DISTRIBUTED GAUSSIAN RVS  LET Y  XNBEGINENUMERATEITEM DETERMINE THE PARAMETERS OF THE DISTRIBUTION OF YITEM IF YY IS MEASURED WE CAN ESTIMATE X BY COMPUTING THE  CONDITIONAL DENSITY FXY  DETERMINE THE MEAN AND VARIANCE OF  THIS CONDITIONAL DENSITY  INTERPRET THESE RESULTS IN TERMS OF  GETTING INFORMATION ABOUT X I SIGMAN2 GG SIGMAX2 AND  II SIGMAN2 LL SIGMAX2ENDENUMERATEITEM SUPPOSE THAT X SIM NCMUX SIGMAX2  AND Y  SIMNCMUY  SIGMAY2 ARE JOINTLY DISTRIBUTED GAUSSIAN RVS  WITH CORRELATION RHO  DETERMINE THE PARAMETERS OF THE  DISTRIBUTION OF Z  A X  BYITEM IF X SIM NC01 SHOW THAT Y  SIGMA X  MUIS DISTRIBUTED AS Y NCSIGMA2MU LABELEXGENGAUSSITEM IF X SIM NCSIGMA2MU DETERMINE EX THE EXPECTED  VALUE OF THE ABSOLUTE VALUE OF XITEM LABELEXGAUSSEST LET X1 X2 LDOTS XN BE N  INDEPENDENT OBSERVATIONS OF A GAUSSIAN RANDOM VARIABLE X WITH  UNKNOWN MEAN AND VARIANCE  WE DESIRE TO ESTIMATE THE MEAN AND  VARIANCE OF X  THE JOINT DENSITY OF N INDEPENDENT GAUSSIAN  RVS CONDITIONED ON KNOWING THE MEAN MU AND THE VARIANCE  SIGMA2 IS FX1X2LDOTSXNMU SIGMA2 FRAC12PIN2SIGMANEXPFRAC12 SIGMA2SUMI1N XI  MU2BEGINENUMERATE  ITEM DETERMINE A EM MAXIMUM LIKELIHOOD ESTIMATE OF MU BY    INDEXMAXIMUM LIKELIHOOD ESTIMATION    MAXIMIZING THIS JOINT DENSITY WITH RESPECT TO MU IE TAKE THE    DERIVATIVE WITH RESPECT TO MU  CALL THE ESTIMATE OF THE MEAN    YOU OBTAIN MUHAT  ITEM SINCE MUHAT IS A FUNCTION OF RANDOM VARIABLES IT IS ITSELF    A RANDOM VARIABLE  DETERMINE THE MEAN EXPECTED VALUE OF    MUHAT  AN ESTIMATE WHOSE EXPECTED VALUE IS EQUAL TO THE VALUE    IT IS ESTIMATED IS SAID TO BE EM UNBIASED INDEXUNBIASED  ITEM DETERMINE THE VARIANCE OF MUHAT    ITEM DETERMINE AN ESTIMATE FOR SIGMA2  ENDENUMERATE  IT IS NATURAL TO ASK IF THERE IS A BETTER ESTIMATOR FOR THE MEAN  THAN THE OBVIOUS ONE JUST OBTAINED  HOWEVER AS WILL BE SHOWN  IN SECTION REFSECCRLB THIS ONE IS DEPENDABLY THE BEST IN  THAT IT HAS THE LOWEST POSSIBLE VARIANCE FOR ANY UNBIASED ESTIMATEENDEXERCISESSECTIONMARKOV AND HIDDEN MARKOV MODELSLABELSECHMM1A HIDDEN MARKOV MODEL HMM IS A STOCHASTIC MODEL THAT IS USED TOMODEL TIMEVARYING RANDOM PHENOMENA  IT IS BASED UPON A MARKOV MODELAND CAN BE UNDERSTOOD IN TERMS OF THE STATESPACE MODELS ALREADYDERIVED  WE NOW PRESENT THE BASIC CONCEPTS PROVIDING RESOLUTION TOTHE ISSUES RAISED HERE IN CHAPTERS REFCHAPEM AND REFCHAPPATHSEARCHPLACEMENT HERE SERVES SEVERAL PURPOSES IT PROVIDES A DEMONSTRATION OFTHE UTILITY OF THE STATESPACE FORMULATION TO YET ANOTHER SYSTEM ITSMOOTHES THE DEVELOPMENT OF HMM ALGORITHMS IN LATER CHAPTERS AND ITPROVIDES INTRODUCTION AND MOTIVATION FOR TWO IMPORTANT ALGORITHMS THEEM ALGORITHM AND THE VITERBI ALGORITHM INDEXEXPECTATIONMAXIMIZATION EM ALGORITHMINDEXVITERBI ALGORITHMSUBSECTIONMARKOV MODELSBEGINFLOATINGFIGURE125ININPUTPICTUREDIRMARKOV1CAPTIONA SIMPLE MARKOV MODELLABELFIGMARKOV1ENDFLOATINGFIGUREINDEXMARKOV MODEL THE MARKOV MODEL IS USED TO MODEL THE EVOLUTIONOF RANDOM PHENOMENA THAT CAN BE IN DISCRETE STATES AS A FUNCTION OFTIME WHERE THE TRANSITION FROM ONE STATE TO THE NEXT IS RANDOMSUPPOSE THAT A SYSTEM CAN BE IN ONE OF S DISTINCT STATES AND THATAT EACH STEP OF DISCRETE TIME IT CAN MOVE TO ANOTHER STATE AT RANDOMWITH THE PROBABILITY OF THE TRANSITION AT TIME T DEPENDENT ONLY UPONTHE STATE OF THE SYSTEM AT TIME T  IT IS CONVENIENT TO REPRESENTTHIS CONCEPT USING A PROBABILISTIC STATE DIAGRAM AS SHOWN IN FIGUREREFFIGMARKOV1  IN THIS FIGURE THE MARKOV MODEL HAS THREE STATESFROM STATE 1 TRANSITIONS TO EACH OF THE STATES ARE POSSIBLE FROMSTATE 1 TO STATE 1 WITH PROBABILITY 05 AND SO FORTH  LET STDENOTE THE STATE AT TIME T WHERE ST TAKES ON ONE OF THE VALUES12LDOTSS  THE INITIAL STATE IS SELECTED ACCORDING TO APROBABILITY PII PII  PS1  IQQUAD I12LDOTSSBY THE FOREGOING DESCRIPTION THE PROBABILITY OF TRANSITION DEPENDSONLY UPON THE CURRENT STATE PST1  JST  I ST1K ST2  L LDOTS  PST1 J ST  ITHIS STRUCTURE ON THE PROBABILITIES IS CALLED THE EM MARKOVPROPERTY AND THE RANDOM SEQUENCE OF STATE VALUES S0 S1S2LDOTS IS CALLED A EM  MARKOV SEQUENCE OR A EM MARKOV  CHAIN   THIS SEQUENCE IS THE OUTPUT OF THE MARKOV MODELWE CAN DETERMINE THE PROBABILITY OF ARRIVING IN THE NEXT STATE BYADDING UP ALL THE PROBABILITIES OF THE WAYS OF ARRIVING THEREBEGINEQUATIONBEGINSPLITPST1  J  PST1JST1 PST1    QQUAD PST1  JST2 PST  2  CDOTS    QQUAD    PST1  JST  S PST  SLABELEQMARKOVP1 ENDSPLITENDEQUATIONTHE COMPUTATION IN REFEQMARKOVP1 CAN BE EXPRESSED CONVENIENTLYUSING MATRIX NOTATION  LET PBFT  BEGINBMATRIX PST  1  PST  2  VDOTS   PST  S ENDBMATRIXBE THE VECTOR OF PROBABILITIES FOR EACH STATE AND LET THE MATRIX ACONTAIN THE TRANSITION PROBABILITIESBEGINEQUATIONA  BEGINBMATRIX P11  P12  CDOTS  P1S P21  P22  CDOTS  P2S  VDOTS PS1  PS2  CDOTS  PSS ENDBMATRIXLABELEQHMMAENDEQUATIONWHERE PIJ IS AN ABBREVIATION FOR PST1ISTJ OR AIJ PST1ISTJ  FOR EXAMPLE FOR THE MARKOV MODEL OF FIGUREREFFIGMARKOV1BEGINEQUATION A  BEGINBMATRIX532  207 371  ENDBMATRIXLABELEQHMMAMATENDEQUATIONA EM STEADYSTATE PROBABILITY ASSIGNMENT IS ONE THAT DOES NOTCHANGE FROM ONE TIME STEP TO THE NEXT SO THE PROBABILITY MUST SATISFYTHE EQUATION A PBF  PBF  THIS IS A PARTICULAR EIGENEQUATIONWITH AN EIGENVALUE OF 1  MORE WILL BE SAID ABOUT EIGENVALUE PROBLEMSIN CHAPTER REFCHAPEIGENBY THE LAW OF TOTAL PROBABILITY EACH COLUMN OF A MUST SUM TO 1BEGINDEFINITION  AN MATSIZEMM MATRIX P SUCH THAT SUMJ1M PIJ 1  EACH ROW SUMS TO 1 AND EACH ELEMENT OF P IS NONNEGATIVE IS  CALLED A BF STOCHASTIC MATRIX  IF THE ROWS AND COLUMNS EACH SUM  TO 1 THEN P IS BF DOUBLY STOCHASTIC INDEXSTOCHASTIC MATRIXENDDEFINITION SEE EXERCISE REFEXSTOCHEIGTHE MATRIX A OF REFEQHMMA IS THE TRANSPOSE OF A STOCHASTIC MATRIXTHE VECTOR PIBF CONTAINS THE INITIAL PROBABILITIES  THUS WE CANWRITE THE PROBABILISTIC UPDATE EQUATION AS PBFT1  APBFTQQUAD TEXTWITH QQUAD PBF0  PIBFOR TO PUT IT ANOTHER WAYBEGINEQUATION PBFT1  APBFT  PIBF DELTATLABELEQMARKOV1ENDEQUATIONWITH PBFT  ZEROBF FOR T LEQ 0  THE SIMILARITY OFREFEQMARKOV1 TO THE FIRST EQUATION OF REFEQSTATE2 SHOULDBE APPARENT  IN COMPARING THESE TWO IT SHOULD BE NOTED THAT THESTATE REPRESENTED BY REFEQMARKOV1 IS ACTUALLY THE VECTOR OFPROBABILITIES PBFT NOT THE STATE OF THE MARKOV SEQUENCE STSUBSECTIONHIDDEN MARKOV MODELSTHE IDEA BEHIND THE HMM CAN BE ILLUSTRATED USING THE URN PROBLEMS OFELEMENTARY PROBABILITY AS SHOWN IN FIGURE REFFIGMARKOV2  SUPPOSEWE HAVE S DIFFERENT URNS EACH OF WHICH CONTAINS ITS OWN SET OFCOLORED BALLS  AT EACH INSTANT OF TIME AN URN IS SELECTED AT RANDOMACCORDING TO THE STATE IT WAS IN AT THE PREVIOUS INSTANT OF TIMETHAT IS ACCORDING TO A MARKOV MODEL  THEN A BALL IS DRAWN ATRANDOM FROM THE URN SELECTED AT TIME T  THE BALL IS WHAT WE OBSERVEAS THE OUTPUT AND THE ACTUAL STATE IS HIDDEN  THROUGHOUT THEREMAINDER OF THIS INTRODUCTION WE WILL CONTINUE TO DEVELOP THENOTATION FOR HMMS WITH DISCRETE OUTPUTS URN PROBLEMS BUT THEDEVELOPMENTS OF LATER CHAPTERS LIFT THIS RESTRICTIONBEGINFIGURECENTERINGINPUTPICTUREDIRMARKOV2CAPTIONTHE CONCEPT OF A HIDDEN MARKOV MODELLABELFIGMARKOV2ENDFIGUREINDEXHIDDEN MARKOV MODEL INDEXHMMSEEHIDDEN MARKOV MODEL THE DISTINCTION BETWEEN MARKOV MODELS AND HIDDEN MARKOV MODELS CAN BEFURTHER CLARIFIED BY CONTINUING THE ANALOGY WITH THE STATESPACEEQUATIONS IN REFEQSTATE2  EQUATION REFEQMARKOV1 PROVIDESFOR THE STATE UPDATE OF THE MARKOV SYSTEM  IN MOST LINEAR SYSTEMSHOWEVER THE STATE VECTOR IS NOT DIRECTLY OBSERVABLE INSTEAD IT ISOBSERVED ONLY THROUGH THE OBSERVATION MATRIX C ASSUMING FOR THEMOMENT THAT D IS ZERO YBFT  C XBFTSO THE STATE IS HIDDEN FROM DIRECT OBSERVATION  SIMILARLY IN THE HMMWE DO NOT OBSERVE THE STATE DIRECTLY  INSTEAD EACH STATE HAS APROBABILITY DISTRIBUTION ASSOCIATED WITH IT  WHEN THE HMM MOVES INTOSTATE ST AT TIME T THE OBSERVED OUTPUT YT IS AN OUTCOME OFA RANDOM VARIABLE YT THAT IS SELECTED ACCORDING TO DISTRIBUTIONFYTSTS WHICH WE WILL REPRESENT USING THE NOTATION FYSTS  FSYTHIS IDEA IS ILLUSTRATED IN FIGURE REFFIGMARKOVFYX  IN THE URNEXAMPLE OF THE PRECEEDING PARAGRAPH THE OUTPUT PROBABILITIES DEPENDON THE CONTENTS OF THE URNS  A SEQUENCE OF OUTPUTS FROM AN HMM ISY0 Y1 Y2LDOTS  THE UNDERLYING STATE INFORMATION IS NOTSEEN DIRECTLY IT IS HIDDENBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRHMM  ENDCENTER  CAPTIONAN HMM WITH FOUR STATESAN HMM WITH FOUR STATES    SHOWING THE STATES THE DISTRIBUTION IN EACH STATE AND    PROBABILISTIC TRANSITIONS BETWEEN STATES  LABELFIGMARKOVFYXENDFIGURETHE PROBABILITY DISTRIBUTION IN EACH STATE CAN BE OF ANY TYPE AND INGENERAL EACH STATE COULD HAVE ITS OWN TYPE OF DISTRIBUTION  MOSTOFTEN IN PRACTICE HOWEVER EACH STATE HAS THE SAME TYPE OFDISTRIBUTION BUT WITH DIFFERENT PARAMETERSLET M DENOTE THE NUMBER OF POSSIBLE OUTCOMES FROM ALL OF THE STATESAND LET YT BE THE RANDOM VARIABLE OUTPUT AT TIME T WITH OUTCOMEYT  WE CAN DETERMINE THE PROBABILITY OF EACH POSSIBLE OUTPUT BYADDING UP ALL THE PROBABILITIESBEGINALIGNED PYT  J  PYTJST  1PST1   QQUAD PYTJST2PST2  CDOTS   QQUAD  PYTJSTSPSTSENDALIGNEDLET QBFT  BEGINBMATRIXPYT1  PYT 2  VDOTS   PYT  M ENDBMATRIXAND  C  BEGINBMATRIX PYT1ST1  CDOTS  PYT1ST  S PYT2ST1  CDOTS  PYT2ST  S VDOTS PYTMST1  CDOTS  PYTMST  S ENDBMATRIXSO CIJ  PYT  IST  JFOR THE URNS SHOWN IN FIGURE REFFIGMARKOV2 WITH THE BALL COLORSBLACK GREEN AND RED CORRESPONDING TO VALUES 1 2 AND 3RESPECTIVELY  C  BEGINBMATRIX12  13    13 13   715   13 16   15    13 ENDBMATRIXEACH OF THE COLUMNS MUST SUM TO ONE  THE OUTPUT PROBABILITIESCAN BE COMPUTED BY QBFT  C PBFTTHE SIMILARITY WITH REFEQSTATE2 SHOULD BE CLEAR  BASED ON THISDISCUSSION THE HMM PARAMETERS ARE DESCRIBED BY THE TRIPLEAPIBFC MUCH LIKE OUR STATESPACE MODELSINDEXPATTERN RECOGNITION INDEXSPEECH PROCESSING THE HMMCAN BE APPLIED TO PATTERN RECOGNITION WHERE THE PATTERNS OCCUR ASEVENTS OCCURRING SEQUENTIALLY IN TIME  PERHAPS THE MOST SUCCESSFULAPPLICATION IS TO SPEECH PROCESSING  EACH WORD OR SOUND PHONEME TOBE RECOGNIZED IS REPRESENTED BY AN HMM WHERE THE OUTPUT IS SOMEFEATURE VECTOR THAT IS DERIVED FROM THE SPEECH DATA  THE RANDOMVARIABILITY IN THE FEATURE VECTOR AND THE AMOUNT OF TIME EACH FEATUREIS PRODUCED IS MODELED BY THE HMM  THE VARIABILITY IN THE DURATION OFTHE WORD IS MODELED BY THE MARKOV MODEL  THE VARIABILITY IN THEOUTPUTS IS MODELED BY THE RANDOM SELECTION FROM WITHIN EACH STATEFOR EXAMPLE IN A SMALL VOCABULARY SYSTEM WITH N WORDS THERE AREN HMMS AI PIBFI CI EACH BEING TRAINED OR ADAPTED TO REPRESENT THE PARAMETERS FOR THAT WORD  THIS IS THE TRAINING PHASE OFTHE PATTERN RECOGNITION PROBLEM  INDEXTRAINING PHASETO PERFORM RECOGNITION OF AN UNKNOWN WORD ITS SEQUENCE OF FEATUREVECTORS IS COMPUTED AND THE LIKELIHOOD PROBABILITY THAT THISSEQUENCE OF FEATURE VECTORS WAS PRODUCED BY THE HMM AI PIBFICI IS COMPUTED FOR EACH I  THAT HMM WHICH PRODUCES THE HIGHESTPROBABILITY SELECTS THE RECOGNIZED WORDTHE HMM HAS ALSO BEEN APPLIED TO HANDWRITING RECOGNITION SPEAKERIDENTIFICATION AND OTHER AREASTHE PATTERN RECOGNITION APPLICATIONIS DIAGRAMMED IN FIGURE REFFIGHMMPATRECBASED ON THIS SIMPLE DISCUSSION THERE ARE SEVERAL QUESTIONS THATCAN BE POSED IN CONJUNCTION WITH HMMSBEGINENUMERATEITEM HOW CAN THE PARAMETERS APIBFC BE ESTIMATED BASED UPON  OBSERVATIONS OF THE DATA  OR MORE GENERALLY HOW CAN THE  PARAMETERS OF OTHER OUTPUT DISTRIBUTIONS BE COMPUTED  IN OTHER  WORDS HOW CAN WE TRAIN THE PARAMETERS OF THE MODELS IN THE PATTERN  RECOGNITION PROBLEM INDEXPARAMETER ESTIMATIONITEM SUPPOSE WE HAVE AN HMM AND WE OBSERVE A SEQUENCE OF DATA  HOWCAN WE DETERMINE HOW WELL THE DATA FITS THE MODEL  IN OTHERWORDS CAN WE EFFICIENTLY DETERMINE THE LIKELIHOOD OF THE DATAINDEXMAXIMUM LIKELIHOODITEM RELATED SOMEWHAT TO THE PREVIOUS SUPPOSE WE HAVE AN HMM AND WE  OBSERVE SOME DATA SUPPOSEDLY GENERATED FROM IT  HOW CAN WE  DETERMINE THE SEQUENCE OF STATES OF THE UNDERLYING MARKOV MODEL  THAT IS WE WANT TO UNCOVER THE HIDDEN STATESENDENUMERATETHESE ISSUES ARE EXPLORED IN CHAPTERS REFCHAPEM ANDREFCHAPPATHSEARCH WHERE THE EM ALGORITHM AND THE VITERBIALGORITHM ARE INTRODUCED AND APPLIED TO THIS PROBLEMBEGINEXERCISESITEM  WRITE A SC MATLAB FUNCTION TT GENMARKOVNABPIIN WHICH  GENERATES N OUTPUTS OF A HIDDEN MARKOV MODEL WITH STATE TRANSITION  MATRIX A AND OUTPUT MATRIX B WITH INITIAL PROBABILITIES  PIINITEM FOR THE STATETRANSITION PROBABILITY MATRIX A GIVEN IN  REFEQHMMAMAT SHOW THAT THE PROBABILITY VECTOR PBF   05720 05441 06138T SATISFIES A PBF  PBFTHIS IS THE EM STEADYSTATE PROBABILITY OF THE MARKOV MODELENDEXERCISESINPUTHOMEDIRINTROPARTPROOFSINPUTHOMEDIRINTROPARTBERLMASSYSETEXSECTREFSECLTIBEGINEXERCISESITEM COMPLEX ARITHMETIC THIS EXERCISE GIVES A BRIEF REFRESHER ON  COMPLEX MULTIPLICATION AS WELL AS MATRIX MULTIPLICATION  LET Z1   AJB AND Z2  C  JD BE TWO COMPLEX NUMBERS  LET Z3  Z1  Z2  EJF  BEGINENUMERATE  ITEM SHOW THAT THE PRODUCT CAN BE WRITTEN AS BEGINBMATRIXE  F  ENDBMATRIX  BEGINBMATRIXCD  D  CENDBMATRIX BEGINBMATRIX A  B ENDBMATRIXIN THIS FORM FOUR REAL MULTIPLIES AND TWO REAL ADDS ARE REQUIREDITEM SHOW THAT THE COMPLEX PRODUCT CAN ALSO BE WRITTEN AS E  ABD  ACD QQUAD F  ABD  BCDIN THIS FORM ONLY THREE REAL MULTIPLICATIONS AND FIVE REAL ADDITIONSARE REQUIRED  IF ADDITION IS SIGNIFICANTLY EASIER THANMULTIPLICATION IN HARDWARE THEN THIS SAVES COMPUTATIONSITEM SHOW THAT THIS MODIFIED SCHEME CAN BE EXPRESSED IN MATRIX  NOTATION AS BEGINBMATRIXE  F ENDBMATRIX  BEGINBMATRIX101  011  ENDBMATRIX BEGINBMATRIXCD00  0CD  0  00D  ENDBMATRIX BEGINBMATRIX10  01  11 ENDBMATRIX  BEGINBMATRIXA  B ENDBMATRIXENDENUMERATEITEM SHOW THAT REFEQPFEZT FOR THE PARTIAL FRACTION EXPANSION OF  A ZTRANSFORM WITH REPEATED ROOTS IS CORRECTITEM DETERMINE THE PARTIAL FRACTION EXPANSION OF THE FOLLOWINGBEGINARRAYLHSPACE8EMLTEXTA  HZ  FRAC13Z1115Z1  56 Z2 TEXTB  HZ  FRAC15Z1  6Z2115Z1  56 Z2TEXTC  HZ  FRAC2  3Z113 Z12 TEXTD  HZ  FRAC56Z113 Z1214Z1ENDARRAYCHECK YOUR RESULTS USING TT RESIDUEZ IN SC MATLABITEM INVERSES OF HIGHERORDER MODES  BEGINENUMERATE  ITEM PROVE THE FOLLOWING PROPERTY FOR ZTRANSFORMS  IF XT LEFTRIGHTARROW XZTHEN TXT LEFTRIGHTARROW Z FRACD XZDZITEM USING THE FACT THAT PT UT LEFTRIGHTARROW 11PZ1  SHOW THAT T PT UT LEFTRIGHTARROW FRACPZ11PZ12ITEM DETERMINE THE ZTRANSFORM OF T2 PT UTITEM BY EXTRAPOLATION DETERMINE THE ORDER OF THE POLE OF A MODE OF  THE FORM TK PT UT  ENDENUMERATEEXSKIPITEM SHOW THAT THE AUTOCORRELATION FUNCTION DEFINED IN  REFEQAUTOCORRDEF HAS THE PROPERTY THAT  RYYK  RBARYYKITEM SHOW THAT REFEQMAAUTOCORR IS CORRECTITEM FOR THE MA PROCESS YT  FT  2FT1  3FT2WHERE FT IS A ZEROMEAN WHITENOISE RANDOM PROCESS WITHSIGMAF2  1 DETERMINE THE MATSIZE33 AUTOCORRELATIONMATRIX RITEM FOR THE FIRSTORDER REAL AR PROCESS YT1  A1 YT   FT1WITH A11 AND EFT  0 SHOW THATBEGINEQUATIONSIGMAY2  EY2T  FRACSIGMAF21A12LABELEQFIRSTARVARENDEQUATIONITEM FOR AN AR PROCESS REFEQAR2 DRIVEN BY A WHITENOISE  SEQUENCE FT WITH VARIANCE SIGMAF2 SHOW THATBEGINEQUATIONSIGMAF2  SUMI0P AI RYYI HAYKIN P 120LABELEQARINPUTVARENDEQUATIONITEM LET YT  7YT1  12 YT2  FT WHERE FT IS A  ZEROMEAN WHITENOISE RANDOM PROCESS WITH SIGMAF2  2    BEGINENUMERATE  ITEM WRITE  THE YULEWALKER EQUATIONS FOR Y  ITEM DETERMINE RYY1 AND RYY2  ITEM FIND SIGMAY2  ENDENUMERATEITEM SECONDORDER AR PROCESSES CONSIDER THE SECONDORDER REAL AR  PROCESS INDEXAUTOREGRESSIVESECONDORDERBEGINEQUATION YT2  A1 YT1  A2 YT  FT2 LABELEQYULEWALKER21ENDEQUATIONWHERE FT IS A ZEROMEAN WHITENOISE SEQUENCE  THE DIFFERENCEEQUATION IN REFEQAR3 HAS A CHARACTERISTIC EQUATION WITH ROOTS P1 P2  FRAC12A1 PM SQRTA12  4A2BEGINENUMERATEITEM USING THE YULEWALKER EQUATIONS SHOW THAT IF THE  AUTOCORRELATION VALUES  RYYLK  EYTKYBARTLARE KNOWN THEN THE MODEL PARAMETERS MAY  BE DETERMINED FROMBEGINEQUATIONBEGINSPLITA1  FRACRYY1RYY0  RYY2 RYY20   RYY21 A2  FRACRYY0RYY2  RYY21 RYY20   RYY21ENDSPLITLABELEQYW5ENDEQUATIONITEM ON THE OTHER HAND IF SIGMAY2  RYY0 AND A1 AND  A2 ARE KNOWN SHOW THAT THE AUTOCORRELATION VALUES CAN BE  EXPRESSED AS  BEGINEQUATION    LABELEQYW6BEGINSPLIT    RYY1  FRACA11A2 SIGMAY2RYY2  SIGMAY2LEFT FRACA121A2  A2RIGHTENDSPLITENDEQUATIONITEM USING REFEQARINPUTVAR AND THE RESULTS OF THIS PROBLEM  SHOW THAT  BEGINEQUATION    LABELEQYW7    RYY0  SIGMAY2  LEFTFRAC1A21A2RIGHT    FRACSIGMAF21A22  A12  ENDEQUATIONITEM USING RYY0  SIGMAY2 AND RYY1  A1  SIGMAY21A2 AS INITIAL CONDITIONS FIND AN EXPLICIT SOLUTION  TO THE YULEWALKER DIFFERENCE EQUATION RYYK  A1 RYYK1  A2 RYYK2  0IN TERMS OF P1 P2 AND SIGMAY2ENDENUMERATE HAYKIN P 121ITEM FOR THE SECONDORDER DIFFERENCE EQUATION YT2  7 YT1 12 YT  FT2WHERE FT IS A ZEROMEAN WHITE SEQUENCE WITH SIGMAF2  1DETERMINE SIGMAY2  RYY0 RYY1 AND RYY2ITEM A RANDOM PROCESS YT HAVING ZEROMEAN AND MATSIZEMM  AUTOCORRELATION MATRIX R IS APPLIED TO AN FIR FILTER WITH IMPULSE  RESPONSE VECTOR HBF  H0H1H2LDOTSHM1T  DETERMINE  THE AVERAGE POWER OF THE FILTER OUTPUT XT  EXSKIPITEM PLACE THE FOLLOWING INTO STATE VARIABLE FORM CONTROLLER  CANONICAL FORM AND DRAW A REALIZATIONBEGINARRAYLHSPACE8EMLTEXTA  HZ  FRAC13Z1115Z1  56 Z2 TEXTB  HZ  FRAC15Z1  6Z2115Z1  56 Z2ENDARRAY ITEM DETERMINE THE FIRST FOUR NONZERO MARKOV PARAMETERS OF THE   SYSTEMS IN THE PREVIOUS EXERCISEITEM IN ADDITION TO THE BLOCK DIAGRAM SHOWN IN FIGURE  REFFIGTRANSFER2 THERE ARE MANY OTHER FORMS  THIS PROBLEM  INTRODUCES ONE OF THEM THE EM OBSERVER CANONICAL FORM INDEXOBSERVER CANONICAL FORM  BEGINENUMERATE  ITEM SHOW THAT THE ZTRANSFORM RELATION IMPLIED BY REFEQARMA    CAN BE WRITTEN ASBEGINEQUATIONBEGINSPLIT YZ   BBAR0 FZ  BBAR1 FZ  ABAR1 YZZ1  BBAR2 FZ  ABAR2 YZ Z2   CDOTS  BBARP FZ  ABARP YZZ1ENDSPLITLABELEQBLOCK2ENDEQUATIONITEM DRAW A BLOCK DIAGRAM REPRESENTING REFEQBLOCK2 CONTAINING  P DELAY ELEMENTS ITEM LABEL THE OUTPUTS OF THE DELAY ELEMENTS FROM RIGHT TO LEFT AS  X1 X2 LDOTS XP  SHOW THAT THE SYSTEM CAN BE PUT INTO STATE  SPACE FORM WITH A  BEGINBMATRIX ABAR1  1  0  CDOTS  0 ABAR2  0  1  CDOTS  0 VDOTS ABARP1  0  0 CDOTS  1 ABARP  0  0  CDOTS  0 ENDBMATRIXQQUAD BBF  BEGINBMATRIX BBAR1  ABAR1 BBAR0  BBAR2  ABAR2 BBAR0 CDOTS BBARP1  ABARP1 BBAR0 BBARP  ABARP BBAR0 ENDBMATRIXQQUAD CBF  BEGINBMATRIX 1  0  0  VDOTS  0 ENDBMATRIXQQUAD D  BBAR0A MATRIX A OF THIS FORM IS SAID TO BE IN EM SECOND COMPANION  FORM INDEXSECOND COMPANION FORMITEM DRAW THE BLOCK DIAGRAM IN OBSERVER CANONICAL FORM FOR HZ  FRAC 2  3Z1  4 Z21  Z1  6Z2  7 Z3AND DETERMINE THE SYSTEM MATRICES ABBF CBFT DENDENUMERATEITEM ANOTHER BLOCK DIAGRAM REPRESENTATION IS BASED UPON THE PARTIAL  FRACTION EXPANSION  ASSUME INITIALLY THAT THERE ARE NO REPEATED ROOTS SO THAT HZ  SUMK1P FRACNK1PK Z1  BEGINENUMERATEITEM DRAW A BLOCK DIAGRAM REPRESENTING THE PARTIAL FRACTION EXPANSION  BY USING THE FACT THAT FRACYZFZ  FRAC11P Z1 HAS THE BLOCK DIAGRAM BEGINCENTERINPUTPICTUREDIRTRANSFER5LATEXINPUTPICTUREDIRTRANSFER5ENDCENTERITEM LET XI I12LDOTS P DENOTE THE OUTPUTS OF THE DELAY  ELEMENTS  SHOW THAT THE SYSTEM CAN BE PUT INTO STATESPACE FORM  WITH  A  BEGINBMATRIX P1  0  0  CDOTS 00P2  0  CDOTS  0  VDOTS 0  0  0  CDOTS  PP ENDBMATRIXQQUAD BBF  BEGINBMATRIX 1  1  VDOTS  1 ENDBMATRIXQQUAD CBF  BEGINBMATRIX  N1  N2  VDOTS  NP ENDBMATRIXQQUAD D  B0A MATRIX A IN THIS FORM IS SAID TO BE A EM DIAGONALMATRIX INDEXDIAGONAL MATRIXITEM DETERMINE THE PARTIAL FRACTION EXPANSION OF  HZ  FRAC 1  2Z1 1  5 Z1  06 Z2AND DRAW THE BLOCK DIAGRAM BASED UPON IT  DETERMINE ABBF CBFDITEM WHEN THERE ARE REPEATED ROOTS THINGS ARE SLIGHTLY MORE  COMPLICATED  CONSIDER FOR SIMPLICITY A ROOT APPEARING ONLY  TWICE  DETERMINE THE PARTIAL FRACTION EXPANSION OF HZ  FRAC1Z11 2 Z115Z12BE CAREFUL ABOUT THE REPEATED ROOT  ITEM DRAW THE BLOCK DIAGRAM CORRESPONDING TO HZ IN PARTIAL  FRACTION FORM USING ONLY THREE DELAY ELEMENTS  ITEM SHOW THAT THE STATE VARIABLES CAN BE CHOSEN SO THAT A  BEGINBMATRIX 5 00 15  0 0  0  2  ENDBMATRIXA MATRIX IN THIS FORM BLOCKS ALONG THE DIAGONAL EACH BLOCK BEINGEITHER DIAGONAL OR DIAGONAL WITH ONES IN IT AS SHOWN IS IN EM JORDANFORM INDEXJORDAN FORMENDENUMERATEITEM SHOW THAT THE SYSTEM IN REFEQSTATE3 HAS THE SAME TRANSFER  FUNCTION AND SOLUTION AS DOES THE SYSTEM IN REFEQSTATE2ITEM LABELEXSTATEOUT FOR A SYSTEM IN STATESPACE REPRESENTATION  BEGINENUMERATE  ITEM SHOW BY INDUCTION THAT REFEQXNDT1 IS CORRECT  ITEM FOR A TIMEVARYING SYSTEM AS IN REFEQSTATE4 DETERMINE    A REPRESENTATION SIMILAR TO REFEQXNDT1  ENDENUMERATEITEM INTERCONNECTION OF SYSTEMS INDEXINTERCONNECTION OF SYSTEMS  CITEKAILATH80 LET A1BBF1CBF1T AND   A2BBF2CBF2T BE TWO SYSTEMS  DETERMINE THE SYSTEM  ABBFCBFT OBTAINED BY CONNECTING THESE TWO SYSTEMS  BEGINENUMERATE  ITEM IN SERIES  ITEM IN PARALLEL  ITEM IN A FEEDBACK CONFIGURATION WITH A1BBF1CBF1T IN    THE FORWARD LOOP AND A2BBF2CBF2T IN THE FEEDBACK LOOP  ENDENUMERATEITEM SHOW THAT  BEGINBMATRIX A A1  0  A2 ENDBMATRIX QQUADBEGINBMATRIXBBF  ZEROBF ENDBMATRIX QQUAD CBFT QBFTAND BEGINBMATRIX A 0  A1  A2 ENDBMATRIX QQUADBEGINBMATRIXBBF  QBF ENDBMATRIX QQUAD CBFT ZEROBFAND ABBFCBFT ALL HAVE THE SAME TRANSFER FUNCTION FOR ALLVALUES OF A1 A2 AND QBF THAT LEAD TO VALID MATRIXOPERATIONS  CONCLUDE THAT REALIZATIONS CAN HAVE DIFFERENT NUMBERS OF STATESEXSKIPITEM CONSIDER THE SYSTEM FUNCTION HZ  FRACZ3  3Z2  2Z Z3  10Z2  31 Z  30BEGINENUMERATEITEM DRAW THE BLOCK DIAGRAM IN CONTROLLER CANONICAL FORMITEM DRAW THE BLOCK DIAGRAM IN JORDAN FORM DIAGONAL  FORM INDEXJORDAN FORMITEM HOW MANY MODES ARE REALLY PRESENT IN THE SYSTEM  THE PROBLEM  HERE IS THAT A EM MINIMAL REALIZATION OF A IS NOT OBTAINED  DIRECTLY FROM THE HZ AS GIVENENDENUMERATEITEM CITEKAILATH80 IF ABBFCBFTD WITH D NEQ 0  DESCRIBES A SYSTEM HS IN STATESPACE FORM SHOW THAT A  BBF CBFTD BBFD CBFTD 1DDESCRIBES A SYSTEM WITH SYSTEM FUNCTION 1HSITEM LABELEXUPDATEDEQ STATESPACE SOLUTIONS  BEGINENUMERATE  ITEM SHOW THAT REFEQSTATEUPDATE IS A SOLUTION TO THE    DIFFERENTIAL EQUATION IN REFEQXNCT1 FOR CONSTANT    ABCDITEM SHOW THAT AN UPDATE FROM XBFTAU TO XBFT IS AS GIVEN  IN REFEQSTATEUPDATEITEM SHOW THAT REFEQXBFT3 IS A SOLUTION TO THE DIFFERENTIAL  EQUATION IN REFEQXNCT1 FOR NONCONSTANT ABCD PROVIDED  THAT PHI SATISFIES THE PROPERTIES GIVEN  ENDENUMERATEITEM FIND A SOLUTION TO THE DIFFERENTIAL EQUATION DESCRIBED BY THE  STATESPACE EQUATIONSBEGINALIGNED XBFDOTT  BEGINBMATRIX 01  10 ENDBMATRIX XBFT EXMATSPYT  1 0 XBFTENDALIGNEDWITH XBF0  XBF0  THESE EQUATIONS DESCRIBE SIMPLE HARMONICMOTIONITEM CONSIDER THE SYSTEM DESCRIBED BYBEGINALIGNED XBFDOTT  BEGINBMATRIX 2  0  1  1 ENDBMATRIX XBFT BEGINBMATRIX 2  1 ENDBMATRIX FT EXMATSPYT  0  2 XBFTENDALIGNEDBEGINENUMERATEITEM DETERMINE THE TRANSFER FUNCTION HSITEM FIND THE PARTIAL FRACTION EXPANSION OF HSITEM VERIFY THAT THE MODES OF HS ARE THE SAME AS THE EIGENVALUES  OF AENDENUMERATEITEM VERIFY REFEQGEOM1 BY LONG DIVISION LABELGEOMETRIC    SERIESEXSKIPITEM SYSTEM IDENTIFICATION INDEXSYSTEM IDENTIFICATIONVIA BODE    PLOTS INDEXBODE PLOTFOR SYSTEM IDENTIFICATION IN THIS  EXERCISE YOU WILL DEVELOP A TECHNIQUE FOR IDENTIFICATION OF THE  PARAMETERS OF A CONTINUOUSTIME SECONDORDER SYSTEM BASED UPON  FREQUENCY RESPONSE MEASUREMENTS BODE PLOTS  ASSUME THAT THE  SYSTEM TO BE IDENTIFIED HAS AN OPENLOOP TRANSFER FUNCTION HOS  FRACBSSA  BEGINENUMERATE  ITEM SHOW THAT WITH THE SYSTEM IN A FEEDBACK CONFIGURATION AS SHOWN    IN FIGURE REFFIGBODEID1 THE TRANSFER FUNCTION CAN BE WRITTEN    AS HCS  FRACYSFS  FRAC11ABS  1BS2 BEGINFIGUREHTBP   BEGINCENTER     LEAVEVMODE     INPUTPICTUREDIRFEEDBACK1     CAPTIONSIMPLE FEEDBACK CONFIGURATION     LABELFIGBODEID1   ENDCENTER ENDFIGUREITEM SHOW THAT FRAC1HCJOMEGA  AJOMEGA ANGLE PHIJOMEGAWHERE AJOMEGA  FRAC1BSQRTBOMEGA22  AOMEGA2 QQUADTEXTANDQQUAD TAN PHIJOMEGA  FRACAOMEGAB  OMEGA2THE QUANTITIES AJOMEGA AND PHIJOMEGA CORRESPOND TO THERECIPROCAL AMPLITUDE AND THE PHASE DIFFERENCE BETWEEN INPUT ANDOUTPUTITEM SHOW THAT IF AMPLITUDEPHASE MEASUREMENTS ARE MADE AT N  DIFFERENT FREQUENCIES OMEGA1ALLOWBREAK OMEGA2 ALLOWBREAK  LDOTS ALLOWBREAK OMEGAN THEN  THE UNKNOWN PARAMETERS A AND B CAN BE ESTIMATED BY SOLVING THE  OVERDETERMINED SET OF EQUATIONS BEGINBMATRIX AJOMEGA1  OMEGA1 SQRT1 1TAN2 PHIJOMEGA1 TAN PHIJOMEGA1  OMEGA1 AJOMEGA2  OMEGA2 SQRT1 1TAN2 PHIJOMEGA2 TAN PHIJOMEGA2  OMEGA2 VDOTS AJOMEGAN  OMEGAN SQRT1 1TAN2 PHIJOMEGAN TAN PHIJOMEGAN  OMEGAN ENDBMATRIXBEGINBMATRIX B  A ENDBMATRIX BEGINBMATRIX 0  OMEGA12 TANPHIJOMEGA1 0  OMEGA22 TANPHIJOMEGA2 VDOTS 0  OMEGAN2 TANPHIJOMEGAN ENDBMATRIX  ENDENUMERATEITEM VERIFY REFEQESD1  LABELEXESD1ITEM SHOW THAT  SUMTINFTYINFTY YT2  FRAC12PI INTPIPIGYYOMEGA DOMEGAHINT RECALL THE INVERSE FOURIER TRANSFORM INDEXFOURIER TRANSFORM YT FRAC12PI INTPIPI YOMEGA EJ OMEGA TDOMEGAITEM SHOW THAT UNDER THE CONDITION THAT REFEQPSDDEC IS TRUE  THE PSD SATISFIES SYOMEGA  LIMNRIGHTARROW INFTY ELEFT FRAC1N  LEFTSUMN1N YN EJOMEGA NRIGHT2 RIGHTHINT  SHOW AND USE THE FACT THAT SUMN1N SUMM1N FNM  SUMLN1N1 NLFLITEM MODAL ANALYSIS THE FOLLOWING DATA ARE MEASURED FROM A  THIRDORDER SYSTEM Y   0320002500010000022200006000120000500001ASSUME THAT THE FIRST TIME INDEX IS 0 SO THAT Y0  032BEGINENUMERATEITEM DETERMINE THE MODES IN THE SYSTEM AND PLOT THEM IN THE COMPLEX PLANEITEM THE DATA CAN BE WRITTEN AS YT  C1P1T  C2P2T  C3P3T QQUAD T GEQ 0DETERMINE THE CONSTANTS C1 C2 AND C3ITEM TO EXPLORE THE EFFECT OF NOISE ON THE SYSTEM ADD RANDOM  GAUSSIAN NOISE TO EACH DATA POINT WITH VARIANCE SIGMA2  001  THEN FIND THE MODES OF THE NOISY DATA  REPEAT SEVERAL TIMES WITH  DIFFERENT NOISE AND COMMENT ON HOW THE MODAL ESTIMATES MOVEENDENUMERATEITEM MODAL ANALYSIS  IF YT HAS TWO REAL SINUSOIDS YT  A COSOMEGA1 T  THETA1  B COSOMEGA2 T THETA2AND THE FREQUENCIES ARE KNOWN DETERMINE A MEANS OF COMPUTING THEAMPLITUDES AND PHASES FROM MEASUREMENTS AT TIME INSTANTS T1 T2LDOTS TNEXSKIPSETEXSECTREFSECMULTGAUSS  ITEM SHOW THAT R1 FROM REFEQINVCOVAR IS CORRECTITEM SHOW THAT REFEQ2GAUSS FOLLOWS FROM REFEQMULTGAUSS AND  REFEQINVCOVARITEM SUPPOSE THAT XSIM NCMUXSIGMAX2 AND N SIM  NC0SIGMAN2 ARE INDEPENDENTLY DISTRIBUTED GAUSSIAN RVS  LET Y  XNBEGINENUMERATEITEM DETERMINE THE PARAMETERS OF THE DISTRIBUTION OF YITEM IF YY IS MEASURED WE CAN ESTIMATE X BY COMPUTING THE  CONDITIONAL DENSITY FXY  DETERMINE THE MEAN AND VARIANCE OF  THIS CONDITIONAL DENSITY  INTERPRET THESE RESULTS IN TERMS OF  GETTING INFORMATION ABOUT IF  X I SIGMAN2 GG SIGMAX2  AND  II SIGMAN2 LL SIGMAX2ENDENUMERATEITEM SUPPOSE THAT X SIM NCMUX SIGMAX2  AND Y  SIMNCMUY  SIGMAY2 ARE JOINTLY DISTRIBUTED GAUSSIAN RVS  WITH CORRELATION RHO  DETERMINE THE PARAMETERS OF THE  DISTRIBUTION OF Z  A X  BYITEM IF X SIM NC01 SHOW THAT Y  SIGMA X  MUIS DISTRIBUTED AS Y NCSIGMA2MU LABELEXGENGAUSS ITEM IF X SIM NCSIGMA2MU DETERMINE EX THE EXPECTED   VALUE OF THE ABSOLUTE VALUE OF XITEM LABELEXGAUSSEST LET X1 X2 LDOTS XN BE N  INDEPENDENT OBSERVATIONS OF A GAUSSIAN RANDOM VARIABLE X WITH  UNKNOWN MEAN AND VARIANCE  WE DESIRE TO ESTIMATE THE MEAN AND  VARIANCE OF X  THE JOINT DENSITY OF N INDEPENDENT GAUSSIAN  RVS CONDITIONED ON KNOWING THE MEAN MU AND THE VARIANCE  SIGMA2 IS FX1X2LDOTSXNMU SIGMA2 FRAC12PIN2SIGMANEXPFRAC12 SIGMA2SUMI1N XI  MU2BEGINENUMERATEITEM DETERMINE A EM MAXIMUM LIKELIHOOD ESTIMATE OF MU BY  INDEXMAXIMUM LIKELIHOOD ESTIMATION MAXIMIZING THIS JOINT DENSITY  WITH RESPECT TO MU IE TAKE THE DERIVATIVE WITH RESPECT TO  MU  CALL THE ESTIMATE OF THE MEAN OBTAINED THUS MUHATITEM SINCE MUHAT IS A FUNCTION OF RANDOM VARIABLES IT IS ITSELF    A RANDOM VARIABLE  DETERMINE THE MEAN EXPECTED VALUE OF    MUHAT  AN ESTIMATE WHOSE EXPECTED VALUE IS EQUAL TO THE VALUE    IT IS ESTIMATED IS SAID TO BE EM UNBIASED INDEXUNBIASED  ITEM DETERMINE THE VARIANCE OF MUHAT    ITEM DETERMINE AN ESTIMATE FOR SIGMA2  ENDENUMERATE  IT IS NATURAL TO ASK IF THERE IS A BETTER ESTIMATOR FOR THE MEAN  THAN THE OBVIOUS ONE JUST OBTAINED  HOWEVER AS WILL BE SHOWN  IN SECTION REFSECCRLB THIS ESTIMATOR IS DEFENDABLY THE BEST IN  THAT IT HAS THE LOWEST POSSIBLE VARIANCE FOR ANY UNBIASED ESTIMATE  EXSKIP SETEXSECTREFSECHMM1ITEM A MARKOV RANDOM PROCESS INDEXMARKOV RANDOM PROCESS XT HAS  THE PROPERTY THAT PXT3  X2XT2X2 XT1X1  PXT3  X3XT2X2WHEN T3  T2  T1 THAT IS THE PROBABILITY DEPENDS ONLY UPON THEMOST RECENT CONDITIONING EVENT  WE WILL ABBREVIATE THIS USING THENOTATION FX3X2X1  FX3X2BEGINENUMERATEITEM FOR A MARKOV PROCESS SHOW THAT FX3X1X2  FX3X2FX2X1THIS IS THE PROPERTY OF CONDITIONAL INDEPENDENCE X3 IS INDEPENDENTOF X1 PROVIDED THAT THEY ARE EACH CONDITIONED ON AN INTERMEDIATEOBSERVATION X2 INDEXMARKOV RANDOM PROCESSCONDITIONAL INDEPENDENCEITEM NOW SUPPOSE THAT XT IS A GAUSSIAN RANDOM PROCESS AND ASSUME  FOR CONVENIENCE ONLY THAT IT IS ZEROMEAN  LET  RXTS  EXT XSIF XT IS ALSO MARKOV SHOW THAT RXT3T1  FRACRXT3T2RXT2T1RXT2T2HINT USE THE FACT THAT EEXT3XT1XT2  EXT3XT1AND USE THE FORMULA FOR CONDITIONAL EXPECTATION DERIVED IN REFEQFXYENDENUMERATE ITEM  WRITE A SC MATLAB FUNCTION TT GENMARKOVNABPIIN THAT   GENERATES N OUTPUTS OF A HIDDEN MARKOV MODEL WITH STATE TRANSITION   MATRIX A AND OUTPUT MATRIX B WITH INITIAL PROBABILITIES   PIINITEM  FOR THE STATETRANSITION PROBABILITY MATRIX A GIVEN IN   REFEQHMMAMAT SHOW THAT THE PROBABILITY VECTOR PBF    05720 05441 06138T SATISFIES  A PBF  PBF  THIS IS THE EM STEADYSTATE PROBABILITY OF THE MARKOV MODELITEM FOR THE STATETRANSITION PROBABILITY MATRIX A GIVEN IN  REFEQHMMAMAT FIND A PROBABILITY VECTOR PBF SUCH THAT SHOW THAT THE PROBABILITY VECTOR PBF    05720 05441 06138T SATISFIES A PBF  PBFSUCH A PROBABILITY VECTOR IS CALLED THE EM STEADYSTATE PROBABILITYOF THE MARKOV MODEL  EXSKIPSETEXSECTREFSECPROOFSITEM SHOW THAT SQRT3 IS IRRATIONALITEM SHOW THAT THERE ARE AN INFINITE NUMBER OF  PRIMES  HINT  USE A PROOF BY CONTRADICTION ASSUMING THAT THERE  ARE ONLY A FINITE NUMBER OF PRIMES  THEN BUILD A NUMBER 2CDOT 3  CDOT 5 CDOT CDOTS CDOT P  1 WHERE P IS THE ASSUMED LAST  PRIME AND SHOW THAT THIS IS NOT DIVISIBLE BY ANY OF THE LISTED PRIMESITEM USING PROOF BY CONTRADICTION SHOW THAT SQRT2 CANNOT BE A  RATIONAL NUMBER  HINT  ASSUME SQRT2  MN FOR SOME INTEGER  M AND N WHERE THE FRACTION IS EXPRESSED IN REDUCED FORM  SHOW  THAT THIS LEADS TO A CONTRADICTIONITEM SHOW THAT IF M2 IS EVEN THEN M MUST BE EVENITEM BY TRIAL AND ERROR DETERMINE A PLAUSIBLE FORMULA FOR SUMI0N 2ITHEN PROVE BY INDUCTION THAT YOUR FORMULA IS CORRECTITEM DETERMINE BY EXPERIMENT A PLAUSIBLE FORMULA FOR THE SUM OF THE  FIRST N ODD INTEGERS 135 CDOTS  2N1THEN PROVE BY INDUCTION THAT YOUR FORMULA IS CORRECTITEM DETERMINE BY EXPERIMENT A PLAUSIBLE FORMULA FOR SUMI1N FRAC1I2 ITHEN PROVE BY INDUCTION THAT YOUR FORMULA IS CORRECTITEM SHOW BY INDUCTION FOR EVERY POSITIVE INTEGER N THAT N3   N IS DIVISIBLE BY 3ITEM THE QUANTITY BOXED BINOMNK  FRACNKNK IS THE NUMBER OF WAYS OF CHOOSING K OBJECTS OUT OF N OBJECTSWHERE N GEQ K  THE QUANTITY BINOMNK IS ALSO KNOWN AS THE EMBINOMIAL COEFFICIENT  WE READ THE NOTATION N CHOOSE K AS NCHOOSE K INDEXBINOMIAL COEFFICIENT INDEXNKBINOMNKSHOW BY INDUCTION THAT FOR 1 LEQ K LEQ NBEGINEQUATIONN1CHOOSEK  NCHOOSEK  NCHOOSEK1LABELEQXSUMBNENDEQUATIONITEM SHOW BY INDUCTION THAT FOR N GEQ 0 SUMK0N N CHOOSE K  2NITEM SHOW BY INDUCTION THATBEGINEQUATION BOXEDXYN  SUMK0N N CHOOSE K XK YNKLABELEQBINOMENDEQUATIONTHIS IMPORTANT FORMULA IS KNOWN AS THE EM BINOMIAL  THEOREM INDEXBINOMIAL THEOREMITEM PROVE THE FOLLOWING BY INDUCTION SUMK1N K2  FRACNN12N16ITEM PROVE THE FOLLOWING BY INDUCTION BOXED SUMK1N RK  FRACRN1  1R1 QQUAD R NEQ 1 INDEXGEOMETRIC SUMITEM PROVE BY INDUCTION THAT FRAC1SQRT4N1  FRAC12CDOT FRAC34 CDOT CDOTSCDOT FRAC2N32N2CDOT FRAC2N12N LEQ  FRAC1SQRT3N1FOR INTEGERS N GEQ 1KAZARINOFF 1961 P 5ITEM PROVE BY INDUCTION THAT FOR XYN IN ZBB WITH X NEQ Y   XY DIVIDES  INDEX  INDEXDIVIDESSEE  XNYN  THIS IS WRITTEN AS XY  XNYNEXSKIPSETEXSECTREFSECLFSR1ITEM PREPARE A TABLE SHOWING THE STORAGE CONTENTS AND OUTPUTS FOR THE  LFSR SHOWN IN THE ACCOMPANYING ILLUSTRATION WITH INITIAL CONDITIONS  SHOWN IN THE DELAY ELEMENTS  THE INITIAL CONDITION IS  0001 AS SHOWN  ALSO DETERMINE THE CONNECTION POLYNOMIAL  CDBEGINCENTERINPUTPICTUREDIRLFSR5LATEXINPUTPICTUREDIRLFSR5ENDCENTERITEM CONSIDER THE LFSR DESCRIBED BY THE THE POLYNOMIAL CD  1D D2  D3BEGINENUMERATEITEM DRAW THE LFSR BLOCK DIAGRAM USING BOTH THE REALIZATION SHOWN IN  FIGURE REFFIGLFSR1 AND THE REALIZATION SHOWN IN REFFIGLFSR12ITEM FOR THE INITIAL CONDITION 001 TRACE THE OPERATION OF  BOTH REALIZATIONS OF THE LFSR AND VERIFY THAT THE OUTPUT SEQUENCE OF  EACH IS THE SAME  HOW MANY DISTINCT STATES ARE THEREENDENUMERATEITEM CONSIDER THE LFSR DESCRIBED BY THE THE POLYNOMIAL 1D2  D3BEGINENUMERATEITEM DRAW THE LFSR BLOCK DIAGRAM USING BOTH THE REALIZATION SHOWN IN  FIGURE REFFIGLFSR1 AND THE REALIZATION SHOWN IN REFFIGLFSR12ITEM FOR THE INITIAL CONDITION 001 TRACE THE OPERATION OF  BOTH REALIZATIONS OF THE LFSR AND VERIFY THAT THE OUTPUT SEQUENCE OF  EACH IS THE SAME  HOW MANY DISTINCT STATES ARE THEREENDENUMERATEITEM GIVEN THE SEQUENCE 0001010  BEGINENUMERATE  ITEM DETERMINE THE SHORTESTLENGTH LFSR THAT COULD PRODUCE THIS    SEQUENCE PERFORMING THE COMPUTATIONS BY HAND  ITEM CHECK YOUR WORK USING ALGORITHM REFALGMASSEY IN SC      MATLAB  ENDENUMERATEITEM SHOW THAT FOR J01LDOTSN THE OUTPUT OF THE LFSR WITH  CONNECTION POLYNOMIAL CN1D AS IN REFEQBERLMASS1 WITH  A  DM1 DN AND LNM SATISFIES DJ  0 NO DISCREPANCYITEM WRITE THE OUTPUT SEQUENCE OF AN LFSR AS A POLYNOMIAL YD  Y0  Y1 D  Y2 D2  CDOTSBEGINENUMERATEITEM USING REFEQLFSR1 SHOW THAT THE JTH COEFFICIENT IN  YDCD VANISHES FOR JPP1 LDOTS WHERE DEGREECD   P  HENCE WE CAN WRITE CDYD  ZDWHERE ZD  Z0  Z1D   CDOTS  ZP1DP1THUS KNOWING ZD WE CAN FIND THE OUTPUT BY POLYNOMIAL LONG DIVISIONBEGINEQUATIONYD  FRACZDCDLABELEQLFSRDIVENDEQUATIONITEM SHOW THAT THE COEFFICIENTS OF ZD CAN BE RELATED TO THE  INITIAL CONDITIONS OF THE LFSR BY BEGINBMATRIX1 0CDOTS  0  C1  1  CDOTS  0 C2  C1  CDOTS  0 VDOTS CP1  CP2  CDOTS  C1  1 ENDBMATRIXBEGINBMATRIXY0  Y1  Y2  VDOTS  YP1 ENDBMATRIX BEGINBMATRIX Z0  Z1  Z2  VDOTS  ZP1ENDBMATRIXENDENUMERATEITEM LET CD  1D2  D3 WITH INITIAL CONTENTS Y0Y1Y2   100 DETERMINE THE FIRST SIX OUTPUTS USING POLYNOMIAL LONG  DIVISION AS IN REFEQLFSRDIV  COMPARE THE RESULTS TO THOSE  OBTAINED DIRECTLY FROM THE LFSR ITEM LET CD  1DD2 WITH INITIAL CONTENTS  Y0Y1     11  DETERMINE THE FIRST SIX OUTPUTS USING POLYNOMIAL LONG   DIVISION AS IN REFEQLFSRDIV  COMPARE THE RESULTS TO THOSE OBTAINED   DIRECTLY FROM THE LFSRITEM DETERMINE THE SEQUENCE YI OF LENGTH SEVEN GENERATED BY  CD  1DD3 AND CALL ITS LENGTH N  THEN COMPUTE THE CYCLIC  AUTOCORRELATION FUNCTION INDEXAUTOCORRELATION RHOK  FRAC1N SUMI0N1 YI YIKWHERE YIK MEANS THAT THE SUBSCRIPT IS COMPUTED MODULO NPLOT THIS AUTOCORRELATION FUNCTIONENDEXERCISESSECTIONREFERENCESTHE LINEAR SYSTEMS THEORY PRESENTED HERE IN BROAD STROKES IS PAINTEDIN CONSIDERABLY FINER DETAIL IN CITERUGH1996 AND CITEKAILATH80OUR BRIEF INTRODUCTION TO LINEAR PREDICTION IS MORE EXTENSIVELYPRESENTED IN CITEDELLER1993HAYKIN1996 WHILE CONSIDERABLY MORE ONSPECTRUM ANALYSIS APPEARS IN CITESTOICAKAY1988MARPLE  THEAPPLICATIONS OF ADAPTIVE FILTERING HIGHLIGHTED HERE ARE DISCUSSED INDEPTH IN CITEHAYKIN1996 AND CITEWIDROW1985  THE HIDDEN MARKOVMODEL IS PRESENTED IN CITERABINER1989DELLER1993 ANDCITERABINERJUANG1993  FOR AN ENJOYABLE AND READABLE INTRODUCTIONTO PROOFS WITH A VARIETY OF SUGGESTIONS AND EXAMPLES AND SOME GOODMATHEMATICAL BACKGROUND CITEVELLEMAN IS RECOMMENDED  ATHOUGHTPROVOKING BOOK ON MATHEMATICAL THINKING IS CITEPOLYA1971MASSEYS ALGORITHM IS PRESENTED IN CITEMASSEY2  AN EXCELLENTPRESENTATION OF THE ALGORITHM IS IN CITEBLAHUT1983  THE BOOKCITEGOLOMB PROVIDES AN INTRODUCTION TO LFSRS AND THE PAPERCITESARWATE1980 AN INTERESTING DISCUSSION OF DECIMATEDINDEXDECIMATION MAXIMALLENGTH SEQUENCES INDEXMAXIMALLENGTH  SEQUENCE APPLICATIONS OF LFSRS TO SPREADSPECTRUM COMMUNICATIONSARE DISCUSSED IN CITEZIEMERPETERSONMEDSKIPIN ADDITION TO THE PRESENT BOOK THERE ARE A NUMBER OF OTHER BOOKSTHAT SHOULD BE CONSIDERED AS PART OF A STANDARD LIBRARY FOR SIGNALPROCESSORS  WE MENTION THE FOLLOWING AS USEFUL REFERENCESBEGINDESCRIPTIONITEMLINEAR ALGEBRA THE BOOKCITESTRANG IS AN EXCELLENT INTRODUCTION TO LINEAR ALGEBRA  THE BOOKCITEGVL PROVIDES EXTENSIVE DETAIL ON ALGORITHMS ASSOCIATED WITH LINEARALGEBRA  IT SHOULD BE A PART OF EVERY SIGNAL PROCESSORS LIBRARYCITEHORNJOHNSON IS A GOOD REFERENCE ON THE THEORY OF MATRICESITEMSTATISTICS A GENERAL BACKGROUND IN STATISTICS IS  CITEHOGGCRAIG1978  AN EXCELLENT RECENT SOURCE ON STATISTICAL  DECISION MAKING IS CITESCHARFL1991  ANOTHER IS  CITEPOOR1988BOOK  A COMPREHENSIVE WORK IS NOCITEVANREES68ITEMALGEBRA INTRODUCTORY BOOKS ON ALGEBRA ARE NOCITEFRALEIGH AND  NOCITEBIRKHOFFMACLANEITEMCALCULUS AND ANALYSIS A STANDARD REFERENCE IS CITEROYDEN  A  MORE INTRODUCTORY LEVEL IS NOCITEBUCK FUNCTIONAL ANALYSIS  TARGETED TOWARD ENGINEERS IS CITENAYLORSELLITEMNUMBER THEORY A GOOD STARTING POINT IS  CITENIVENZUCKERMAN  AN ENTERTAINING LOOK AT A VARIETY OF  APPLICATIONS IS CITESCHROEDER  A BOOK WITH A LITTLE MORE DEPTH  IS CITEHUAITEMOPTIMIZATION SOME EXCELLENT SOURCES ARECITEFLETCHER1980 CITELUENBERGER CITELUENBERGER1984ITEMNUMERICAL ANALYSIS THE CLASSIC WORK CITERALSTON IS STILL  EXCELLENT  A MORE RECENT WORK IS CITECHENEYENDDESCRIPTION LOCAL VARIABLES TEXMASTER TEST END COMPLETING THE SQUARECHAPTERCOMPLETING THE SQUARELABELAPPDXCTSINDEXCOMPLETING THE SQUARECOMPLETING THE SQUARE IS A SIMPLE ALGEBRAIC TECHNIQUE THAT ARISESFREQUENTLY ENOUGH IN BOTH SCALAR AND VECTOR PROBLEMS THAT IT IS WORTHILLUSTRATINGSECTIONTHE SCALAR CASELABELSECB1THE QUADRATIC EXPRESSIONBEGINEQUATIONJX   AX2  BX  CLABELEQCTS0ENDEQUATIONCAN BE WRITTEN AS AX2  FRACBAX  CIN COMPLETING THE SQUARE WE WRITE THIS AS A PERFECT SQUARE WITH ACONSTANT OFFSET  TAKING THE COEFFICIENT OF X AND DIVIDING BY 2 ITIS STRAIGHTFORWARD TO VERIFY THATBEGINEQUATIONBOXEDAX2FRACBAX  C  AXFRACB2A2  FRACB24A   CLABELEQCTS1ENDEQUATIONBY MEANS OF COMPLETING THE SQUARE WE CAN OBTAIN BOTH THE MINIMIZINGVALUE OF X AND THE MINIMUM VALUE OF JX IN REFEQCTS0EXAMINATION OF REFEQCTS1 REVEALS THAT THE MINIMUM MUST OCCURWHEN X  FRACB2AA RESULT ALSO READILY OBTAINED VIA CALCULUS  IN THIS CASE WE ALSOGET THE MINIMUM VALUE AS WELL SINCE IF X  FRACB2A THEN JMIN  C  FRACB24ABEGINEXAMPLE  WE DEMONSTRATE AN EXAMPLE OF THE THE USE OF COMPLETING THE SQUARE  FOR A PROBLEM OF ESTIMATING A GAUSSIAN RANDOM VARIABLE OBSERVED IN  GAUSSIAN NOISE  SUPPOSE X SIM NCMUXSIGMAX2 AND N SIM  NC0SIGMAN2 WHERE X IS REGARDED AS A SIGNAL AND N IS  REGARDED AS NOISE  WE MAKE AN OBSERVATION OF THE SIGNAL IN THE  NOISE Y  XNGIVEN A MEASUREMENT OF YY WE DESIRE TO FIND FXY  BY BAYESTHEOREM FXY  FRACFYXFXINTINFTYINFTY FYXFXDXTHE DENSITY FYX CAN BE OBTAINED BY OBSERVING THAT FOR A GIVENVALUE OF XX Y  XNIS SIMPLY A SHIFT OF THE RANDOM VARIABLE N AND HENCE IS GAUSSIANWITH VARIANCE SIGMAN2 AND MEAN X  THAT IS FYX  FNYXWHERE FN IS THE DENSITY OF N  WE CAN THEREFORE WRITE FXY  FRACFNXYFXINTINFTYINFTY FYXFXDXTHE CONSTANT IN THE DENOMINATOR IS SIMPLY A NORMALIZING VALUE TO MAKETHE DENSITY FXY INTEGRATE TO 1 WE WILL CALL IT C AND PAYLITTLE ATTENTION TO IT  WE CAN WRITE FXY  FRAC1C FRAC1SIGMAN SQRT2PIEYX22SIGMAN2 FRAC1SIGMAX SQRT2PIEXMUX22SIGMAX2LET US FOCUS OUR ATTENTION ON THE EXPONENT WHICH WE DENOTE BY E E  FRAC12SIGMAN2YX2 FRAC12SIGMAX2XMUX2THIS CAN BE WRITTEN AS E  X2LEFTFRAC12SIGMAN2  FRAC1SSIGMAX2RIGHT X LEFTFRACYSIGMAN2  FRACMUXSIGMAX2RIGHT C1WHERE C1 DOES NOT DEPEND UPON X  BY COMPLETING THE SQUARE WEHAVE E  FRACSIGMAX2  SIGMAN22SIGMAN2SIGMAX2LEFT X   FRACYSIGMAX2  MUX SIGMAN2SIGMAX2     SIGMAN2RIGHT  C2WHERE C2 DOES NOT DEPEND UPON X  THE DENSITY CAN THUS BE WRITTEN FXY  LEFTFRAC12PI CSIGMANSIGMAXEC2RIGHTEXPLEFTFRAC12LEFTX  FRACYSIGMAX2  MUX      SIGMAN2SIGMAX2      SIGMAN2RIGHT1LEFTSIGMAN2SIGMAX2SIGMAX2SIGMAN2RIGHTRIGHTTHIS HAS THE FORM OF A GAUSSIAN DENSITY SO THE CONSTANTS IN FRONT OFTHE EXPONENTIAL MUST BE SUCH THAT THIS INTEGRATES TO 1  THE MEAN OFTHIS GAUSSIAN DENSITY IS MUXY  FRACSIGMAX2SIGMAX2SIGMAN2 Y FRACSIGMAN2SIGMAX2  SIGMAN2MUXAND THE VARIANCE IS SIGMAXY2  FRACSIGMAN2SIGMAX2SIGMAX2SIGMAN2LET IS CONSIDER AN INTERPRETATION OF THIS RESULT  IF SIGMAN2 GGSIGMAX2 THEN AN OBSERVATION OF  Y DOES NOT TELL US MUCH ABOUTX BECAUSE THE INTERFERING NOISE N IS TOO STRONG  THE INFORMATIONWE HAVE ABOUT X GIVEN Y IS THUS ABOUT THE SAME AS THE INFORMATIONWE HAVE ABOUT X ALONE  THIS OBSERVATION IS VALIDATED IN THEANALYSIS IF SIGMAN2 GG SIGMAX2 THEN MUXY APPROX MUXAND  SIGMAXY2 APPROX SIGMAX2ON THE OTHER HAND IF THE NOISE VARIANCE IS SMALL SO THAT SIGMAN2LL SIGMAX2 THEN AN OBSERVATION OF Y IS ALMOST THE SAME AS ANOBSERVATION ON X ITSELF IN THIS CASE WE HAVE MUXY APPROX Y QQUAD TEXTANDQQUADSIGMAXY2 APPROX SIGMAN2ENDEXAMPLESECTIONTHE MATRIX CASELABELSECB2THE EQUATION  XBFT A XBF  XBFTYBF  CWHERE A IS SYMMETRIC AND INVERTIBLE CAN BE WRITTEN ASBEGINEQUATION XBFZBFT A XBFZBF  DLABELEQCTS2ENDEQUATIONWHERE ZBF  FRAC12 A1 YBFAND D  C  FRAC12YBFT ZBFSETEXSECTREFSECB1BEGINEXERCISESITEM THE CHARACTERISTIC FUNCTION INDEXCHARACTERISTIC FUNCTION OF A  INDEXCHARACTERISTIC FUNCTIONGAUSSIAN OF A RANDOM VARIABLE X IS  THE CONJUGATE OF THE FOURIER TRANSFORM OF ITS DENSITY PHIXOMEGA  INT FXX EJOMEGA XDXBEGINENUMERATEITEM SHOW THAT FOR A GAUSSIAN DENSITY WITH FXF  FRAC1SQRT2PI SIGMA  EXMU22SIGMA2HAS THE CHARACTERISTIC FUNCTION PHIXOMEGA  EXPLEFTJMU OMEGA  FRAC12OMEGA2  SIGMA2RIGHTITEM TO FOLLOW UP THE CHARACTERISTIC FUNCTION IDEA SHOW THAT THE  NTH MOMENT INDEXMOMENTS OF A RV OF X CAN BE OBTAINED FROM  ITS CHARACTERISTIC FUNCTION BY EXN  LEFTFRAC1JN FRACDN PHIXOMEGADOMEGAN  RIGHTOMEGA0INDEXCHARACTERISTIC FUNCTIONMOMENTS USINGENDENUMERATEITEM SHOW THAT THE CONDITIONAL DENSITY IN REFEQFXY IS  CORRECTEXSKIPSETEXSECTREFSECB2ITEM USING REFEQCTS2 DETERMINE THE FXBFYBF  WHEN Y  XNAND X AND N ARE GAUSSIANDISTRIBUTED RANDOM VECTORS WITH X SIM NCMUBFRX QQUADTEXTANDQQUAD   N SIM NCZEROBFRNENDEXERCISES LOCAL VARIABLES TEXMASTER TEST ENDSECTIONAN APPLICATION LFSRS AND MASSEYS ALGORITHMLABELSECLFSR1IN THIS SECTION WE INTRODUCE THE LINEAR FEEDBACK SHIFT REGISTER LFSRINDEXLINEAR FEEDBACK SHIFT REGISTER LFSR WHICHIS NOTHING MORE THAN A DETERMINISTIC AUTOREGRESSIVE SYSTEM  THECONCEPTS PRESENTED HERE WILL ILLUSTRATE SOME OF THE LINEAR SYSTEMSTHEORY PRESENTED IN THIS CHAPTER PROVIDE A DEMONSTRATION OF SOMEMETHODS OF PROOF AND INTRODUCE OUR FIRST ALGORITHMINPUTINTRODIRALGTEXTBOXINPUTINTRODIRGF2BOXAN LFSR IS SIMPLY AN AUTOREGRESSIVE FILTER OVER A FIELD FINDEXFIELD SEE BOX REFBOXALGEBRA THAT HAS NO INPUTSIGNAL  AN LFSR IS SHOWN IN FIGURE REFFIGLFSR1  AN ALTERNATIVEREALIZATION PREFERRED IN HIGHSPEED IMPLEMENTATIONS BECAUSE THEADDITION OPERATIONS ARE NOT CASCADED IS SHOWN IN FIGURE REFFIGLFSR12THE INTERNAL STATE SEQUENCE OF THIS ALTERNATIVE REALIZATION IS NOTNECESSARILY BUT WITH APPROPRIATE INITIAL CONDITIONS THE OUTPUTSEQUENCE IS THE SAMEIF THE CONTENTS ARE BINARY IT IS HELPFUL TO VIEW THE STORAGE ELEMENTSAS D FLIPFLOPS SO THAT THE MEMORY OF THE LFSR IS SIMPLY A SHIFTREGISTER AND THE LFSR IS A DIGITAL STATE MACHINE  FOR A BINARY LFSRTHE CONNECTIONS ARE EITHER 1 OR 0 CONNECTION OR NO CONNECTION ANDALL OPERATIONS ARE CARRIED OUT IN GF2 INDEXFIELDFINITE FIELDTHAT IS MODULO 2 SEE BOX REFBOXGF2  MASSEYS ALGORITHM APPLIESOVER ANY FIELD BUT MOST COMMONLY IT IS USED IN CONNECTION WITH THEBINARY FIELDTHE OUTPUT OF THE LFSR ISBEGINEQUATION YJ  SUMI1P CI YJIQQUAD  JPP1P2LDOTSLABELEQLFSR1ENDEQUATIONBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRLFSR1    CAPTIONLFSR REALIZATION    LABELFIGLFSR1  ENDCENTERENDFIGUREBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRLFSR2    CAPTIONALTERNATIVE LFSR REALIZATION    LABELFIGLFSR12  ENDCENTERENDFIGURETHE NUMBER OF FEEDBACK COEFFICIENTS P IS CALLED THE EM LENGTH OFTHE LFSRBEGINEXAMPLE LABELEXMMASS1  THE LFSR OVER GF2 SHOWN IN FIGURE REFFIGLFSR2A SATISFIES YJ  YJ1  YJ3WITH INITIAL REGISTER CONTENTS 001 THE LFSR OUTPUT SEQUENCE ISSHOWN IN FIGURE REFFIGLFSR2B WHERE THE NOTATION DZ1 ISEMPLOYED  THE ALTERNATIVE REALIZATION IS SHOWN IN FIGUREREFFIGLFSR2C WITH ITS CORRESPONDING OPERATION SHOWN IN FIGUREREFFIGLFSR2DBEGINCENTERENDCENTERAFTER J6 THE SEQUENCE REPEATS SO THAT SEVEN DISTINCT STATES OCCURIN THIS DIGITAL STATE MACHINE  NOTE THAT FOR THIS LFSR THE REGISTERCONTENTS ASSUME ALL POSSIBLE NONZERO SEQUENCES OF THREE DIGITSENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    SUBFIGUREBLOCK DIAGRAMINPUTPICTUREDIRLFSR4A INPUTPICTUREDIRLFSR4ALATEXQQUADSUBFIGUREOUTPUT SEQUENCEBEGINTABULARCCC HLINEJ  STATE  YJ OUTPUT  HLINE0  001 1 1  100 1 2  110 1 3  111 0 4  011 1 5  101 0 6  010 0 HLINE7  001 1 VDOTS  VDOTS  VDOTS HLINEENDTABULARSUBFIGUREALTERNATE BLOCK DIAGRAMINPUTPICTUREDIRLFSR4 QQUAD INPUTPICTUREDIRLFSR4LATEXSUBFIGUREOUTPUT SEQUENCE FOR ALTERNATIVEREALIZATIONBEGINTABULARCCC HLINE J  STATE  YJ OUTPUT  HLINE0  001 1 1  101 1 2  111 1 3  110 0 4  011 1 5  100 0 6  010 0 HLINE7  001 1 VDOTS  VDOTS  VDOTS HLINEENDTABULARCAPTIONA BINARY LFSR AND ITS OUTPUT    LABELFIGLFSR2  ENDCENTERENDFIGURETAKING THE ZTRANSFORM OF REFEQLFSR1 WE OBTAINBEGINEQUATION YZ1C1 Z1  C2 Z2  CDOTS  CP ZP  0LABELEQLFSR10ENDEQUATIONIT WILL BE CONVENIENT TO REPRESENT THE LFSR USING THE POLYNOMIAL INREFEQLFSR10 IN THE FORM CD  1 C1 D  C2 D2  CDOTS  CP DPWHERE D  Z1 IS A DELAY OPERATOR  WE NOTE THAT THE OUTPUTSEQUENCE PRODUCED BY THE LFSR DEPENDS UPON BOTH THE FEEDBACKCOEFFICIENTS AND THE INITIAL CONTENTS OF THE STORAGE REGISTERSSUBSECTIONISSUES AND APPLICATIONS OF LFSRSWITH A CORRECTLY DESIGNED FEEDBACK POLYNOMIAL CD THE OUTPUTSEQUENCE OF A BINARY LFSR IS A MAXIMALLENGTH SEQUENCE PRODUCING2P1 OUTPUTS BEFORE THE SEQUENCE REPEATS INDEXMAXIMALLENGTH  SEQUENCE THIS SEQUENCE ALTHOUGH NOT TRULY RANDOM EXHIBITS MANY OFTHE CHARACTERISTICS OF NOISE SUCH AS PRODUCING RUNS OF ZEROS AND ONESOF DIFFERENT LENGTHS HAVING A CORRELATION FUNCTION THAT APPROXIMATESA DELTA FUNCTION AND SO FORTH  THE SEQUENCE PRODUCED IS SOMETIMESCALLED A PSEUDONOISE SEQUENCE INDEXPSEUDONOISE SEQUENCEPSEUDONOISE SEQUENCES ARE EMPLOYED IN A VARIETY OF APPLICATIONSINCLUDING SPREADSPECTRUM COMMUNICATIONS ERROR DETECTION ANDRANGING  THE GLOBAL POSITIONING SYSTEM BASED ON AN ARRAY OFSATELLITES IN GEOSYNCHRONOUS ORBIT EMPLOYS PSEUDONOISE SEQUENCES TOCARRY TIMING INFORMATION USED FOR NAVIGATIONAL PURPOSESIN SOME OF THESE APPLICATIONS THE FOLLOWING PROBLEM ARISES GIVEN ASEQUENCE Y0ALLOWBREAK Y1ALLOWBREAK LDOTSALLOWBREAKYN1 DEEMED TO BE THE OUTPUT OF AN LFSR DETERMINE THE FEEDBACKCONNECTION POLYNOMIAL CD AND THE INITIAL REGISTER CONTENTS OF THESHORTEST LFSR THAT COULD PRODUCE THE SEQUENCE  SOLVING THIS PROBLEMIS THE FOCUS OF THE REMAINDER OF THIS SECTION  THE ALGORITHM WEDEVELOP IS KNOWN AS MASSEYS INDEXMASSEYS ALGORITHM ALGORITHMNOT ONLY DOES IT SOLVE THE PARTICULAR PROBLEM STATED HERE BUT AS WESHALL SEE IT PROVIDES AN EFFICIENT ALGORITHM FOR SOLVING A PARTICULARSET OF TOEPLITZ EQUATIONSAN LFSR THAT PRODUCES THE SEQUENCE Y0Y1 LDOTS YN1COULD CLEARLY BE OBTAINED FROM AN LFSR OF LENGTH N EACH STORAGEELEMENT CONTAINING ONE OF THE VALUES  HOWEVER THIS MAY NOT BE THESHORTEST POSSIBLE LFSR  ANOTHER APPROACH TO THE SYSTEM SYNTHESIS ISTO SET UP A SYSTEM OF EQUATIONS OF THE FOLLOWING FORM ASSUMING FORTHIS EXAMPLE THAT THE LENGTH OF THE LFSR IS P3 BEGINBMATRIX Y2  Y1  Y0 Y3 Y2  Y1 Y4Y3Y2ENDBMATRIXBEGINBMATRIXC1  C2  C3ENDBMATRIX  BEGINBMATRIXY3  Y4  Y5 ENDBMATRIXTHESE EQUATIONS ARE IN THE SAME FORM AS THE YULEWALKERINDEXYULEWALKER EQUATIONS EQUATIONS IN REFEQYW3 INPARTICULAR THE MATRIX ON THE LEFT IS A TOEPLITZ MATRIXINDEXTOEPLITZ MATRIX WHEREAS THE YULEWALKER EQUATIONS WEREORIGINALLY DEVELOPED IN THIS BOOK IN THE CONTEXT OF A STOCHASTICSIGNAL MODEL WE OBSERVE THAT THERE IS A DIRECT PARALLEL WITHDETERMINISTIC AUTOREGRESSIVE SIGNAL MODELSKNOWING THE VALUE OF P THE YULEWALKER EQUATIONS COULD BE SOLVED BYANY MEANS AVAILABLE TO SOLVE P EQUATIONS IN P UNKNOWNS  HOWEVERDIRECTLY SOLVING THIS SET OF EQUATIONS IS INEFFICIENT IN AT LEAST TWOWAYSBEGINENUMERATEITEM A GENERAL SOLUTION OF A MATSIZEPP SET OF EQUATIONS  REQUIRES OP3 OPERATIONS  WE ARE INTERESTED IN DEVELOPING AN  ALGORITHM THAT REQUIRES FEWER OPERATIONS  THE ALGORITHM WE DEVELOP  REQUIRES OP2 OPERATIONSITEM FREQUENTLY THE ORDER P IS NOT KNOWN IN ADVANCE  THE VALUE OF  P COULD BE DETERMINED BY STARTING WITH A SMALL VALUE OF P AND  INCREASING THE SIZE OF THE MATRIX UNTIL AN LFSR IS OBTAINED THAT  PRODUCES THE ENTIRE SEQUENCE  THIS COULD BE DONE WITHOUT TAKING  INTO ACCOUNT THE RESULT FOR SMALLER VALUES OF P  MORE DESIRABLE  WOULD BE AN ALGORITHM THAT BUILDS RECURSIVELY ON PREVIOUSLYOBTAINED  SOLUTIONS TO OBTAIN A NEW SOLUTION  THIS IS IN FACT HOW WE  PROCEEDENDENUMERATESINCE WE BUILD UP THE LFSR USING INFORMATION FROM PRIOR COMPUTATIONSWE NEED A NOTATION TO REPRESENT THE FEEDBACK CONNECTION POLYNOMIALUSED AT DIFFERENT STAGES OF THE ALGORITHM  LET CND  1 C1ND  CDOTS  CLNNDLNDENOTE THE FEEDBACK CONNECTION POLYNOMIAL FOR THE LFSR CAPABLE OFPRODUCING THE OUTPUT SEQUENCE  Y0 Y1 LDOTS YN1 WHERELN IS THE DEGREE OF THE FEEDBACK CONNECTION POLYNOMIALTHE ALGORITHM WE OBTAIN PROVIDES AN EFFICIENT WAY OF SOLVING THEYULEWALKER EQUATIONS WHEN P IS NOT KNOWN  IN CHAPTERREFCHAPSPECIALMAT WE ENCOUNTER AN ALGORITHM FOR SOLVING TOEPLITZMATRIX EQUATIONS WITH FIXED P THE LEVINSONDURBIN ALGORITHM  ATHIRD GENERAL APPROACH BASED UPON THE EUCLIDEAN ALGORITHM IS ALSOKNOWN SEE EG CITEBLAHUT1992  EACH OF THESE ALGORITHMS HASOP2 COMPLEXITY BUT THEY HAVE TENDED TO BE USED IN DIFFERENTAPPLICATION AREAS THE LEVINSONDURBIN ALGORITHM BEING MOST COMMONLYUSED WITH LINEAR PREDICTION AND SPEECH PROCESSING AND THE MASSEY OREUCLIDEAN ALGORITHM BEING USED IN FINITEFIELD APPLICATIONS SUCH ASERRORCORRECTION CODINGSUBSECTIONMASSEYS ALGORITHMWE BUILD THE LFSR THAT PRODUCES THE ENTIRE SEQUENCE BY SUCCESSIVELYMODIFYING AN EXISTING LFSR IF NECESSARY TO PRODUCE INCREASINGLYLONGER SEQUENCES  WE START WITH AN LFSR THAT COULD PRODUCE Y0  WEDETERMINE IF THAT LFSR COULD ALSO PRODUCE THE SEQUENCE Y0Y1IF IT CAN THEN NO MODIFICATIONS ARE NECESSARY  IF THE SEQUENCECANNOT BE PRODUCED USING THE CURRENT LFSR CONFIGURATION WE DETERMINEA NEW LFSR THAT CAN PRODUCE THE ENTIRE SEQUENCE  WE PROCEED THIS WAYINDUCTIVELY EVENTUALLY CONSTRUCTING AN LFSR CONFIGURATION THAT CANPRODUCE THE ENTIRE SEQUENCE  Y0 Y1 LDOTS YN1  BY THISPROCESS WE OBTAIN A SEQUENCE OF POLYNOMIALS AND THEIR DEGREES BEGINMATRIXC1DL1 C2DL2 VDOTS CNDLN ENDMATRIXWHERE THE LAST LFSR PRODUCES Y0LDOTSYN1AT SOME INTERMEDIATE STEP SUPPOSE WE HAVE AN LFSR CND OFDEGREE LN THAT PRODUCES Y0ALLOWBREAK Y1ALLOWBREAKLDOTSALLOWBREAK YN1 FOR SOME N  N  WE CHECK IF THIS LFSRWILL ALSO PRODUCE YN BY COMPUTING THE OUTPUT YHATN  SUMI1LN CNI YNIIF YHATN IS EQUAL TO YN THEN THERE IS NO NEED TO UPDATE THELFSR AND CN1D  CND  OTHERWISE THERE IS SOMENONZERO EM DISCREPANCY DN  YN YHATN  YN  SUMI1LN CNI YNI SUMI0LN CNI YNIIN THIS CASE WE WILL UPDATE OUR LFSR USING THE FORMULABEGINEQUATIONCN1D  CND  A DL CMDLABELEQBERLMASS1ENDEQUATIONWHERE A IS SOME ELEMENT IN THE FIELD L IS AN INTEGER ANDCMD IS ONE OF THE PRIOR LFSRS PRODUCED BY OUR PROCESS THATALSO HAD A NONZERO DISCREPANCY DM  USING THIS NEW LFSR WECOMPUTE THE NEW DISCREPANCY DENOTED BY DN ASBEGINALIGNDN SUMI0LN1 CN1I YNI  NONUMBER  SUMI0LN CNI YNI  A SUMI0LMCMI YNILLABELEQBM2ENDALIGNNOW LET LNM  THEN THE SECOND SUMMATION GIVES A SUMI0LM CMI YMI  A DMTHUS IF WE CHOOSE A  DM1 DN THEN THE SUMMATION INREFEQBM2 GIVES DN1  DN1  DM1DN DM  0SO THE NEW LFSR PRODUCES THE SEQUENCE Y0Y1LDOTSYNSUBSECTIONCHARACTERIZATION OF LFSR LENGTH IN MASSEYS ALGORITHMTHE UPDATE IN REFEQBERLMASS1 IS IN FACT THE HEART OF MASSEYSALGORITHM  FROM AN OPERATIONAL POINT OF VIEW NO FURTHER ANALYSIS ISNECESSARY  HOWEVER THE PROBLEM WAS TO FIND THE SHORTEST LFSRPRODUCING A GIVEN SEQUENCE  WE HAVE PRODUCED A MEANS OF FINDING ANLFSR BUT HAVE NO INDICATION YET THAT IT IS THE SHORTESTESTABLISHING THIS WILL REQUIRE SOME ADDITIONAL EFFORT IN THE FORM OFTWO THEOREMS  THE PROOFS ARE CHALLENGING BUT IT IS WORTH THE EFFORTTO THINK THEM THROUGHIN GENERAL CONSIDERABLE SIGNAL PROCESSING RESEARCH FOLLOWS THISGENERAL PATTERN AN ALGORITHM MAY BE ESTABLISHED THAT CAN BE SHOWN TOWORK EMPIRICALLY FOR SOME PROBLEM BUT CHARACTERIZING ITS PERFORMANCELIMITS OFTEN REQUIRES SIGNIFICANT ADDITIONAL EFFORTBEGINTHEOREM  SUPPOSE THAT AN LFSR OF LENGTH LN PRODUCES THE SEQUENCE Y0  Y1 ALLOWBREAK LDOTS ALLOWBREAK YN1 BUT NOT THE  SEQUENCE Y0Y1ALLOWBREAKLDOTS ALLOWBREAK YN  THEN ANY  LFSR THAT PRODUCES THE LATTER SEQUENCE MUST HAVE A LENGTH LN1  SATISFYING LN1 GEQ N1  LNENDTHEOREMBEGINPROOF  THE THEOREM IS ONLY OF PRACTICAL INTEREST IF LN  N  OTHERWISE IT IS TRIVIAL TO PRODUCE THE SEQUENCE  LET US TAKE  THEN LN  N  LET CND  1C1N D  CDOTS  CLNNDLNREPRESENT THE CONNECTIONS FOR THE LFSR WHICH PRODUCES Y0Y1LDOTS YN1 AND LET CN1D  1C1N1 D  CDOTS CLN1N1DLN1DENOTE THE CONNECTIONS FOR THE LFSR WHICH PRODUCES Y0Y1 LDOTSYN  NOW WE DO A PROOF BY CONTRADICTION INDEXPROOFBY  CONTRADICTION ASSUME CONTRARY TO THE THEOREM THATBEGINEQUATIONLN1 LEQ N  LNLABELEQLFSRCONTENDEQUATIONFROM THE DEFINITIONS OF THE CONNECTION POLYNOMIALS WE OBSERVE THATBEGINEQUATION SUMI1LN CIN YJI QUAD BEGINCASES  YJ  JLNLN1 LDOTS N1 NEQ YN  JNENDCASESLABELEQLFSR3ENDEQUATIONANDBEGINEQUATION  LABELEQLFSR4  SUMI1LN1 CIN1 YJI  YJ QQUAD JLN1  LN11 LDOTS NENDEQUATIONFROM REFEQLFSR4 WE HAVE YN  SUMI1LN1 CIN1 YNITHE INDICES IN THIS SUMMATION RANGE FROM N1 TO NLN1 WHICHBECAUSE OF THE CONTRARY ASSUMPTION MADE IN REFEQLFSRCONT IS ASUBSET OF THE RANGE LN LN1 LDOTS N1  THUS THE EQUALITY INREFEQLFSR3 APPLIES AND WE CAN WRITE YN  SUMI1LN1 CIN1 YNI SUMI1LN1 CN1I SUMK1LN CNK YNIKINTERCHANGING THE ORDER OF SUMMATION WE HAVEBEGINEQUATIONYN  SUMK1LN CNK SUMI1LN1 CN1IYNIK LABELEQLFSR5ENDEQUATIONSETTING JN IN REFEQLFSR3 WE OBTAIN YN NEQ SUMK1LN CKNYNKIN THIS SUMMATION THE INDICES RANGE FROM N1 TO NLN WHICHBECAUSE OF REFEQLFSRCONT IS A SUBSET OF THE RANGELN1LN11LDOTSN OF REFEQLFSR4  THUS WE CAN WRITEBEGINEQUATIONYN NEQ SUMK1LN CNK SUMI1LN1CIN1 YNKILABELEQLFSR6ENDEQUATIONCOMPARING REFEQLFSR5 WITH REFEQLFSR6 WE OBSERVE ACONTRADICTION  HENCE THE ASSUMPTION ON THE LENGTH OF THE LFSRS MUSTHAVE BEEN INCORRECTBY THIS CONTRADICTION WE MUST HAVE LN1 GEQ N  1  LNENDPROOFSINCE THE SHORTEST LFSR THAT PRODUCES THE SEQUENCE Y0Y1LDOTSYNMUST ALSO PRODUCE THE FIRST PART OF THAT SEQUENCE WE MUST HAVELN1 GEQ LN  COMBINING THIS WITH THE RESULT OF THE THEOREMWE OBTAINBEGINEQUATION LN1 GEQ MAXLN N1LNLABELEQLNP1ENDEQUATIONWE CONCLUDE THAT THE SHIFT REGISTER CANNOT BECOME SHORTER AS MOREOUTPUTS ARE PRODUCEDWE HAVE SEEN HOW TO UPDATE THE LFSR TO PRODUCE A LONGER SEQUENCE USINGREFEQBERLMASS1 AND ALSO HAVE SEEN THAT THERE IS A LOWER BOUNDON THE LENGTH OF THE LFSR  WE NOW SHOW THAT THIS LOWER BOUND CAN BEACHIEVED WITH EQUALITY THUS PROVIDING THE EM SHORTEST LFSR WHICHPRODUCES THE DESIRED SEQUENCEBEGINTHEOREM  LET LI CIDI02LDOTSN BE A  SEQUENCE OF MINIMUMLENGTH HBOXLFSRS THAT PRODUCE THE SEQUENCE  Y0Y1LDOTS YI1  IF CN1D NEQ CND  THEN A NEW LFSR CAN BE FOUND THAT SATISFIES  LET THE CONNECTION POLYNOMIAL CND PRODUCE THE SEQUENCE  Y0 Y1 LDOTS YN1  ALSO LET CM M  N  DENOTE THE EM LAST CONNECTION POLYNOMIAL BEFORE CND  WHICH CAN PRODUCE THE SEQUENCE  Y0 Y1 LDOTS YM1 BUT  NOT THE SEQUENCE  Y0Y1 LDOTS YM  LET LM AND LN  BE THE LENGTHS OF THE LFSRS DESCRIBED BY CND AND  CMD RESPECTIVELY   LET DN  DENOTE THE EM DISCREPANCY BETWEEN THE YN AND THE NTH OUTPUT  OF THE LFSR WITH CND DN  YN  SUMI1LN CIN YNIIF THE DISCREPANCY DN0 THEN CN1D  CNDOTHERWISE IF DN NEQ 0 THAT IS IF CND FAILSTO PRODUCE THE SEQUENCE  Y0Y1 LDOTS YN A NEW POLYNOMIALCN1D WILL PRODUCE THE SEQUENCE WHEREBEGINEQUATION CN1D  CND  DN DM1 DNMCMDLABELEQCUPDATEENDEQUATIONFURTHERMORE THE DEGREE OF THE NEW POLYNOMIAL SATISFIESREFEQLNP1 WITH EQUALITYLN1  MAXLNN1  LNENDTHEOREMBEGINPROOF  WE WILL DO A PROOF BY INDUCTION INDEXPROOFBY INDUCTION TAKING  AS THE INDUCTIVE HYPOTHESIS THATBEGINEQUATIONLK1  MAXLK K1LKLABELEQLUPDATEENDEQUATIONFOR K01LDOTSN  THIS CLEARLY HOLDS WHEN K0 SINCE L00  TO GET STARTED LET M1C1D  1 D1  1 AND L1  0 AN LFSR OF LENGTH0  WE ALSO ASSUME AS AN INITIAL CONDITION THAT Y1  1THE OUTPUT OF THE INITIAL LFSR IS SUMI1L1 C1I SI  0THE SUM IS EMPTY  IF Y0 IS 0 THE LFSR CORRECTLY PRODUCES THEOUTPUT AND WE SET C0D  C1D  1OTHERWISE THERE IS A EM DISCREPANCY D0 BETWEEN THEOUTPUT OF THE INITIAL LFSR AND Y0 D0  Y0 SUMI1L1 CI1 YI  Y0THEN BY REFEQCUPDATEBEGINALIGNEDC0D  C1D  D0 D  1 Y0 DENDALIGNEDTHE OUTPUT OF THIS LFSR IS SUMI11  Y01  Y0SO C0D CORRECTLY PRODUCES THE OUTPUTNOW WE TAKE AS THE INDUCTIVE HYPOTHESIS THAT THERE IS A LFSRCND SATISFYINGBEGINEQUATIONYJ  SUMI1LN CIN YJI  BEGINCASES  0  JLN LN1 LDOTS N1 DN  JNENDCASESLABELEQLFSR5ENDEQUATIONWHERE DN NEQ 0 IS THE DISCREPANCY  LET CM M  N DENOTE THE EM LAST CONNECTION POLYNOMIALBEFORE CND THAT CAN PRODUCE THE SEQUENCE  Y0 Y1LDOTS YM1 BUT NOT THE SEQUENCE  Y0Y1 LDOTS YMSUCH THAT LM  LN LM  LN THEN LM1  LNHENCE IN LIGHT OF REFEQLUPDATEBEGINEQUATION LM1  LN  M1LMLABELEQLUPDATE2ENDEQUATIONIF CN1D IS UPDATED FROM CND ACCORDING TOREFEQBERLMASS1 WITH LNM WE HAVE ALREADY OBSERVED THAT ITIS CAPABLE OF PRODUCING THE SEQUENCE Y0Y1LDOTSYN  BY THEUPDATE FORMULA REFEQBERLMASS1 WE NOTE THAT LN1  MAXLNNM  LMUSING REFEQLUPDATE2 WE FIND THATLN1  MAXLNN1 LNTHE LFSR BEFORE THE LASTLENGTH CHANGE ALSO SATISFIES BY THE INDUCTIVE HYPOTHESISBEGINEQUATIONYJ  SUMI1LM CIM YJI  BEGINCASES  0  JLM LM1 LDOTS M1 DM  JMENDCASESLABELEQLFSR6ENDEQUATIONTHE OUTPUT OF THE LFSR CN1D WHERE CN1D ISOBTAINED REFEQCUPDATE CAN BE WRITTEN ASYJ  SUMI1LN1 CN1YJI  YJ  SUMI1LN  CNI YJI  DN DM1LEFTYJNM  SUMI1LM CIM YJNMIRIGHTTHEN USING REFEQLFSR5 AND REFEQLFSR6 WE HAVE YJ  SUMI1LN1 CN1YJI  BEGINCASES  0  JLN1 LN11 LDOTS N1 DN  DNDM1DM  0  JNENDCASESSO THAT THE LFSR PRODUCES Y0Y1 LDOTS YNFROM REFEQCUPDATE AND REFEQLUPDATE2 IT FOLLOWS THAT THEDEGREE OF CN1DMUST SATISFY LN1  MAXLNNMLM  MAXLNN1LNENDPROOFIN THE UPDATE STEP WE OBSERVE THAT IF  2LN  NTHEN USING REFEQLUPDATE CN1 HAS LENGTH LN1 LN THAT IS THE POLYNOMIAL IS UPDATED BUT THERE IS NO CHANGE INLENGTH  THE SHIFTREGISTER SYNTHESIS ALGORITHM KNOWN AS MASSEYS ALGORITHMIS PRESENTED FIRST IN PSEUDOCODE AS ALGORITHM REFALGMASSEYP WHEREWE USE THE NOTATIONS CD  CND QQUAD PD  CMDBEGINPROGENVMASSEYS    ALGORITHM PSEUDOCODEMASSEYPMASSEYS ALGORITHM PSEUDOCODESMALLBEGINPROGTABSQUAD  QUAD  QUAD  QUAD QUAD  QUAD  QUAD KILLINPUT  Y0Y1LDOTSYN1 INITIALIZE L  0 CD  1  THE CURRENT CONNECTION POLYNOMIAL PD  1 THE CONNECTION POLYNOMIAL BEFORE LAST LENGTH CHANGE S  1 S IS NM THE AMOUNT OF SHIFT IN UPDATE DM  1 PREVIOUS DISCREPANCY FOR N0 TO N1     D  YN  SUMI1L CI YNI    IF D  0       S  S1  ELSE  IF 2 L  N THEN NOLENGTH CHANGE IN UPDATE     CD  CD D DM1 DS PD     S  S1       ELSE      UPDATE C WITH LENGTH CHANGE    TD  CD TEMPORARY STORE    CD  CD D DM1 DS PD    L  N1L    PD  TD    DM  D    S  1  END ENDENDPROGTABSENDPROGENVA SC MATLAB IMPLEMENTATION OF MASSEYS ALGORITHM WITH COMPUTATIONSOVER GF2 IS SHOWN IN ALGORITHM REFALGMASSEY  THE VECTORIZEDSTRUCTURE OF SC MATLAB ALLOWS THE PSEUDOCODE IMPLEMENTATION TO BEEXPRESSED ALMOST DIRECTLY IN EXECUTABLE CODE  THE STATEMENT TT C   MODC ZEROS1LMSLN  ZEROS1S P2 SIMPLY ALIGNS THEPOLYNOMIALS REPRESENTED IN TT C AND TT P BY APPENDING ANDPREPENDING THE APPROPRIATE NUMBER OF ZEROS AFTER WHICH THEY CAN BEADDED DIRECTLY ADDITION IS MOD 2 SINCE OPERATIONS ARE IN GF2RENEWCOMMANDEXPLICITPROG1RENEWCOMMANDPROGDIRMATLABDIRBEGINNEWPROGENVMASSEYS ALGORITHMMASSEYMMASSEYMASSEYS    ALGORITHMENDNEWPROGENVRENEWCOMMANDEXPLICITPROGRENEWCOMMANDPROGDIRBECAUSE THE SC MATLAB CODE SO CLOSELY FOLLOWS THE PSEUDOCODE ONLYA FEW OF THE ALGORITHMS THROUGHOUT THE BOOK WILL BE SHOWN USINGPSEUDOCODE WITH PREFERENCE GIVEN TO SC MATLAB CODE TO ILLUSTRATEAND DEFINE THE ALGORITHMS  TO CONSERVE PAGE SPACE SUBSEQUENT ALGORITHMS ARE NOT EXPLICITLYDISPLAYED  INSTEAD THE ICON PAR  NOINDENT INCLUDEGRAPHICSPICTUREDIRPICON1PS NOINDENT INCLUDEGRAPHICSPICON EPSFIGFILEPICOM EPSFIGFILEPICTUREDIRPICON1PS NOINDENTIS USED TO INDICATE THAT THE ALGORITHM IS TO BE FOUND ON THE CDROMBEGINEXAMPLEFOR THE SEQUENCE OF EXAMPLE REFEXMMASS1 Y   1110100THE FEEDBACK CONNECTION POLYNOMIAL OBTAINED BY A CALL TO TT MASSEYIS C  1101WHICH CORRESPONDS TO THE POLYNOMIAL CD  1DD3THUS YZ1Z1Z3  0OR YJ  YJ1  YJ3AS EXPECTEDENDEXAMPLEBEGINEXERCISES  ITEM PREPARE A TABLE SHOWING THE STORAGE CONTENTS AND OUTPUTS FOR    THE LFSR SHOWN BELOW WITH INITIAL CONDITIONS SHOWN IN THE DELAY    ELEMENTS  ALSO DETERMINE THE CONNECTION POLYNOMIAL CDBEGINCENTERINPUTPICTUREDIRLFSR5LATEXENDCENTERITEM FOR THE LFSR DESCRIBED BY THE THE POLYNOMIAL 1D D2  D3BEGINENUMERATEITEM DRAW THE LFSR BLOCK DIAGRAMITEM FOR THE INITIAL CONTENTS Y3  1 Y2  0 Y1  0  DETERMINE THE OUTPUT SEQUENCE  HOW MANY DISTINCT STATES ARE THEREENDENUMERATEITEM FOR THE LFSR DESCRIBED BY THE THE POLYNOMIAL 1D2  D3BEGINENUMERATEITEM DRAW THE LFSR BLOCK DIAGRAMITEM FOR THE INITIAL CONTENTS Y3  1 Y2  0 Y1  0  DETERMINE THE OUTPUT SEQUENCE HOW MANY DISTINCT STATES ARE THEREENDENUMERATEITEM GIVEN THE SEQUENCE 0001010  BEGINENUMERATE  ITEM DETERMINE THE SHORTESTLENGTH LFSR WHICH COULD PRODUCE THIS    SEQUENCE PERFORMING THE COMPUTATIONS BY HAND  ITEM CHECK YOUR WORK USING ALGORITHM REFALGMASSEY IN SC      MATLAB  ENDENUMERATEITEM WRITE THE OUTPUT SEQUENCE AS A POLYNOMIAL YD  Y0  Y1 D  Y2 D2  CDOTSBEGINENUMERATEITEM ITEM USING REFEQLFSR1 SHOW THAT THE JTH COEFFICIENT IN  YDCD VANISHES FOR JLL1 LDOTS WHERE DEGREECD   L  HENCE WE CAN WRITE CDYD  ZDWHERE ZD  Z0  Z1D   CDOTS  ZL1DL1THUS KNOWING ZD WE CAN FIND THE OUTPUT BY POLYNOMIAL LONG DIVISIONBEGINEQUATIONYD  FRACZDCDLABELEQLFSRDIVENDEQUATIONITEM SHOW THAT THE COEFFICIENTS OF ZD CAN BE RELATED TO THE  INITIAL CONDITIONS OF THE LFSR BY BEGINBMATRIX1 0CDOTS  0  C1  1  CDOTS  0 C2  C1  CDOTS  0 VDOTS CL1  CL2  CDOTS  C1  1 ENDBMATRIXBEGINBMATRIXY0  Y1  Y2  VDOTS  YL1 ENDBMATRIX BEGINBMATRIX Z0  Z1  Z2  VDOTS  ZL1ENDBMATRIXENDENUMERATEITEM LET CD  1D2 WITH INITIAL CONTENTS Y0Y1   10  DETERMINE THE FIRST 6 OUTPUTS USING POLYNOMIAL LONG  DIVISION REFEQLFSRDIV  COMPARE THE RESULTS TO THOSE OBTAINED  DIRECTLY FROM THE LFSRITEM LET CD  1DD2 WITH INITIAL CONTENTS  Y0Y1    11  DETERMINE THE FIRST 6 OUTPUTS USING POLYNOMIAL LONG  DIVISION REFEQLFSRDIV  COMPARE THE RESULTS TO THOSE OBTAINED  DIRECTLY FROM THE LFSRITEM DETERMINE THE SEQUENCE YI OF LENGTH SEVEN GENERATED BY  CD  1DD3 AND CALL ITS LENGTH N  THEN COMPUTE THE CYCLIC  AUTOCORRELATION FUNCTION INDEXAUTOCORRELATION RHOK  FRAC1N SUMI0N1 YI YIKWHERE YIK IS DETERMINED CYCLICLY   PLOT THISAUTOCORRELATION FUNCTIONENDEXERCISES LOCAL VARIABLES TEXMASTER TEST ENDSECTIONSOME ASPECTS OF PROOFSLABELSECPROOFSBEGINQUOTESOURCEHPP FERGUSONMATHEMATICS IS SIMPLY SUSTAINED LOGICAL THINKINGENDQUOTESOURCEBEGINQUOTESOURCEPLATOTHERE IS NO ROYAL ROAD TO GEOMETRYENDQUOTESOURCEBEGINQUOTESOURCERICHARD W HAMMINGEM CODING AND INFORMATION     THEORY P 164    SOME PEOPLE BELIEVE THAT A THEOREM IS PROVED WHEN A LOGICALLY  CORRECT PROOF IS GIVEN BUT SOME PEOPLE BELIEVE IT IS PROVED ONLY  WHEN THE STUDENTS SEES WHY IT IS INEVITABLY TRUEENDQUOTESOURCEIN ENGINEERING CLASSES THAT REQUIRE PROOFS IT ALMOST INEVITABLYARISES THAT A STUDENT WILL COMPLAIN THAT HE OR SHE DOES NOT KNOW HOWTO DO PROOFS  THE WAY IT IS USUALLY STATED OF DOING PROOFSSEEMS TO SUGGEST THAT THE STUDENT PERHAPS BELIEVES THERE IS SOMEUNIVERSALLY APPLICABLE METHOD OF DOING PROOFS THAT WILL PROVE ALLPROBLEMS  ON THE ONE HAND THERE IS NO ONE THAT KNOWS HOW TO DOPROOFS OF EVERYTHING  A PROOF REQUIRES INSIGHT UNDERSTANDINGBACKGROUND AND CREATIVITY AND SOME PLAUSIBLE CONJECTURES HAVE THUSFAR ELUDED PROOF AND WILL CONTINUE TO DO SO THAT ITSELF IS ATHEOREM  SOME PROOFS HAVE THE SUBTLETY AND BEAUTY OF A WELLCRAFTEDSONNET  ON THE OTHER HAND MOST PROOFS CONSIST OF CLARIFICATIONS OFPATTERNS THAT HAVE BEEN PREVIOUSLY OBSERVED OR ARE PRECISE STATEMENTSOF SOME FACT  EVERY ENGINEERING STUDENT SHOULD BE ABLE TO DOPROOFS TO SOME EXTENTSIGNAL PROCESSING EMPLOYING MATHEMATICAL CONCEPTS TO ACCOMPLISHENGINEERING PURPOSES OFTEN PRESENTS A DIFFICULT CHALLENGE TOENGINEERING STUDENTS WHO WANT TO KNOW HOW TO USE THE MATERIAL BUTRESIST THE MATHEMATICAL FORMALITIES  IN PARTICULAR THEOREMS ANDPROOFS  NEVERTHELESS THROUGHOUT THIS BOOK MANY OF THE CONCEPTS AREPRESENTED IN A THEOREMPROOF FORMAT AS A MEANS OF ORGANIZATION ANDOPPORTUNITIES FOR PROVING MANY CONCEPTS ARE PROVIDED IN THE EXERCISESTHE FOLLOWING JUSTIFICATIONS ARE PROVIDED FOR REQUIRING PROOFS OFENGINEERING STUDENTSBEGINENUMERATEITEM BECAUSE AN ENGINEER PUTS THINGS TOGETHER WITH AN EYE TO DESIGN  AND UTILITY THE ABILITY TO MOVE FROM A REQUIREMENT SPECIFICATION TO  A FINISHED DESIGN IS AN IMPORTANT SKILL  IN ITS RESTRICTED DOMAIN  PROVING A THEOREM IS NOTHING MORE THAN DESIGN TAKING SPECIFICATIONS  AND USING AVAILABLE COMPONENTS TO PRODUCE A RESULT  THE  SPECIFICATIONS ARE THE HYPOTHESES OF THE THEOREM AND THE AVAILABLE  COMPONENTS ARE WHATEVER KNOWLEDGE CAN BE BROUGHT TO BEAR ON THE  PROBLEM  LIKE MOST DESIGN PROBLEMS THERE MAY BE MANY CORRECT  SOLUTIONS AND MANY INCORRECT APPROACHES  IT IS PERHAPS THE  FLEXIBILITY OF CHOICE EXERCISED AGAINST INFLEXIBLE LOGIC THAT MAKES  PROOFS CHALLENGING  LIKE DESIGN A PROOF MAY REQUIRE TRYING MANY  DIFFERENT AVENUES BEFORE A FRUITFUL APPROACH IS ENCOUNTERED  ITEM A PROOF PROVIDES AN OPPORTUNITY TO REVIEW AND DEEPEN  UNDERSTANDING OF CONCEPTS AND DEFINITIONS THAT HAVE BEEN PRESENTED  TOOLS THAT DONT GET USED OR ARE NOT UNDERSTOOD CORRECTLY WILL NEVER  BECOME USEFUL TOOLS  ITEM AS NEW ALGORITHMS ARE DEVELOPED THEY MUST BE EVALUATED  OFTEN  THIS IS DONE EMPIRICALLY BY MEANS OF COMPUTER SIMULATION OR BY  TESTING OF PROTOTYPES  HOWEVER IT IS BETTER TO HAVE A SENSE OF THE  CORRECTNESS OF A DESIGN BEFORE TOO MANY RESOURCES ARE EXPENDED IN  ITS PROTOTYPING  THE SKILLS DEVELOPED IN LEARNING TO DO PROOFS OF  THEOREMS MAY ASSIST IN EVALUATING AND IMPROVING SIGNAL PROCESSING  ALGORITHMS   ITEM THERE IS NO ESCAPING THE FACT THAT THE SIGNAL PROCESSING  LITERATURE IS VERY MATHEMATICAL  A BROAD MATHEMATICAL VOCABULARY  AND THE ABILITY TO READ MATHEMATICS ARE NECESSARY TO DRAW MEANINGFUL  INFORMATION FROM THE LITERATURE  SHOULD THE OCCASION ARISE WHEN  STUDENTS WISH TO PUBLISH THEIR OWN RESULTS IN SIGNAL PROCESSING  LITERATURE THEY WILL NEED TO SPEAK THE LANGUAGE  ITEM DOING A PROOF IS A GOOD CHANCE TO STRETCH SOME INTELLECTUAL  MUSCLESENDENUMERATETHE INTENT OF THIS SECTION IS TO PROVIDE SOME SUGGESTIONS ON METHODSOF PROOF THAT APPEAR IN THE LITERATURE  THIS IS BY NO MEANS ANEXHAUSTIVE LIST NEW AND IMPORTANT CONCEPTS CAN ARISE AS NEW WAYS OFANSWERING QUESTIONS ARE CREATED  AS AN EXAMPLE CONSIDER SHANNONSINDEXCHANNELCODING THEOREM CHANNELCODING THEOREM WHICH STATESBASICALLY THAT THERE IS A CODE WHICH CAN BE USED TO TRANSMIT DATAOVER A CHANNEL WITH ARBITRARILY LOW PROBABILITY OF ERROR PROVIDEDTHAT THE RATE OF TRANSMISSION IS LESS THAN THE CAPACITY OF THECHANNEL  IN PROVING THE THEOREM SHANNON TOOK AN UNPRECEDENTED STEPINSTEAD OF LOOKING FOR A PARTICULAR CODE TO ANSWER THE QUESTION HEINSTEAD AVERAGED OVER ALL POSSIBLE CODES  THIS PARTICULAR TRICK MADETHE ANALYSIS FALL RIGHT INTO PLACE  SUCH TRICKS OR CREATIVEINSIGHTS CANNOT BE TAUGHT  THERE ARE HOWEVER SOME LOGICALAPPROACHES WHICH CAN BE TAUGHT AND EXERCISEDA THEOREM MAY BE STATED SOMETHING LIKE IF P THEN Q  IN THISP IS CALLED THE EM HYPOTHESIS AND Q IS CALLED THE EM  CONCLUSION  WE SAY THAT P IMPLIES Q AND MAY WRITE PRIGHTARROW Q  INDEXIMPLICATION THE STATEMENT IF P THENQ IS NOT LOGICALLY EQUIVALENT TO SAYING THAT BECAUSE Q OCCURSP MUST ALSO OCCUR  FOR EXAMPLE CONSIDER THE FOLLOWING SYLLOGISMSMALLSKIPINDENT IF A BOOK FALLS ON FRANKS HEAD HIS HEAD WILL HURT INDENT  FRANKS HEAD HURTS SMALLSKIPNOINDENT WE CANNOT CONCLUDE THAT A BOOK HAS FALLEN ON FRANKS HEADHE MAY SIMPLY HAVE A HEADACHE  IN THE IMPLICATION P RIGHTARROW QWE SAY THAT P IS SUFFICIENT FOR Q KNOWLEDGE THAT P OCCURS ISSUFFICIENT TO ESTABLISH THE PRESENCE OF Q  HOWEVER P IS NOTNECESSARY FOR Q Q COULD PERHAPS HAVE HAPPENED ANOTHER WAYINDEXSUFFICIENT INDEXNECESSARYNOTE THAT IF P RIGHTARROW Q AND IF Q IS NOT TRUE THEN P CANNOTBE TRUE  BASED ON THE SYLLOGISM ABOVE IF FRANKS HEAD DOES NOT HURTWE EM CAN CONCLUDE THAT A BOOK DID NOT FALL ON HIS HEADEQUIVALENT WAYS OF EXPRESSING THIS IMPLICATION ARE SMALLSKIPP IMPLIES Q INDENT IF P THEN Q INDENT P RIGHTARROW Q INDENT Q IF P INDENT P ONLY IF Q  INDENT P IS A SUFFICIENT BUT NOT NECESSARY CONDITION FOR Q  INDENT NOT Q IMPLIES NOT P THIS IS THE EM  CONTRAPOSITIVEINDEXCONTRAPOSITIVE  INDENT Q IS A NECESSARY CONDITION FOR P SMALLSKIPFOR THE STATEMENT P RIGHTARROW QTHE STATEMENT OBTAINED BY REVERSING THE ROLES OF P AND Q Q RIGHTARROW PIS CALLED THE EM CONVERSE  INDEXCONVERSE THAT FACT THAT PRIGHTARROW Q AND ITS CONVERSE Q RIGHTARROW P ARE BOTH TRUECAN BE STATED IN A VARIETY OF EQUIVALENT WAYS SMALLSKIPP IMPLIES Q EM AND Q IMPLIES P  INDENT P IMPLIES Q AND CONVERSELY INDENT P IF AND ONLY IF Q INDEXIFFSEEIF AND ONLY IF INDEXIF AND ONLY IF INDENT P IS A NECESSARY AND SUFFICIENT CONDITION FOR Q INDENT P LEFTRIGHTARROW Q SMALLSKIPNOINDENT THE STATEMENT P IF AND ONLY IF Q IS OFTEN ABBREVIATEDP IFF Q SMALLSKIPWE NOW PRESENT SOME COMMENTS ABOUT PROOFS IN A GENERAL FRAMEWORKTHESE SUGGESTIONS DO NOT PROVIDE AN EXHAUSTIVE BAG OF TRICKS BUT AREMERELY INTENDED TO SUGGEST SOME APPROACHES THAT MIGHT WORK SUBSECTIONPROOF BY COMPUTATION DIRECT PROOFLABELSECPROOFCOMPINDEXPROOFBY COMPUTATION PROOFS OF SOME STATEMENTS MAY BEMOSTLY COMPUTATIONAL INVOLVING SUCH TECHNIQUES AS INTEGRATION OFTENUSING CHANGE OF VARIABLES PROPERTIES OF INTEGRATION LINEAR ALGEBRATAYLOR SERIES ETC  AS A SIMPLE EXAMPLE TO PROVE THAT CONVOLUTIONCOMMUTES THAT IS THAT INTINFTYINFTY FTTAUHTAUDTAU INTINFTYINFTY FTAUHTTAUDTAUIT SUFFICES TO MAKE A CHANGE OF VARIABLE XTTAU IN THE FIRSTINTEGRAL  IF YOU WERE APPROACHING THE PROBLEM WITHOUT KNOWING THETRICK THE BEST THING TO DO WOULD BE TO SIMPLY TRY SEVERALAPPROACHES  IF WHAT YOU ARE TRYING TO PROVE IS TRUE SOONER OR LATERYOU MAY STUMBLE ACROSS THE CORRECT APPROACH  WHILE THIS MAY LACKPOLISH IT MIRRORS THE WAY THINGS ARE DISCOVERED IN THE REAL WORLDRARELY DOES A USEFUL CONCEPT OR PRODUCT SPRING FORTH FULLBLOWN AS IFFROM THE HEAD OF ZEUS  DISCOVERY REQUIRES EXPLORATION THOUGHT ANDTRIALANDERROR  OF COURSE EXPERIENCE IN AN AREA CAN SHORTEN THETIME BETWEEN CONCEPT AND EXECUTION  TO EXPERIENCED MATHEMATICIANSSOME THINGS BECOME TRANSPARENTLY OBVIOUS BECAUSE THEY HAVE SOLVED SO MANYRELATED PROBLEMS  A STUDENT STARTING OUT IN AN AREA MAY NOT HAVE THEBENEFIT OF THAT INSIGHT  WHAT IS OFTEN REQUIRED IS THE DETERMINATIONTO TRY THINGS OUT POSSIBLY WITHOUT BEING ABLE TO FORESEE AT THEOUTSET WHAT WILL RESULT  EXPERIENCE WILL LENGTHEN THE NUMBER OF STEPSYOU CAN SEE AHEADBEGINEXAMPLEHERE IS AN EXAMPLE OF A DIRECT PROOF  IT NOT ONLY ILLUSTRATES AUSEFUL PROOF BUT INTRODUCES SEVERAL CONCEPTS WHICH WILL BE MORETHOROUGHLY EXPLORED LATER IN THE BOOK SUCH AS DISTANCE MEASURESTRIANGLE INEQUALITY AND NORMS OF VECTORSLET X  XBF1 XBF2 LDOTS XBFM BE A SET OF DISCRETE POINTSIN RBBN  LET DXBFIXBFJ INDICATE THE DISTANCE BETWEEN THEVECTORS XBFI AND XBFJ  THE SETS DEFINED BY VI   XBF IN RBBNMC  XBF TEXT IS CLOSER TO XBFI THAN TO  ANY OTHER XBFJ I NEQ JTHAT IS  VI   XBF IN RBBNMC   DXBFXBFI  DXBFXBFJ I NEQJARE CALLED THE EM VORONOI REGIONS OF X  THE VECTOR XBFI INVI IS CALLED THE CELL REPRESENTATIVE  INDEXVORONOI REGIONVORONOI REGIONS ARISE IN VECTOR QUANTIZATION INDEXVECTOR QUANTIZATION AND DATA COMPRESSION INDEXDATA COMPRESSIONSEE SECTION REFSECCLUSTAPP  WE WILL PROVE THAT VORONOIREGIONS ARE CONVEX SETS INDEXCONVEX SET  PICK A VORONOI CELLWITHOUT LOSS OF GENERALITY WE WILL CALL THE CELL V1 WITH ITS CELLREPRESENTATIVE XBF1  LET PBF AND QBF BE ARBITRARY POINTS IN V1 AND LET US  DESIGNATE PBF AS THE POINT WHICH IS FURTHER FROM XBF1  IF EVERY POINT  ON THE LINE BETWEEN PBF AND QBF IS IN V1 THEN THE SET IS  CONVEX  LET XBF BE A POINT ON THE LINE BETWEEN PBF AND  QBF XBF  LAMBDA PBF  1LAMBDA QBF QQUAD 0 LEQ LAMBDA LEQ 1WE WILL DENOTE DXBF1XBF AS XBF1  XBF THE NORM OFTHE DIFFERENCE  THEN BEGINALIGNED DXBF1XBF    XBF1  LAMBDA PBF  1LAMBDA QBF    LAMBDAXBF1  PBF  1LAMBDAXBF1  QBF LEQ LAMBDA  XBF1  PBF   1LAMBDA XBF1  QBF LEQ LAMBDA  XBF1  PBF LEQ  XBF1  PBFENDALIGNEDWHERE THE FIRST INEQUALITY FOLLOWS FROM THE TRIANGLE INEQUALITYINDEXINEQUALITIESTRIANGLE  THUS XBF IS CLOSER TO XBF1 THANIS PBF WHICH IS IN THE VORONOI CELL  BY THE DEFINITION OF THEVORONOI CELL IF PBF IS IN THE VORONOI CELL THEN XBF MUST ALSOBEENDEXAMPLEOF COURSE THE TRIALANDERROR ASPECT OF FINDING THE CORRECTCOMPUTATION IN THIS EXAMPLE IS NOT SHOWN ONLY THE FINISHED PRODUCTSOME STANDARD TRICKS THAT ARE EMPLOYED IN PROOFS ARE WORTH MENTIONINGBEGINENUMERATEITEM COUNTING AND LISTS  MAKE AN EXHAUSTIVE LIST OF ALL THE  ELEMENTS AND CONSIDER WHAT YOU ARE TRYING TO DO APPLIED TO ALL OF  THEMITEM TO SHOW THAT A AND B ARE THE SAME IT MAY WORK TO SHOW THAT  A SUBSET B AND B SUBSET A  SIMILARLY TO SHOW THAT XY  SHOW THAT X GEQ Y AND Y GEQ X  SEE FOR EXAMPLE THE PROOF  TO THEOREM REFTHMBASISSAMEITEM IN ANALYTICAL WORK THE TAYLOR SERIES OR THE MEAN VALUE  THEOREM ARE EXCELLENT TOOLSITEM EXHAUSTIVE CHECKING  FOR EXAMPLE TO VERIFY THAT A SET  SATISFIES CERTAIN PROPERTIES SIMPLY VALIDATE THAT THE PROPERTIES  HOLD INDIVIDUALLYENDENUMERATESUBSECTIONPROOF BY CONTRADICTIONLABELSECPROOFCONTBEGINQUOTESOURCEAYN RANDEM ATLAS SHRUGGED  P 188CONTRADICTIONS DO NOT EXIST  WHENEVER YOU THINK THAT YOU ARE FACING ACONTRADICTION CHECK YOUR PREMISES  YOU WILL FIND THAT ONE OF THEM ISWRONG INDEXPROOFBY CONTRADICTIONENDQUOTESOURCEA POWERFUL PROOF TECHNIQUE IS PROOF BY CONTRADICTION  IN ORDER TOSHOW THAT P RIGHTARROW Q WE TAKE AS TRUE THE HYPOTHESIS P ANDEM ASSUME THAT Q IS NOT TRUE  THE PROOF FOLLOWS BY SHOWING THATTHIS ASSUMPTION LEADS TO A LOGICAL CONTRADICTION  BEGINEXAMPLE  WE WILL PROVE A MILLENNIAOLD THEOREM KNOWN TO THE PYTHAGOREANS OF  GREECE  RECALL THAT A RATIONAL NUMBER IS A NUMBER  THAT CAN BE EXPRESSED AS A RATIO OF INTEGERS  THUS 37 IS A  RATIONAL NUMBERNOINDENT BF THEOREM EM SQRT2 IS IRRATIONAL SMALLSKIPINDEXIRRATIONALPRIOR TO ESTABLISHING THIS THEOREM THE PYTHAGOREANS HELD THEVIEWPOINT THAT THE HARMONIES OF COSMOS COULD BE EXPRESSED AS RATIOS OFINTEGERS  THIS THEOREM LEAD TO CONSIDERABLE RELIGIOUS UPHEAVAL IN ITSDAY SMALLSKIP NOINDENT BF PROOF WE WILL ASSUME A RESULT CONTRARY TO THESTATEMENT OF THE THEOREM AND SHOW THAT THIS LEADS TO A CONTRADICTIONWE ASSUME THAT SQRT2 EM IS RATIONAL THAT IS THATBEGINEQUATIONSQRT2  MNLABELEQSQRT2ENDEQUATIONFOR SOME INTEGERS M AND N  NOW WE SHOW THAT THIS LEADS TO ACONTRADICTION  SQUARING REFEQSQRT2 WE OBTAINBEGINEQUATION2  FRACM2N2LABELEQPROOFCONT1ENDEQUATIONSO 2N2  M2FROM THIS WE SEE THAT M2 MUST BE AN EVEN NUMBER AND HENCE THAT MMUST BE EVEN SHOW THIS  LET US WRITE M  2K FOR SOME INTEGERK  SUBSTITUTING THIS INTO REFEQPROOFCONT1 WE OBTAIN 2  FRAC4K2N2OR 2  FRACN2K2THIS IS EQUIVALENT TO SQRT2  FRACNKNOW WE HAVE RETURNED BACK AN EXPRESSION HAVING THE SAME FORM ASREFEQSQRT2 BUT WITH K  N  BEING NOW IN A POSITION TO REPEATTHE OPERATION WE HAVE REACHED THE PRECIPICE LEADING TO ACONTRADICTION BECAUSE THE NUMBERS IN THE RATIO WILL BE REDUCED BYITERATION OF THESE SAME STEPS DOWN TO ABSURDLY SMALL VALUES  BY THISCONTRADICTION WE MUST CONCLUDE THAT THE ORIGINAL ASSUMPTIONREFEQSQRT2 IS FALSEENDEXAMPLE EXAMPLES OF PROOF BY CONTRADICTION ARE GIVEN IN THEOREMS REFONE OF THE ISSUES OVER WHICH MATHEMATICIANS SOMETIMES FRET IS THEUNIQUENESS OF A SOLUTION TO A GIVEN PROBLEM  PROVING UNIQUENESS ISVERY COMMONLY DONE USING CONTRADICTION  TWO DISTINCT SOLUTIONS TO THEPROBLEM ARE PROPOSED AND IT IS SHOWN THAT THESE SOLUTIONS ARE EQUALA CONTRADICTION WHICH POINTS OUT THAT ONLY ONE SOLUTION IS POSSIBLETHIS METHOD IS EXEMPLIFIED IN THE PROOF OF THEOREM REFTHMUNIQBASSUBSECTIONPROOF BY INDUCTIONLABELSECPROOFINDBEGINQUOTESOURCEHENRI POINCAREEM SCIENCE AND HYPOTHESIS  THE ESSENTIAL CHARACTERISTIC OF REASONING BY RECURRENCE IS THAT IT  CONTAINS CONDENSED SO TO SPEAK IN A SINGLE FORMULA AN INFINITE  NUMBER OF SYLLOGISMSENDQUOTESOURCEINDEXPROOFBY INDUCTIONPROOF BY INDUCTION ALLOWS ONE TO ESTABLISH GENERAL CONCLUSIONS FROM ALIMITED SET OF TEST CASES  SUPPOSE YOU HAVE SOME STATEMENT THATDEPENDS UPON AN INTEGER N  WE WILL DENOTE THIS STATEMENT BY SN STATEMENT S IS A FUNCTION OF N  YOU BEGIN BY SHOWING THATSN IS TRUE FOR N1  SOMETIMES ANOTHER SMALL VALUE OF N ISTHE STARTING POINT  THEN YOU SHOW THAT ASSUMING SN IS TRUE LEADSTO AN IMPLICATION THAT SN1 IS ALSO TRUE  WHAT IS AMAZING ANDPOWERFUL IS THAT YOU GET TO EM ASSUME THE TRUTH OF SN AND USETHIS TO SHOW THE TRUTH OF SN1  THE ASSUMED HYPOTHESIS SN ISCALLED THE EM INDUCTIVE HYPOTHESISBEGINEXAMPLE  THE FIRST EXAMPLE SHOULD BE FAMILIAR  WE WANT TO SHOW THAT THE SUM  OF THE FIRST N INTEGERS IS SUMK0N K  FRACNN12CLEARLY THIS IS TRUE FOR N0 AND ALSO CLEARLY IT IS TRUE FOR N1LET US ASSUME ITS TRUTH FOR N  THAT IS WE NOW EM ASSUME THAT SUMK0N K  FRACNN12AND SHOW THAT THIS IMPLIESTHE TRUTH FOR N1  THAT IS WE NEED TO SHOW THAT SUMK0N1   FRACN1N22WE HAVEBEGINALIGNEDSUMK0N1 K  LEFTSUMK0N KRIGHT  N1    FRACNN12  N1  FRACN23N22  FRACN1N22ENDALIGNEDWHERE THE SECOND EQUALITY COMES BY ASSUMPTION OF THE INDUCTIVEHYPOTHESISENDEXAMPLE  WE WILL DO ANOTHER INDUCTIVE PROOF OF MATHEMATICAL FLAVOR TO  ILLUSTRATE ANOTHER POINTBEGINEXAMPLE  WE WILL SHOW THAT TEXTIF  N GEQ 5  TEXT THEN  2N  N2WHAT MAKES THIS EXAMPLE FUNDAMENTALLY DIFFERENT FROM THE PREVIOUS ISTHAT THE STARTING POINT IS NOT N0 BUT N5THE STATEMENT IS CLEARLYTRUE WHEN N5  LET US ASSUME THAT IT HOLDS FOR N THAT IS OURINDUCTIVE HYPOTHESIS IS 2N  N2AND SHOW THAT IT MUST BE TRUE FOR N1 THAT IS 2N1  N12WE HAVE BEGINALIGN2N1  2CDOT 2N NONUMBER  2N2 QQUADTEXTBY THE INDUCTIVE HYPOTHESIS NONUMBER  INTERTEXTHFILLBY THE INDUCTIVE HYPOTHESIS N2N2 GEQ N2  5N QQUADTEXTBECAUSE N GEQ 5 NONUMBER  N2  2N3N  N2  2N1 NONUMBER  N12 NONUMBERENDALIGNENDEXAMPLEWE NOW OFFER AN EXAMPLE WITH A LITTLE MORE OF AN ENGINEERING FLAVORBEGINEXAMPLE  SUPPOSE THERE IS A COMMUNICATION LINK IN WHICH ERRORS CAN BE MADE  WITH PROBABILITY P  THIS LINK IS DIAGRAMMED IN FIGURE  REFFIGBSC1A  WHEN A 0 IS SENT IT IS RECEIVED AS A 0 WITH  PROBABILITY 1P AND AS A 1 WITH PROBABILITY P  THIS  COMMUNICATIONLINK MODEL IS CALLED A BINARY SYMMETRIC CHANNEL  BSC INDEXBINARY SYMMETRIC CHANNEL  NOW SUPPOSE THAT N BSCS ARE PLACED END TO END AS IN FIGURE  REFFIGBSC1B  DENOTE THE PROBABILITY OF ERROR AFTER N  CHANNELS BY PNE  WE WISH TO SHOW THAT THE ENDTOEND  PROBABILITY OF ERROR ISBEGINEQUATION PNE  FRAC12112PNLABELEQPNEENDEQUATIONBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    SUBFIGUREA SINGLE CHANNELINPUTPICTUREDIRBSC1 QQUADSUBFIGUREN CHANNELS ENDTOENDINPUTPICTUREDIRBSC3    CAPTIONBINARY SYMMETRIC CHANNEL MODEL    LABELFIGBSC1  ENDCENTERENDFIGUREWHEN N1 WE COMPUTE P1E  P AS EXPECTED  LET US NOW ASSUMETHAT PNE AS GIVEN IN REFEQPNE IS TRUE FOR N AND SHOW THATTHIS PROVIDES A TRUE FORMULA FOR PN1E  IN N1 STAGES WE CAN MAKE AN ERROR IF THERE ARE NO ERRORS IN THEFIRST N STAGES AND AN ERROR OCCURS IN THE LAST STAGE OR IF AN ERRORHAS OCCURRED OVER THE FIRST N STAGES AND NO ERROR OCCURS IN THE LASTSTAGE  THUS BEGINALIGNEDPN1E  1PPNE  P1PNE  1PFRAC12112PN  P1FRAC12112PN  QQUADQQUAD TEXTBY THE INDUCTIVE HYPOTHESIS  FRAC12112PN1ENDALIGNEDENDEXAMPLEPROOF BY INDUCTION IS VERY POWERFUL AND WORKS IN A REMARKABLE NUMBEROF CASES  IT REQUIRES THAT YOU BE ABLE TO STATE THE THEOREM YOU MUSTSTART WITH THE INDUCTIVE HYPOTHESIS WHICH IS USUALLY THE DIFFICULTPART  IN PRACTICE STATEMENT OF THE THEOREM MUST COME BY SOME INITIALGRIND SOME INSIGHT AND A LOT OF WORK  THEN INDUCTION IS USED TOPROVE THAT THE RESULT IS CORRECT  SOME SIMPLE OPPORTUNITIES FORSTATING THE INDUCTIVE HYPOTHESIS AND THEN PROVING IT ARE PROVIDED INTHE EXERCISESBEGINEXERCISESITEM SHOW THAT SQRT3 IS IRRATIONALITEM SHOW THAT THERE ARE AN INFINITE NUMBER OF  PRIMES  HINT  USE A PROOF BY CONTRADICTION ASSUMING THAT THERE  ARE ONLY A FINITE NUMBER OF PRIMES  THEN BUILD A NUMBER 2CDOT 3  CDOT 5 CDOT CDOTS CDOT P  1 WHERE P IS THE ASSUMED LAST  PRIME AND SHOW THAT THIS IS NOT DIVISIBLE BY ANY OF THE LISTED PRIMESITEM USING PROOF BY CONTRADICTION SHOW THAT SQRT2 CANNOT BE A  RATIONAL NUMBER  HINT  ASSUME SQRT2  MN FOR SOME INTEGER  M AND N WHERE THE FRACTION IS EXPRESSED IN REDUCED FORM  SHOW  THAT THIS LEADS TO A CONTRADICTIONITEM SHOW THAT IF M2 IS EVEN THEN M MUST BE EVENITEM BY TRIAL AND ERROR DETERMINE A PLAUSIBLE FORMULA FOR SUMI0N 2ITHEN PROVE BY INDUCTION THAT YOUR FORMULA IS CORRECTITEM DETERMINE BY EXPERIMENT A PLAUSIBLE FORMULA FOR THE SUM OF THE  FIRST N ODD INTEGERS 135 CDOTS  2N1THEN PROVE BY INDUCTION THAT YOUR FORMULA IS CORRECTITEM DETERMINE BY EXPERIMENT A PLAUSIBLE FORMULA FOR SUMI1N FRAC1I2 1THEN PROVE BY INDUCTION THAT YOUR FORMULA IS CORRECTITEM SHOW BY INDUCTION FOR EVERY POSITIVE INTEGER N THAT N3  N  IS IS DIVISIBLE BY 3ITEM THE QUANTITY BOXED NCHOOSEK  FRACNKNK IS THE NUMBER OF WAYS OF CHOOSING K OBJECTS OUT OF N OBJECTSWHERE N GEQ K  THE QUANTITY NCHOOSE K IS ALSO KNOWN AS THE EMBINOMIAL COEFFICIENT  WE READ THE NOTATION N CHOOSE K AS NCHOOSE K INDEXBINOMIAL COEFFICIENT INDEXNKN CHOOSE K  SHOW BY INDUCTION THAT N1CHOOSEK  NCHOOSEK  NCHOOSEK1ITEM SHOW BY INDUCTION THAT FOR N GEQ 0 SUMI0N N CHOOSE K  2NITEM SHOW BY INDUCTION THAT BOXEDXYN  SUMK0N N CHOOSE K XK YNKTHIS IMPORTANT FORMULA IS KNOWN AS THE EM BINOMIAL  THEOREM INDEXBINOMIAL THEOREMITEM PROVE THE FOLLOWING BY INDUCTION SUMK1N J2  FRACNN12N16ITEM PROVE THE FOLLOWING BY INDUCTION BOXED SUMK1N RN  FRACRN  1R1 QQUAD R NEQ 1 INDEXGEOMETRIC SUMITEM PROVE BY INDUCTION THAT FRAC1SQRT4N1  FRAC12CDOT FRAC34 CDOT CDOTSCDOT FRAC2N32N2CDOT FRAC2N12N   FRAC1SQRT3N1FOR INTEGERS N GEQ 2KAZARINOFF 1961 P 5ITEM PROVE BY INDUCTION THAT FOR XYN IN ZBB XY DIVIDES  INDEX INDEXDIVIDESSEE  XNYN  THIS IS WRITTEN AS XY  XNYNENDEXERCISESINPUTLINALGDIRLININTRO  INTRODUCT TO LINEAR ALGEBRA CHAPTER VECTOR SPACESINPUTLINALGDIRVECTSP    VECTOR SPACES FINITE DIMENSIONAL ANDINPUTLINALGDIRVECT    VECTOR SPACES APPLICATIONS CHAPTER MATRICES AND LINEAR OPERATORSINPUTLINALGDIRMATEQ     MATRIX EQUATIONS   INCLUDES NORM AND RANK CHAPTER COMPUTING MATRIX SOLUTIONSINPUTLINALGDIRMATINV    PROPERTIES OF MATRIX INVERSESINPUTLINALGDIRMATCOND   MATRIX CONDITION NUMBER LU FACTORIZATIONINPUTLINALGDIRMATPROJ   PROJECTION MATRICESINPUTLINALGDIRLINTRANS  TRANSFORMATION OF BASES SIMILARITY INPUTLINALGDIREIGEN     STUFF ON EIGENVECTORSINPUTLINALGDIRMATFACT    MATRIX FACTORIZATIONSINPUTLINALGDIRSVD       SINGULAR VALUE DECOMPOSITIONINPUTLINALGDIRCANON     CANONICAL FORMS FOR MATRICESINPUTLINALGDIRMODALMATRIX  MODAL MATRICES AND EXPONENTIAL MODELSINPUTLINALGDIRSPECIALMAT  SPECIAL MATRIX FORMSINPUTLINALGDIRKRONECKER  KRONECKER PRODUCT AND ITS APPLICATIONSINPUTLINALGDIRVECOPAPPENDIXINPUTLINALGDIRLINBASICSMYENDAPPENDIX LOCAL VARIABLES TEXMASTER TEST ENDCHAPTERSIGNAL SPACESLABELCHAPVECTSPBEGINQUOTESOURCEEDWARD ABBEYEM DESERT SOLITAIRE  LANGUAGE MAKES A MIGHTY LOOSE NET WITH WHICH TO GO FISHING FOR  SIMPLE FACTS WHEN FACTS ARE INFINITEENDQUOTESOURCEBEGINQUOTESOURCEHENRI POINCAREEM SCIENCE AND HYPOTHESIS  BEGINNERS ARE NOT PREPARED FOR REAL MATHEMATICAL RIGOR THEY WOULD  SEE IN IT NOTHING BUT EMPTY TEDIOUS SUBTLETIES  IT WOULD BE A  WASTE OF TIME TO TRY TO MAKE THEM MORE EXACTING THEY HAVE TO PASS  RAPIDLY AND WITHOUT STOPPING OVER THE ROAD WHICH WAS TRODDEN SLOWLY  BY THE FOUNDERS OF THE SCIENCEENDQUOTESOURCEBEGINQUOTESOURCECHAIM POTOKEM IN THE BEGINNING  ALL BEGINNINGS ARE HARDENDQUOTESOURCETHIS CHAPTER IS MOSTLY ABOUT TWO KINDS OF MATHEMATICAL OBJECTS METRICSPACES AND LINEAR VECTOR SPACES  THE IDEA BEHIND A METRIC SPACE ISSIMPLY THAT WE PROVIDE A WAY OF MEASURING THE DISTANCE BETWEENMATHEMATICAL OBJECTS SUCH AS SETS POINTS FUNCTIONS OR SEQUENCESWITH THIS NOTION OF DISTANCE WE WILL BE ABLE TO GENERALIZE SOME OF THEFAMILIAR CONCEPTS OF CALCULUS SUCH AS CONTINUITY OR CONVERGENCEBEYOND OPERATIONS ON A SINGLE DIMENSION TO OPERATIONS IN HIGHERDIMENSIONS  THE CONCEPT OF A VECTOR SPACE IS ALSO SIMPLE IT IS A SET OF OBJECTSTHAT CAN BE COMBINED TOGETHER USING LINEAR COMBINATIONS  BUT THETHEORY OF VECTOR SPACES HAS FARREACHING RAMIFICATIONS COVERING ASIGNIFICANT PORTION OF THE THEORY OF SIGNAL PROCESSING  A KEY INSIGHTIN VECTOR SPACE THEORY IS THAT IN A GEOMETRICALLY USEFUL SENSE BF  FUNCTIONS IE SIGNALS CAN BE REGARDED AS VECTORS  THISGEOMETRIC UNDERSTANDING PROVIDES A POWERFUL TOOL FOR SIGNAL ANALYSISIN THIS CHAPTER THE BASIC THEORY AND NOTATION OF VECTOR SPACES ISDEVELOPED  IN CHAPTER REFCHAPVECTAP WE PUT THIS NOTION TO WORK INA VARIETY OF APPLICATIONS INCLUDING OPTIMAL FILTERING BOTH LEASTSQUARES AND MINIMUM MEAN SQUARES TRANSFORMS DATA COMPRESSIONSAMPLING AND INTERPOLATIONIN OUR STUDY OF METRIC SPACES AND VECTOR SPACES THE INTENT IS TOPROVIDE A FRAMEWORK FOR THE GENERAL DISCUSSION OF SIGNALS  BEFOREEMBARKING ON THIS CHAPTER THE READER IS ENCOURAGED TO REVIEW THEBASIC DEFINITIONS OF FUNCTIONS AND SETS APPEARING IN APPENDIXREFAPPDXSETFUNCT  MATRIX NOTATION IS HEAVILY EMPLOYED IN THISSTUDY SO REVIEW OF THE BASIC MATRIX NOTATIONS PRESENTED IN APPENDIXREFAPPDXLINBASICS IS ALSO RECOMMENDEDIN THE DEVELOPMENT OF THIS CHAPTER WE BUILD SUCCESSIVELY FROM BF METRICSPACES TO BF VECTOR SPACES TO BF NORMED VECTOR SPACES TOBF NORMED INNERPRODUCT SPACES  THIS WILL LEAD US TO THE IMPORTANT IDEA OFPROJECTIONS AND ORTHOGONAL PROJECTIONS  ORTHOGONAL PROJECTION WILL BEA TOOL OF TREMENDOUS IMPORTANCE TO US IN THE NEXT CHAPTER WHERE ITWILL BE USED AS THE GEOMETRICAL BASIS FOR BOTH LEASTSQUARES ANDMINIMUM MEANSQUARES FILTERING AND PREDICTIONINPUTHOMEDIRLINALGPARTBASICSSECTIONSOME ALGEBRAIC DEFINITIONSTHE DEFINITIONS IN THIS SECTION ARE PROVIDED TO BE ABLE TO STATECLEARLY IN WHAT FOLLOWS WHERE THE COMPUTATIONS TAKE PLACE  IN SOMEAPPLICATIONS COMPUTATIONS ARE NOT DONE USING THE FAMILIAR REALNUMBERS BUT ARE DONE USING NUMBERS MODULO N SUCH AS N256 FOR8BIT REPRESENTATIONS  IN THIS CASE WE MUST PAY ATTENTION TO THEPARTICULAR ALGEBRAIC PROPERTIES OF THE OBJECTS THAT ARE USED  THEALGEBRAIC PROPERTIES OF INTEREST ARE WHETHER THE COMPUTATIONS TAKEPLACE IN A GROUP A RING OR A FIELD  IF THE PARTICULAR APPLICATIONSOF INTEREST TO THE READER WILL ALWAYS BE COMPUTED USING REAL NUMBERSRBB THEN THESE DEFINITIONS CAN BE SKIPPED SINCE THE SET OF REALNUMBERS IS A GROUP UNDER BOTH ADDITION AND MULTIPLICATION AFTERREMOVING 0  IT IS ALSO A RING AND A FIELDTO INTRODUCE GROUPS RINGS AND FIELDS WE NEED THE NOTION OF A BINARYOPERATOR  IN THE INTEREST OF BREVITY WE INTRODUCE THIS BY SEVERALEXAMPLESBEGINEXAMPLE   BEGINENUMERATE  ITEM THE OPERATOR  IS A BINARY OPERATOR  ITEM THE OPERATOR  IS A BINARY OPERATOR  ITEM THE OPERATOR CDOT MULTIPLICATION IS A BINARY OPERATOR  ITEM THE FUNCTION COMPOSITION OPERATOR CIRC IS A BINARY OPERATOR  ENDENUMERATEENDEXAMPLEIN SHORT A BINARY OPERATOR TAKES TWO OPERANDS AND RETURNS THEOPERATION ON THOSE TWO OPERANDSSUBSECTIONGROUPSLABELSECGROUPSBEGINDEFINITION LABELDEFGROUP  A SET S EQUIPPED WITH SINGLE BINARY OPERATION  IS A GROUP IF IT  SATISFIES THE FOLLOWINGBEGINENUMERATEG1ITEM THE BINARY OPERATION IS CLOSED IN S  THIS IS DIFFERENT THEN  THE TOPOLOGICAL NOTION OF CLOSURE  THAT IS FOR ANY A B IN S  THE ELEMENTS AB AND BA ARE ALSO IN S INDEXCLOSEDOPERATIONITEM THERE IS AN IDENTITY ELEMENT EIN S SUCH THAT FOR ANY AIN  S  INDEXIDENTITY ELEMENT AE  EA  ATHAT IS THE IDENTITY ELEMENT LEAVES EVERY ELEMENT UNCHANGED UNDER THEOPERATION ITEM FOR EVERY ELEMENT A IN S THERE IS AN ELEMENT B IN S CALLED  ITS EM INVERSE SUCH THAT AB  E QQUAD BA  EITEM THE BINARY OPERATION IS ASSOCIATIVE INDEXASSOCIATIVE  FOR  EVERY AB C IN S ABC  ABCENDENUMERATEWE DENOTE THE GROUP BY LA SRAENDDEFINITIONIF IT IS TRUE THAT AB  BA FOR EVERY AB IN S THEN THE GROUPIS SAID TO BE A BF COMMUTATIVE  INDEXCOMMUTATIVEINDEXABELIAN IF THE OPERATION  IS AN ADDITION OPERATOR ACOMMUTATIVE GROUP IS REFERRED TO AS AN BF ABELIAN GROUP  NOTE THATIT IS NOT NECESSARY FOR EVERY GROUP TO BE COMMUTATIVEEXAMPLES OF GROUPSBEGINENUMERATEITEM THE INTEGERS UNDER ADDITION  THE GROUP IS DENOTED LA  ZBBRA  NOTE THAT THE INTEGERS UNDER MULTIPLICATION DO EM  NOT FORM A GROUP THERE IS NO MULTIPLICATIVE INVERSE FOR EVERY ELEMENTITEM THE INTEGERS MODULO 7 UNDER ADDITION  THIS GROUP IS DENOTED LA  ZBB7RA  ALSO THE INTEGERS MODULO 7 UNDER MULTIPLICATION  DENOTED AS LA ZBB7CDOTRA  HOWEVER INTEGERS MODULO 6 UNDER  MULTIPLICATION DO NOT FORM A GROUP THERE IS NO MULTIPLICATIVE  INVERSE TO THE NUMBER 2 FOR EXAMPLEITEM THE SET OF REAL NUMBERS UNDER EITHER ADDITION OR MULTIPLICATION  LA RBBRA OR LA RBBCDOT RAITEM THE SET OF POLYNOMIALS WITH COEFFICIENTS FROM A GROUPENDENUMERATESUBSECTIONRINGSLABELSECRINGSBEGINDEFINITION LABELDEFRINGINDEXRING A SET R EQUIPPED WITH TWO OPERATIONS WHICH WE WILLDENOTE AS  AND  IS A RING IF IT SATISFIES THE FOLLOWINGBEGINENUMERATER1ITEM LA RRA IS AN ABELIAN GROUPITEM THE OPERATION  IS ASSOCIATIVEITEM LEFT AND RIGHT DISTRIBUTED LAWS HOLD  FOR ALL ABC IN R ABC  ABAC QQUADQQUAD ABC  ACBCENDENUMERATEWE DENOTE THE RING BY LARRAENDDEFINITIONWE NOTE IN PARTICULAR THAT EM MULTIPLICATIVE INVERSES NEED NOT EXIST  IN A RING  IN FACT THE RING MIGHT NOT EVEN HAVE A MULTIPLICATIVE  IDENTITY  THE OPERATOR  IS NOT NECESSARILY COMMUTATIVE NOR IS AN IDENTITY ORINVERSE REQUIRED FOR THE OPERATION   IF THERE IS AN ELEMENT 1 IN R SUCH THAT FOR ANY R IN R 1R  R1  RTHIS ELEMENT IS SAID TO BE AN IDENTITY AND THE RING IS SAID TO BE A RINGWITH IDENTITYEXAMPLES OF RINGSBEGINENUMERATEITEM THE SET OF SQUARE MATRICES WITH REAL ELEMENTS NOT COMMUTATIVE  MULTIPLICATION HAS AN IDENTITY NOT EVERY MATRIX HAS AN INVERSEITEM THE SET OF RATIONAL NUMBERS QBBITEM THE SET OF REAL NUMBERS RBBITEM THE SET OF COMPLEX NUMBERS CBBITEM THE SET OF POLYNOMIALS WHOSE COEFFICIENTS COME FROM A RING  COMMUTATIVE HAS AN IDENTITY POLYNOMIALS MAY NOT HAVE  MULTIPLICATIVE INVERSESITEM THE SET OF POLYNOMIALS WITH MULTIPLICATION DONE MODULO ANOTHER  POLYNOMIALITEM INTEGERS MODULO 6 LA ZBB6CDOTRA  NOT EVERY ELEMENT  HAS A MULTIPLICATIVE INVERSEENDENUMERATESUBSECTIONFIELDSLABELSECFIELDS FIELDS INCORPORATE THE ALGEBRAIC OPERATIONS WE ARE FAMILIAR WITHFROM WORKING WITH REAL AND COMPLEX NUMBERSBEGINDEFINITION LABELDEFFIELD  A F EQUIPPED WITH TWO OPERATIONS  AND  IS A FIELD IF IT  SATISFIES THE FOLLOWINGBEGINENUMERATEF1ITEM LA FRA IS AN ABELIAN GROUPITEM THE SET F EXCLUDING 0 THE ADDITIVE IDENTITY IS A COMMUTATIVE  GROUP UNDER ITEM THE OPERATIONS  AND  DISTRIBUTEENDENUMERATEWE MAY DENOTE THE FIELDS AS LA FCDOTRAENDDEFINITIONEXAMPLES OF FIELDSBEGINENUMERATEITEM THE FAMILIAR OPERATIONS ON THE RATIONALS REALS AND COMPLEX  NUMBERS ITEM THE INTEGERS MODULO 2  THIS FORMS A FIELD THAT ARISES  FREQUENTLY IN DIGITAL OPERATIONS SINCE THE ELEMENTS ARE EITHER 0 OR  1  THIS FIELD IS REFERRED TO AS GF2  INDEXGF2GF2ITEM THE INTEGERS MODULO 7 LA ZBB7CDOTRA  THIS FORMS A  EM FINITE FIELD IT TURNS OUT WE WONT SHOW THIS HERE THAT  FIELD HAVING A FINITE NUMBER OF ELEMENTS HAS PM ELEMENTS IN IT  WHERE P IS PRIME  HOWEVER IF M1 THE OPERATIONS ARE NOT DONE  SIMPLY USING OPERATIONS MODULO PMITEM AS AN EXAMPLE OF SOMETHING EM NOT A FIELD INTEGER OPERATIONS  MODULO 4 DOES NOT FORM A FIELDENDENUMERATESECTIONVECTOR SPACESLABELSECVS1A FINITEDIMENSIONAL VECTOR XBF MAY BE WRITTEN AS XBF  LEFTBEGINARRAYC X1X2 VDOTS XNENDARRAYRIGHTTHE ELEMENTS OF THE VECTOR ARE XI I12LDOTSN  EACH OF THEELEMENTS OF THE VECTOR LIES IN SOME SET SUCH AS THE SET OF REALNUMBERS XI IN RBB OR THE SET OF INTEGERS XI IN ZBB   THISSET OF NUMBERS IS CALLED THE SET OF SCALARS OF THE VECTOR SPACETHE FINITEDIMENSIONAL VECTOR REPRESENTATION IS WIDELY USEDESPECIALLY FOR DISCRETETIME SIGNALS IN WHICH THE DISCRETETIMESIGNAL COMPONENTS FORM ELEMENTS IN A VECTOR  HOWEVER FORREPRESENTING AND ANALYZING CONTINUOUSTIME SIGNALS A MOREENCOMPASSING UNDERSTANDING OF VECTOR CONCEPTS IS USEFUL  IT ISPOSSIBLE TO REGARD THE FUNCTION XT AS A VECTOR AND TO APPLY MANYOF THE SAME TOOLS TO THE ANALYSIS OF XT THAT MIGHT BE APPLIED TOTHE ANALYSIS OF A MORE CONVENTIONAL VECTOR XBF  WE WILL THEREFOREUSE THE SYMBOL X OR XT ALSO TO REPRESENT VECTORS AS WELL ASTHE SYMBOL XBF PREFERRING THE SYMBOL XBF FOR THE CASE OFFINITEDIMENSIONAL VECTORS  ALSO IN INTRODUCING NEW VECTOR SPACECONCEPTS VECTORS ARE INDICATED IN BOLD FONT TO DISTINGUISH THEVECTORS FROM THE SCALARS  NOTE IN HANDWRITTEN NOTATION SUCH AS ON ABLACKBOARD THE BOLD FONT IS USUALLY DENOTED IN THE SIGNAL PROCESSINGCOMMUNITY BY AN UNDERSCORE AS IN XUL OR FOR BREVITY BY NOADDITIONAL NOTATION  DENOTING HANDWRITTEN VECTORS WITH ASUPERSCRIPTED ARROW VECX IS MORE COMMON IN THE PHYSICS COMMUNITYBEGINDEFINITION  A BF LINEAR VECTOR SPACE INDEXVECTOR SPACE S OVER A SET OF  SCALARS R IS A COLLECTION OF OBJECTS KNOWN AS VECTORS TOGETHER  WITH AN ADDITIVE OPERATION  AND A SCALAR MULTIPLICATION OPERATION  CDOT THAT SATISFY THE FOLLOWING PROPERTIESBEGINENUMERATEVS1ITEM S FORMS A GROUP INDEXGROUP UNDER ADDITION   THAT IS  THE FOLLOWING PROPERTIES ARE SATISFIED  BEGINENUMERATE  ITEM FOR ANY XBF AND YBF IN S XBF  YBF IN S  THE    ADDITION OPERATION IS CLOSEDFOOTNOTEA CLOSED OPERATION IS A      DISTINCT CONCEPT FROM A CLOSED SET  A CLOSED BINARY OPERATION      INDEXCLOSED OPERATION  ON A SET S IS SUCH THAT FOR ANY      X Y IN S THEN XY IN SITEM THERE IS AN IDENTITY ELEMENT IN S WHICH WE WILL DENOTE AS  ZEROBF SUCH THAT FOR ANY XBF IN S XBF  ZEROBF  ZEROBF  XBF  XBFITEM FOR EVERY ELEMENT XBF IN S THERE IS ANOTHER ELEMENT YBF  IN S SUCH THAT XBF  YBF  ZEROBFTHE ELEMENT YBF IS THE ADDITIVE INVERSE OF XBF AND IS USUALLYDENOTED AS XBFITEM THE ADDITION OPERATION IS ASSOCIATIVE FOR ANY XBF YBF AND  ZBF IN S XBFYBF ZBF  XBF  YBFZBF  ENDENUMERATEITEM FOR ANY A B IN R AND ANY XBF AND YBF IN S  AXBF IN S  ABXBF  ABXBF  ABXBF  A XBF  BXBF  AXBF  YBF  A XBF  A YBFITEM THERE IS A MULTIPLICATIVE IDENTITY ELEMENT 1 IN R SUCH THAT  1XBF  XBF  THERE IS AN ELEMENT 0 IN R SUCH THAT 0XBF  0ENDENUMERATETHE SET R IS THE SET OF SCALARS OF THE VECTOR SPACEENDDEFINITIONTHE SET OF SCALARS IS MOST FREQUENTLY TAKEN TO BE THE SET OF REALNUMBERS OR COMPLEX NUMBERS  HOWEVER IN SOME APPLICATIONS OTHER SETSOF SCALARS ARE USED SUCH AS POLYNOMIALS OR NUMBERS MODULO 256  THEONLY REQUIREMENT ON THE SET OF SCALARS IS THAT THE OPERATIONS OFADDITION AND MULTIPLICATION CAN BE USED AS USUAL ALTHOUGH NOMULTIPLICATIVE INVERSE IS NEEDED AND THAT THERE IS A NUMBER 1 THATIS A MULTIPLICATIVE IDENTITY  IN THIS CHAPTER WHEN WE TALK ABOUTISSUES SUCH AS CLOSED SUBSPACES COMPLETE SUBSPACES AND SO FORTH ITIS ASSUMED THAT THE SET OF SCALARS IS EITHER THE REAL NUMBERS RBBOR THE COMPLEX NUMBERS CBB SINCE THESE ARE COMPLETEWE WILL REFER INTERCHANGEABLY TO EM LINEAR VECTOR SPACE OR EM  VECTOR SPACEBEGINEXAMPLE THE MOST FAMILIAR VECTOR SPACE IS RBBN THE SET OF  NTUPLES INDEXRNRBBN FOR EXAMPLE  IF XBF1 XBF2 IN RBB4 AND XBF1  BEGINBMATRIX 1  5  4  2 ENDBMATRIXQQUADQQUADXBF2  BEGINBMATRIX 5  2   0  2 ENDBMATRIXTHEN XBF1  XBF2  BEGINBMATRIX6  7  4  0 ENDBMATRIXQQUADQQUAD3 XBF1  2 XBF2  BEGINBMATRIX1319122 ENDBMATRIXENDEXAMPLESEVERAL OTHER FINITEDIMENSIONAL VECTOR SPACES EXIST OF WHICH WEMENTION A FEWBEGINEXAMPLE   BEGINENUMERATE  ITEM THE SET OF MATSIZEMN MATRICES WITH REAL ELEMENTS  ITEM THE SET OF POLYNOMIALS OF DEGREE UP TO N WITH REAL COEFFICIENTS  ITEM THE SET OF POLYNOMIALS WITH REAL COEFFICIENTS WITH THE USUAL    ADDITION AND MULTIPLICATION MODULO THE POLYNOMIAL PT  1T8    FORMS A LINEAR VECTOR SPACE  WE DENOTE THIS VECTOR SPACE AS    RBBTT81  ENDENUMERATEENDEXAMPLEIN ADDITION TO THESE EXAMPLES WHICH WILL BE SHOWN SUBSEQUENTLY TOHAVE FINITE DIMENSIONALITY THERE ARE MANY IMPORTANT VECTOR SPACESTHAT ARE INFINITEDIMENSIONAL IN A MANNER TO BE MADE PRECISE BELOWBEGINEXAMPLE   BEGINENUMERATE  ITEM LP THE SET OF ALL INFINITELYLONG SEQUENCES XN FORMS AN    INFINITEDIMENSIONAL VECTOR SPACE  ITEM CAB THE SET OF CONTINUOUS FUNCTIONS DEFINED    OVER THE INTERVAL AB FORMS A VECTOR SPACE  INDEXCABCAB  ITEM LPAB THE FUNCTIONS IN LP FORM THE ELEMENTS OF AN    INFINITEDIMENSIONAL VECTOR SPACEENDENUMERATEENDEXAMPLEBEGINDEFINITION  LET S BE A VECTOR SPACE  IF V SUBSET S IS A SUBSET SUCH THAT  V IS ITSELF A VECTOR SPACE THEN V IS SAID TO BE A BF    SUBSPACE OF SENDDEFINITIONBEGINEXAMPLE   BEGINENUMERATE  ITEM LET S BE THE SET OF ALL POLYNOMIALS AND LET V BE THE SET    OF POLYNOMIALS OF DEGREE LESS THAN 6  THE V IS A SUBSPACE OF    S  ITEM LET S CONSIST OF THE SET OF 5TUPLES S   00000010011000111000AND LET V BE THE SET V   0000001001 WHERE THE ADDITION IS DONE MODULO 2  THEN S IS A VECTOR SPACECHECK THIS AND V IS A SUBSPACE  ENDENUMERATEENDEXAMPLETHROUGHOUT THIS CHAPTER AND THE REMAINDER OF THE BOOK WE WILL USEINTERCHANGEABLY THE WORDS VECTOR AND SIGNAL  FOR ADISCRETETIME SIGNAL WE MAY THINK OF THE VECTOR COMPOSED OF THESAMPLES OF THE FUNCTION AS A VECTOR IN RBBN OR CBBN  FOR ACONTINUOUSTIME SIGNAL ST THE VECTOR IS THE SIGNAL ITSELF ANELEMENT OF A SPACE SUCH AS LPAB  THUS BOXEDTEXTTHE STUDY OF SIGNALS IS THE STUDY OF VECTOR SPACESBEGINEXERCISES  ITEM SHOW THAT IF V AND W ARE SUBSPACES OF A VECTOR SPACE S  THEN THEIR INTERSECTION V CAP W IS A SUBSPACEITEM SHOW THAT IF V AND W ARE SUBSPACES OF A VECTOR SPACE S  THEN THEIR SUM VW IS A SUBSPACEENDEXERCISESSUBSECTIONLINEAR COMBINATIONS OF VECTORSLET S BE A VECTOR SPACE OVER R AND LETPBF1PBF2LDOTSPBFM BE VECTORS IN S THEN FOR CI IN RTHE LINEAR COMBINATION XBF  C1PBF1  C2 PBF2  LDOTS CM PBFM IS IN S  THE SET OF VECTORS PBFI CAN BE REGARDED AS EM  BUILDING BLOCKS OR INGREDIENTS FOR OTHER SIGNALS AND THE LINEARCOMBINATION SYNTHESIZES XBF FROM THESE COMPONENTS  IF THE SET OFINGREDIENTS IS SUFFICIENTLY RICH THAN A WIDE VARIETY OF SIGNALSVECTORS CAN BE CONSTRUCTED  IF THE INGREDIENT VECTORS ARE KNOWNTHEN THE VECTOR XBF IS ENTIRELY CHARACTERIZED BY THE REPRESENTATIONC1C2LDOTSCM SINCE KNOWING THESE TELLS HOW TO SYNTHESIZEXBF  BEGINDEFINITION LABELDEFLC  LET S BE A VECTOR SPACE OVER R AND LET T SUBSET S PERHAPS  WITH INFINITELY MANY ELEMENTS  A POINT XBF IN S IS SAID TO BE  A BF LINEAR COMBINATION INDEXLINEAR COMBINATION OF POINTS IN  T IF THERE IS A EM FINITE SET OF POINTS PBF1PBF2LDOTS  PBFM IN T AND A FINITE SET OF SCALARS C1C2LDOTS CM IN  R SUCH THAT XBF  C1 PBF1  C2 PBF2  CDOTS  CM PBFMENDDEFINITIONIT IS SIGNIFICANT THAT THE LINEAR COMBINATION ENTAILS ONLY A FINITESUMBEGINEXAMPLE LABELEXMLINCOMB1LET S  CRBB THE SET OF CONTINUOUS FUNCTIONS  DEFINED ON THE REAL NUMBERS  LET P1T  1 P2T  T AND  P3T  T2  THEN A LINEAR COMBINATION OF THESE FUNCTIONS IS XT  C1  C2T  C3 T2 THESE FUNCTIONS CAN BE USED AS BUILDING BLOCKS TO CREATE ANYSECONDDEGREE POLYNOMIAL  AS WILL BE SEEN IN THE FOLLOWING THEREARE FUNCTIONS BETTER SUITED TO THE TASK OF BUILDING POLYNOMIALSIF THE FUNCTION P4T  T2  1 IS ADDED TO THE SET OF FUNCTIONSTHEN OTHER FUNCTIONS OF THE FORM XT  C1  C2 T  C3T2  C4 T21  C1C4  A2 T C3C4 T2CAN BE CONSTRUCTED WHICH IS STILL JUST A QUADRATIC POLYNOMIAL  THATIS THE NEW FUNCTION DOES NOT EXPAND THE SET OF FUNCTIONS THAT CAN BECONSTRUCTED SO P4T IS IN SOME SENSE REDUNDANT  THIS MEANSTHAT THERE IS MORE THAN ONE WAY TO REPRESENT A POLYNOMIAL  FOREXAMPLE THE POLYNOMIAL XT  6  5T  T2CAN BE REPRESENTED AS XT  8P1T  5 P2T   P3T  2P4T OR AS XT  9P1T  5 P2T   2P3T  3P4T ENDEXAMPLEBEGINEXAMPLELET PBF1PBF2 IN RBB3 WITH PBF1  101T  PBF2  110T  THEN XBF  C1XBF1  C2 XBF2  LEFTBEGINARRAYCC1C2  C2 1ENDARRAYRIGHT THE SET OF VECTORS THAT CAN BE CONSTRUCTED WITH PBF1PBF2DOES NOT COVER THE SET OF ALL VECTORS IN RBB3  FOR EXAMPLE THEVECTOR XBF  BEGINBMATRIX 5  2  6 ENDBMATRIXCANNOT BE FORMED AS A LINEAR COMBINATION OF PBF1 AND PBF2ENDEXAMPLESEVERAL QUESTIONS RELATED TO LINEAR COMBINATIONS ARE ADDRESSED IN THISAND SUCCEEDING SECTIONS AMONG THEMBEGINITEMIZEITEM IS THE REPRESENTATION OF A VECTOR AS A LINEAR COMBINATION OF  OTHER VECTORS UNIQUEITEM WHAT IS THE SMALLEST SET OF VECTORS THAT CAN BE USED TO  SYNTHESIZE ANY VECTOR IN SITEM GIVEN THE SET OF VECTORS PBF1PBF2LDOTSPBFM HOW ARE  THE COEFFICIENTS C1C2LDOTSCM FOUND TO REPRESENT THE VECTOR  XBF IF IN FACT IT CAN BE REPRESENTEDITEM WHAT ARE THE REQUIREMENTS ON THE VECTORS PBFI IN  ORDER TO BE ABLE TO SYNTHESIZE ANY VECTOR X IN SITEM SUPPOSE THAT XBF CANNOT BE REPRESENTED EXACTLY USING THE SET OF  VECTORS PBFI  WHAT IS THE BEST APPROXIMATION THAT CAN BE MADE  WITH A GIVEN SET OF VECTORSENDITEMIZEIN THIS CHAPTER WE EXAMINE THE FIRST TWO QUESTIONS LEAVING THEREST OF THE QUESTIONS TO THE APPLICATIONS OF THE NEXT CHAPTERSUBSECTIONLINEAR INDEPENDENCEWE WILL FIRST EXAMINE THE QUESTION OF THE UNIQUENESS OF THEREPRESENTATION AS A LINEAR COMBINATIONBEGINDEFINITION  LABELDEFLININD  LET S BE A VECTOR SPACE AND LET T BE A SUBSET OF S  THE SET  T IS BF LINEARLY INDEPENDENT IF FOR EACH FINITE NONEMPTY SUBSET  OF T SAY PBF1PBF2LDOTSPBFM THE ONLY SET OF  SCALARS SATISFYING THE EQUATION C1PBF1  C2PBF2  LDOTS  CMPBFM  0 IS THE TRIVIAL SOLUTION C1  C2  CDOTS  CM 0 INDEXLINEARLY INDEPENDENT  THE SET OF VECTORS PBF1PBF2LDOTSPBFM IS SAID TO BE BF   LINEARLY DEPENDENT IF THERE EXISTS A SET OF SCALAR COEFFICIENTS C1C2LDOTSCM WHICH ARE NOT ALL ZERO SUCH THAT C1PBF1  C2PBF2  LDOTS  CMPBFM  0 ENDDEFINITIONBEGINEXAMPLE   BEGINENUMERATE  ITEM THE FUNCTIONS P1T P2T P3T P4T IN S OF    EXAMPLE REFEXMLINCOMB1 ARE LINEARLY DEPENDENT BECAUSE P4T  P1T  P3T  0THAT IS THERE IS A NONZERO LINEAR COMBINATION OF THE FUNCTIONS WHICHIS EQUAL TO ZEROITEM THE VECTORS PBF1  234T PBF2  162 AND  PBF3  162T ARE LINEARLY DEPENDENT SINCE 4PBF1  5PBF2  3PBF3  0ITEM THE FUNCTIONS P1T  T AND P2T  1T ARE LINEARLY  INDEPENDENT  ENDENUMERATEENDEXAMPLEBEGINDEFINITION  LET T BE A SET OF VECTORS IN A VECTOR SPACE S OVER A SET OF  SCALARS R THE NUMBER OF VECTORS IN T COULD BE INFINITE  THE  SET OF VECTORS V THAT CAN BE REACHED BY ALL POSSIBLE FINITE  LINEAR COMBINATIONS OF VECTORS IN T IS THE BF SPAN OF THE  VECTORS  THIS IS DENOTED BY V  LSPANTTHAT IS FOR ANY XBF IN V THERE IS SOME SET OF COEFFICIENTSCI IN R SUCH THAT XBF  SUMI1M CI PBFI WHERE EACH PBFI IN TENDDEFINITIONIT MAY BE OBSERVED THAT V IS A SUBSPACE OF S  WE ALSO OBSERVETHAT V  LSPANT IS THE SMALLEST SUBSPACE OF S CONTAINING TIN THE SENSE THAT FOR EVERY SUBSPACE MSUBSET S SUCH THAT T SUBSETM THEN V SUBSET MTHE SPAN OF A SET OF VECTORS CAN BE THOUGHT OF AS A LINE IF ITOCCUPIES ONE DIMENSION OR AS A PLANE IF IT OCCUPIES TWO DIMENSIONSOR AS A HYPERPLANE IF IT OCCUPIES MORE THAN TWO DIMENSIONS  IN THISBOOK WE WILL SPEAK OF THE EM PLANE SPANNED BY A SET REGARDLESS OFITS DIMENSIONALITYBEGINEXAMPLE BEGINENUMERATEITEM LET PBF1  110T AND PBF2  010T BE IN  RBB3  LINEAR COMBINATIONS OF THESE VECTORS ARE XBF  BEGINBMATRIXC1C2  C2  0 ENDBMATRIXFOR C1C2 IN RBB  THE SPACE U  LSPANPBF1PBF2 IS ASUBSET OF THE SPACE RBB3 IT IS THE PLANE IN WHICH THE VECTORS110T AND 010T LIE WHICH IS THE XY PLANE IN THE USUALCOORDINATE SYSTEM AS SHOWN IN FIGURE REFFIGPLAN1BEGINFIGUREHTBP  BEGINCENTERINPUTPICTUREDIRPLAN1  CAPTIONA SUBSPACE OF RBB3  LABELFIGPLAN1ENDCENTERENDFIGUREITEM LET P1T  1  T AND P2T  T  THEN V   LSPANP1P2 IS THE SET OF ALL POLYNOMIALS UP TO DEGREE 1  THE  SET V COULD BE ENVISIONED ABSTRACTLY AS A PLANE LYING IN THE  SPACE OF ALL POLYNOMIALSENDENUMERATEENDEXAMPLEBEGINDEFINITION  LET  T BE A SET OF VECTORS IN A VECTOR SPACE S AND LET V  SUBSET S BE A SUBSPACE  IF EVERY VECTOR XBF IN V  CAN BE WRITTEN AS A LINEAR COMBINATION OF VECTORS IN T THEN T  IS A BF SPANNING SET OF VENDDEFINITIONBEGINEXAMPLE   BEGINENUMERATE  ITEM THE VECTORS PBF1  1 6 5T PBF2  242T    PBF3  110T PBF4  752T FORM A SPANNING SET OF    RBB3  ITEM THE FUNCTIONS P1T  1T P2T  1T2 P3T     T2 AND P4T  2 FORM A SPANNING SET OF THE SET OF    POLYNOMIALS UP TO DEGREE 2  ENDENUMERATEENDEXAMPLELINEAR INDEPENDENCE PROVIDES US WITH WHAT WE NEED FOR A UNIQUEREPRESENTATION AS A LINEAR COMBINATION AS THE FOLLOWING THEOREMSHOWSBEGINTHEOREM LABELTHMUNIQBAS  LET S BE A VECTOR SPACE AND LET T BE A NONEMPTY SUBSET OF S  THE SET T IS LINEARLY INDEPENDENT IF AND ONLY IF FOR EACH NONZERO  XBF IN LSPANT THERE IS EXACTLY ONE FINITE SUBSET OF T  WHICH WE WILL DENOTE AS PBF1PBF2LDOTSPBFM AND A  UNIQUE SET OF SCALARS C1C2LDOTSCM SUCH THAT XBF  C1 PBF1  C2 PBF2  CDOTS  CM PBFMENDTHEOREMBEGINPROOF  WE WILL FIRST SHOW THAT T LINEARLY INDEPENDENT IMPLIES A UNIQUE  REPRESENTATION  SUPPOSE THAT THERE ARE TWO SETS OF VECTORS IN T PBF1 PBF2 LDOTS PBFM QQUAD TEXTANDQQUAD    QBF1 QBF2 LDOTS QBFNAND CORRESPONDING NONZERO COEFFICIENTS SUCH THAT XBF  C1 PBF1  C2 PBF2  CDOTS  CM PBFM QQUADTEXTANDQQUADXBF  D1 QBF1  D2 QBF2  CDOTS  DN QBFNWE NEED TO SHOW THAT NM AND PBFI  QBFI FOR I12LDOTSMAND THAT CI  DI  WE NOTE THAT C1 PBF1  C2 PBF2  CDOTS  CM PBFM  D1 QBF1  D2QBF2  CDOTS  DN QBFN  0SINCE C1 NEQ 0 BY THE DEFINITION OF LINEAR INDEPENDENCE THEVECTOR PBF1 MUST BE AN ELEMENT OF THE SET QBF1QBF2LDOTSQBFN AND THE CORRESPONDING COEFFICIENTS MUST BE EQUAL SAYPBF1  QBF1 AND C1  D1  SIMILARLY SINCE C2 NEQ 0 WECAN SAY THAT PBF2  QBF2 AND C2  D2  PROCEEDING SIMILARLYWE MUST HAVE PBFIQBFI FOR I12LDOTSM AND CI  DI  CONVERSELY SUPPOSE THAT FOR EACH XBF IN LSPANT THEREPRESENTATION XBF  C1 PBF1  CDOTS CM PBFM IS UNIQUEASSUME TO THE CONTRARY THAT T IS LINEARLY DEPENDENT SO THAT THEREARE VECTORS PBF1PBF2LDOTS PBFM SUCH THATBEGINEQUATION PBF1   A2PBF2 A3 PBF3  CDOTS  AM  PBFMLABELEQLININD2ENDEQUATIONBUT THIS GIVES TWO REPRESENTATIONS OF THE VECTOR PBF1 ITSELF ANDTHE LINEAR COMBINATION  REFEQLININD2  SINCE THIS CONTRADICTSTHE UNIQUE REPRESENTATION T MUST BE LINEARLY INDEPENDENTENDPROOFSUBSECTIONBASIS AND DIMENSIONLABELSECHAMELBASISUP TO THIS POINT WE HAVE USED THE TERM DIMENSION FREELY ANDWITHOUT A FORMAL DEFINITION  WE HAVE NOT CLARIFIED WHAT IS MEANT BYFINITEDIMENSIONAL AND INFINITEDIMENSIONAL VECTOR SPACESIN THIS SECTION WE AMEND THIS OMISSION BY DEFINING THE HAMEL BASISOF A VECTOR SPACE  BEGINDEFINITION  INDEXHAMEL BASIS  LET S BE A VECTOR SPACE AND LET T BE A SET OF VECTORS FROM S  SUCH THAT LSPANT  S  IF T IS LINEARLY INDEPENDENT THEN T  IS SAID TO BE A BF HAMEL BASIS FOR SENDDEFINITIONBEGINEXAMPLE BEGINENUMERATEITEM THE SET OF VECTORS IN THE LAST EXAMPLE IS NOT LINEARLY  INDEPENDENT SINCE   4 PBF1  5 PBF2 21 PBF3  5 PBF4  0HOWEVER THE SET T  PBF1PBF2PBF3 IS LINEARLYINDEPENDENT AND SPANS THE SPACE RBB3  HENCE T IS A HAMELBASIS FOR RBB3ITEM THE VECTORS EBF1  BEGINBMATRIX1  0  0 ENDBMATRIXQQUADEBF2  BEGINBMATRIX 0  1  0  ENDBMATRIXQQUADEBF3  BEGINBMATRIX 0  0  1 ENDBMATRIXFORM ANOTHER HAMEL BASIS FOR RBB3  THIS BASIS IS OFTEN CALLEDTHE BF NATURAL BASISITEM THE VECTORS P1T  1 P2TT  P3T  T2 FORM A  HAMEL BASIS FOR THE SET S  MBOXALL POLYNOMIALS OF DEGREE LEQ 2ANOTHER HAMEL BASIS FOR S IS THE SET OF POLYNOMIALS Q1T  2Q2T  TT2 Q3T  TENDENUMERATEENDEXAMPLEAS THIS EXAMPLE SHOWS THERE IS NOT NECESSARILY A UNIQUE HAMEL BASISFOR A VECTOR SPACE  HOWEVER THE FOLLOWING THEOREM SHOWS THAT EVERYBASIS FOR A VECTOR SPACE HAVE A COMMON ATTRIBUTE THE CARDINALITY ORNUMBER OF ELEMENTS IN THE BASISBEGINTHEOREM LABELTHMBASISSAME  IF T1 AND T2 ARE HAMEL BASES FOR A VECTOR SPACE S THEN  T1 AND T2 HAVE THE SAME CARDINALITYENDTHEOREMTHE PROOF OF THIS THEOREM IS SPLIT INTO TWO PIECES THEFINITEDIMENSIONAL CASE AND THE INFINITEDIMENSIONAL CASE  THELATTER MAY BE OMITTED ON A FIRST READINGBEGINPROOF  FINITEDIMENSIONAL CASE SUPPOSE  T1  PBF1PBF2LDOTS PBFMQQUADTEXTANDQQUAD T2  QBF1QBF2LDOTSQBFNARE TWO HAMEL BASES OF S  EXPRESS THE POINT QBF1 IN T2 AS QBF1  C1 PBF1  C2 PBF2  CDOTS  CM PBFMAT LEAST ONE OF THE COEFFICIENTS CI MUST BE NONZERO LET US TAKETHIS AS C1  WE CAN THEN WRITE PBF1  FRAC1C1QBF1  C2 PBF2  CDOTS  CM PBFMBY THIS MEANS WE CAN ELIMINATE PBF1 AS A BASIS VECTOR IN T1 ANDUSE INSTEAD THE SET QBF1ALLOWBREAK PBF2ALLOWBREAK LDOTSALLOWBREAK PBFM AS A BASISSIMILARLY WE WRITE QBF2  D1 QBF1  D2 PBF2  CDOTS  DM PBFMAND AS BEFORE ELIMINATE PBF2 SO THAT  QBF1QBF2PBF3LDOTS PBFM FORMS A BASIS  CONTINUING IN THIS WAY WECAN ELIMINATE EACH PBFI SHOWING THAT QBF1 LDOTS QBFMSPANS THE SAME SPACE AS PBF1 LDOTS PBFM  WE CAN CONCLUDETHAT M GEQ N  SUPPOSE TO THE CONTRARY THAT N  M  THEN AVECTOR SUCH AS QBFM1 WHICH DOES NOT FALL IN THE BASIS SETQBF1LDOTS QBFM WOULD HAVE TO BE LINEARLY DEPENDENT WITHTHAT SET WHICH VIOLATES THE FACT THAT T2 IS ITSELF A BASISREVERSING THE ARGUMENT WE FIND THAT N GEQ M  IN COMBINATIONTHEN WE CONCLUDE THAT MNINFINITEDIMENSIONAL CASE  LET T1 AND T2 BE BASES  FOR ANXBF IN T1 LET T2XBF DENOTE THE UNIQUE FINITE SET OF POINTSIN T2 NEEDED TO EXPRESS XBF  CLAIM IF YBF IN T2 THEN YBF IN T2XBF FOR SOME XBF INT1  PROOF SINCE A POINT YBF IS IN S THEN YBF MUST BE AFINITE LINEAR COMBINATION OF VECTORS IN T1 SAY YBF  C1 XBF1  C2 XBF2  CDOTS  CM XBFMFOR SOME SET OF VECTORS XBFI IN T1  THEN FOR EXAMPLE XBF1  FRAC1C1YBF  C2 XBF2  CDOTS  CM XBFMSO THAT BY THE UNIQUENESS OF THE REPRESENTATION YBF IN B2XBFSINCE FOR EVERY YBF IN T2 THERE IS SOME XBF IN T1 SUCH THAT YBFIN T2XBF IT FOLLOWS THAT T2  BIGCUPXBF IN T1 T2XBFNOTING THAT THERE ARE T1  INDEX  CDOT INDEXBAR CDOT  SETS IN THIS UNIONFOOTNOTERECALL THAT THE NOTATION S  INDICATES THE CARDINALITY OF THE SET S SEE SECTION  REFSECFUNDAMENTALS EACH OF WHICH CONTRIBUTES AT LEAST ONE  ELEMENT TO T2 WE CONCLUDE THAT T2 GEQ T1  NOW TURNING THE ARGUMENT AROUND WE CONCLUDE THAT T1 GEQ  T2  BY THESE TWO INEQUALITIES WE CONCLUDE THAT T1  T2ENDPROOFON THE STRENGTH OF THIS THEOREM WE CAN STATE A CONSISTENT DEFINITIONFOR THE DIMENSION OF A VECTOR SPACEBEGINDEFINITION  LET T BE A HAMEL BASIS FOR A VECTOR SPACE S  THE CARDINALITY OF  T IS THE BF DIMENSION OF S  THIS IS DENOTED AS  DIMENSIONS  IT IS THE EM NUMBER OF LINEARLY INDEPENDENT    VECTORS REQUIRED TO SPAN THE SPACEENDDEFINITIONSINCE THE DIMENSION OF A VECTOR SPACE IS UNIQUE WE CAN CONCLUDE THATA BASIS T FOR A SUBSPACE S IS A EM SMALLEST SET OF VECTORSWHOSE LINEAR COMBINATIONS CAN FORM EVERY VECTOR IN A VECTOR SPACE SIN THE SENSE THAT A BASIS OF T VECTORS IS CONTAINED IN EVERY OTHERSPANNING SET FOR STHE LAST REMAINING FACT WHICH WE WILL NOT PROVE SHOWS THE IMPORTANCEOF THE HAMEL BASIS EM EVERY VECTOR SPACE HAS A HAMEL BASIS  SOFOR MANY PURPOSES WHATEVER WE WANT TO DO WITH A VECTOR SPACE CAN BEDONE TO THE HAMEL BASISBEGINEXAMPLE  LET S BE THE SET OF ALL POLYNOMIALS  THEN A POLYNOMIAL XT IN  S CAN BE WRITTEN AS A LINEAR COMBINATION OF THE FUNCTIONS  1TT2LDOTS  IT CAN BE SHOWN SEE EXERCISE  REFEXLINIDPOLY THAT THIS SET OF FUNCTIONS IS LINEARLY  INDEPENDENT  HENCE THE DIMENSION OF S IS INFINITEENDEXAMPLEBEGINEXAMPLE  CITEFRIEDMAN TO ILLUSTRATE THAT INFINITE DIMENSIONAL VECTOR  SPACES CAN BE DIFFICULT TO WORK WITH AND THAT PARTICULAR CARE IS  REQUIRED WE DEMONSTRATE THAT FOR AN INFINITEDIMENSIONAL VECTOR  SPACE S AN INFINITE SET OF LINEARLY INDEPENDENT VECTORS WHICH  SPAN S NEED NOT FORM A BASIS FOR S    LET X BE THE INFINITESEQUENCE SPACE WITH ELEMENTS OF THE FORM  X1X2X3LDOTS WHERE EACH XI IN RBB  THE SET OF  VECTORS PBFJ  100LDOTS010LDOTS QQUAD J23LDOTSWHERE THE SECOND 1 IS IN THE JTH POSITION FORMS A SET OF LINEARLYINDEPENDENT VECTORS  WE FIRST SHOW THE SET PBFJJ23LDOTS SPANS X  LET X X1X2X3LDOTS BE AN ARBITRARY ELEMENT OF X  LET  SIGMAN  X1  X2  CDOTS  XNAND LET TAUN BE AN INTEGER LARGER THAN NSIGMAN2  NOWCONSIDER THE SEQUENCE OF VECTORS YBFN  X2 PBF2  X3 PBF3  CDOTS  XN PBFN FRACSIGMANTAUNPBFN1  CDOTS  PBFPWHERE P  NTAUN  FOR EXAMPLE BEGINALIGNEDYBF3  XBF2 PBF2  XBF3 PBF3  FRACX1  X2X3TAUNPBF4  PBF5  CDOTS  PBF4TAUN  XBF2 PBF2  XBF3 PBF3  X1 X2X31FRAC1TAUNFRAC1TAUNLDOTSFRAC1TAUNENDALIGNED IN THE LIMIT AS N RIGHTARROW INFTY THE RESIDUAL TERM BECOMES X1  X2  CDOTS100LDOTSAND YBFNRIGHTARROW XBF  SO THERE IS A REPRESENTATION FOR XBFUSING THIS INFINITE SET OF BASIS FUNCTIONSHOWEVER  THIS IS THE SUBTLE BUT IMPORTANT POINT  THEREPRESENTATION EXISTS AS A RESULT OF A LIMITING PROCESS  THERE IS NOEM FINITE SET OF FIXED SCALARS C2 C3 LDOTS CN SUCH THATTHE SEQUENCE XBF  100LDOTS CAN BE WRITTEN IN TERMS OF THEBASIS FUNCTIONS AS XBF  100LDOTS  C2 PBF2  C3 PBF3  LDOTS  CN PBFNWHEN WE INTRODUCED THE CONCEPT OF LINEAR COMBINATIONS IN DEFINITIONREFDEFLC ONLY EM FINITE SUMS WERE ALLOWED  SINCE REPRESENTINGXBF WOULD REQUIRE AN INFINITE SUM THE SET OF FUNCTIONSPBF2PBF3 LDOTS DOES EM NOT FORM A BASISIT MAY BE OBJECTED THAT IT WOULD BE STRAIGHTFORWARD TO SIMPLY EXPRESSAN INFINITE SUM SUMJ2INFTY C2 PBF2 AND HAVE DONE WITH THEMATTER  BUT DEALING WITH INFINITE SERIES ALWAYS REQUIRES MORE CARETHAN DOES FINITE SERIES SO WE CONSIDER THIS AS A DIFFERENT CASEENDEXAMPLESUBSECTIONFINITEDIMENSIONAL VECTOR SPACES AND MATRIX NOTATIONTHE MAJOR FOCUS OF OUR INTEREST IN VECTOR SPACES WILL BE ONFINITEDIMENSIONAL VECTOR SPACES  EVEN WHEN DEALING WITHINFINITEDIMENSIONAL VECTOR SPACES WE SHALL FREQUENTLY BE INTERESTEDIN FINITEDIMENSIONAL REPRESENTATIONS  IN THE CASE OFFINITEDIMENSIONAL VECTOR SPACES EM WE SHALL REFER TO THE HAMEL  BASIS SIMPLY AS THE BASISONE PARTICULARLY USEFUL ASPECT OF FINITEDIMENSIONAL VECTOR SPACES ISTHAT MATRIX NOTATION CAN BE USED FOR CONVENIENT REPRESENTATION OFLINEAR COMBINATIONS  LET THE MATRIX A BE FORMED BY STACKING THEVECTORS PBF1 PBF2 LDOTSPBFM SIDE BY SIDE  A  BEGINBMATRIXPBF1 PBF2 CDOTSPBFM ENDBMATRIXFOR A VECTOR CBF  BEGINBMATRIXC1  C2  VDOTS  CM ENDBMATRIXTHE PRODUCT XBF  ACBF COMPUTES THE LINEAR COMBINATION XBF  C1 PBF1  C2 PBF2  CDOTS  CM PBFMTHE QUESTION OF THE LINEAR DEPENDENCE OF THE VECTORS PBFI CANBE EXAMINED BY LOOKING AT THE RANK OF THE MATRIX A AS DISCUSSED INSECTION REFSECRANKBEGINEXERCISES  ITEM AN EQUIVALENT DEFINITION FOR LINEAR INDEPENDENCE FOLLOWS  A    SET T IS LINEARLY INDEPENDENT IF FOR EACH VECTOR XBF IN T    XBF IS NOT A LINEAR COMBINATION OF THE POINTS IN THE SET T      XBF THAT IS THE SET T WITH THE VECTOR XBF REMOVED          SHOW THAT THIS DEFINITION IS EQUIVALENT TO THAT OF DEFINITION    REFDEFLININD  ITEM LET S BE A FINITEDIMENSIONAL VECTOR SPACE WITH    DIMENSIONS  M  SHOW THAT EVERY SET CONTAINING M1 POINTS    IS LINEARLY DEPENDENT  ITEM SHOW THAT IF T IS A SUBSET OF A VECTOR SPACE S WITH    LSPANTS SHOW THAT T CONTAINS A HAMEL BASIS OF S  ITEM LET S DENOTE THE SET OF ALL SOLUTIONS OF THE DIFFERENTIAL    EQUATION DEFINED ON C30INFTY  SEE DEFINITION    REFDEFCLASSCK  INDEXCKCLASS CK FRACD3 XDT3  B FRACDX2DT2  C FRACDXDT  DX     0SHOW THAT S HAS DIMENSION 3  ITEM LET S BE L202PI AND LET T BE THE SET OF ALL    FUNCTIONS XNT  EJNT FOR N01LDOTS  SHOW THAT T IS    LINEARLY INDEPENDENT      CONCLUDE THAT L202PI IS AN    INFINITE DIMENSIONAL SPACE    HINT ASSUME THAT C1 EJ N1 T  C2    EJ N2 T  CDOTS  CM EJ NM T  0  DIFFERENTIATE    M1 TIMES  ITEM LABELEXLINIDPOLY SHOW THAT THE SET 1TT2LDOTSTM    IS A LINEARLY INDEPENDENT SET HINT THE FUNDAMENTAL THEOREM OF    ALGEBRA STATES THAT A POLYNOMIAL FX OF DEGREE M HAS EXACTLY    M ROOTS COUNTING MULTIPLICITYKEENER P 3ENDEXERCISESSECTIONNORMS AND NORMED VECTOR SPACESLABELSECNORMVSWHEN DEALING WITH VECTOR SPACES IT IS COMMON TO TALK ABOUT THE LENGTHAND DIRECTION OF THE VECTOR AND THERE IS AN INTUITIVE GEOMETRICCONCEPT AS TO WHAT THE LENGTH AND DIRECTION ARE  THERE ARE A VARIETYOF WAYS OF DEFINING THE LENGTH OF A VECTOR  THE MATHEMATICAL CONCEPTASSOCIATED WITH THE LENGTH OF A VECTOR IS THE BF NORM WHICH ISDISCUSSED IN THIS SECTION  IN SECTION REFSECINNERPROD1 WEINTRODUCE THE CONCEPT OF AN INNER PRODUCT WHICH IS USED TO PROVIDE ANINTERPRETATION OF ANGLE BETWEEN VECTORS AND HENCE DIRECTIONBEGINDEFINITION  LET S BE A VECTOR SPACE WITH ELEMENTS XBF  A REALVALUED  FUNCTION  XBF IS SAID TO BE A BF NORM INDEXNORM IF  XBF SATISFIES THE FOLLOWING PROPERTIES  BEGINENUMERATEN1  ITEM XBF GEQ 0 FOR ANY XBF IN S  ITEM XBF  0 IF AND ONLY IF XBF  ZEROBF  ITEM ALPHA XBF  ALPHA  XBF WHERE ALPHA IS AN    ARBITRARY SCALAR  ITEM XBF  YBF LEQ  XBF   YBF TRIANGLE    INEQUALITY  ENDENUMERATETHE REAL NUMBER  XBF IS SAID TO BE THE NORM OF XBF OR THELENGTH OF XBFENDDEFINITIONTHE TRIANGLE INEQUALITY N4 CAN BE INTERPRETED GEOMETRICALLY USINGFIGURE REFFIGTRIINEQ2 WHERE XBF YBF AND ZBF ARETHE SIDES OF A TRIANGLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRTRIINEQ2    CAPTIONA TRIANGLE INEQUALITY INTERPRETATION    LABELFIGTRIINEQ2  ENDCENTERENDFIGUREA NORM FEELS A LOT LIKE A METRIC BUT ACTUALLY REQUIRES MORESTRUCTURE THAN A METRIC  FOR EXAMPLE THE DEFINITION OF A NORMREQUIRES THAT ADDITION XBF  YBF AND SCALAR MULTIPLICATION ALPHAXBF ARE DEFINED WHICH WAS NOT THE CASE FOR A METRICNEVERTHELESS BECAUSE OF THEIR SIMILAR PROPERTIES NORMS AND METRICSCAN BE DEFINED IN TERMS OF EACH OTHER  FOR EXAMPLE IF  XBF ISA NORM THEN  DXBFYBF   XBF  YBFIS A METRIC  THE TRIANGLE INEQUALITY FOR METRICS IS ESTABLISHED BYNOTING THAT  XBF  YBF   XBF  ZBF  ZBF  XBF LEQ  XBF ZBF   YBF  ZBFTHIS TRICK OF ADDING AND SUBTRACTING THE QUANTITY TO MAKE THE ANSWERCOME OUT RIGHT IS OFTEN USED IN ANALYSIS  ALTERNATIVELY GIVEN AMETRIC D DEFINED ON A VECTOR SPACE A NORM CAN BE WRITTEN AS  XBF  DXBFZEROBFTHE DISTANCE THAT XBF IS FROM THE ORIGIN OF THE VECTOR SPACEBEGINEXAMPLE  BASED UPON THE METRICS WE HAVE ALREADY SEEN WE CAN READILY DEFINE  SOME USEFUL NORMS FOR NDIMENSIONAL VECTORS  BEGINENUMERATE  ITEM THE L1 NORM  XBF1  SUMI1N XI  ITEM THE L2 NORM XBFP  LEFTSUMI1N      XIPRIGHT1P  ITEM THE LINFTY NORM  XBFINFTY  MAXI12LDOTSN    XI  ENDENUMERATEEACH OF THESE NORMS INTRODUCES ITS OWN GEOMETRY  CONSIDER FOREXAMPLE THE UNIT SPHERE DEFINED BY SP   XBF IN RBB2MC  XBFP LEQ 1FIGURE REFFIGSPHERES ILLUSTRATES THE SHAPE OF SUCH SPHERES FORVARIOUS VALUES OF PENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRSPHERES    CAPTIONUNIT SPHERES IN RBB2 UNDER VARIOUS    PROTECTLPROTECTPPROTECT NORMS    LABELFIGSPHERES  ENDCENTERENDFIGUREBEGINEXAMPLE  WE CAN ALSO DEFINE NORMS OF FUNCTIONS DEFINED OVER THE INTERVAL  AB  BEGINENUMERATE  ITEM THE L1 NORM  XT1  INTAB XTDT  ITEM THE LP NORM XT2  LEFTINTAB XTPDT    RIGHT1P FOR 1 LEQ P  INFTY  ITEM THE LINFTY NORM XTINFTY  SUPT IN AB XT  ENDENUMERATEENDEXAMPLETHE LINFTY AND LINFTY NORMS ARE REFERRED TO AS THE EM  UNIFORM NORMSBEGINDEFINITION  A BF NORMED LINEAR SPACE IS A PAIR S CDOT WHERE S IS  A VECTOR SPACE AND  CDOT IS A NORM DEFINED ON S  A NORMED  LINEAR SPACE IS OFTEN DENOTED SIMPLY BY SENDDEFINITIONWHEN DISCUSSING THE METRICAL PROPERTIES OF A NORMED LINEAR SPACE THEMETRIC IS DEFINED IN TERMS OF THE NORM DXBFYBF  XBFYBFBEGINDEFINITION  A VECTOR XBF IS SAID TO BE BF NORMALIZED IF XBF    1 INDEXNORMALIZED VECTOR  IT IS POSSIBLE TO NORMALIZE ANY VECTOR EXCEPT THE ZERO VECTOR YBF   XBFXBF HAS YBF  1  A NORMALIZED VECTOR IS ALSO  REFERRED TO AS A BF UNIT VECTOR INDEXUNIT VECTORENDDEFINITIONWITH A VARIETY OF NORMS TO CHOOSE FROM IT IS NATURAL TO ADDRESS THEISSUE OF WHICH NORM SHOULD BE USED IN A PARTICULAR  OFTEN THE L2OR L2 NORM IS USED FOR REASONS WHICH BECOME CLEAR SUBSEQUENTLYHOWEVER OCCASIONS MAY ARISE IN WHICH OTHER NORMS OR NORMLIKEFUNCTIONS ARE USED  FOR EXAMPLE IN A HIGHSPEED SIGNALPROCESSINGALGORITHM IT MAY BE NECESSARY TO USE THE L1 NORM SINCE IT MAY BEEASIER IN THE AVAILABLE HARDWARE TO COMPUTE AN ABSOLUTE VALUE THAN ITIS TO COMPUTE A SQUARE  OR IN A PROBLEM OF DATA REPRESENTATION OFAUDIO INFORMATION QUANTIZATION IT MAY BE APPROPRIATE TO USE A NORMFOR WHICH THAT REPRESENTATION IS CHOSEN THAT IS BEST AS PERCEIVED BYHUMAN LISTENERS  IDEALLY A NORM THAT MEASURED EXACTLY THE DISTORTIONPERCEIVED BY THE HUMAN EAR WOULD BE DESIRED IN SUCH AN APPLICATIONTHIS IS ONLY APPROXIMATELY ACHIEVABLE SINCE IT DEPENDS UPON SO MANYPSYCHOACOUSTIC EFFECTS OF WHICH ONLY A FEW ARE UNDERSTOOD  SIMILARCOMMENTS COULD BE MADE REGARDING NORMS FOR VIDEO CODING  IN SHORTTHE NORM SHOULD BE CHOSEN THAT IS BEST SUITED TO THE PARTICULARAPPLICATIONTHE EXACT NORM VALUES COMPUTED FOR A VECTOR XBF CHANGE DEPENDING ONTHE PARTICULAR NORM USED BUT A VECTOR THAT IS SMALL WITH RESPECT TOONE NORM IS ALSO SMALL WITH RESPECT TO ANOTHER NORM  NORMS ARE THUSEQUIVALENT IN THE SENSE DESCRIBED IN THE FOLLOWING THEOREMBEGINTHEOREMNORM EQUIVALENCE THEOREM  IF  CDOT  AND   CDOT  ARE TWO NORMS ON RBBN OR CBBN THEN  XBFK  RIGHTARROW 0 TEXT AS  KRIGHTARROW INFTY QUADTEXTIF AND ONLY IF  QUAD  XBFK RIGHTARROW 0 TEXT AS  KRIGHTARROW INFTYENDTHEOREMTHE  PROOF OF THIS THEOREM MAKES USE OF THE CAUCHYSCHWARZ INEQUALITYWHICH IS INTRODUCED IN SECTION REFSECCS  YOU MAY WANT TO COMEBACK TO THIS PROOF AFTER READING THAT SECTIONBEGINPROOFIT SUFFICES TO SHOW THAT THERE ARE CONSTANTS C1 C2  0 SUCH THATBEGINEQUATIONC1  XBF  LEQ  XBF  LEQ C2  XBF LABELEQNORM2ENDEQUATIONTO PROVE REFEQNORM2 IT SUFFICES TO ASSUME THAT  CDOT IS THE L2 NORM  TO SEE THIS OBSERVE THAT IF D1  XBF  LEQ XBF2 LEQ D2  XBF QQUAD TEXTANDQQUAD D1  XBF  LEQ XBF2 LEQ D2  XBF THEN BEGINALIGND1XBF LEQ D2  XBF  INTERTEXTANDD1 XBF   LEQ D2  XBF ENDALIGNSO REFEQNORM2 HOLDS WITH C1  D1D2 AND C2  D2D1LET XBF BE EXPRESSED AS A LINEAR COMBINATION OF BASIS VECTORS XBF  SUMI1N XI EBFITHEN BY THE PROPERTIES OF THE NORM  XBF  LEFT SUMI1N XI EBFI RIGHT LEQ SUMI1NXI  EBFITHE SUM ON THE RIGHT IS SIMPLY THE INNER PRODUCT OF THE VECTORCOMPOSED OF THE MAGNITUDES OF THE XIS AND THE VECTOR COMPOSED OFTHE MAGNITUDES OF THE BASIS VECTORS  BEING AN INNER PRODUCT THECAUCHYSCHWARZ INEQUALITY APPLIES AND   XBF  LEQ  XBF 2LEFTSUMI1N  EBFI  2RIGHT12 LET BETA  LEFTSUMI1N  EBFI  2RIGHT12THEN THE LEFT INEQUALITY OF REFEQNORM2 APPLIES WITH C1 1BETAFOR POINTS XBF ON THE UNIT SPHERE S   XBF  XBF 2 1 THE NORM  CDOT  MUST BE GREATER THAN 0 BY THE PROPERTIES OFNORMS AND HENCE  XBF  GEQ ALPHA FOR SOME ALPHA  0 FORXBF IN S  THEN  XBF   LEFT FRACXBF XBF 2  RIGHT  XBF 2 GEQALPHA  XBF 2SO THE RIGHTHAND INEQUALITY HOLDS WITH C2  1ALPHAENDPROOFFOR EXAMPLEBEGINEQUATIONBEGINSPLIT XBF 2 LEQ XBF1 LEQ SQRTNXBF 2 XBF INFTY LEQ XBF2 LEQ SQRTNXBF INFTY XBF INFTY LEQ XBF1 LEQ NXBF INFTYENDSPLITLABELEQNORMCOMPENDEQUATIONFINALLY WE END WITH AN IMPORTANT DEFINITIONBEGINDEFINITION  FOR A SEQUENCE XN IN A NORMED LINEAR SPACE SPACE S CDOT   IF THERE EXISTS  A NUMBER M INFTY SUCH THAT XN  M QQUAD FORALL NTHEN THE SEQUENCE IS SAID TO BE BF BOUNDED INDEXBOUNDED SEQUENCEENDDEFINITION BEGINDEFINITION   A SEQUENCE XN IS BF MONOTONIC IF  X1 LEQ X2 LEQ X3 LEQ CDOTS  OR  X1 GEQ X2 GEQ X3 GEQ CDOTS  ENDDEFINITION FOR SEQUENCES OVER THE REAL NUMBERS THE FOLLOWING FACT IS CLEAR EVERY BOUNDED MONOTONIC SEQUENCE IS CONVERGENT  SINCE THE SEQUENCE IS BOUNDED THE MONOTONIC SEQUENCE RUNS OUT OF ROOM AND HENCE MUST HAVE A LIMIT POINT WHICH BECAUSE THE SEQUENCE IS MONOTONIC MUST BE UNIQUESUBSECTIONFINITEDIMENSIONAL NORMED LINEAR SPACESTHE NOTION OF A CLOSED SET AND A COMPLETE SET WERE INTRODUCED INSECTION REFSECSEQUENCES   AS POINTED OUT HAVING COMPLETE SETS ISADVANTAGEOUS BECAUSE ALL CAUCHY SEQUENCES CONVERGE SO THATCONVERGENCE OF A SEQUENCE CAN BE ESTABLISHED SIMPLY BY DETERMININGWHETHER A SEQUENCE IS CAUCHYFINITEDIMENSIONAL NORMED LINEAR SPACES HAVE SEVERAL VERY USEFUL PROPERTIESBEGINENUMERATEITEM EVERY FINITEDIMENSIONAL SUBSPACE OF A VECTOR SPACE IS CLOSEDITEM EVERY FINITEDIMENSIONAL SUBSPACE OF A VECTOR SPACE IS COMPLETEITEM IF LMC X RIGHTARROW Y IS A LINEAR OPERATOR AND X IS A  FINITE DIMENSIONAL NORMED VECTOR SPACE THEN L IS CONTINUOUS  THIS IS TRUE EVEN IF Y IS NOT FINITE DIMENSIONAL  AS WE SHALL  SEE IN CHAPTER REFCHAPMATINV THIS MEANS THAT THE OPERATOR IS  ALSO BOUNDEDITEM AS OBSERVED ABOVE DIFFERENT NORMS ARE EQUIVALENT ON RBBN OR  CBBN  IN FACT IN ANY FINITEDIMENSIONAL SPACE ANY TWO NORMS  ARE EQUIVALENTENDENUMERATEA LOT OF THE ISSUES OVER WHICH A MATHEMATICIAN WOULD FRET ENTIRELYDISAPPEAR IN FINITEDIMENSIONAL SPACES  THIS IS PARTICULARLY USEFULSINCE MANY OF THE PROBLEMS OF INTEREST IN SIGNAL PROCESSING ARE FINITEDIMENSIONALWE WILL NOT PROVE THESE USEFUL FACTS HERE  INTERESTED READERS SHOULDCONSULT FOR EXAMPLE CITESECTION 510NAYLORSELLBEGINEXERCISESITEM SHOW THAT IN A NORMED LINEAR SPACE BOXED X  Y LEQ XYITEM USING THE TRIANGLE INEQUALITY SHOW THAT ZX LEQ ZY  YXITEM LET P BE IN THE RANGE 0  P  1 AND CONSIDER THE SPACE  LP01 OF ALL FUNCTIONS WITH   X  INT01 XTPDT  INFTYSHOW THAT X IS NOT A NORM ON LP01  HOWEVER SHOW THATDXY   XY IS A METRIC  HINT FOR A REAL NUMBER ALPHASUCH THAT 0 LEQ ALPHA LEQ 1 NOTE THAT ALPHA LEQ ALPHAP LEQ1ITEM SHOW THAT THE NORM FUNCTION CDOTMC  S RIGHTARROW RBB  IS CONTINUOUS  HINT USE THE TRIANGLE INEQUALITYITEM SHOW THAT A NORM IS A CONVEX FUNCTION  SEE SECTION  REFSECCONVFUNCITEM FOR EACH OF THE INEQUALITY RELATIONSHIPS BETWEEN NORMS IN  REFEQNORMCOMP DETERMINE A VECTOR XBF FOR WHICH EACH  INEQUALITY IS ACHIEVED WITH EQUALITYENDEXERCISESSECTIONINNER PRODUCTS AND INNER PRODUCT SPACESLABELSECINNERPROD1AN INNER PRODUCT IS AN OPERATION ON TWO VECTORS THAT RETURNS A SCALARVALUE  INNER PRODUCTS CAN BE USED TO PROVIDE THE GEOMETRICINTERPRETATION OF THE DIRECTION OF A VECTOR IN AN ARBITRARY VECTORSPACE  THEY CAN ALSO BE USED TO DEFINE A NORM KNOWN AS THE INDUCEDNORMWE WILL DEFINE THE INNER PRODUCT IN THE GENERAL CASE IN WHICH THEVECTOR SPACE S HAS ELEMENTS THAT ARE COMPLEXBEGINDEFINITION  LET S BE A VECTOR SPACE DEFINED OVER A SCALAR  FIELD R  AN BF INNER PRODUCT IS A FUNCTION  LACDOTCDOTRAMC STIMES S RIGHTARROW R WITH THE FOLLOWING  PROPERTIES INDEXINNER PRODUCT  BEGINENUMERATEIP1  ITEM LA XBFYBFRA  OVERLINELA YBFXBFRA WHERE THE    OVERBAR INDICATES COMPLEX CONJUGATION  FOR VECTORS DEFINED OVER A    FIELD OTHER THAN COMPLEX NUMBERS THIS SIMPLIFIES TO LA    XBFYBFRA  LA YBFXBFRA  ITEM LA ALPHAXBFYBF RA  ALPHALA XBFYBFRA  ITEM LA XBFYBFZBFRA  LA XBFZBFRA  LA    YBFZBFRA  ITEM LA XBFXBFRA  0 IF XBF NEQ 0 AND LA XBFXBF    RA  0 IF AND ONLY IF XBF  0  ENDENUMERATEENDDEFINITIONBEGINDEFINITION    A VECTOR SPACE EQUIPPED WITH AN INNER PRODUCT IS CALLED AN BF  INNERPRODUCT SPACE   ENDDEFINITION  INNERPRODUCT SPACES ARE SOMETIMES CALLED PREHILBERT SPACES  WE  ENCOUNTER IN SECTION REFSECHILBERT WHAT A HILBERT SPACE ISTHERE ARE A VARIETY OF WAYS THAT AN INNER PRODUCT CAN BEDEFINED  NOTATIONAL ADVANTAGE AND ALGORITHMIC EXPEDIENCY CAN BEOBTAINED BY SUITABLE SELECTION OF AN INNER PRODUCT  WE BEGIN WITHTHE MOST STRAIGHTFORWARD EXAMPLES OF INNER PRODUCTSBEGINEXAMPLE  FOR FINITEDIMENSIONAL VECTORS XBF YBF IN RBBN THE  CONVENTIONAL INNER PRODUCT BETWEEN THE VECTORS XBF  BEGINBMATRIXX1  X2  VDOTS  XNENDBMATRIXQQUAD TEXTANDQQUAD YBF  BEGINBMATRIX  Y1  Y2  VDOTS   YNENDBMATRIXISBEGINALIGNEDLA XBFYBF RA  X1Y1  X2 Y1  CDOTS  XN YN  SUMI1N XI YI  YBFT XBF  XBFT YBFENDALIGNEDTHIS INNER PRODUCT IS THE BF EUCLIDEAN INNER PRODUCT THIS IS ALSOTHE BF DOT PRODUCT INDEXDOT PRODUCTSEEINNER PRODUCT USED INVECTOR CALCULUS AND IS SOMETIMES WRITTEN LA XBFYBFRA  XBFCDOT YBFIF THE VECTORS ARE IN CBBN WITH COMPLEX ELEMENTS THEN THEEUCLIDEAN INNER PRODUCT IS LA XBFYBFRA  SUMK1N XK OVERLINEYK  YBFH XBF ENDEXAMPLEBEGINEXAMPLE  EXTENDING THE SUM OF PRODUCTS IDEA TO FUNCTIONS THE  FOLLOWING IS AN INNER PRODUCT FOR THE SPACE OF FUNCTIONS DEFINED ON  01  LA XTYTRA  INT01 XTOVERLINEYT DT FOR FUNCTIONS DEFINED OVER RBB AN INNER PRODUCT IS LA XTYTRA  INTINFTYINFTY XTYT DT ENDEXAMPLEBEGINEXAMPLE  CONSIDER A CAUSAL SIGNAL XT WHICH IS PASSED THROUGH A CAUSAL  FILTER WITH IMPULSE RESPONSE HT  THE OUTPUT AT A TIME  T IS YT  XTHTBIGGTT  INT0T XTAUHTTAUDTAULET GTAU  HTTAU  THEN YT  INT0T XTAU GTAU  DTAU  LA X G RAWHERE THE INNER PRODUCT IS LA FG RA  INT0T FTGT  DTSO THE OPERATION OF FILTERING AND TAKING THE OUTPUT AT A FIXED TIMEIS EQUIVALENT TO COMPUTING AN INNER PRODUCTENDEXAMPLEAN INNER PRODUCT CAN ALSO BE DEFINED ON MATRICES  LET S BE THEVECTOR SPACE OF MATSIZEMN MATRICES  THEN WE CAN DEFINE ANINNER PRODUCT ON THIS VECTOR SPACE BY LA AB RA  TRACEAH BBEGINEXERCISES KEENER P 8ITEM COMPUTE THE INNER PRODUCTS LA FG RA FOR THE FOLLOWINGUSING THE DEFINITION LA FG RA  INT01 FTGTDTBEGINENUMERATEITEM FT  T2  2T GT  T1ITEM FT  ET GT T1ITEM FT  COS2PI T GT  SIN2PI TENDENUMERATEITEM COMPUTE THE INNER PRODUCTS XBFT YBF OF THE FOLLOWING USING  THE EUCLIDEAN INNER PRODUCT  BEGINENUMERATE  ITEM XBF  1234T YBF  2341T  ITEM XBF  23 YBF  12T  ENDENUMERATEENDEXERCISESSUBSECTIONWEAK CONVERGENCEPROTECTFOOTNOTETHE CONCEPTS IN THIS SECTION ARE  USED BRIEFLY IN SECTION  REFSECORTHOSUB AND MOSTLY IN CHAPTER REFCHAPCOMPMAP IT ISRECOMMENDED THAT THIS SECTION BE SKIPPED ON A FIRST READING CHECK THE COMMENTEDOUT STUFF IN COMPMAPTEX IN ITERWHEN WE HAVE A SEQUENCE OF VECTORS XBFN AS WE SAW IN SECTIONREFSECSEQUENCES WE CAN TALK ABOUT CONVERGENCE OF THE SEQUENCETO SOME VALUE SAY XBFN RIGHTARROW XBF WHICH MEANS THAT  XBFN  XBF RIGHTARROW 0FOR SOME NORM  CDOT   IT IS INTERESTING TO EXAMINE THEQUESTION OF CONVERGENCE IN THE CONTEXT OF INNER PRODUCTSBEGINLEMMA LABELLEMCONTIP THE INNER PRODUCT IS CONTINUOUS  THAT IS IF XBFN RIGHTARROW XBF IN SOME INNER PRODUCT SPACE S THEN  LA XBFNYBFRA RIGHTARROW LA XBFYBFRA FOR ANY YBF IN SENDLEMMABEGINPROOF  SINCE XBFN IS CONVERGENT IT MUST BE BOUNDED SO THAT   XBFN LEQ M  INFTY  THEN BEGINALIGNED LA XBFNYBF RA  LA XBFYBFRA   LA XBFNXBF YBF RA  LEQ  XBFN  XBF   YBFENDALIGNEDSINCE  XBFN  XBF RIGHTARROW 0 THE CONVERGENCE OF LAXBFNYBFRA  IS ESTABLISHEDENDPROOFFROM THIS WE NOTE THAT CONVERGENCE XBFN RIGHTARROW XBF CALLEDEM STRONG CONVERGENCE IMPLIES LA XBFNYBF RA RIGHTARROWLA XBFYBFRA WHICH IS CALLED EM WEAK CONVERGENCE  ON THE OTHERHAND IT DOES NOT FOLLOW NECESSARILY THAT IF A SEQUENCE CONVERGESWEAKLY SO THAT INDEXSTRONG CONVERGENCE INDEXWEAK CONVERGENCE LA XBFN YBF RA RIGHTARROW LAXBFYBFRATHAT IT ALSO CONVERGES STRONGLYBEGINEXAMPLE  LET XBFN  000LDOTS100LDOTS BE THE SEQUENCE THAT IS  ALL 0 EXCEPT FOR A 1 AT POSITION N AND LET YBF   1121418LDOTS  THEN LA XBFN YBFRA RIGHTARROW 0BUT THE SEQUENCE  XBFN HAS NO LIMIT  THE SEQUENCE THUSCONVERGES WEAKLY BUT NOT STRONGLYENDEXAMPLEBEGINEXERCISESITEM LABELEXSTRCON SHOW THAT STRONG CONVERGENCE IMPLIES WEAK  CONVERGENCE  849 82 ENDEXERCISESSECTIONINDUCED NORMSLABELSECINDNORMWE HAVE SEEN THAT THE EUCLIDEAN NORM OF A VECTOR XBF IN RBBN IS DEFINED AS XBF22  X12  X22  CDOTS  XN2WE OBSERVE THAT THE INNER PRODUCT OF XBF WITH ITSELF IS LA XBFXBFRA  X12  X22  CDOTS  XN2HENCE WE CAN USE THE INNER PRODUCT TO PRODUCE A SPECIAL NORM CALLEDTHE BF INDUCED NORM  MORE GENERALLY GIVEN AN INNER PRODUCT LACDOTCDOTRA IN A VECTOR SPACE S WE HAVE THE INDUCED NORMDEFINED BY BOXED XBF   LA XBFXBFRA12 FOR EVERY X IN SIT SHOULD BE POINTED OUT THAT NOT EVERY NORM IS AN INDUCED NORM  FOREXAMPLE THE LP AND LP NORMS ARE ONLY INDUCED NORMS WHEN P2BEGINEXAMPLE  ANOTHER EXAMPLE OF AN INDUCED NORM IS FOR FUNCTIONS IN L2AB  XT2  LA XTXTRA12  LEFTINTAB XT2DTRIGHT12ENDEXAMPLEFOR AN INDUCED NORM WE HAVE THE FOLLOWING USEFUL FACT FOR AN INNERPRODUCT OVER A COMPLEX VECTOR SPACE BEGINALIGNED XBF  YBF2  LA XBFYBFXBFYBFRA  LA XBF XBF RA LA XBFYBFRA  LA YBFXBF RA  LA YBFYBFRA   XBF2  2 REAL LA XBFYBF RA  YBF2ENDALIGNEDFOR A VECTOR OVER A REAL VECTOR SPACE THIS SIMPLIFIES TO  XBF  YBF2   XBF2  2 LA XBFYBFRA  YBF2SECTIONTHE CAUCHYSCHWARZ INEQUALITYLABELSECCSIN THE DEFINITION OF A NORM ONE OF THE KEY REQUIREMENTS OF THEFUNCTION  CDOT IS THAT  XBF  YBF LEQ  XBF   YBFUP TO THIS POINT WE HAVE ASSUMED THAT THE METRICS MENTIONED DOSATISFY THIS PROPERTY  WE ARE NOW READY TO PROVE THIS RESULT FOR THEIMPORTANT SPECIAL CASE OF THE L2 OR L2 NORM OR MORE GENERALLYFOR A NORM INDUCED FROM ANY INNER PRODUCT  IN THE INTEREST OFGENERALITY WE SHALL EXPRESS THIS RESULT IN TERMS OF INNER PRODUCTSFIRSTTHE KEY INEQUALITY IN OUR PROOF IS THE EM CAUCHYSCHWARZ  INEQUALITY  INDEXCAUCHYSCHWARZ INEQUALITYINDEXINEQUALITIESCAUCHYSCHWARZ THIS INEQUALITY WILL PROVE TO BEONE OF THE CORNERSTONES OF SIGNAL PROCESSING ANALYSIS  IT WILLPROVIDE THE BASIS FOR THE IMPORTANT PROJECTION THEOREM AND BE THE KEYSTEP IN THE DERIVATION OF THE MATCHED FILTER  IT CAN BE USED TO PROVETHE IMPORTANT GEOMETRICAL FACT THAT THE GRADIENT OF A FUNCTION POINTSIN THE DIRECTION OF STEEPEST INCREASE WHICH IS THE KEY FACT USED INTHE DEVELOPMENT OF GRADIENT DESCENT OPTIMIZATION TECHNIQUES  NOT ONLYIS IT SPECIFICALLY USEFUL BUT THE ANALYSIS AND OPTIMIZATION PERFORMEDUSING THE CAUCHYSCHWARZ INEQUALITY PROVIDES A POWERFUL ARCHETYPE FORMANY OTHER OPTIMIZATION PROBLEMS OPTIMIZING VALUES CAN OFTEN BEOBTAINED BY ESTABLISHING AN INEQUALITY THEN SATISFYING THE CONDITIONSFOR WHICH THE INEQUALITY ACHIEVES EQUALITY  IF THE CAUCHYSCHWARZINEQUALITY DOES NOT SERVE THE PURPOSE OTHER INEQUALITIES OFTEN WILLSUCH AS THE CAUCHYSCHWARZS BIG BROTHERS THE HOLDER AND MINKOWSKIINEQUALITIESBEGINTHEOREM LABELTHMCS CAUCHYSCHWARZ INEQUALITY  IN AN  INNER PRODUCT SPACE S WITH INDUCED NORM CDOTBEGINEQUATIONBOXED LA XBFYBFRA LEQ  XBF  YBFLABELEQSW1ENDEQUATIONFOR ANY XBF YBF IN S WITH EQUALITY IF AND ONLY IF YBF ALPHA XBF FOR SOME ALPHAENDTHEOREMBEGINPROOF  BY EXPRESSING OUR PROOF IN TERMS OF INNER PRODUCTS WE  COVER BOTH THE CASE OF FINITE AND INFINITEDIMENSIONAL VECTORS  FOR GENERALITY WE ASSUME COMPLEX VECTORS    FIRST NOTE THAT IF XBF  0 OR YBF0 THE THEOREM IS TRIVIAL  SO WE EXCLUDE THESE CASES  FORM THE QUANTITYBEGINEQUATION  XBF  ALPHA YBF 2  XBF   2 REALLA  XBFALPHA YBFRA  ALPHA2  YBF 2LABELEQSW2ENDEQUATIONTHIS IS ALWAYS POSITIVE  WE WANT TO CHOOSE ALPHA TO MAKE THIS ASSMALL AS POSSIBLE  FOR REAL VECTORS THIS CAN BE DONE SIMPLY BYTAKING THE DERIVATIVE WITH RESPECT TO ALPHA AND EQUATING THEDERIVATIVE TO ZERO  WE DEMONSTRATE ANOTHER TECHNIQUE BY COMPLETINGTHE SQUARE INDEXCOMPLETING THE SQUARE SEE APPENDIX REFAPPDXCTSWE CAN WRITE 0 LEQ  XBF  ALPHA YBF 2   YBF2LEFT ALPHA  FRACLA    XBFYBFRA YBF2 ALPHABAR  FRACOVERLINELA    XBFYBFRA      YBF2RIGHT  FRACLA XBFYBFRA2 YBF2   XBF2THEN THE MINIMUM VALUE OF  XBFALPHA YBF2 IS OBTAINED WHEN ALPHA  FRACLA XBFYBFRAYBF2IN WHICH CASE THE COMPLETION OF THE SQUARE LEAVES  FRACLA XBFYBFRA2 YBF2   XBF2 GEQ 0FROM WHICH THE DESIRED INEQUALITY FOLLOWSNOW EXAMINE THE CONDITION FOR EQUALITY  IF YBFALPHA XBF THEN EQUALITYIN REFEQSW1 IS IMMEDIATE  ON THE OTHER HAND SUPPOSE THAT THEEQUALITY IN REFEQSW1 IS SATISFIED  THEN WORKING BACKWARDTHROUGH REFEQSW2 INDICATES THAT XBF  ALPHA YBF  0  BUT BYTHE PROPERTIES OF A NORM THIS MEANS THAT XBF  ALPHA YBF FOR SOMEALPHAENDPROOFTHIS THEOREM APPLIES TO EM ANY NORMED LINEAR VECTOR SPACE WITH ANINDUCED NORM  FOR THE VECTOR SPACE RBBN WITH THE EUCLIDEAN INNERPRODUCT THE CAUCHYSCHWARZ INEQUALITY IS BOXEDXBFT YBF2 LEQ XBFT XBFYBFT YBFFOR THE VECTOR SPACE CBBN WITH THE EUCLIDEAN INNERPRODUCT THE CAUCHYSCHWARZ INEQUALITY IS XBFH YBF2 LEQXBFH XBFYBFH YBF FOR THE VECTOR SPACE OF REAL FUNCTIONS DEFINED OVER AB THECAUCHYSCHWARZ INEQUALITY IS BOXEDLEFTINTAB FTGTDTRIGHT2 LEQ INTAB  F2TDT INTAB  G2TDTUSING THE CAUCHYSCHWARZ INEQUALITY WE CAN NOW SHOW THAT THE INDUCEDNORM SATISFIES THE REQUIRED TRIANGLE INEQUALITY PROPERTY  FOR VECTORSXBF AND YBF WHICH WE ASSUME FOR CONVENIENCE TO BE REAL WE HAVE BEGINALIGNED   XBF  YBF 2  LA XBFYBFXBFYBFRA  LA XBFXBFRA  2 LA XBFYBFRA  LA YBFYBF RA   LEQ LA XBFXBFRA  2 XBF   YBF  LA YBFYBF RA   XBF  YBF2ENDALIGNEDSECTIONDIRECTION OF VECTORS ORTHOGONALITYLABELSECDIRVECTHE INNER PRODUCT CAN BE USED TO DEFINE A DIRECTION OF ANGULARSEPARATION BETWEEN VECTORS AND HENCE A CONCEPT OF DIRECTIONFOR VECTORS XBF AND YBF IN RBB3 OR RBB2 IT IS WELLKNOWN THAT THE COSINE OF THE ANGLE BETWEEN THE VECTORS IS COS THETA  FRACLA XBFYBFRAXBF2 YBF2 NOTE THAT THE 2NORM  WHICH IS THE INDUCED NORM  IS USED INDEFINING THE LENGTH  USING THE CAUCHYSCHWARZ INEQUALITY IT CAN BESHOWN THATBEGINEQUATION 1 LEQ FRACLA XBFYBFRAXBF2 YBF2 LEQ 1LABELEQANGLEBOUNDENDEQUATIONSO THE ANGLE THETA IS REAL  THIS SAME EXPRESSION WITH THEAPPROPRIATE INNER PRODUCT DEFINES DIRECTION IN ANY INNER PRODUCT SPACEBEGINEXAMPLE  CONSIDER THE VECTORS XBF  1 2 3 4TQQUAD YBF  4 2 4 5TTHEN THE ANGLE THETA BETWEEN THE VECTORS IS DETERMINED BY COS THETA  FRACLA XBFYBFRAXBF YBF  0935ENDEXAMPLEBEGINEXAMPLE FOR FUNCTIONS DEFINED ON 01 FIND THE ANGLE  BETWEEN THE FUNCTIONS  X1T  1T2QQUADTEXTANDQQUADX2T  T22TFIRST COMPUTE  X1   LEFTINT01 X1T2DTRIGHT12 SQRT2815AND  X2   LEFTINT01 X2T2DTRIGHT12 SQRT815THEN COS THETA  FRACINT01 X1TX2TDTX1 X2 FRAC298SQRT14ENDEXAMPLEBEGINDEFINITION  IF XBF AND YBF ARE NONZERO VECTORS AND XBFALPHA YBF FOR  SOME SCALAR ALPHA THEN XBF AND YBF ARE SAID TO BE BF    COLINEAR  INDEXCOLINEAR IN AN INNERPRODUCT SPACE THIS  MEANS THAT THE ANGLE BETWEEN XBF AND YBF SATISFIES COS  THETA  PM 1ENDDEFINITIONA GEOMETRIC CONCEPT WHICH WILL BE OF CONSIDERABLE IMPORTANCE TO US ISTHE IDEA OF ORTHOGONAL VECTORSBEGINDEFINITION  VECTORS X AND Y IN AN INNER PRODUCT SPACE ARE SAID TO BE BF    ORTHOGONAL INDEXORTHOGONAL IF LA XY RA  0 NOTATIONALLY THIS IS DENOTED AS X PERP YINDEXPERPPERPSEEORTHOGONAL THE WORDS PERPENDICULARINDEXPERPENDICULARSEEORTHOGONAL AND NORMALINDEXNORMALSEEORTHOGONAL ARE SYNONYMOUS WITH ORTHOGONALENDDEFINITIONTHE ZERO VECTOR IS ORTHOGONAL TO EVERY OTHER VECTORBEGINDEFINITIONA SET OF VECTORS  PBF1PBF2LDOTSPBFM IS SAID TO BE BF  ORTHONORMAL INDEXORTHONORMAL IF THEY ARE MUTUALLY PAIRWISEORTHOGONAL AND EACH HAVE UNIT LENGTH LA PBFIPBFJ RA  DELTAIJWHERE DELTAIJ IS THE BF KRONECKER DELTA INDEXKRONECKER DELTA FUNCTION DEFINED BY INDEXDELTA FUNCTION DELTAIJ  BEGINCASES  1  I J  0  TEXTOTHERWISEENDCASESENDDEFINITIONFOR ORTHOGONAL VECTORS REGARDLESS OF THE INNER PRODUCT THE FAMILIARPYTHAGOREAN THEOREM HOLDSBEGINLEMMA LABELLEMPYTH THE PYTHAGOREAN THEOREM  INDEXPYTHAGOREAN THEOREM IF XBF PERP YBF AND  CDOT IS  AN INDUCED NORM THEN FOR THE NORM  CDOT  INDUCED FROM THE  INNER PRODUCTBEGINEQUATION XBF  YBF2  XBF2  YBF2LABELEQPYTHAG1ENDEQUATIONCONVERSELY IF REFEQPYTHAG1 HOLDS THEN XBF PERP YBFENDLEMMATHE  PROOF IS STRAIGHTFORWARDBEGINEXAMPLE  CONSIDER THE SET OF POLYNOMIALS  P0T1 QQUAD P1T  T QQUAD P2T FRAC123T21 QQUADP3T  FRAC125T3  3TP4T  FRAC1835T4  30T2  3THEN IT MAY BE VERIFIED BY DIRECT COMPUTATION THAT WHEN THE INNERPRODUCT IS DEFINED AS LA FG RA INT11 FTGTDTTHESE POLYNOMIALS ARE ORTHOGONAL LA PMPN RA  BEGINCASES  0  M NEQ N   FRAC22N1  M  NENDCASESTHESE POLYNOMIALS ARE THE FIRST FEW EM LEGENDRE POLYNOMIALS ALL OFWHICH ARE ORTHOGONAL OVER 11 INDEXLEGENDRE POLYNOMIALENDEXAMPLEGEOMETRIC INSIGHT CAN OFTEN BE OBTAINED BY DRAWING QUALITATIVECOORDINATE SYSTEMS THAT DEMONSTRATE SUBSPACES ORTHOGONALITY ETCWITHOUT NECESSARY REGARD TO THE DETAILS OF THE LENGTHS OF VECTORS ORTHE ANGLES BETWEEN VECTORS  BASED ON THE GEOMETRIC UNDERSTANDING SUCHCOORDINATE SYSTEMS AFFORD IT MAY BE EASIER TO PROVIDE MATHEMATICALSTATEMENTS FOR THE GEOMETRIC CONSTRUCTIONSBEGINEXAMPLE  LET X1T  1  X2T  T AND X3T  T2  FOR T IN 01  THEN LA X1X2 RA  0QUADQUAD LA X1X3 RA  13 QUADQUADLA X2X3 RA  14SO X1 AND X2 ARE ORTHOGONAL BUT THE OTHER PAIRS OF FUNCTIONSARE NOT  THIS MAY BE DIAGRAMMED AS SHOWN IN FIGURE REFFIGQG1WHERE THE ORTHOGONALITY HAS BEEN EXPLICITLY SHOWN BUT THE PARTICULARANGLES BETWEEN OTHER VECTORS HAS NOT BEENENDEXAMPLEBEGINEXERCISES  ITEM SHOW THAT FOR AN INDUCED NORM CDOT    BEGINEQUATION      LABELEQPARALLELOGRAM       XY 2   XY2  2X2 2Y2    ENDEQUATIONTHIS EQUATION IS KNOWN AS THE PARALLELOGRAM LAW  IN TWODIMENSIONALGEOMETRY AS SHOWN IN FIGURE REFFIGPARALLELOGRAM THE RESULT SAYSTHAT THE SUM OF  SQUARES OF THE LENGTHS OF THE DIAGONALS IS EQUAL TOTWICE THE SUM OF THE SQUARES OF THE ADJACENT SIDES A SORT OF TWOFOLDPYTHAGOREAN THEOREMBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRPARALLEL    CAPTIONTHE PARALLELOGRAM LAW    LABELFIGPARALLELOGRAM  ENDCENTERENDFIGUREITEM PROVE LEMMA REFLEMPYTHITEM SHOW THAT  LA XBFYBFRA  FRACXBF  YBF22  XBF  YBF224 THIS IS KNOWN AS THE POLARIZATION IDENTITYITEM PROVE REFEQPYTHAG1ITEM SHOW THAT THE INEQUALITY REFEQANGLEBOUND IS TRUE  ITEM LET X1T  3T2  1 X2T 5T3  3T AND X3T     2T2  T AND DEFINE THE INNER PRODUCT AS LA FGRA     INT11 FTGTDT  COMPUTE THE ANGLES EACH PAIRWISE    COMBINATION OF THESE FUNCTIONS AND IDENTIFY FUNCTIONS THAT ARE    ORTHOGONAL  ITEM LET  BEGINALIGNEDXBF1  1 2 4 2T XBF2  5231T XBF3  1212TENDALIGNEDAND COMPUTE THE ANGLES BETWEEN THESE VECTORS USING THE EUCLIDEANINNER PRODUCT AND IDENTIFY WHICH VECTORS ARE ORTHOGONALITEM SHOW THAT A SET OF VECTORS P1P2LDOTSPM WHICH ARE  MUTUALLY ORTHOGONAL SO THAT LA PIPJ RA  0 TEXT IF  I NEQ JIS LINEARLY INDEPENDENT  ORTHOGONALITY IMPLIES LINEAR INDEPENDENCEITEM LET S BE A VECTOR SPACE WITH AN INDUCED NORM  SHOW THAT  LA XBFYBF RA   XBF   YBFIF AND ONLY IF A XBF   B YBF  0 FOR SOME SCALARS A AND BENDEXERCISESSECTIONWEIGHTED INNER PRODUCTSLABELSECWIPFOR A FINITEDIMENSIONAL VECTOR SPACE A BF WEIGHTED INNER PRODUCTINDEXWEIGHTED INNER PRODUCTCAN BE OBTAINED BY INSERTING A HERMITIAN WEIGHTING MATRIX W BETWEENTHE ELEMENTS INNERPXBFYBFW  XBFH W YBF  YBFH W XBF THE CONCEPT OF ORTHOGONALITY IS DEFINED WITH RESPECT TO THE PARTICULARINNER PRODUCT USED CHANGING THE INNER PRODUCT MAY CHANGE THEORTHOGONALITY RELATIONSHIP BETWEEN VECTORSBEGINEXAMPLE  CONSIDER THE VECTORS XBF1  BEGINBMATRIX11 ENDBMATRIXQQUADQQUAD XBF2 BEGINBMATRIX2 1 ENDBMATRIXIT IS EASILY VERIFIED THAT THESE VECTORS ARE NOT ORTHOGONAL WITHRESPECT TO THE USUAL INNER PRODUCT XBF1TXBF2  HOWEVER FORTHE WEIGHTED INNER PRODUCT LA XBFYBFRAW  XBFTBEGINBMATRIX2  2  2  2ENDBMATRIX YBFTHE VECTORS XBF1 AND XBF2 ARE ORTHOGONALENDEXAMPLEIN ORDER FOR THE WEIGHTED INNER PRODUCT TO BE USED TO DEFINE A NORMAS IN  XBF W2  INNERPXBFXBFW  XBFH W XBFIT IS NECESSARY THAT XBFH W XBF  0 FOR ALL XBF NEQ 0  AMATRIX W WITH THIS PROPERTY IS SAID TO BE BF POSITIVE DEFINITEINDEXPOSITIVE DEFINITEBEGINEXAMPLE  THE WEIGHTED INNER PRODUCT OF THE PREVIOUS EXAMPLE CANNOT BE USED AS  A NORM BECAUSE FOR ANY VECTOR OF THE FORM XBF  BEGINBMATRIX ALPHA  ALPHA ENDBMATRIXTHE PRODUCT XBFT W XBF  0 WHICH VIOLATES THE CONDITIONS FOR A NORMENDEXAMPLEWEIGHTING CAN ALSO BE APPLIED TO INTEGRAL INNER PRODUCTS  IF THERE ISSOME FUNCTION WT GEQ 0 OVER AB THEN AN INNER PRODUCT CAN BEDEFINED AS LA FGRAW  INTAB WT FT GT DTTHE WEIGHTING CAN BE USED TO PLACE MORE EMPHASIS ON CERTAIN PARTS OFTHE FUNCTION  MORE PRECISELY WE MUST HAVE WT GEQ 0 WITHWT0 ONLY ON A SET OF MEASURE ZEROBEGINEXAMPLE   LABELEXMCHEBYPOL LET US DEFINE A SET OF POLYNOMIALS BY TNT  COSN COS1TFOR T IN 11  THE FIRST FEW OF THESE OBTAINED BY APPLICATIONOF TRIGONOMETRIC IDENTITIES ARE T0T 1 QQUAD T1T  T QQUAD T2T  2T21 QQUAD T3T 4T3  3TA PLOT OF THE FIRST FEW OF THESE IS SHOWN IN FIGURE REFFIGCHEBPOLYINDEXCHEBYSHEV POLYNOMIAL INDEXORTHOGONAL POLYNOMIALTHESE POLYNOMIALS ARE THE EM CHEBYSHEV POLYNOMIALS  THEY HAVE THEINTERESTING PROPERTY THAT OVER THE INTERVAL 11 ALL THE EXTREMAOF THE FUNCTIONS HAVE THE VALUES 1 OR 1  THIS PROPERTY MAKES THEMVERY USEFUL FOR APPROXIMATION OF FUNCTIONS AS DISCUSSED IN CHAPTERREFCHAPPROXFURTHERMORE THE CHEBYSHEV POLYNOMIALS AREORTHOGONAL WITH WEIGHT FUNCTION WT  FRAC1SQRT1T2OVER THE INTERVAL 11  THE ORTHOGONALITY RELATIONSHIP BETWEENTHE CHEBYSHEV POLYNOMIALS IS INT11 FRAC1SQRT1T2 TNT TMT  DT  PIDELTANMENDEXAMPLEBEGINFIGUREHTBP  CENTERLINEEPSFIGFILEPICTUREDIRCHEBY1EPS CHEBYPLOTM  CAPTIONCHEBYSHEV POLYNOMIALS PROTECTTPROTECT0TPROTECT    THROUGH PROTECTTPROTECT5TPROTECT FOR T IN     11   LABELFIGCHEBPOLYENDFIGUREWE CAN DEFINE A WEIGHTED INNER PRODUCT ON THE VECTOR SPACE OFMATSIZEMN MATRICESBY LA AB RA  TRACEAH W BWHERE W IS A SYMMETRIC POSITIVEDEFINITE MATSIZEMM MATRIXUSING A NORM INDUCED FROM A WEIGHTED INNER PRODUCT WE CAN DEFINE AWEIGHTED DISTANCE BETWEEN TWO VECTORSBEGINEQUATION DWXBFYBF2   XBF  YBFW2  XBFYBFH WXBFYBFLABELEQMAHAL1ENDEQUATIONBEGINEXAMPLE  A WEIGHTED DISTANCE ARISES NATURALLY IN MANY SIGNAL DETECTION  ESTIMATION AND PATTERN RECOGNITION PROBLEMS IN NONWHITE GAUSSIAN  NOISE  IN THIS EXAMPLE A DETECTION PROBLEM IS CONSIDERED  DETECTION PROBLEMS ARE DISCUSSED MORE FULLY IN CHAPTER  REFCHAPDETECTION    LET SBF IN RBBN BE A SIGNAL WHICH TAKES ON ONE OF TWO  DIFFERENT VALUES EITHER SBF  SBF0 OR SBF  SBF1  ONE OF  THESE SIGNALS IS CHOSEN AT RANDOM WITH EQUAL PROBABILITY  EITHER  BY A BINARY DATA TRANSMITTER OR BY NATURE  THE SIGNAL SBF IS  OBSERVED IN THE PRESENCE OF ADDITIVE GAUSSIAN NOISE NBF WHICH HAS  MEAN ZEROBF AND COVARIANCE MATRIX R  THE OBSERVATION YBF  CAN BE MODELED AS YBF  SBF  NBF FROM THE OBSERVATION OF YBFYBF WE DESIRE TO DETERMINE WHICH VALUEOF SBF ACTUALLY OCCURRED  THIS IS THE BF DETECTION PROBLEMCONDITIONED UPON A VALUE OF SBFSBF THE OBSERVATION IS GAUSSIAN WITHMEAN SBF AND THE SAME COVARIANCE FYBFSBF  SBF  FRAC12PIN2DETR12EXPFRAC12 YBFSBFT R1 YBFSBFWHERE EITHER SBFSBF0 OR SBF  SBF1  FROM THE OBSERVATIONYBF WE CAN COMPUTE THE EM LIKELIHOOD THAT THE SIGNAL WASPRODUCED BY SBF FOR EACH OF THE POSSIBLE VALUES OF SBF THENSELECT THE ONE WITH THE HIGHEST LIKELIHOOD  THAT IS WE COMPAREBEGINEQUATION FYBFSBFSBF0QQUAD TEXTWITHQQUADFYBFSBFSBF1LABELEQDETECT1ENDEQUATIONAND DETERMINE OUR DECISION ABOUT SBF ON THE BASIS OF WHICHLIKELIHOOD FUNCTION IS LARGEST  THIS IS THE MAXIMUM LIKELIHOODDECISION RULE  CANCELING COMMON FACTORS IN THE COMPARISON THIS ISEQUIVALENT TO COMPARINGBEGINEQUATION YBFSBF0T R1 YBFSBF0QQUAD TEXTWITHQQUADYBFSBF1T R1YBFSBF1LABELEQDETECT2ENDEQUATIONAND CHOOSING EITHER SBF0 OR SBF1 DEPENDING UPON WHICHQUANTITY IS SMALLER  THESE QUANTITIES CAN BE OBSERVED TO BE WEIGHTEDDISTANCES OF THE FORM REFEQMAHAL1  LET W  R1 AND DEFINETHE WEIGHTED INNER PRODUCT IN RBBN BY LA XBFYBFRAW  XBFT W YBFTHIS INDUCES A WEIGHTED NORM  XBFW2  XBFT W XBFTHE COMPARISON IN REFEQDETECT2 CORRESPONDS TO COMPUTING  YBF  SBF0W QQUAD TEXTAND QQUAD    YBF  SBF1WWITH THE MAXIMUM LIKELIHOOD CHOICE BEING THAT WHICH HAS THE MINIMUMWEIGHT DISTANCE  THIS WEIGHED DISTANCE MEASURE ARISES COMMONLY INPATTERN RECOGNITION PROBLEMS AND IS KNOWN AS THE EM MAHALONOBIS  DISTANCE INDEXPATTERN RECOGNITION INDEXMAHALONOBIS DISTANCEFURTHER SIMPLIFICATIONS ARE OFTEN POSSIBLE IN THIS COMPARISONBEGINALIGNYBFSBF0W  YBFT W YBF  YBFT W SBF0  SBF0T W YBF SBF0T W SBF0  YBFT W YBF  2YBFT W SBF0  SBF0T W SBF0ENDALIGNAND SIMILARLY FOR YBF  SBF1W  IF SBF0 AND SBF1HAVE THE SAME INNER PRODUCT NORM SO SBF0T W SBF0  SBF1T WSBF1 THEN WHEN COMPARING YBFSBF0W WITHYBFSBF1W THESE TERMS CANCEL AS WELL AS THE YBFTWYBFTERM  THE CHOICE IS MADE DEPENDING ON WHETHER YBFT WSBF0 QQUAD TEXTORQQUAD YBFT W SBF1 IS LARGER THAT IS DEPENDING ON WHICH WEIGHTED INNER PRODUCT ISLARGEST  THE INNER PRODUCT IS THUS SEEN TO BE A SIMILARITY MEASURETHE SIGNAL SBF IS CHOSEN THAT IS MOST SIMILAR TO THE RECEIVEDSIGNAL VECTOR WHERE THE SIMILARITY IS DETERMINED BY THE WEIGHTEDINNER PRODUCTENDEXAMPLESUBSECTIONEXPECTATION AS AN INNER PRODUCTTHE EXAMPLES OF WEIGHTED INNER PRODUCTS UP UNTIL NOW HAVE BEEN OFDETERMINISTIC FUNCTIONS  AN IMPORTANT GENERALIZATION DEVELOPS WHEN AA JOINT DENSITY IS USED AS A WEIGHTING FUNCTION IN THE INNER PRODUCTLET X AND Y BE RANDOM VARIABLES WITH JOINT DENSITY FXYXYWE DEFINE AN INNER PRODUCT BETWEEN THEM AS LA XYRA  INT X Y FXYXY  DXDYTHIS INNER PRODUCT IS OF COURSE AN EXPECTATION AND INTRODUCTION OFTHIS INNER PRODUCT ALLOWS THE CONCEPTUAL POWER OF VECTOR SPACES TO BEAPPLIED TO MEANSQUARE ESTIMATION THEORY  THUS LA XY RA  EXYE IS THE EXPECTATION OPERATOR  ORTHOGONALITY IS DEFINED FORRANDOM VARIABLES AS IT IS FOR DETERMINISTIC QUANTITIES THE RANDOMVARIABLES X AND Y ARE ORTHOGONAL IF EXY  0 THE INNER PRODUCTINDUCES A NORM LA XX RA  E X2IF X IS A ZEROMEAN RV THEN LA XXRA  VARX IS AN INDUCEDNORMFOOTNOTEAS WITH OTHER FUNCTION SPACES THERE ARE SOME TECHNICAL  PROBLEMS ASSOCIATED WITH VECTOR SPACES OVER PROBABILITY SPACES  SINCE THERE MAY BE RANDOM VARIABLES X AND Y SUCH THAT  XY    0 BUT X NEQ Y ALWAYS  HOWEVER IT CAN BE SHOWN THAT IF   XY  0 THEN XY AS ALMOST SURELY THAT IS EXCEPT ON A  SET OF PROBABILITY MEASURE 0  WE CAN ALSO DEFINE AN INNER PRODUCT BETWEEN RANDOM EM VECTORS  LETXBF  X1X2LDOTSXNT AND YBF  Y1Y2LDOTSYNT BENDIMENSIONAL RANDOM VECTORS  THEN WE CAN DEFINE AN INNER PRODUCTBETWEEN THESE VECTORS AS LA XBF YBF RA  E SUMI1N XI YBARINOTE THAT WE CAN WRITE THIS INNER PRODUCT AS LA YBF YBF RA    EYBFH YBFANOTHER NOTATION THAT IS SOMETIMES CONVENIENT IS TO WRITE LA YBF YBF RA  TRACE EYBF YBFHWHERE THE TRACEX IS THE TRACE OPERATOR INDEXTRACE THE SUMOF THE ELEMENTS ON THE DIAGONAL OF THE SQUARE MATRIX X  SEESECTION REFSECTPOSETRACEWHEN THE VECTORSPACE VIEWPOINT IS APPLIED TO PROBLEMS OFMINIMIZATION AS DISCUSSED SUBSEQUENTLY THERE ARE TWO MAJORAPPROACHES TO THE PROBLEM  IN THE FIRST CASE AN INNER PRODUCT ISUSED THAT IS NOT BASED ON AN EXPECTATION MINIMIZATION OF THIS SORTIS REFERRED TO AS EM LEASTSQUARES LS INDEXLEASTSQUARES INTHE SIGNAL PROCESSING LITERATURE  WHEN AN INNER PRODUCT IS USED THATIS DEFINED AS AN EXPECTATION THEN THE APPROXIMATION OBTAINED ISREFERRED TO AS A EM MINIMUM MEANSQUARES MMS APPROXIMATIONINDEXMINIMUM MEANSQUARE  IN FACT BOTH APPROXIMATION TECHNIQUESRELY ON PRECISELY THE SAME THEORY BUT SIMPLY EMPLOY INNER PRODUCTSSUITED TO THE NEEDS OF THE PARTICULAR PROBLEMBEGINEXERCISESITEM PERFORM THE SIMPLIFICATIONS TO GO FROM THE COMPARISON IN    REFEQDETECT1 TO THE COMPARISON IN REFEQDETECT2  ITEM SHOW BY INTEGRATION THAT INT11 FRAC1SQRT1T2 TNT TMT  BEGINCASES  PI  NM0   PI2  NM NNEQ 0   0  N NEQ MENDCASESHINT USE T  COS X IN THE INTEGRALENDEXERCISESSECTIONHILBERT AND BANACH SPACESLABELSECHILBERTWITH THE DEFINITIONS OF METRIC SPACES AND INNERPRODUCT SPACES BEHINDUS WE ARE NOW READY TO INTRODUCE THE SPACES IN WHICH MOST OF THE WORKIN SIGNAL PROCESSING IS PERFORMEDBEGINDEFINITION  A COMPLETE NORMED VECTOR SPACE IS CALLED A BF BANACH SPACE  INDEXBANACH SPACE A COMPLETE NORMED VECTOR SPACE WITH AN INNER  PRODUCT IN WHICH THE NORM IS THE INDUCED NORM IS CALLED A BF    HILBERT SPACE INDEXHILBERT SPACEENDDEFINITIONSOME EXAMPLES OF BANACH AND HILBERT SPACESBEGINENUMERATEITEM THE SPACE OF CONTINUOUS FUNCTIONS CABDINFTY FORMS A  BANACH SPACE  RECALL THAT IN EXAMPLE REFEXMXINF  C11DINFTY WAS SHOWN TO BE COMPLETEITEM HOWEVER THE SPACE OF FUNCTIONS CAB WITH THE LP NORM P   INFTY DOES EM NOT FORM A BANACH SPACE SINCE IT IS NOT  COMPLETE  WE SAW IN EXAMPLE REFEXMFNSEQ A SEQUENCE OF  CONTINUOUS FUNCTIONS THAT DOES NOT HAVE A LIMIT IN C11ITEM THE SEQUENCE SPACE LP0INFTY IS A BANACH SPACE  WHEN  P2 IT IS A HILBERT SPACEITEM THE SPACE LPAB IS A BANACH SPACE  WHEN P2 IT IS A  HILBERT SPACE  THE HILBERT SPACE OF FUNCTIONS WITH DOMAIN OVER THE  WHOLE REAL LINE IS DENOTED LPRBBENDENUMERATEBECAUSE OF THE UTILITY OF HAVING THE NORM INDUCED FROM AN INNERPRODUCT THE EMPHASIS IN THIS AND SUCCEEDING CHAPTERS IS ON HILBERTSPACES  INPUTLINALGDIRHILBERTBOXTEXIT CAN BE SHOWN CITEP 267NAYLORSELL THAT IF A NORMED VECTORSPACE IS FINITE DIMENSIONAL THEN IT IS COMPLETE  HENCE EVERY NORMEDFINITE DIMENSIONAL SPACE IS A BANACH SPACE IF THE NORM IS INDUCEDFROM AN INNER PRODUCT THEN IT IS ALSO A HILBERT SPACE  FURTHERMORE EVERY FINITEDIMENSIONAL SUBSPACE OF A SPACE ISCOMPLETESECTIONORTHOGONAL SUBSPACESLABELSECORTHOSUBBEGINDEFINITION  LET S BE A VECTOR SPACE AND LET V AND W BE SUBSPACES OF S  V AND W ARE BF ORTHOGONAL IF EVERY VECTOR VBF IN V IS  ORTHOGONAL TO EVERY VECTOR WBF IN WMC LA VBFWBF RA   0 INDEXORTHOGONAL SUBSPACEENDDEFINITIONBEGINDEFINITION  FOR A SUBSPACE V OF AN INNER PRODUCT SPACE S THE SPACE OF ALL  VECTORS ORTHOGONAL TO V IS CALLED THE BF ORTHOGONAL COMPLEMENT  OF V  THIS IS DENOTED AS VPERPENDDEFINITIONBEGINEXAMPLELET V BE THE PLANE SHOWN IN FIGURE REFFIGORTHOG1  THEN THEORTHOGONAL SPACE WVPERP IS SPANNED BY THE VECTOR WBF  ENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRORTHOGSPACE1    CAPTIONA SPACE AND ITS ORTHOGONAL COMPLEMENT    LABELFIGORTHOG1  ENDCENTERENDFIGURETHE ORTHOGONAL COMPLEMENT OF A SUBSPACE IS ITSELF A SUBSPACE SEEEXERCISE REFEXORTHOGCOMP1  THE ORTHOGONAL COMPLEMENT HAS THEFOLLOWING PROPERTIES SEE LUENBERGER P 52 NS P 294295BEGINTHEOREM CITELUENBERGER1969NAYLORSELL  LABELTHMORTHOGCOMP LET V  AND W BE SUBSETS OF AN INNER PRODUCT SPACE S NOT NECESSARILY  COMPLETE  THEN  BEGINENUMERATE  ITEM VPERP IS A CLOSED SUBSPACE OF S  ITEM V SUBSET VPERPPERP  ITEM IF V SUBSET W THEN WPERP SUBSET VPERP  ITEM VPERPPERPPERP  VPERP  ITEM IF X IN V CAP VPERP THEN X  0  ITEM VPERPPERP IS THE SMALLEST CLOSED SUBSPACE CONTAINING    S  THAT IS VPERPPERP  CLOSUREV  ENDENUMERATEENDTHEOREMBEGINPROOF  WE WILL PROVE PART 1  THE REST OF THE PROPERTIES ARE TO BE PROVED  AS AN EXERCISE SEE EXERCISE  REFEXORTHOCOMP  TO SHOW CLOSURE  OF VPERP LET  XBFN BE A CONVERGENT SEQUENCE IN  VPERP SO THAT XBFN RIGHTARROW XBF  THEN BY THE  CONTINUITY OF THE INNER PRODUCT SHOWN IN LEMMA REFLEMCONTIP WE  HAVE FOR ANY V IN V 0  LA XBFN VBFRA RIGHTARROW LA XBFVBFRASO THAT XBF IN VPERPENDPROOFWHAT IS PERHAPS A LITTLE SURPRISING AT FIRST ABOUT THIS THEOREM IS THEFACT THAT IT MAY EM NOT BE THE CASE THAT VPERPPERP  V  WHAT IS LACKING IS THE COMPLETENESS VPERPPERP MAY HAVE CAUCHYSEQUENCES IN IT THAT V DOES NOTBEGINEXERCISES    ITEM LABELEXORTHOGCOMP1 SHOW THAT THE ORTHOGONAL COMPLEMENT OF A SUBSPACE IS A SUBSPACEITEM LABELEXORTHOCOMP PROVE ITEMS 2 THROUGH 6 OF THEOREM  REFTHMORTHOGCOMPENDEXERCISESSECTIONLINEAR TRANSFORMATIONS RANGE AND NULLSPACELABELSECLINTRANSWE PAUSE IN OUR DEVELOPMENT OF VECTOR SPACES TO REINTRODUCE A CONCEPTTHAT SHOULD BE FAMILIARBEGINDEFINITIONINDEXTRANSFORMATIONLINEAR  A TRANSFORMATION LMC X RIGHTARROW Y FROM A VECTOR SPACE X TO A  VECTOR SPACE Y WHERE X AND Y HAVE THE SAME SCALAR FIELD R  IS A BF LINEAR TRANSFORMATION IF FOR ALL VECTORS X X1 X2 IN  X   BEGINENUMERATE  ITEM LALPHA X  ALPHA LX FOR ALL XBF IN X AND ALL    SCALARS ALPHA IN R AND  ITEM LX1  X2  LX1  LX2  ENDENUMERATEENDDEFINITIONWE WILL THINK OF LINEAR TRANSFORMATIONS AS EM OPERATORS INDEXOPERATORBEGINEXAMPLE WE WILL BEGIN WITH SEVERAL EXAMPLES FROM VECTOR SPACES  OF FUNCTIONS BEGINENUMERATEITEM LET X BE THE SET OF CONTINUOUS REALVALUED FUNCTIONS AND  DEFINE LMC X RIGHTARROW X BY LXT  INT0T HTAU XTTAUDTAUFOR ALL XT IN X  THEN L IS A LINEAR TRANSFORMATION WHICHCONVOLVES THE SIGNAL X WITH THE SIGNAL H INDEXCONVOLUTIONITEM LET X BE THE SET OF CONTINUOUS REALVALUED FUNCTIONS DEFINED  ON 01  THEN LMC X RIGHTARROW RBB DEFINED BY LXT  INT01 HTAU XTAUDTAUIS A LINEAR TRANSFORMATION AN INNER PRODUCTITEM LET X BE THE SET OF CONTINUOUS REALVALUED FUNCTIONS AND LET  TT0MC X RIGHTARROW X BE DEFINED BY TT0XT  BEGINCASES  XT  T  T0  0  TEXTOTHERWISEENDCASESWHERE T0 IS A PARAMETER OF THE TRANSFORMATION  THEN TT0 IS ALINEAR TRANSFORMATION  THIS TRANSFORMATION TRUNCATES A SIGNAL INTIME INDEXTRUNCATIONIN TIMEITEM LET X BE THE SET OF ALL FOURIER TRANSFORMABLE FUNCTIONS AND  LET Y BE THE SET OF FOURIER TRANSFORMS OF ELEMENTS IN X  DEFINE  FMC X RIGHTARROW Y BY FXT  INTINFTYINFTY XT EJOMEGA T DTTHE OPERATOR F IS A LINEAR OPERATORITEM LET BMC X RIGHTARROW X BE DEFINED BY BB0XT  FC1 TB0 XOMEGAWHERE XOMEGA IS THE FOURIER TRANSFORM OF XT FC1 ISTHE INVERSE FOURIER TRANSFORM OPERATOR AND TB0XOMEGATRUNCATES THE FOURIER TRANSFORM  THUS BB0XT IS A BANDLIMITEDSIGNAL INDEXTRUNCATIONIN FREQUENCY INDEXBANDLIMITED SIGNALENDENUMERATEENDEXAMPLEBEGINEXAMPLE PERHAPS MORE COMMONLY WE SEE LINEAR TRANSFORMATIONS  BETWEEN VECTOR SPACES OF FINITE DIMENSION  IN GENERAL A LINEAR  TRANSFORMATION L FROM THE VECTOR SPACE RN TO RM CAN BE  EXPRESSED USING THE NOTATION OF AN MATSIZEMN MATRIX L  THAT IS THE MATRIX BECOMES THE LINEAR TRANSFORMATION  BEGINENUMERATE  ITEM LET LMC RBB3 RIGHTARROW RBB2 BE DEFINED BY LX1X2X3  X1  2X2 3X2  4X3THIS LINEAR TRANSFORMATION CAN BE PLACED IN MATRIX NOTATION  BYWRITING AN ELEMENT IN RBB3 IN VECTOR FORM AS X1X2X3T INRBB3 WE CAN WRITE L  BEGINBMATRIX 120 034 ENDBMATRIXTHEN L XBF  BEGINBMATRIX X12X2  3X2  4X3 ENDBMATRIXITEM LET LMC RBB3RIGHTARROW RBB3 BE DEFINED BY THE MATRIX L  BEGINBMATRIX 001  010  100ENDBMATRIXTHEN L IS THE LINEAR TRANSFORMATION THAT REVERSES THE COORDINATES OFA VECTOR XBF IN RBB3  ENDENUMERATEENDEXAMPLECONSIDERABLY MORE IS SAID ABOUT LINEAR TRANSFORMATIONS BETWEENFINITEDIMENSIONAL VECTORS SPACES IN CHAPTER REFCHAPMATINVASSOCIATED WITH ANY OPERATOR LINEAR OR OTHERWISE ARE TWO IMPORTANTSPACES  THESE SPACES ARE THE RANGE AND THE NULLSPACE  TWO MORESPACES ASSOCIATED WITH LINEAR OPERATORS ARE PRESENTED IN SECTIONREFSEC4SUBOPBEGINDEFINITION  LET LMC XRIGHTARROW Y BE AN OPERATOR LINEAR OR OTHERWISE  THE BF RANGE SPACE INDEXRANGE RANGEL IS RANGEL  YBF  LXBFMC  XBF IN XTHAT IS IT IS THE SET OF VALUES IN Y THAT ARE REACHED FROM X BYAPPLICATION OF L  THE BF NULLSPACE INDEXNULLSPACE NULLSPACEL IS NULLSPACEL  XBF IN X LXBF  ZEROBFTHAT IS IT IS THE SET OF VALUES IN XBF THAT ARE TRANSFORMED TO ZEROBFIN Y BY L  THE NULLSPACE OF AN OPERATOR IS ALSO CALLED THE BF  KERNEL OF THE OPERATOR INDEXKERNELENDDEFINITIONLET A BE AN MATSIZENM MATRIX A  PBF1PBF2LDOTSPBFMWHICH WE REGARD AS A LINEAR OPERATOR  THEN A POINT XBF IN RBBMIS TRANSFORMED AS A XBF  X1 PBF1  X2 PBF2  CDOTS  XM PBFMWHICH IS A LINEAR COMBINATION OF THE COLUMNS OF A  THUS THE RANGEMAY BE EXPRESSED AS RANGEA  LSPANPBF1PBF2LDOTSPBFMTHE RANGE OF A MATRIX IS ALSO REFERRED TO AS THE EM COLUMN SPACEINDEXCOLUMN SPACESEERANGE OF A  THE NULLSPACE IS THAT SET OFVECTORS SUCH THAT AXBF  ZEROBFBEGINEXAMPLE  LET  A  BEGINBMATRIX 1  0  0 0  0  0 1  0  1 ENDBMATRIXTHEN THE RANGE OF A IS LSPAN101T 001TTHE NULLSPACE OF A IS  NULLSPACEA  LSPAN010TENDEXAMPLEBEGINEXAMPLE BEGINENUMERATEITEM LET LXT  INT0T XTAU HTTAUDTAU  THEN THE  NULLSPACE OF L IS THE SET OF ALL FUNCTIONS XT THAT RESULT IN  ZERO WHEN CONVOLVED WITH HT  FROM SYSTEMS THEORY WE REALIZE  THAT WE CAN TRANSFORM THE CONVOLUTION OPERATION AND MULTIPLY IN THE  FREQUENCY DOMAIN  FROM THIS PERSPECTIVE WE PERCEIVE THAT THE  NULLSPACE OF L IS THE SET OF FUNCTIONS WHOSE FOURIER TRANSFORMS DO  NOT SHARE ANY SUPPORT WITH THE SUPPORT INDEXSUPPORT OF THE  FOURIER TRANSFORM OF HITEM LET LXT  INT0T XTAU HTAU DTAU WHERE X IS THE  SET OF CONTINUOUS FUNCTIONS  THEN RANGEL IS THE SET OF REAL  NUMBERS UNLESS HT EQUIV 0ITEM THE RANGE OF THE OPERATOR A  BEGINBMATRIX 10 0 0 ENDBMATRIXIS THE SET OF ALL VECTORS OF THE FORM C0T  THE NULLSPACE OFTHIS OPERATOR IS LSPAN01ENDENUMERATEENDEXAMPLEBEGINEXERCISESITEM   LET X AND Y BE VECTOR SPACES OVER THE SAME SET OF  SCALARS  LET LTXY DENOTE THE SET OF ALL LINEAR TRANSFORMATIONS  FROM X TO Y  LET L AND M BE OPERATORS FROM LTXY  DEFINE AN ADDITION OPERATOR BETWEEN L AND M AS LMX  LX MXFOR ALL X IN X  ALSO DEFINE SCALAR MULTIPLICATION BY ALX  ALXSHOW THAT LTXYIS A LINEAR VECTOR SPACEITEM LET X Y AND Z BE LINEAR VECTOR SPACES OVER THE SAME SET  OF SCALARS AND LET L1MC XRIGHTARROW Y AND L2MC Y  RIGHTARROW Z BE LINEAR OPERATORS SHOW THAT THE COMPOSITION  L2L1MC XRIGHTARROW Z IS A LINEAR OPERATORENDEXERCISESSECTIONINNERSUM AND DIRECTSUM SPACESLABELSECISDSPBEGINDEFINITION  IF V AND W ARE LINEAR SUBSPACES THE SPACE V  W IS THE BF    INNER SUM INDEXINNER SUM SPACE CONSISTING OF ALL  POINTS XBF  VBF  WBF WHERE VBF IN V AND WBF IN WENDDEFINITIONBEGINEXAMPLE LABELEXMVS1  CONSIDER S  GF23 INDEXGF2GF2 THAT IS THE SET OF  ALL 3TUPLES OF ELEMENTS OF GF2 SEE BOX REFBOXGF2  THEN  FOR EXAMPLE XBF  101 IN SQQUAD TEXTANDQQUAD YBF  001IN SAND XBF  YBF  100LET W  LSPAN010 AND V LSPAN100 BE TWO SUBSPACES IN S  THEN W  000010AND V  000100THESE TWO SUBSPACES ARE  ORTHOGONALTHE ORTHOGONAL COMPLEMENT TO V IS  VPERP  000010001011THUS W SUBSET VPERPTHE INNER SUM SPACE OF V AND W IS VW  000010100110ENDEXAMPLEBEGINDEFINITION  TWO LINEAR SUBSPACES V AND W OF THE SAME DIMENSIONALITY ARE BF    DISJOINT INDEXDISJOINT IF V CAP W   0   THAT IS THE  ONLY VECTOR THEY HAVE IN COMMON IS THE ZERO VECTOR  DISJOINT  SUBSPACES ARE SLIGHTLY DIFFERENT FROM DISJOINT SETS SINCE DISJOINT  SUBSPACES MUST HAVE THE ZERO VECTOR IN COMMON WHEREAS DISJOINT SETS  HAVE NO ELEMENTS IN COMMONENDDEFINITIONBEGINEXAMPLE  IN FIGURE REFFIGDISJOINT1 THE PLANE S IS A VECTOR SPACE IN  TWO DIMENSIONS AND V AND W ARE TWO ONEDIMENSIONAL SUBSPACES  INDICATED BY THE LINES IN THE FIGURE  THE ONLY POINT THEY HAVE IN  COMMON IS THE ORIGIN SO THEY ARE DISJOINT  NOTE THAT THEY ARE NOT  NECESSARILY ORTHOGONALENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRDISJOINT1    CAPTIONDISJOINT LINES IN RBB2    LABELFIGDISJOINT1  ENDCENTERENDFIGUREWHEN S  VW AND V AND W ARE DISJOINT W IS SAID TO BE THEEM ALGEBRAIC COMPLEMENT OF V  INDEXALGEBRAIC COMPLEMENT THELAST EXAMPLE ILLUSTRATES AN ALGEBRAIC COMPLEMENT THE INNER SUM OF THETWO LINES GIVES THE ENTIRE VECTOR SPACE S  ON THE OTHER HAND THESETS V AND W IN EXAMPLE REFEXMVS1 ARE NOT ALGEBRAICCOMPLEMENTS SINCE V  W IS NOT THE SAME AS S  AN ALGEBRAICCOMPLEMENT TO THE SET V OF THAT EXAMPLE WOULD BE THE SET Z  LSPAN010001 000010001011IT IS STRAIGHTFORWARD TO SHOW THAT IN ANY VECTOR SPACE S  EVERYLINEAR SUBSPACE HAS AN ALGEBRAIC COMPLEMENT  LET B BE A HAMELINDEXHAMEL BASISBASIS FOR S AND LET B1 SUBSET B BE A HAMEL BASIS FOR VTHEN LET B2  B  B1 THE SET DIFFERENCE SO THAT B1 CAP B2 EMPTYSET  THEN W  LSPANB2IS A HAMEL BASIS FOR THE ALGEBRAIC COMPLEMENT OF VTHE DIRECT SUM OF DISJOINT SPACES CAN BE USED TO PROVIDE A UNIQUEREPRESENTATION OF A VECTORBEGINLEMMA LABELLEMVWUNIQUE CITENAYLORSELL  LET V AND W BE  SUBSPACES OF A VECTOR SPACE S  THEN FOR  EACH XBF IN VW THERE IS A EM UNIQUE VBF IN V AND A EM    UNIQUE WBF IN W SUCH THAT XBF  VBF  WBF IF AND ONLY IF  V AND W ARE DISJOINTENDLEMMABEGINPROOFASSUME THAT V AND W ARE DISJOINT  THEN IF THERE ARE TWOREPRESENTATIONS FOR XBF XBF  VBF1  WBF1  VBF2  WBF2THEN VBF1 VBF2  WBF1  WBF2  BUT SINCE VBF1VBF2 INV AND WBF1WBF2 IN W AND V CAP W  0 WE MUST HAVE VBF1VBF2  0 AND WBF1  WBF2  0CONVERSELY SUPPOSE THAT THERE IS A UNIQUE REPRESENTATION XBF  VBF WBF FOR EACH XBF IN VW  ASSUME AS A CONTRADICTION THAT VAND W ARE NOT DISJOINT SO THAT THERE IS A NONZERO ELEMENT ZBF IN VCAP W  THEN WE CAN WRITE XBF  VBF  C ZBF  WBF  C ZBFWHERE C IS ANY SCALAR VALUE  BUT THIS LEADS TO A NONUNIQUEREPRESENTATION ENDPROOFANOTHER WAY OF COMBINING VECTOR SPACES IS BY THE DIRECT SUM  BEGINDEFINITION  THE BF DIRECT SUM INDEXDIRECT SUM OF LINEAR SPACES V AND  W DENOTED V OPLUS W IS DEFINED ON THE CARTESIAN PRODUCT  INDEXCARTESIAN PRODUCT V TIMES W SO A POINT IN VOPLUS W IS  AN ORDERED PAIR VW WITH V IN V AND W IN W  ADDITION IS  DEFINED COMPONENTWISE V1W1  V2W2  V1V2W1W2  SCALAR MULTIPLICATION IS DEFINED AS ALPHAVW  ALPHA VALPHA  WENDDEFINITIONTHE SUM VW AND THE DIRECT SUM VOPLUS W ARE DIFFERENT LINEARSPACES  HOWEVER IF V AND W ARE EM DISJOINT THEN VW AND VOPLUSW HAVE EXACTLY THE SAME STRUCTURE MATHEMATICALLY THEY ARE SIMPLYDIFFERENT REPRESENTATIONS OF THE SAME THING   WHEN DIFFERENTMATHEMATICAL OBJECTS BEHAVE THE SAME ONLY VARYING IN THE NAME THEOBJECTS ARE SAID TO BE EM ISOMORPHIC SEE BOX REFBOXISOMORPHBEGINTEXTBOX09TEXTWIDTHISOMORPHISMLABELBOXISOMORPHBEGINQUOTESOURCEWILLIAM SHAKESPEAREWHATS IN A NAME THAT WHICH WE CALL A ROSE BY ANY OTHER NAME WOULD SMELL AS SWEETENDQUOTESOURCEISOMORPHISM DENOTES THE FACT THAT TWO OBJECTS MAY HAVE THE SAMEOPERATIONAL BEHAVIOR EVEN IF THEY HAVE DIFFERENT NAMES INDEXISOMORPHISMAS AN EXAMPLE CONSIDER THE FOLLOWING TWO OPERATIONS FOR TWO GROUPSCALLED LA G1RA AND LA G2RABEGINCENTER    BEGINTABULARCCCCC    00011011 HLINE0000011011 0101001110 1010110001 1111100100    ENDTABULARQQUADQQUAD  BEGINTABULARCCCCCABCD HLINEAABCD BBADC CCDAB DDCBA  ENDTABULARENDCENTERCAREFUL COMPARISON OF THESE ADDITION TABLES REVEALS THAT THE SAMEOPERATION OCCURS IN BOTH TABLES BUT THE NAMES OF THE ELEMENTS AND THEOPERATOR HAVE BEEN CHANGEDMORE GENERALLY WE DESCRIBE AN ISOMORPHISM AS FOLLOWS  LET G1 ANDG2 BE TWO ALGEBRAIC OBJECTS EG GROUPS FIELDS VECTOR SPACESETC  LET  BE A BINARY OPERATION ON G1 AND LET CIRC BE THECORRESPONDING OPERATION ON G2  LET PHIMC  G1 RIGHTARROW G2BE A BF ONETOONE AND ONTO INVERTIBLE FUNCTION  FOR ANY XYIN G1 LET S  PHIX QQUAD TEXTANDQQUAD T  PHIYWHERE S IN G2 AND T IN G2  THEN PHI IS AN ISOMORPHISM IF PHIX  Y  PHIX CIRC PHIYNOTE THAT THE OPERATION ON THE LEFT TAKES PLACE IN G1 WHILE THEOPERATION ON THE RIGHT TAKES PLACE IN G2  ENDTEXTBOXBEGINEXAMPLE LABELEXMISO  USING THE VECTOR SPACE OF EXAMPLE REFEXMVS1 WE FIND  V OPLUS W 000000100000000010100010UNDER THE MAPPING PHIVBFWBF  VBF WBF WE FIND PHIV OPLUS W 000100010110WHICH IS THE SAME AS FOUND IN VW IN EXAMPLE REFEXMVS1  VECTORSPACE OPERATIONS ADDITION MULTIPLICATION BY A SCALAR ETC ON VOPLUS W HAVE EXACTLY ANALOGOUS RESULTS ON PHIVOPLUS W SO VOPLUS W AND VW ARE ISOMORPHICENDEXAMPLETHE DIRECT SUM V OPLUS W IS COMMONLY EMPLOYED BETWEEN ORTHOGONALVECTOR SPACES IN THE ISOMORPHIC FORM THAT IS AS THE SUM OF THEELEMENTS  THIS IS JUSTIFIED BECAUSE ORTHOGONAL SPACES ARE DISJOINTSEE EXERCISE REFEXORTHODISTHE FOLLOWING THEOREM INDICATES WHEN VW AND V OPLUS W AREISOMORPHICBEGINTHEOREM LABELTHMVWISO CITEPAGE 199NAYLORSELL LET V  AND W BE LINEAR   SUBSPACES OF A LINEAR SPACE S  THEN VW AND V OPLUS W ARE  ISOMORPHIC IF AND ONLY IF V AND W ARE DISJOINTENDTHEOREMBECAUSE OF THIS THEOREM WHEN V AND W ARE DISJOINT IT IS FREQUENTTO WRITE VW IN PLACE OF V OPLUS W AND VICE VERSA  CARE SHOULDBE TAKEN HOWEVER TO UNDERSTAND WHAT SPACE IS ACTUALLY INTENDEDBEGINEXERCISESITEM SHOW THAT THE SET VOPLUS W HAS THE SAME ALGEBRAIC STRUCTURE  AS DOES THE SET V  W ESTABLISHING THAT THE ISOMORPHISM HOLDITEM CITEP 200NAYLORSELL LET X  L2PIPI AND LET S1  LSPAN1COS TCOS 2TLDOTSQQUAD QQUAD S2 LSPANSIN T SIN 2T LDOTSITEM SHOW THAT S1 OPLUS S2 AND S1  S2 ARE ISOMORPHICITEM SHOW THAT DIMENSIONS1 OPLUS S2  DIMENSIONS1  DIMENSIONS2ITEM LET S BE A LINEAR SPACE AND ASSUME THAT S  S1  S2   CDOTS  SN WHERE THE SI ARE ARE MUTUALLY DISJOINT LINEAR  SUBSPACES OF S  LET BI BE A HAMEL BASIS OF SI  SHOW THAT  B  B1 CUP B2 CDOTS CUP BN IS A HAMEL BASIS FOR SITEM LABELEXORTHODIS SHOW THAT  BEGINENUMERATE  ITEM IF V AND W ARE ORTHOGONAL SUBSPACES THEN THEY ARE    DISJOINT  ITEM IF V AND W ARE DISJOINT THEY ARE NOT NECESSARILY    ORTHOGONAL  ENDENUMERATEITEM PROVE LEMMA REFLEMVWUNIQUEENDEXERCISESSECTIONPROJECTIONS AND ORTHOGONAL PROJECTIONSLABELSECPROJECTIONSAS POINTED OUT IN LEMMA REFLEMVWUNIQUE IF V AND W AREDISJOINT SUBSPACES OF A LINEAR SPACE S THEN ANY VECTOR XBF IN SCAN BE UNIQUELY WRITTEN AS  XBF   VBF  WBFWHERE VBF IN V AND WBF IN W  THIS REPRESENTATION ISILLUSTRATED IN FIGURE REFFIGDISJOINT2BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRDISJOINT2    CAPTIONDECOMPOSITION OF XBF INTO DISJOINT COMPONENTS    LABELFIGDISJOINT2  ENDCENTERENDFIGURELET US INTRODUCE PROJECTION INDEXPROJECTION OPERATOR PMC SRIGHTARROW V WITH THE FOLLOWING OPERATION FOR ANY XBF IN S WITHTHE DECOMPOSITION XBF  VBF  WBFLET P XBF  VBFTHAT IS THE PROJECTION OPERATOR RETURNS THAT COMPONENT OF XBFWHICH LIES IN V  IF XBFIN V TO BEGIN WITH THEN OPERATION BYP DOES NOT CHANGE THE VALUE OF XBF  THUS SINCE PXBF IN VWE SEE THAT PPXBF  PXBF  THIS MOTIVATES THE FOLLOWINGDEFINITIONBEGINDEFINITION  A LINEAR TRANSFORMATION P OF A LINEAR SPACE INTO ITSELF IS A BF    PROJECTION IF P2  P INDEXPROJECTIONENDDEFINITIONNOINDENT AN OPERATOR P SUCH THAT P2  P IS SAID TO BE EM  IDEMPOTENT INDEXIDEMPOTENTIF V IS A LINEAR SUBSPACE AND P IS AN OPERATOR THAT PROJECTS ONTOV THE PROJECTION OF A VECTOR XBF ONTO V IS SOMETIMES DENOTED ASXPROJ VTHE RANGE AND NULLSPACE OF A PROJECTION OPERATOR PROVIDE ADISJOINT DECOMPOSITION OF A VECTOR SPACE AS THE FOLLOWING THEOREMSHOWSBEGINTHEOREM LABELTHMRNP  LET P BE A PROJECTION OPERATOR DEFINED ON A LINEAR SPACE S  THEN THE RANGE AND NULLSPACE OF P ARE DISJOINT LINEAR SUBSPACES OF  S AND S  RANGEP  NULLSPACEP  THAT IS RANGEP AND  NULLSPACEP ARE ALGEBRAIC COMPLEMENTS  INDEXALGEBRAIC COMPLEMENTENDTHEOREMBEGINEXAMPLE  LET XT BE A SIGNAL WITH FOURIER TRANSFORM XOMEGA  THEN THE  TRANSFORMATION POMEGA0 OMEGA0 GEQ 0 DEFINED BY PXOMEGA  BEGINCASES  XOMEGA  TEXTFOR  OMEGA0 LEQ OMEGA LEQ OMEGA0 0  TEXTOTHERWISEENDCASESWHICH CORRESPONDS TO FILTERING THE SIGNAL WITH A BRICKWALLLOWPASS FILTER IS A PROJECTION OPERATIONENDEXAMPLEBEGINEXAMPLE  LET PT T GEQ 0 BE THE TRANSFORMATION ON THE FUNCTION XT DEFINED BY PTXT  BEGINCASES  XT  TEXTFOR T LEQ T LEQ T 0  TEXTOTHERWISEENDCASESTHIS IS A TIMETRUNCATION OPERATION AND IS A PROJECTIONENDEXAMPLEBEGINEXAMPLE  A MATRIX A IS SAID TO BE A EM SMOOTHING MATRIX IF THERE IS A  SPACE OF SMOOTH VECTORS V SUCH THAT FOR A VECTOR XBF IN V AXBF  XBFTHAT IS A SMOOTH VECTOR UNAFFECTED BY A SMOOTHING OPERATION  ALSOTHE LIMIT AINFTY  LIMPRIGHTARROW INFTY APEXISTS  AS AN ARBITRARY VECTOR THAT IS NOT ALREADY SMOOTH ISREPEATEDLY SMOOTHED IT BECOMES INCREASINGLY SMOOTH  BY THEREQUIREMENT THAT AXBF  XBF FOR XBF IN V IT IS CLEAR THAT THESET OF SMOOTH VECTORS IS IN FACT RANGEA AND A IS A PROJECTIONMATRIX  SMOOTHING MATRICES ARE DISCUSSED FURTHER INCITEGREVILLE1957GREVILLE1966ENDEXAMPLELET P BE A PROJECTION ONTO A CLOSED SUBSPACE V OF STHEN IP IS ALSO A PROJECTION SEE EXERCISEREFEXIMP   THEN WE CAN WRITE XBF  PXBF  IPXBFTHIS DECOMPOSES XBF INTO THE TWO PARTS P XBF IN VAND IPXBF IN WAS FIGURE REFFIGDISJOINT2 SUGGESTS THE SUBSPACES V AND WINVOLVED IN THE PROJECTION ARE NOT NECESSARILY ORTHOGONAL  HOWEVERIN MOST APPLICATIONS ORTHOGONAL SUBSPACES ARE NEEDED  THIS LEADS TOTHE FOLLOWING DEFINITIONBEGINDEFINITION  LET P BE A PROJECTION OPERATOR ON AN INNER PRODUCT SPACE S  P  IS SAID TO BE AN EM ORTHOGONAL PROJECTION IF ITS RANGE AND  INDEXORTHOGONAL PROJECTION  NULLSPACE ARE ORTHOGONAL RANGEP PERP NULLSPACEPENDDEFINITIONTHE NEED FOR AN ORTHOGONAL PROJECTION MATRIX IS PROVIDED BY THEFOLLOWING PROBLEM GIVEN A POINT XBF IN A VECTOR SPACE S AND ASUBSPACE V SUBSET S WHAT IS THE NEAREST POINT IN V TO XBFCONSIDER THE VARIOUS REPRESENTATIONS OF XBF SHOWN IN FIGUREREFFIGORTHOGPROJ1  AS SUGGESTED BY THE FIGURE DECOMPOSITION OFXBF AS XBF  VBF0  WBF0PROVIDES THE POINT VBF0IN V THAT IS CLOSEST TO XBF  THEVECTOR WBF0 IS ORTHOGONAL TO V WITH RESPECT TO THE INNERPRODUCT APPROPRIATE TO THE PROBLEM  OF THE VARIOUS WBF VECTORSTHAT MIGHT BE USED IN THE REPRESENTATION THE VECTORS WBF0WBF1 OR WBF2 IN THE FIGURE THE VECTOR WBF0 IS THE VECTOROF THE SHORTEST LENGTH AS DETERMINED BY THE NORM INDUCED BY THE INNERPRODUCT   PROOF OF THIS GEOMETRICALLY APPEALING AND INTUITIVE NOTIONIS PRESENTED IN THE NEXT SECTION AS THE PROJECTION THEOREM  BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODEINPUTPICTUREDIRORTHOGPROJ1    CAPTIONORTHOGONAL PROJECTION FINDS THE CLOSEST POINT IN V TO      XBF    LABELFIGORTHOGPROJ1  ENDCENTERENDFIGUREIT IS DIFFICULT TO OVERSTATE THE IMPORTANCE OF THE NOTION OFPROJECTION  PROJECTION IS THE KEY CONCEPT OF MOST STOCHASTICFILTERING AND PREDICTION THEORY IN SIGNAL PROCESSING  CHAPTERREFCHAPVECTAP CONTAINS SEVERAL APPLICATIONS OF THIS IMPORTANTCONCEPTANOTHER VIEWPOINT OF THE PROJECTION THEOREM IS REPRESENTED IN FIGUREREFFIGORTHOGPROJ2  SUPPOSE THAT V IS THE SPAN OF THE BASISVECTORS PBF1PBF2 AS SHOWN  THEN THE NEAREST POINT TOXBF IN V IS THE POINT VBF0 AND THE VECTOR WBF0 IS THEDIFFERENCE  IF WBF0 IS ORTHOGONAL TO VBF0 THEN IT MUST BEORTHOGONAL TO PBF1 AND PBF2  BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODEINPUTPICTUREDIRORTHOGPROJ2    CAPTIONORTHOGONAL PROJECTION ONTO THE SPACE SPANNED BY SEVERAL VECTORS    LABELFIGORTHOGPROJ2  ENDCENTERENDFIGUREIF WE REGARD VBF0 AS AN APPROXIMATION TO XBF THAT MUST LIE INTHE SPAN OF PBF1 AND PBF2 THEN WBF0  XBF  VBF0IS THE APPROXIMATION ERROR  CONSIDER THE VECTORS PBF1 ANDPBF2 AS THE DATA FROM WHICH THE APPROXIMATION IS TO BE FORMEDTHEN EM THE LENGTH OF THE APPROXIMATION ERROR VECTOR WBF0 IS  MINIMIZED WHEN THE ERROR IS ORTHOGONAL TO THE DATA SUBSECTIONPROJECTION MATRICESLABELSECPROJMATINDEXPROJECTION MATRIXLET US RESTRICT OUR ATTENTION FOR THE MOMENT TO FINITEDIMENSIONALVECTOR SPACES  LET A BE AN MATSIZEMN MATRIX WRITTEN AS A  PBF1PBF2LDOTSPBFNAND LET THE SUBSPACE V BE THE COLUMN SPACE OF A  V  LSPANPBF1 PBF2 LDOTS PBFN  RANGEAASSUME THAT WE ARE USING THE USUAL INNER PRODUCT LA XBF YBFRA XBFH YBF  THEN AS WE SEE IN THE NEXT CHAPTER THEPROJECTION MATRIX PA THAT PROJECTS ORTHOGONALLY ONTO THE COLUMNSPACE OF A IS BEGINEQUATION PA  AAHA1AHLABELEQPROJMAT1ENDEQUATIONINDEXPROJECTION MATRIXBEGINTHEOREM LABELTHMSYMPROJ  ANY HERMITIAN SYMMETRIC MATRIX WITH P2  P IS AN ORTHOGONAL  PROJECTION MATRIXENDTHEOREMLOOKING AHEAD TO WHERE THESE CONCEPTS ARE DEFINED IT CAN BE SHOWNTHAT ANY SELFADJOINT BOUNDED LINEAR OPERATOR P WITH P2P IS APROJECTION OPERATORBEGINPROOF  THE OPERATION PXBF IS A LINEAR COMBINATION OF THE COLUMNS OF P  TO SHOW THAT P IS AN ORTHOGONAL PROJECTION WE MUST SHOW THAT XBF   PXBF IS ORTHOGONAL TO THE COLUMN SPACE OF P  FOR ANY VECTOR  PCBF IN THE COLUMN SPACE OF P XBF PXBFH PCBF  XBFHPP2CBF  0SO XBF  PXBF IS ORTHOGONAL TO THE COLUMN SPACE OF PENDPROOFIT WILL OCCASIONALLY BE USEFUL TO DO THE PROJECTION USING A WEIGHTEDINNER PRODUCT  LET THE INNER PRODUCT BEBEGINEQUATION LA XBF YBFRAW  XBFH W YBFLABELEQWIPPRENDEQUATIONWHERE W IS A POSITIVEDEFINITE HERMITIAN MATRIX  THE INDUCED NORMIS  XBFW2  LA XBF XBFRAW  XBFH W XBFLET A BE AN MATSIZEMN MATRIX AS BEFORE  THEN THE PROJECTIONMATRIX WHICH PROJECTS ORTHOGONALLY ONTO THE COLUMN SPACE OF A WHERETHE ORTHOGONALITY IS ESTABLISHED USING THE INNER PRODUCTREFEQWIPPR IS THE MATRIXBEGINEQUATION PAW  AAH W A1 AH WLABELEQPROJMAT2ENDEQUATIONBEGINEXERCISESITEM IF VBF IS A VECTOR SHOW THAT THE MATRIX WHICH PROJECTS ONTO LSPANVBF IS PV  FRACVBF VBFHVBFHVBFITEM SHOW THAT THE MATRIX PA IN REFEQPROJMAT1 IS A PROJECTION  MATRIXITEM FOR THE PROJECTION MATRIX PAW INREFEQPROJMAT2  BEGINENUMERATE  ITEM SHOW THAT PAW2  PAW  ITEM SHOW THAT PAWPERP  IPAW IS ORTHOGONAL TO    PAW USING THE WEIGHTED INNER PRODUCT  THAT IS    PAWH W PAWPERP  0  ENDENUMERATE  ITEM LET  PBF1  BEGINBMATRIX 123  4 ENDBMATRIX QQUADPBF2  BEGINBMATRIX 4  2  6  7 ENDBMATRIX QQUADPBF3  BEGINBMATRIX 3  4  2  1 ENDBMATRIXAND  XBF  BEGINBMATRIX 1  2  3  7 ENDBMATRIXDETERMINE THE NEAREST VECTOR XBFHAT INLSPANPBF1PBF2PBF3  ALSO DETERMINE THE ORTHOGONALCOMPLEMENT OF XBF IN LSPANPBF1PBF2PBF3ITEM LET A BE A MATRIX WHICH CAN BE FACTORED ASBEGINEQUATIONA  U SIGMA VHLABELEQSVDEXER1ENDEQUATIONSUCH THAT UAH UA  I QQUAD VAH VA  I AND SIGMAA IS  DIAGONAL MATRIX WITH REAL VALUESTHE FACTORIZATION IN REFEQSVDEXER1 IS THE SINGULAR VALUEDECOMPOSITION SEE CHAPTER REFCHASVDSHOW THAT PA  PUA  ITEM TWO ORTHOGONAL PROJECTION OPERATORS PA AND PB ARE SAID TO  BE ORTHOGONAL IF PAPB  0  SHOW THAT  BEGINENUMERATE  ITEM PA AND PB ARE ORTHOGONAL IF AND ONLY IF THEIR RANGES ARE    ORTHOGONAL  ITEM PAPB IS A PROJECTION OPERATOR IF AND ONLY IF PA AND    PB  ARE ORTHOGONAL  ENDENUMERATEITEM SHOW THAT THE CBF WHICH MINIMIZES REFEQWPROJ1 IS CBF  AHWA1AHW XBFSO THAT THE WEIGHTED PROJECTION OF XBF IS ACBF AAHWA1AHW XBFITEM PROVE THEOREM REFTHMRNPITEM LABELEXIMP IF P IS A PROJECTION OPERATOR SHOW THAT IP  IS A PROJECTION OPERATOR  DETERMINE THE RANGE AND NULLSPACE OF  IPITEM LET S BE A VECTOR SPACE AND LET V1V2LDOTS VN BE  LINEAR SUBSPACES SUCH THAT VI IS DISJOINT FROM SUMJ NEQ I  VJ FOR EACH I  LET PJ BE THE PROJECTION ON S FRO WHICH  RANGEPJ  VJ AND NULLSPACEPJ  SUMJ NEQ K VK  DEFINE AN OPERATOR T  LAMBDA1 P1  LAMBDA2 P2  CDOTS  LAMBDAN PNBEGINENUMERATEITEM SHOW THAT IF XBF IN VJ THEN T XBF  LAMBDAJ XBFITEM SHOW THAT T IS A PROJECTION IF AND ONLY LAMBDAJ IS EITHER  0 OR 1ENDENUMERATELET P1P2LDOTS PM BE A SET OF ORTHOGONAL PROJECTIONS WITH PIPJ  0 FOR I NEQ JBEGINENUMERATEITEM SHOW THAT Q  P1  P2  CDOTS  PM IS AN ORTHOGONAL  PROJECTIONITEM WHAT HAPPENS IF P1 P2 NEQ 0ENDENUMERATEITEM LET A AND B BE MATRICES SUCH THAT AHB  0 SEEREFEQABORTHOG  THEN V  RANGEA AND W  RANGEB AREORTHOGONAL  BEGINENUMERATEITEM SHOW THAT PA  I  PBITEM SHOW THAT A XBF CAN BE DECOMPOSED AS XBF  PA XBF  PBXBF  PA XBF  IPAXBFENDENUMERATEENDEXERCISESSECTIONTHE PROJECTION THEOREMBEGINQUOTESOURCEMICHAEL SPIVAKEM CALCULUS ON MANIFOLDS  IMPORTANT ATTRIBUTES OF MANY FULLY EVOLVED MAJOR THEOREMS  BEGINENUMERATE  ITEM IT IS TRIVIAL  ITEM IT IS TRIVIAL BECAUSE THE TERMS APPEARING IN IT HAVE BEEN    PROPERLY DEFINED  ITEM IT HAS SIGNIFICANT CONSEQUENCES  ENDENUMERATEENDQUOTESOURCETHE MAIN PURPOSE OF THIS SECTION IS TO PROVE THE GEOMETRICALLYINTUITIVE NOTION INTRODUCED IN THE PREVIOUS SECTION THE POINT VBF0IN V THAT IS CLOSEST TO A POINT XBF IS THE ORTHOGONAL PROJECTIONOF XBF ONTO VBEGINTHEOREM LABELTHMPROJ THE PROJECTION THEOREM  INDEXPROJECTION THEOREM CITELUENBERGER1969 LET S BE A  HILBERT SPACE AND LET V BE A CLOSED SUBSPACE OF S  FOR ANY  VECTOR XBF IN S THERE EXISTS A EM UNIQUE VECTOR VBF0 IN  V CLOSEST TO XBF THAT IS  XBF VBF0 LEQ  XBF  VBF   FOR ALL VBF IN V  FURTHERMORE THE POINT VBF0 IS THE  MINIMIZER OF  XBF VBF0 IF AND ONLY IF XBF  VBF0 IS  ORTHOGONAL TO VENDTHEOREMTHE IDEA BEHIND THE THEOREM IS SHOWN IN FIGURE REFFIGPROJ1BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRPROJ1    CAPTIONTHE PROJECTION THEOREM    LABELFIGPROJ1  ENDCENTERENDFIGUREBEGINPROOF  THERE ARE SEVERAL ASPECTS OF THIS THEOREM    BEGINENUMERATE  ITEM THE FIRST AND MOST TECHNICAL ASPECT IS THE EM EXISTENCE    OF THE MINIMIZING POINT VBF0  ASSUME XBF NOT IN V AND    LET DELTA  INFVBF IN V XBF  VBF  WE NEED TO SHOW    THAT THERE IS A VBF0 IN V WITH XBF  VBF0 DELTA    LET VBFI BE A SEQUENCE OF VECTORS IN V SUCH THAT X     VI RIGHTARROW DELTA  WE WILL SHOW THAT VI IS A    CAUCHY SEQUENCE HENCE HAS A LIMIT IN S  BY    REFEQPARALLELOGRAM VBFJXBF  XBF  VBFI 2  VBFJ XBF XBFVBFI2  2VBFJ  XBF2  2XBFVBFI2THE LATTER CAN BE REARRANGED AS VBFJ  VBFI2  2VBFJ  XBF2  2XBFVBFI2 4XBF  VBFIVBFJ22SINCE S IS A VECTOR SPACE VBFIVBFJ2 IN S  ALSO BY THEDEFINITION OF DELTA XBF VBFIVBFJ2 GEQ DELTASO THAT  VBFI  VBFJ2 LEQ 2VBFJ  XBF  2XBFVBFI 4DELTA2 THEN SINCE VBFI IS DEFINED SO THAT VBFJXBF RIGHTARROWDELTA2 WE CONCLUDE THAT VBFIVBFJ2 RIGHTARROW 0SO VBFI IS A CAUCHY SEQUENCE  SINCE V IS A HILBERT SPACE ASUBSPACE OF S THE LIMIT EXISTS AND V0 IN VITEM LET US NOW SHOW THAT IF VBF0 MINIMIZES  XBF  VBF0  THEN XBF    VBF0 PERP V  LET VBF0 BE THE NEAREST  VECTOR TO XBF IN V LET VBF BE  A UNITNORM VECTOR IN V SUCH THAT CONTRARY TO THE STATEMENT OF  THE THEOREM LA XBF  VBF0 VBFRA  DELTA NEQ 0LET ZBF  VBF0  DELTA VBF IN V FOR SOME NUMBER DELTATHEN BEGINALIGNED XBF  ZBF 2   XBF  VBF0 2  2 REALLA XBF VBF0 DELTA VBF RA   DELTA VBF2    XBF  VBF02  DELTA2   XBF  VBF02ENDALIGNEDTHIS IS A CONTRADICTION HENCE DELTA  0ITEM CONVERSELY SUPPOSE THAT XBFVBF0 PERP V  THEN FOR ANY VBFIN V WITH VBF NEQ VBF0BEGINALIGN XBF  VBF2   XBF  VBF0  VBF0  VBF2 NONUMBER   XBF  VBF02   VBF0  VBF2 LABELEQHOTH GEQ  XBF  VBF02ENDALIGNWHERE ORTHOGONALITY IS USED TO OBTAIN REFEQHOTHITEM INDEXUNIQUENESS UNIQUENESS OF THE NEAREST POINT IN V TO  XBF MAY BE SHOWN AS FOLLOWS  SUPPOSE THAT XBF  VBF1   WBF1  VBF2  WBF2 WHERE WBF1  XBF  VBF1 PERP V AND  WBF2  XBF  VBF2 PERP V FOR SOME VBF1 VBF2 IN V  THEN 0  VBF1  VBF2  WBF1  WBF2 OR VBF2  VBF1  WBF1WBF2BUT SINCE VBF2VBF1 IN V IT FOLLOWS THAT WBF1WBF2 INV SO WBF1  W2 HENCE VBF1 VBF2  ENDENUMERATEENDPROOFBASED ON THE PROJECTION THEOREM EVERY VECTOR IN A HILBERT SPACE SCAN BE EXPRESSED UNIQUELY AS THAT PART WHICH LIES IN A SUBSPACE VAND THAT PART WHICH IS ORTHOGONAL TO VBEGINTHEOREM CITELUENBERGER1969 LABELTHMHDECOMP  LET V BE A CLOSED LINEAR SUBSPACE OF A HILBERT SPACE S  THEN S  V OPLUS VPERPAND V  VPERPPERPENDTHEOREMTHE ISOMORPHIC INTERPRETATION OF THE DIRECT SUM IS IMPLIED IN THISNOTATIONBEGINPROOF  LET XBF IN S  THEN BY THE PROJECTION THEOREM THERE IS A UNIQUE  VBF0 IN V SUCH THAT  XBF  VBF0 LEQ XBF  VBF   FOR ALL VBF IN V AND WBF0  XBF VBF0 IN VPERP  WE CAN THUS  DECOMPOSE ANY VECTOR IN S INTO XBF  VBF0  WBF0 QQUADTEXTWITH  VBF0 IN V WBF0 INVPERPTO SHOW THAT V  VPERPPERP WE NEED TO SHOW ONLY THATVPERPPERP SUBSET V SINCE WE ALREADY KNOW BY THEOREMREFTHMORTHOGCOMP THAT V SUBSET VPERPPERP  LET XBF INVPERPPERP  WE WILL SHOW THAT IT IS ALSO TRUE THAT XBF IN VBY THE FIRST PART WE CAN WRITE XBF  VBF  WBF WHERE VBF INV AND WBF IN VPERP  BUT SINCE V SUBSET VPERPPERP WEHAVE VBF IN VPERPPERP SO THAT WBF  XBF  VBF IN VPERPPERPSINCE WBF IN VPERP AND WBF IN VPERPPERP WE MUST HAVEWBF PERP W OR W  ZEROBF THUS XBF  VBF IN VENDPROOFTHIS THEOREM APPLIES TO HILBERT SPACES WHERE BOTH COMPLETENESS AND ANINNER PRODUCT DEFINING ORTHOGONALITY ARE AVAILABLEBEGINEXERCISES ITEM PROVE LEMMA REFLEMISO1ITEM SHOW THAT IF V AND W ARE CLOSED SUBSPACES OF A HILBERT  SPACE S AND IF V PERP W THEN VW IS A CLOSED SUBSPACE OF  S NS P 297ENDEXERCISESSECTIONORTHOGONALIZATION OF VECTORSLABELSECGRAMSCHMITINDEXGRAMSCHMIDT PROCESS IN MANY APPLICATIONS COMPUTATIONSINVOLVING BASIS VECTORS ARE EASIER IF THE VECTORS ARE ORTHOGONAL  ITIS THEREFORE USEFUL TO BE ABLE TO TAKE A SET OF VECTORS T ANDPRODUCE AN ORTHOGONAL SET OF VECTORS T WITH THE SAME SPAN AS TTHIS IS WHAT THE GRAMSCHMIDT ORTHOGONALIZATION PROCEDURE DOES  THEGRAMSCHMIDT PROCEDURE CAN ALSO BE USED TO DETERMINE THE DIMENSION OFTHE SPACE SPANNED BY A SET OF VECTORS SINCE A VECTOR LINEARLYDEPENDENT ON OTHER VECTORS EXAMINED PRIOR IN THE PROCEDURE YIELDS AZERO VECTORGIVEN A SET OF VECTORS T   PBF1PBF2LDOTSPBFN WE WANT TO FINDA SET OF VECTORS T  QBF1QBF2ALLOWBREAKLDOTSALLOWBREAK QBFN  WITH N LEQ N SO THATLSPANQBF1QBF2LDOTSQBFN   LSPANPBF1PBF2LDOTSPBFN AND  LA QBFIQBFJ RA  DELTAIJASSUME THAT NONE OF THE PBFI VECTORS ARE ZERO VECTORSTHE PROCESS WILL BE DEVELOPED STEPWISE  THE NORM  CDOT  INTHIS SECTION IS THE INDUCED NORMBEGINENUMERATEITEM NORMALIZE THE FIRST VECTOR QBF1  FRACPBF1PBF1ITEM COMPUTE THE DIFFERENCE BETWEEN THE PROJECTION OF PBF2 ONTO  QBF1 AND PBF2  BY THE ORTHOGONALITY THEOREM THIS IS  ORTHOGONAL TO P1 EBF2  PBF2  FRACLA PBF2QBF1 RA  QBF1 2 QBF1   PBF2  LA PBF2QBF1 RA QBF1IF EBF2  0 THEN QBF2 IN LSPANQBF1 AND CAN BEDISCARDED WE WILL ASSUME THAT SUCH DISCARDS ARE DONE AS NECESSARY INWHAT FOLLOWS  IF EBF2 NEQ 0 THEN NORMALIZE QBF2  FRACEBF2EBF2THESE STEPS ARE SHOWN IN FIGURE REFFIGGS1BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRGRAMSCHMIDT1    CAPTIONTHE FIRST STEPS OF THE GRAMSCHMIDT PROCESS    LABELFIGGS1  ENDCENTERENDFIGUREITEM AT THE NEXT STAGE A VECTOR ORTHOGONAL TO QBF1 AND QBF2  IS OBTAINED FROM THE ERROR BETWEEN PBF3 AND ITS PROJECTION ONTO  LSPANQBF1QBF2 EBF3  PBF3  LA PBF3QBF1 QBF1  LA PBF3 QBF2RA  QBF2THIS IS NORMALIZED TO PRODUCE QBF3 QBF3  FRACQBF3QBF3SEE FIGURE REFFIGGS2BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRGRAMSCHMIDT2    CAPTIONTHIRD STEP OF THE GRAMSCHMIDT PROCESS    LABELFIGGS2  ENDCENTERENDFIGUREITEM NOW PROCEED INDUCTIVELY  TO FORM THE NEXT ORTHOGONAL VECTOR  USING PBFK DETERMINE THE COMPONENT ORTHOGONAL TO ALL PREVIOUSLY  FOUND VECTORSBEGINEQUATION EBFK  PBFK  SUMI1K1 LA PBFKQBFIRA  QBFILABELEQGSFORMENDEQUATIONAND NORMALIZEBEGINEQUATION QBFK  FRACEBFKEBFK2LABELEQGSFORM2ENDEQUATIONENDENUMERATEBEGINEXAMPLE LABELEXMLEGENDREPOLY  THE SET OF FUNCTIONS 1TT2LDOTSTM DEFINED OVER 11  FORMS A LINEARLY INDEPENDENT SET LET THE INNER PRODUCT BE LA FGRA  INT11 FTGTDTBY THE GRAMSCHMIDT PROCEDURE WE FINDBEGINALIGNED Q0T  FRAC1SQRT2  Q1T  SQRT32 T  Q2T  FRAC3SQRT522T213  Q3T  FRAC5SQRT722T3  3T5  VDOTSENDALIGNEDTHE FUNCTIONS SO OBTAINED ARE KNOWN AS THE EM LEGENDRE POLYNOMIALSTHEY ARE FREQUENTLY WRITTEN WITHOUT THENORMALIZATION ASBEGINALIGNED Y0T  1  Y1T  T  Y2T  T213  Y3T  T3  3T5  VDOTSENDALIGNEDINDEXLEGENDRE POLYNOMIALIF WE CHANGE THE INNER PRODUCT TO INCLUDE A WEIGHTING FUNCTION LA FGRA  INT11 FRAC1SQRT1T2 FTGTDTTHEN THE ORTHOGONAL POLYNOMIALS OBTAINED BY APPLYING THE GRAMSCHMIDTPROCESS TO THE POLYNOMIALS 1TLDOTSTN ARE THE CHEBYSHEVPOLYNOMIALS DESCRIBED IN EXAMPLE REFEXMCHEBYPOL INDEXCHEBYSHEV POLYNOMIAL INDEXORTHOGONAL POLYNOMIALENDEXAMPLESUBSUBSECTIONA MATRIXBASED IMPLEMENTATIONFOR FINITEDIMENSIONAL VECTORS THE GRAMSCHMIDT PROCESS CAN BEREPRESENTED IN A MATRIX FORM  LET A  PBF1PBF2LDOTSPBFN BE A MATSIZEMN MATRIX  THE ORTHOGONALVECTORS OBTAINED BY THE GRAMSCHMIDT PROCESS ARE STACKED IN A MATRIXQ  QBF1QBF2LDOTSQBFN TO BE DETERMINED WHERE N N  WE LET THE UPPER TRIANGULAR MATRIX R HOLD THE INNER PRODUCTSAND NORMS FROM REFEQGSFORM AND REFEQGSFORM2 R  BEGINBMATRIX PBF1  LA PBF2QBF1RA  LA PBF3QBF1 RA  CDOTS LA PBFN QBF1 RA              EBF2           LA PBF3QBF2 RA  CDOTS            LA PBFNQBF2 RA                                     EBF3           CDOTS            LA PBFNQBF3 RA                                                                 VDOTS                                                          CDOTS  EBFN ENDBMATRIXTHE INNER PRODUCTS IN THE SUMMATION SUMI1K1 LAPBFKQBFIRA QBFI ARE REPRESENTED BY RSUBRANGE1K1K QMCSUBRANGE1K1HAMCK AND THE SUM IS THENQMCSUBRANGE1K1RSUBRANGE1K1K  WE THUS OBTAIN THEFACTORIZATION  INDEXMATRIX FACTORIZATIONSQR INDEXQR FACTORIZATIONUSING GRAMSCHMIDT INDEXFACTORIZATIONSSEEMATRIX FACTORIZATIONS A  QRALGORITHM REFALGQR1 ILLUSTRATES A SC MATLAB IMPLEMENTATION OFTHIS GRAMSCHMIDT PROCESSBEGINNEWPROGENVGRAMSCHMIDT    ALGORITHM QR    FACTORIZATIONGRAMSCHMIDT1MQR1GRAMSCHMIDT    ALGORITHMENDNEWPROGENVWITH THE OBSERVATION FROM REFEQGSFORM THAT PBFK  QBFK RKK  SUMI1K1 RIKQBFIWE NOTE THAT WE CAN WRITE A IN A FACTORED FORM AS A  QR AND THATQ SATISFIES QHQ  I  NOTE THAT BOXEDTEXTTHE MATRIX Q PROVIDES AN ORTHOGONAL BASIS TO THE COLUMN  SPACE OF A FOR FINITEDIMENSIONAL VECTORS THE COMPUTATIONS OF THE GRAMSCHMIDTPROCESS MAY BE NUMERICALLY UNSTABLE FOR POORLY CONDITIONED MATRICESEXERCISE REFEXMGS DISCUSSES A MODIFIED GRAMSCHMIDT WHILE OTHERMORE NUMERICALLY STABLE METHODS OF ORTHOGONALIZATION ARE EXPLORED INCHAPTER REFCHAPMATFACTBEGINEXERCISESITEM MODIFY ALGORITHM REFALGQR1 SO THAT IT ONLY RETAINS COLUMNS  OF Q THAT ARE NONZERO MAKING CORRESPONDING ADJUSTMENTS TO R  COMMENT ON THE PRODUCT QR IN THIS CASE GRAMSCHMIDT2MITEM MODIFY ALGORITHM REFALGQR1 TO COMPUTE A SET OF ORTHOGONAL  VECTORS WITH RESPECT TO THE WEIGHTED INNER PRODUCT LA XBF YBF  RA  XBFT W YBF FOR A POSITIVE DEFINITE SYMMETRIC MATRIX WITEM DETERMINE THE FIRST FOUR POLYNOMIALS ORTHOGONAL OVER 01  A  SYMBOLIC MANIPULATION PACKAGE IS RECOMMENDEDITEM USING A SYMBOLIC MANIPULATION PACKAGE WRITE A FUNCTION WHICH  PERFORMS THE GRAMSCHMIDT ORTHOGONALIZATION OF A SET OF FUNCTIONSITEM FOR THE FUNCTIONS SHOWN IN FIGURE REFFIGGSEX DETERMINE A  SET OF ORTHOGONAL FUNCTIONS SPANNING THE SAME SPACE USING THE  FUNCTIONS IN THE ORDER SHOWN   SOMETHING LIKE THE PROAKIS EXERCISE  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODE      INPUTPICTUREDIRGRAMSCHMIDT3      CAPTIONFUNCTIONS TO ORTHOGONALIZE      LABELFIGGSEX    ENDCENTER  ENDFIGUREITEM MODIFIED GRAMSCHMIDT INDEXMODIFIED GRAMSCHMIDT THE  COMPUTATIONS OF THE GRAMSCHMIDT ALGORITHM CAN BE REORGANIZED TO BE  MORE STABLE NUMERICALLY  IN THESE MODIFIED COMPUTATIONS A COLUMN  OF Q AND A EM ROW OF R IS PRODUCED AT EACH ITERATION  THE  REGULAR GRAMSCHMIDT PROCESS PRODUCES A COLUMN OF Q AND A COLUMN  OF R AT EACH ITERATION  LET THE ROWS OF R BE DENOTED AS  RBFIT  BEGINENUMERATE  ITEM SHOW THAT FOR AN MATSIZEMN MATRIX A A  SUMI1K1  SUMIKN QBFI RBFIT  ZEROBFAKWHERE AK IS MATSIZEMNK1ITEM LET AK  EBFK B AND EXPLAIN WHY THE KTH COLUMN OF  Q AND THE KTH ROW OF R ARE GIVEN BY RKK   EBFK QQUAD QBFK  EBFKRKK QQUADRKK1LDOTSRKN  QBFKTBITEM THEN SHOW THAT THE NEXT ITERATION CAN BE STARTED BY COMPUTING A  SUMI1K QBFI RBFIT  ZEROBF AK1WHERE AK1  B  QBFKRKK1LDOTSRKKNITEM CODE THE MODIFIED GRAMSCHMIDT ALGORITHM IN SC MATLAB  ENDENUMERATEENDEXERCISESSECTIONSOME FINAL TECHNICALITIES FOR INFINITEDIMENSIONAL SPACESTHE CONCEPT OF BASIS THAT WAS INTRODUCED IN SECTIONREFSECHAMELBASIS WAS BASED UPON THE STIPULATION THAT LINEARCOMBINATIONS ARE EM FINITE SUMS  WITH THE ADDITIONAL CONCEPTS OFORTHOGONALITY AND NORMALITY WE CAN INTRODUCE A SLIGHTLY MODIFIEDNOTION OF A BASIS  A SET T  PBF1PBF2LDOTS IS SAID TO BEORTHONORMAL INDEXORTHONORMAL BASIS IF LA XBFI XBFJRA DELTAIJ  FOR AN ORTHONORMAL SET T IT CAN BE SHOWN THAT THEINFINITE SUM SUMI1INFTY CI PBFICONVERGES IF AND ONLY IF THE SERIES SUMI1N CI2 CONVERGESAN ORTHONORMAL SET OF BASIS FUNCTIONS PBF1PBF2LDOTS ISSAID TO BE A BF COMPLETEFOOTNOTEA COMPLETE SET OF FUNCTIONS IS  DIFFERENT FROM COMPLETE SPACE IN WHICH EVERY CAUCHY SEQUENCE HAS A  LIMIT  SET INDEXCOMPLETE SET FOR A HILBERT SPACE S IF EVERYXBF IN S CAN BE REPRESENTED AS XBF  SUMI1INFTY CI PBFIFOR SOME SET OF COEFFICIENTS CI  SEVERAL SETS OF COMPLETE BASISFUNCTIONS ARE PRESENTED IN CHAPTER REFCHAPVECTAP AFTER A MEANSHAS BEEN PRESENTED FOR FINDING THE COEFFICIENTS CI  A COMPLETESET OF FUNCTIONS WILL BE CALLED A BF BASIS MORE STRICTLY ANORTHONORMAL BASIS  THE BASIS AND THE HAMEL BASIS ARE NOT IDENTICALFOR INFINITEDIMENSIONAL SPACES  IN PRACTICE IT IS THE BASIS NOTTHE HAMEL BASIS WHICH IS OF MOST USE  IT CAN BE SHOWN THAT ANYORTHONORMAL BASIS IS A SUBSET OF A HAMEL BASISIN FINITE DIMENSIONS NONE OF THESE ISSUES HAVE ANY BEARING  ANORTHONORMAL HAMEL BASIS EM IS  AN ORTHONORMAL BASIS  ONLY THENOTION OF BASIS NEEDS TO BE RETAINED FOR FINITE DIMENSIONALSPACES  IN THE FUTURE WE WILL DROP THE ADJECTIVE HAMEL AND REFERONLY TO A BASIS FOR A FINITEDIMENSIONAL VECTOR SPACEANOTHER CONCEPT THAT WE HAVE DANCED AROUND UP TO THIS POINT BUT FORWHICH THE STUDENT SHOULD HAVE SUFFICIENT MATURITY BY NOW IS THE NOTIONOF A DENSE SETBEGINDEFINITION  LET XD BE A CLOSED METRIC SPACE AND LET D SUBSET X  THEN  D IS BF DENSE IN X IF FOR EACH X IN X AND EVERY EPSILON   0 THERE IS A POINT D IN D SUCH THAT  X  D  EPSILONENDDEFINITIONTHE POINT OF A DENSE SET D IS THAT EVERY ELEMENT IN THE LARGER SETX IS SUFFICIENTLY CLOSE FOR ANY MEASURE OF SUFFICIENCY TO ANELEMENT OF D  ANOTHER DEFINITION OF A DENSE SET IS A SET DSUBSET X IS DENSE IN X IF THE CLOSE OF D IS XTHE MOST FAMOUS EXAMPLE OF DENSE SETS IS THE SET OF RATIONAL NUMBERSAS A SUBSET OF THE REAL NUMBERS  EVERY REAL NUMBER IS ARBITRARILYCLOSE TO SOME RATIONAL NUMBER  THE POINT OF DENSE SETS IS THAT IN MANY CASES WE CAN FOCUS ATTENTIONON THE DENSE SUBSET D WHICH IS USUALLY MUCH SMALLER THAN THEORIGINAL SET X  STATEMENTS WHICH ARE TRUE ON D CAN OFTEN BEEXTENDED TO STATEMENTS WHICH ARE TRUE ON XWHILE ON THIS TOPIC WE ROUND OUT OUR DISCUSSION WITH THE FOLLOWINGDEFINITION  BEGINDEFINITION  A NORMED SPACE X IS BF SEPARABLE IF IT CONTAINS A COUNTABLE  DENSE SETENDDEFINITIONTHE SET RBB IS SEPARABLE THE RATIONAL NUMBERS ARE COUNTABLE  ASSOME OTHER EXAMPLES OF SEPARABLE AND NONSEPARABLE SETS WE PRESENT THEFOLLOWING FOR PROOFS SEE CITEPAGE 43LUENBERGER1968BEGINENUMERATEITEM LET LP SPACES WITH P  INFTY ARE SEPARABLE LINFTY IS  NOT SEPARABLEITEM THE LP SPACES WITH P  INFTY ARE SEPARABLE  LINFTY  IS NOT SEPARABLEITEM THE SPACE CAB IS SEPARABLEENDENUMERATESETEXSECTREFSECNORM1BEGINEXERCISESEXSKIPITEM WE WILL EXAMINE THE LINFTY METRIC TO GET A SENSE AS TO WHY  IT SELECTS THE MAXIMUM VALUE  GIVEN THE VECTOR XBF  12345  6 COMPUTE THE LP METRIC DPXBFZEROBF FOR  P12410100INFTY  COMMENT ON WHY DPXBFZEROBF  RIGHTARROW MAXXI AS PRIGHTARROW INFTYITEM LET X BE AN ARBITRARY SET  SHOW THAT THE FUNCTION DEFINED BY DXY  BEGINCASES  1  X  Y   0  X NEQ 0ENDCASESIS A METRICITEM VERIFY THAT THE HAMMING DISTANCE DHXBFYBF INTRODUCED IN  EXAMPLE REFEXMHD1 IS A METRICITEM VERIFY THAT THE CODE SPACE OF EXAMPLE REFEXMCODESPACE IS A  METRIC SPACEITEM PROOF OF THE TRIANGLE INEQUALITY  BEGINENUMERATE  ITEM FOR XY IN RBB PROVE THE TRIANGLE INEQUALITY IN THE FORM XY LEQ X  YWHAT IS THE CONDITION FOR EQUALITY  ITEM FOR XBF YBF IN RBBN PROVE THE TRIANGLE INEQUALITY  XBFYBF  LEQ  XBF    YBF WHERE  CDOT  IS THE USUAL EUCLIDEAN NORM  HINT USE THE FACT THATSUMI1N XI YI LEQ  XBF  YBF  IE THE CAUCHYSCHWARZ INEQUALITY INDEXCAUCHYSCHWARZ INEQUALITY  ENDENUMERATEITEM LET XD BE A METRIC SPACE  SHOW THAT DBXY  FRACDXY1DXYIS A METRIC ON X  WHAT SIGNIFICANT FEATURE DOES THIS METRICPOSSESSITEM LET XD BE A METRIC SPACE  SHOW THAT DMXY  MIN1DXYIS A METRIC ON X  WHAT SIGNIFICANT FEATURE DOES THIS METRICPOSSESSITEM IN DEFINING THE METRIC OF THE SEQUENCE SPACE  LINFTY0INFTY IN REFEQLINFSEQ SUP WAS USED INSTEAD  OF MAX  TO SEE THE NECESSITY OF THIS DEFINITION DEFINE THE  SEQUENCES XBF AND YBF BY XN  FRAC1N1 QQUAD YN  FRACNN1SHOW THAT DINFTYXY  XNYN FOR ALL N GEQ 1ITEM FOR THE METRIC SPACE RBBNDP SHOW THAT DPXBFYBF  IS DECREASING WITH P  THAT IS DPXBFYBF GEQ  DQXBFYBF IF P LEQ Q  HINT TAKE THE DERIVATIVE WITH  RESPECT TO P AND SHOW THAT IT IS LEQ 0  USE THE EM LOG SUM    INEQUALITY CITECOVERTM1991 INDEXINEQUALITIESLOG SUM  WHICH STATES INDEXLOG SUM INEQUALITY THAT FOR NONNEGATIVE  SEQUENCES A1 A2 LDOTS AN AND B1 B2 LDOTS BN SUMI1N AI LOG LEFTFRACAIBIRIGHT GEQLEFTSUMI1N AIRIGHT LOG FRACSUMI1N AISUMI1N BIUSE BI  1 AND AI  XI  YIP  ALSO USE THE FACT THAT FOR ANYNONNEGATIVE SEQUENCE ALPHAI SUCH THAT SUMI1N ALPHAI 1 THE MAXIMUM VALUE OF  SUMI1N ALPHAI LOG ALPHAIIS 0ITEM IF REQUIREMENT M3 IN THE DEFINITION OF A METRIC IS RELAXED TO  THE REQUIREMENT DXY 0 TEXT IF  XYALLOWING THE POSSIBILITY THAT DXY0 EVEN WHEN X NEQ Y THEN AEM PSEUDOMETRIC IS OBTAINED  INDEXPSEUDOMETRIC  LET FMC  X RIGHTARROW RBB BE AN ARBITRARY FUNCTION DEFINED ON ASET X  SHOW THAT DXY  FXFY IS A PSEUDOMETRICEXSKIPITEM SHOW THAT IF A AND B ARE OPEN SETS  BEGINENUMERATE  ITEM A CUP B IS OPEN  ITEM A CAP B IS OPEN  ENDENUMERATEITEM DEVISE AN EXAMPLE TO SHOW THAT THE UNION OF AN INFINITE NUMBER  OF CLOSED SETS NEED NOT BE CLOSEDITEM LET  B  TEXTALL POINTS  P INRBB2 TEXT WITH  0  P LEQ 2 CUP TEXTTHE POINT  04BEGINENUMERATEITEM DRAW THE SET BITEM DETERMINE THE BOUNDARY OF BITEM DETERMINE THE INTERIOR OF BENDENUMERATEITEM EXPLAIN WHY THE SET OF REAL NUMBERS IS BOTH OPEN AND CLOSEDITEM DETERMINE INF AND SUP FOR THE FOLLOWING SETS OF REAL NUMBERS A  04 QQUAD B  0INFTY QQUAD C  INFTY5ITEM SHOW THAT THE BOUNDARY OF A SET S IS A CLOSED SETITEM SHOW THAT THE BOUNDARY OF A SET  S IS THE INTERSECTION OF THE  CLOSURE OF S AND THE CLOSURE OF THE COMPLEMENT OF SITEM SHOW THAT S SUBSET RBBN IS CLOSED IF AND ONLY IF EVERY  CLUSTER POINT FOR S BELONGS TO S  EXSKIPITEM FIND LIMSUPNRIGHTARROW INFTY AN AND  LIMINFNRIGHTARROW INFTY AN FOR  BEGINENUMERATE  ITEM AN  COSFRAC2PI3 N  ITEM AN  COSSQRT2 N  ITEM AN  2 1N3  2N  ITEM AN  N21N  ENDENUMERATEITEM IF LIMSUPNRIGHTARROW INFTY AN  A AND  LIMSUPNRIGHTARROW INFTY BN  B THEN IS IT NECESSARILY TRUE  THAT LIMSUPNRIGHTARROW INFTY ANBN  ABITEM SHOW THAT IF XN IS A SEQUENCE SUCH THAT  DXN1XN  C RNFOR 0 LEQ R  1 THEN XN IS A CAUCHY SEQUENCE  BUCK P 53ITEM LET PBFN  XNYNZN IN RBB3  SHOW THAT IF  PBFN IS A CAUCHY SEQUENCE USING THE METRIC DPBFJ PBFK  SQRTXJ  XK2  YJ  YK2  ZJ   ZK2THEN SO ARE THE SEQUENCES  XN YN AND ZN USINGTHE METRIC DXJXK  XJ  XKITEM SHOW THAT IF A SEQUENCE XN IS CONVERGENT THEN IT IS A CAUCHY  SEQUENCE INDEXCAUCHY SEQUENCECONVERGENCEITEM SHOW THAT THE SEQUENCE XN  INT1N FRACCOS TT2  DT  IS CONVERGENT USING THE METRIC DXY  XY  HINT SHOW THAT  XN IS A CAUCHY SEQUENCE  USE THE FACT THAT INTFRACCOS TT2DT LEQ INTFRAC1T2DTNOTE THIS IS AN EXAMPLE OF KNOWING THAT A SEQUENCE CONVERGESWITHOUT KNOWING WHAT IT CONVERGES TO ITEM SHOW THAT IF XN IS A CAUCHY SEQUENCE THEN XN IS   CONVERGENT PROVIDED THAT THE LIMIT EXISTSITEM THE FACT THAT A SEQUENCE IS CAUCHY DEPENDS UPON THE METRIC  EMPLOYED  LET FNT BE THE SEQUENCE OF FUNCTIONS DEFINED IN  REFEQFNSEQ IN THE METRIC SPACE CABDINFTY WHERE DINFTYFG  SUPT FT  GTSHOW THAT DINFTYFNFM  FRAC12FRACN2M QQUAD MNHENCE CONCLUDE THAT IN THIS METRIC SPACE FN IS NOT A CAUCHYSEQUENCEITEM IN THIS PROBLEM WE WILL SHOW THAT THE SET OF CONTINUOUS  FUNCTIONS IS COMPLETE WITH RESPECT TO THE UNIFORM SUP NORM  LET   FNT BE A CAUCHY SEQUENCE OF CONTINUOUS FUNCTIONS  LET  FT BE THE POINTWISE LIMIT OF FNT  FOR ANY EPSILON  0 LET N BE CHOSEN SO THAT MAX FNTFMT LEQ  EPSILON3  SINCE FKT IS CONTINUOUS THERE IS A D0  SUCH  THAT FTDELTA  FT  EPSILON3 WHENEVER DELTA LEQ D    FROM THIS CONCLUDE THAT  FTDELTA  FT  EPSILONAND HENCE THAT FT IS CONTINUOUSITEM FIND THE ESSENTIAL SUPREMUM OF THE FUNCTION XT DEFINED BY XT  BEGINCASES  SINPI T  T IN 11 T NEQ 0   3  T0ENDCASESEXSKIPSETEXSECTREFSECVS1ITEM AN EQUIVALENT DEFINITION FOR LINEAR INDEPENDENCE FOLLOWS A SET  T IS LINEARLY INDEPENDENT IF FOR EACH VECTOR XBF IN T XBF  IS NOT A LINEAR COMBINATION OF THE POINTS IN THE SET T    XBF THAT IS THE SET T WITH THE VECTOR XBF REMOVED  SHOW  THAT THIS DEFINITION IS EQUIVALENT TO THAT OF DEFINITION  REFDEFLININD  ITEM LET S BE A FINITEDIMENSIONAL VECTOR SPACE WITH    DIMENSIONS  M  SHOW THAT EVERY SET CONTAINING M1 POINTS    IS LINEARLY DEPENDENT  HINT USE INDUCTION  ITEM SHOW THAT IF T IS A SUBSET OF A VECTOR SPACE S WITH    LSPANTS THEN T CONTAINS A HAMEL BASIS OF S  ITEM LET S DENOTE THE SET OF ALL SOLUTIONS OF THE FOLLOWING DIFFERENTIAL    EQUATION DEFINED ON C30INFTY  SEE DEFINITION    REFDEFCLASSCK  INDEXCKCLASS CK FRACD3 XDT3  B FRACD2XDT2  C FRACDXDT  DX     0SHOW THAT S IS A LINEAR SUBSPACE OF C30INFTYITEM LET S BE L202PI AND LET T BE THE SET OF ALL  FUNCTIONS XNT  EJNT FOR N01LDOTS  SHOW THAT T IS  LINEARLY INDEPENDENT  CONCLUDE THAT L202PI IS AN INFINITE  DIMENSIONAL SPACE  HINT ASSUME THAT C1 EJ N1 T  C2 EJ    N2 T  CDOTS  CM EJ NM T  0 FOR NI NEQ NJ WHEN I  NEQ J  DIFFERENTIATE M1 TIMES AND USE THE PROPERTIES OF  VANDERMONDE MATRICES SECTION REFSECVANDERMONDE  ITEM LABELEXLINIDPOLY SHOW THAT THE SET 1TT2LDOTSTM    IS A LINEARLY INDEPENDENT SET HINT THE FUNDAMENTAL THEOREM OF    INDEXFUNDAMENTAL THEOREM OF ALGEBRA    ALGEBRA STATES THAT A POLYNOMIAL FX OF DEGREE M HAS EXACTLY    M ROOTS COUNTING MULTIPLICITYKEENER P 3EXSKIPSETEXSECTREFSECNORMVSITEM LABELEXTINEQBK SHOW THAT IN A NORMED LINEAR SPACE BOXED X  Y LEQ XYITEM SHOW THAT A NORM IS A CONVEX FUNCTION  SEE SECTION  REFSECCONVFUNCITEM SHOW THAT EVERY CAUCHY SEQUENCE  XN IN A NORMED LINEAR  SPACE IS BOUNDED INDEXCAUCHY SEQUENCEBOUNDED INDEXBOUNDED    SEQUENCEITEM USING THE TRIANGLE INEQUALITY SHOW THAT ZX LEQ ZY  YXITEM LET X BE THE SPACE OF EM FINITELY NONZERO SEQUENCES  XBF  X1X2X3LDOTSXN00LDOTS  DEFINE THE NORM ON  X AS XBF  MAXIXI  LET XBFN BE A POINT IN X A  SEQUENCE DEFINED BY XBFN   1FRAC12FRAC13 LDOTS    FRAC1N100LDOTSBEGINENUMERATEITEM SHOW THAT THE SEQUENCE XBFN IS A CAUCHY SEQUENCEITEM SHOW THAT X IS NOT COMPLETE INDEXCOMPLETE METRIC SPACEENDENUMERATEITEM LET P BE IN THE RANGE 0  P  1 AND CONSIDER THE SPACE  LP01 OF ALL FUNCTIONS WITH   X  INT01 XTPDT  INFTYSHOW THAT X IS NOT A NORM ON LP01   HOWEVER SHOW THAT DXY   XY IS A METRIC  HINT FOR A REAL NUMBER ALPHASUCH THAT 0 LEQ ALPHA LEQ 1 NOTE THAT ALPHA LEQ ALPHAP LEQ1ITEM LET S BE A NORMED LINEAR SPACE  SHOW THAT THE NORM FUNCTION  CDOTMC  S RIGHTARROW RBB IS CONTINUOUS  HINT SEE  EXERCISE REFEXTINEQBK  ITEM FOR EACH OF THE INEQUALITY RELATIONSHIPS BETWEEN NORMS IN  REFEQNORMCOMP DETERMINE A VECTOR XBF IN RBBN FOR WHICH  EACH INEQUALITY IS ACHIEVED WITH EQUALITY  EXSKIPSETEXSECTREFSECINNERPROD1 KEENER P 8ITEM COMPUTE THE INNER PRODUCTS LA FG RA FOR THE FOLLOWINGUSING THE DEFINITION LA FG RA  INT01 FTGTDTBEGINENUMERATEITEM FT  T2  2T GT  T1ITEM FT  ET GT T1ITEM FT  COS2PI T GT  SIN2PI TENDENUMERATEITEM COMPUTE THE INNER PRODUCTS XBFT YBF OF THE FOLLOWING USING  THE EUCLIDEAN INNER PRODUCT  BEGINENUMERATE  ITEM XBF  1234T YBF  2341T  ITEM XBF  23 YBF  12T  ENDENUMERATEITEM DETERMINE WHICH OF THE FOLLOWING DETERMINES AN INNER PRODUCT  OVER THE SPACE OF REAL CONTINUOUS FUNCTIONS WITH CONTINUOUS FIRST  DERIVATIVES I  LA FGRA  INT01 FTGTDT  F0G0 QQUADQQUADII   LA FGRA  INT01 FTGTDTEXSKIPSETEXSECTREFSECINDNORM  ITEM SHOW THAT FOR AN INDUCED NORM CDOT OVER A REAL VECTOR SPACE    BEGINENUMERATE    ITEM THE EM PARALLELOGRAM LAW IS TRUE    BEGINEQUATION      LABELEQPARALLELOGRAM       XY 2   XY2  2X2 2Y2    ENDEQUATION    IN TWODIMENSIONAL GEOMETRY AS SHOWN IN FIGURE    REFFIGPARALLELOGRAM THE RESULT SAYS THAT THE SUM OF SQUARES    OF THE LENGTHS OF THE DIAGONALS IS EQUAL TO TWICE THE SUM OF THE    SQUARES OF THE ADJACENT SIDES A SORT OF TWOFOLD PYTHAGOREAN    THEOREMBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRPARALLEL    CAPTIONTHE PARALLELOGRAM LAW    LABELFIGPARALLELOGRAM  ENDCENTERENDFIGUREITEM SHOW THAT  LA XYRA  FRACX  Y2  X  Y24 THIS IS KNOWN AS THE EM POLARIZATION IDENTITYINDEXPOLARIZATION  IDENTITYENDENUMERATEEXSKIPSETEXSECTREFSECCSITEM FOR THE INNER PRODUCE LA FGRA  INT01 FTGTDT  VERIFY THE CAUCHYSCHWARZ INEQUALITY IF  BEGINENUMERATE  ITEM FT  ET GT T1  ITEM FT  ET GT  5 ET  ENDENUMERATEITEM SHOW THAT THE INEQUALITY REFEQANGLEBOUND IS TRUEEXSKIPSETEXSECTREFSECDIRVECITEM PROVE LEMMA REFLEMPYTH  ITEM LET X1T  3T2  1 X2T 5T3  3T AND X3T     2T2  T AND DEFINE THE INNER PRODUCT AS LA FGRA     INT11 FTGTDT  COMPUTE THE ANGLES OF EACH PAIRWISE    COMBINATION OF THESE FUNCTIONS AND IDENTIFY FUNCTIONS THAT ARE    ORTHOGONAL  ITEM LET  BEGINALIGNEDXBF1  1 2 4 2T XBF2  5231T XBF3  1212TENDALIGNEDAND COMPUTE THE ANGLES BETWEEN THESE VECTORS USING THE EUCLIDEANINNER PRODUCT AND IDENTIFY WHICH VECTORS ARE ORTHOGONALITEM SHOW THAT A SET OF NONZERO VECTORS P1P2LDOTSPM THAT  ARE MUTUALLY ORTHOGONAL SO THAT LA PIPJ RA  0 TEXT IF  I NEQ JIS LINEARLY INDEPENDENT  ORTHOGONALITY IMPLIES LINEAR INDEPENDENCE ITEM LET S BE A VECTOR SPACE WITH AN INDUCED NORM  SHOW THAT   LA XBFYBF RA   XBF   YBF  IF AND ONLY IF A XBF   B YBF  0 FOR SOME SCALARS A AND BEXSKIPSETEXSECTREFSECWIPITEM PERFORM THE SIMPLIFICATIONS TO GO FROM THE COMPARISON IN    REFEQDETECT1 TO THE COMPARISON IN REFEQDETECT2  ITEM SHOW BY INTEGRATION THAT INT11 FRAC1SQRT1T2 TNT TMT  BEGINCASES  PI  NM0   PI2  NM NNEQ 0   0  N NEQ MENDCASESHINT USE T  COS X IN THE INTEGRALEXSKIPSETEXSECTREFSECORTHOSUB    ITEM LABELEXORTHOGCOMP1 SHOW THAT THE ORTHOGONAL COMPLEMENT OF A SUBSPACE IS A SUBSPACEITEM LABELEXORTHOCOMP PROVE ITEMS 2 THROUGH 5 OF THEOREM  REFTHMORTHOGCOMP  HINT FOR ITEM 5 USE THEOREM REFTHMHDECOMPEXSKIPSETEXSECTREFSECLINTRANSITEM DETERMINE THE RANGE AND NULLSPACE OF THE FOLLOWING LINEAR  OPERATORS MATRICES A  BEGINBMATRIX10  54  2 4 ENDBMATRIXQQUADQQUADB  BEGINBMATRIX101  549  246 ENDBMATRIXITEM   LET X AND Y BE VECTOR SPACES OVER THE SAME SET OF  SCALARS  LET LTXY DENOTE THE SET OF ALL LINEAR TRANSFORMATIONS  FROM X TO Y  LET L AND M BE OPERATORS FROM LTXY  DEFINE AN ADDITION OPERATOR BETWEEN L AND M AS LMX  LX MXFOR ALL X IN X  ALSO DEFINE SCALAR MULTIPLICATION BY ALX  ALXSHOW THAT LTXYIS A LINEAR VECTOR SPACE ITEM LET X Y AND Z BE LINEAR VECTOR SPACES OVER THE SAME SET   OF SCALARS AND LET L1MC XRIGHTARROW Y AND L2MC Y   RIGHTARROW Z BE LINEAR OPERATORS SHOW THAT THE COMPOSITION   L2L1MC XRIGHTARROW Z IS A LINEAR OPERATOREXSKIPSETEXSECTREFSECISDSPITEM PROVE LEMMA REFLEMVWUNIQUE  ITEM SHOW THAT IF V AND W ARE SUBSPACES OF A VECTOR SPACE S  THEN THEIR INTERSECTION V CAP W IS A SUBSPACEITEM SHOW THAT IF V AND W ARE SUBSPACES OF A VECTOR SPACE S  THEN THEIR SUM VW IS A SUBSPACE ITEM SHOW THAT THE SET VOPLUS W HAS THE SAME ALGEBRAIC STRUCTURE   AS DOES THE SET V  W ESTABLISHING THAT THE ISOMORPHISM HOLDSITEM CITEP 200NAYLORSELL LET X  L2PIPI AND LET S1  LSPAN1COS TCOS 2TLDOTSQQUAD QQUAD S2 LSPANSIN T SIN 2T LDOTSBEGINENUMERATEITEM SHOW THAT S1 OPLUS S2 AND S1  S2 ARE ISOMORPHICITEM SHOW THAT DIMENSIONS1 OPLUS S2  DIMENSIONS1S2ENDENUMERATEITEM LABELEXORTHODIS SHOW THAT  BEGINENUMERATE  ITEM IF V AND W ARE ORTHOGONAL SUBSPACES THEN THEY ARE    DISJOINT  ITEM IF V AND W ARE DISJOINT THEY ARE NOT NECESSARILY    ORTHOGONAL  ENDENUMERATEITEM LET S BE A LINEAR SPACE AND ASSUME THAT S  S1  S2   CDOTS  SN WHERE THE SI ARE MUTUALLY DISJOINT LINEAR  SUBSPACES OF S  LET BI BE A HAMEL BASIS OF SI  SHOW THAT  B  B1 CUP B2CUP CDOTS CUP BN IS A HAMEL BASIS FOR SITEM PROVE THEOREM REFTHMVWISOITEM LET V AND W BE LINEAR SUBSPACES OF A FINITEDIMENSIONAL  LINEAR SPACE S  SHOW THAT DIMENSIONVW  DIMENSIONV  DIMENSIONW  DIMENSIONV CAPWTHEN CONCLUDE THAT DIMENSIONVOPLUS W  DIMENSIONV  DIMENSIONWEXSKIPSETEXSECTREFSECPROJECTIONSITEM IF VBF IS A VECTOR SHOW THAT THE MATRIX WHICH PROJECTS ONTO LSPANVBF IS PV  FRACVBF VBFHVBFHVBFITEM SHOW THAT THE MATRIX PA IN REFEQPROJMAT1 IS A PROJECTION  MATRIXITEM FOR THE PROJECTION MATRIX PAW INREFEQPROJMAT2  BEGINENUMERATE  ITEM SHOW THAT PAW2  PAW  ITEM SHOW THAT PAWPERP  IPAW IS ORTHOGONAL TO    PAW USING THE WEIGHTED INNER PRODUCT THAT IS    PAWH W PAWPERP  0  ENDENUMERATE  ITEM LET  PBF1  BEGINBMATRIX 123  4 ENDBMATRIX QQUADPBF2  BEGINBMATRIX 4  2  6  7 ENDBMATRIX QQUADPBF3  BEGINBMATRIX 3  4  2  1 ENDBMATRIXAND  XBF  BEGINBMATRIX 1  2  3  7 ENDBMATRIXDETERMINE THE NEAREST VECTOR XBFHAT INLSPANPBF1PBF2PBF3  ALSO DETERMINE THE ORTHOGONALCOMPLEMENT OF XBF IN LSPANPBF1PBF2PBF3ITEM LET A BE A MATRIX WHICH CAN BE FACTORED ASBEGINEQUATIONA  U SIGMA VHLABELEQSVDEXER1ENDEQUATIONSUCH THAT UH U  I QQUAD VH V  I AND SIGMA IS A DIAGONAL MATRIX WITH REAL VALUESTHE FACTORIZATION IN REFEQSVDEXER1 IS THE SINGULAR VALUEDECOMPOSITION SEE CHAPTER REFCHAPSVDSHOW THAT PA  PU  ITEM TWO ORTHOGONAL PROJECTION MATRICES PA AND PB ARE SAID TO  BE ORTHOGONAL IF PAPB  0  THIS IS DENOTED AS PA PERP PB  INDEXPROJECTION OPERATORSORTHOGONAL SHOW THAT  BEGINENUMERATE  ITEM PA AND PB ARE ORTHOGONAL IF AND ONLY IF THEIR RANGES ARE    ORTHOGONAL  ITEM PAPB IS A PROJECTION OPERATOR IF AND ONLY IF PA AND    PB  ARE ORTHOGONAL  ENDENUMERATEITEM SHOW THAT THE CBF WHICH MINIMIZES REFEQWPROJ1 IS CBF  AHWA1AHW XBFSO THAT THE WEIGHTED PROJECTION OF XBF IS ACBF AAHWA1AHW XBFITEM PROVE THEOREM REFTHMRNPITEM LET P1P2LDOTS PM BE A SET OF ORTHOGONAL PROJECTIONS WITH PIPJ  0 FOR I NEQ JSHOW THAT Q  P1  P2  CDOTS  PM IS AN ORTHOGONAL  PROJECTIONITEM LABELEXIMP IF P IS A PROJECTION OPERATOR SHOW THAT IP  IS A PROJECTION OPERATOR  DETERMINE THE RANGE AND NULLSPACE OF  IPITEM LET S BE A VECTOR SPACE AND LET V1V2LDOTS VN BE  LINEAR SUBSPACES SUCH THAT VI IS ORTHOGONAL FROM SUMJ NEQ I  VJ FOR EACH I AND WHERE S  V1  V2  CDOTS  VNLET PJ BE THE PROJECTION ON S FOR WHICH RANGEPJ  VJ ANDNULLSPACEPJ  SUMJ NEQ K VK  DEFINE AN OPERATOR P  LAMBDA1 P1  LAMBDA2 P2  CDOTS  LAMBDAN PNBEGINENUMERATEITEM SHOW THAT IF XBF IN VJ THEN P XBF  LAMBDAJ XBFITEM SHOW THAT P IS A PROJECTION IF AND ONLY IF LAMBDAJ IS  EITHER 0 OR 1ENDENUMERATEITEM LET A AND B BE MATRICES SUCH THAT AHB  0  THEN V   RANGEA AND W  RANGEB ARE ORTHOGONALBEGINENUMERATESHOW THAT PA  I  PB ITEM SHOW THAT A VECTOR XBF CAN BE DECOMPOSED AS  XBF  PA XBF  PBXBF  PA XBF  IPAXBF ENDENUMERATE ITEM SHOW THAT IF V AND W ARE CLOSED SUBSPACES OF A HILBERT   SPACE S AND IF V PERP W THEN VW IS A CLOSED SUBSPACE OF   S NS P 297EXSKIPSETEXSECTREFSECGRAMSCHMITITEM USING A SYMBOLIC MANIPULATION PACKAGE WRITE A FUNCTION WHICH  PERFORMS THE GRAMSCHMIDT ORTHOGONALIZATION OF A SET OF FUNCTIONSITEM DETERMINE THE FIRST FOUR POLYNOMIALS ORTHOGONAL OVER 01  ASYMBOLIC MANIPULATION PACKAGE IS RECOMMENDEDITEM FOR THE FUNCTIONS SHOWN IN FIGURE REFFIGGSEX DETERMINE A  SET OF ORTHOGONAL FUNCTIONS SPANNING THE SAME SPACE USING THE  FUNCTIONS IN THE ORDER SHOWN   SOMETHING LIKE THE PROAKIS EXERCISE  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODE      INPUTPICTUREDIRGRAMSCHMIDT3      CAPTIONFUNCTIONS TO ORTHOGONALIZE      LABELFIGGSEX    ENDCENTER  ENDFIGUREITEM MODIFY ALGORITHM REFALGQR1 SO THAT IT ONLY RETAINS COLUMNS  OF Q THAT ARE NONZERO MAKING CORRESPONDING ADJUSTMENTS TO R  COMMENT ON THE PRODUCT QR IN THIS CASE GRAMSCHMIDT2MITEM MODIFY ALGORITHM REFALGQR1 TO COMPUTE A SET OF ORTHOGONAL  VECTORS WITH RESPECT TO THE WEIGHTED INNER PRODUCT LA XBF YBF  RA  XBFT W YBF FOR A POSITIVE DEFINITE SYMMETRIC MATRIX WITEM LABELEXMGS MODIFIED GRAMSCHMIDT INDEXMODIFIED    GRAMSCHMIDT INDEXGRAMSCHMIDTMODIFIED THE  COMPUTATIONS OF THE GRAMSCHMIDT ALGORITHM CAN BE REORGANIZED TO BE  MORE STABLE NUMERICALLY  IN THESE MODIFIED COMPUTATIONS A COLUMN  OF Q AND A EM ROW OF R IS PRODUCED AT EACH ITERATION  THE  REGULAR GRAMSCHMIDT PROCESS PRODUCES A COLUMN OF Q AND A COLUMN  OF R AT EACH ITERATION   LET THE KTH COLUMN OF Q BE DENOTED AS  QBFK AND LET THE KTH ROW OF R BE DENOTED AS RBFKT  BEGINENUMERATE  ITEM SHOW THAT FOR AN MATSIZEMN MATRIX A A  SUMI1K1 QBFI RBFIT  SUMIKN QBFIRBFIT  ZEROBF AKWHERE AK IS MATSIZEMNK1ITEM LET AK  ZBFK B WHERE B IS MATSIZEMNK  AND EXPLAIN WHY THE KTH COLUMN OF Q AND THE KTH ROW OF R ARE  GIVEN BY RKK   EBFK QQUAD QBFK  EBFKRKK QQUADRKK1LDOTSRKN  QBFKTBITEM THEN SHOW THAT THE NEXT ITERATION CAN BE STARTED BY COMPUTING A  SUMI1K QBFI RBFIT  ZEROBF AK1WHERE AK1  B  QBFKRKK1LDOTSRKNITEM CODE THE MODIFIED GRAMSCHMIDT ALGORITHM IN SC MATLAB  ENDENUMERATEENDEXERCISESSECTIONREFERENCESMUCH OF THE MATERIAL ON METRIC SPACES HILBERT SPACES AND BANACHSPACES PRESENTED HERE IS SIGNIFICANTLY COMPRESSED FROMCITENAYLORSELL  IN THEIR EXPANDED TREATMENT THEY PROVIDE PROOFS OFSEVERAL POINTS THAT WE HAVE MERELY MENTIONED  ANOTHER SOURCE ONVECTOR SPACES IS CITEFRIEDMAN  AN EXCELLENT HISTORICAL SOURCE ONVECTOR SPACES AND THEIR APPLICATIONS TO SIGNAL PROCESSING ANDENGINEERING IS CITELUENBERGER1969  FUNCTION SPACES WITH ANEMPHASIS ON SERIES REPRESENTATIONS ARE DISCUSSED IN CITEKEENERSIMILAR TREATMENT OF METRIC AND VECTOR SPACES IS CITEFRANKS  OURDISCUSSION OF THE MODIFIED GRAMSCHMIDT PROCESSINDEXGRAMSCHMIDTMODIFIED IS DRAWN FROM CITEGVLEXTENSIVE PROPERTIES OF THE ORTHOGONAL POLYNOMIALS INTRODUCED HERE AREDISCUSSED AND TABULATED IN CITEABRAMOWITZ SEE ALSO CITEWALTER1994AN EXTENSION OF THE CONCEPT OF A BASIS IS THAT OF A EM  FRAMEINDEXFRAME WHICH PROVIDES AN OVERDETERMINED SET OF  REPRESENTATIONAL FUNCTIONS  A TUTORIAL INTRODUCTION TO FRAMES WITH  APPLICATIONS IN SIGNAL PROCESSING APPEARS IN CITEPEI1997 LOCAL VARIABLES TEXMASTER TEST ENDCHAPTERREPRESENTATION AND APPROXIMATION IN VECTOR SPACESLABELCHAPVECTAPBEGINQUOTESOURCEMICHAEL SPIVAKEM CALCULUS ON MANIFOLDS  ANY GOOD MATHEMATICAL COMMODITY IS WORTH GENERALIZINGENDQUOTESOURCESECTIONTHE APPROXIMATION PROBLEM IN HILBERT SPACELABELSECHILBAPPROXLET SCDOT BE A NORMED LINEAR VECTOR SPACE FOR SOME NORM CDOT   LET T   PBF1ALLOWBREAK PBF2ALLOWBREAK LDOTSALLOWBREAK PBFM SUBSET S BE A SET OFLINEARLY INDEPENDENT VECTORS IN A VECTOR SPACE S AND LET V LSPANT   THE ANALYSIS PROBLEM IS THIS GIVEN A VECTOR XBFIN S FINDTHE COEFFICIENTS C1C2LDOTSCM SO THATBEGINEQUATION XHAT  C1 PBF1  C2 PBF2   CDOTS CM PBFMLABELEQAPPROX1ENDEQUATIONAPPROXIMATES XBF AS CLOSELY AS POSSIBLE  THE HAT  CARETINDICATES THAT THIS IS OR MAY BE AN APPROXIMATIONINDEXAPPROXIMATION THAT IS WE WISH TO WRITEBEGINALIGNEDXBF  XBFHAT  EBF  C1 PBF1  C2 PBF2   CDOTS CM PBFM  EBFENDALIGNEDWHERE EBF IS THE APPROXIMATION ERROR SO THAT  XBF  XBFHAT   EBF IS AS SMALL AS POSSIBLE  THE PROBLEM IS DIAGRAMMED IN FIGUREREFFIGAPPROX1 FOR M2BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRORTHOGPROJ3    CAPTIONTHE APPROXIMATION PROBLEM    LABELFIGAPPROX1  ENDCENTERENDFIGUREOF COURSE IF XBF IN V THEN IT IS POSSIBLE TO FIND COEFFICIENTS SOTHAT  XBF  XBFHAT   0  THE PARTICULAR NORM CHOSEN INPERFORMING THE MINIMIZATION AFFECTS THE ANALYTIC APPROACH TO THEPROBLEM AND THE FINAL ANSWER  IF THE L1 OR L1 NORM IS CHOSENTHEN THE ANALYSIS INVOLVES ABSOLUTE VALUES WHICH MAKES AN ANALYTICALSOLUTION INVOLVING DERIVATIVES DIFFICULT  IF THE LINFTY ORLINFTY NORM IS CHOSEN THE ANALYSIS MAY INVOLVE DERIVATIVES OFTHE MAX FUNCTION WHICH IS ALSO DIFFICULT  THIS APPROXIMATIONPROBLEM IS DISCUSSED IN CHAPTER REFCHAPAPPROX  IF THE L2 OR L2 NORM IS CHOSEN MANY OF THE ANALYTICALDIFFICULTIES DISAPPEAR  THE NORM IS THE INDUCED NORM AND THEPROPERTIES OF THE PROJECTION THEOREM CAN BE USED TO FORMULATE THESOLUTION  ALTERNATIVELY THE SOLUTION CAN BE OBTAINED USING CALCULUSTECHNIQUES  ACTUALLY FOR PROBLEMS POSED USING THE LP NORMS AGENERALIZATION OF THE PROJECTION THEOREM CAN BE USED OPTIMIZING INBANACH SPACE RATHER THAN HILBERT SPACE BUT THIS LIES BEYOND THE SCOPEOF THIS BOOK  CHOOSING THE L2 NORM ALLOWS FAMILIAR EUCLIDEANGEOMETRY TO BE USED TO DEVELOP INSIGHT  THE APPROXIMATION PROBLEMWHEN THE INDUCED NORM IS USED FOR EXAMPLE EITHER AN L2 OR L2NORM IS KNOWN AS THE HILBERT SPACE APPROXIMATION PROBLEMTO DEVELOP GEOMETRIC INSIGHT INTO THE APPROXIMATION PROBLEM THEANALYSIS FORMULAS ARE PRESENTED BY STARTING WITH THE APPROXIMATIONPROBLEM WITH ONE ELEMENT IN T AIDED BY A KEY OBSERVATION THE ERRORIS ORTHOGONAL TO THE DATA  THE ANALYSIS IS THEN EXTENDED TO TWODIMENSIONS THEN TO ARBITRARY DIMENSIONS  WE WILL BEGIN FIRST WITHGEOMETRIC PLAUSIBILITY AND CALCULUS THEN PROVE THE RESULT USING THECAUCHYSCHWARZ INEQUALITYTO BEGIN LET T IN RBB2 CONSIST OF ONLY ONE VECTORTPBF1  FOR A VECTOR XBF IN RBB2 WE WISH TO REPRESENTXBF AS A LINEAR COMBINATION OF T XBF  C1 PBF1  EBFIN SUCH A WAY AS TO MINIMIZE THE NORM OF THE APPROXIMATION ERROR EBF  IN THIS SIMPLEST CASE THERE IS ONLY THE PARAMETER C1 TOIDENTIFY  THE SITUATION IS ILLUSTRATED IN FIGURE REFFIGAPPROX1AABEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODESUBFIGUREONE VECTOR IN TINPUTPICTUREDIRORTHOGPROJ4 QQUAD SUBFIGURETWO VECTORS IN T INPUTPICTUREDIRORTHOGPROJ5    CAPTIONAPPROXIMATION WITH ONE AND TWO VECTORS    LABELFIGAPPROX1A  ENDCENTERENDFIGUREIF THE L2 OR L2 NORM IS USED IT MAY BE OBSERVED GEOMETRICALLYTHAT BF THE ERROR IS MINIMIZED WHEN THE ERROR IS ORTHOGONAL TO VTHAT IS WHEN THE ERROR IS ORTHOGONAL TO THE DATA THAT FORMS OURESTIMATE  WRITTEN MATHEMATICALLY THE NORM OF THE ERROR  EBF IS MINIMIZED WHEN  EBF PERP PBF1 OR LA XBF  C1 PBF1 PBF1 RA  0USING THE PROPERTIES OF INNER PRODUCTSBEGINEQUATIONC1  FRAC LA XBFPBF1 RA  PBF1 22LABELEQNORM3ENDEQUATIONGEOMETRICALLY THE QUANTITY FRAC LA XBFPBF1 RA  PBF1 22IS THE BF PROJECTION OF THE VECTOR XBF IN THE DIRECTION OFPBF1 IT IS THE LENGTH OF THE SHADOW THAT XBF CASTS ONTOPBF1 EXPRESSED AS A PROPORTION OF THE LENGTH OF PBF1THE SAME APPROXIMATION FORMULA MAY ALSO BE OBTAINED BY CALCULUS  WEFIND C1 TO MINIMIZE  XBF  C1 PBF122  LA XBFC1PBF1XBFC1PBF1RABY TAKING THE DERIVATIVE WITH RESPECT TO C1 AND EQUATING THE RESULTTO ZERO  THIS GIVES THE SAME ANSWER AS REFEQNORM3CONTINUING OUR DEVELOPMENT WHEN T CONTAINS TWO VECTORS WE CAN WRITETHE APPROXIMATION AS XBF  C1 PBF1  C2 PBF2  EBFFIGURE REFFIGAPPROX1AB ILLUSTRATES THE CONCEPT FOR VECTORS INRBB3  IT IS CLEAR FROM THIS FIGURE THAT IF EUCLIDEANDISTANCE IS USED THE ERROR IS ORTHOGONAL TO THE DATA PBF1 ANDPBF2  THIS GIVES THE FOLLOWING ORTHOGONALITY CONDITIONS BEGINALIGNEDLA XBF  C1 PBF1  C2 PBF2PBF1 RA  0LA XBF  C1 PBF1  C2 PBF2PBF2 RA  0ENDALIGNEDEXPANDING THESE USING THE PROPERTIES OF INNER PRODUCTS GIVESBEGINALIGNED LA XBFPBF1 RA  C1 LA PBF1PBF1 RA  C2 LAPBF2PBF1 RA LA XBFPBF2 RA  C1 LA PBF1PBF2 RA  C2 LAPBF2PBF2 RAENDALIGNEDWHICH CAN BE WRITTEN MORE CONCISELY IN MATRIX FORMBEGINEQUATIONBEGINBMATRIX LA PBF1PBF1RA  LA PBF2PBF1RA LA PBF1PBF2RA  LA PBF2PBF2RAENDBMATRIXBEGINBMATRIX C1  C2 ENDBMATRIX BEGINBMATRIX LA XBFPBF1RA  LA XBFPBF2RA ENDBMATRIXLABELEQPROJ1ENDEQUATIONSOLUTION OF THIS MATRIX EQUATION PROVIDES THE DESIRED COEFFICIENTSBEGINEXAMPLE  SUPPOSE XBF  123T PBF1  110T AND PBF2   210T  IT IS CLEAR THAT XBFHAT  C1 PBF1  C2 PBF2 CANNOT BE AN EXACT REPRESENTATION OF XBF SINCE THERE IS NO WAY TOMATCH THE THIRD ELEMENT  USING REFEQPROJ1 WE OBTAINLEFTBEGINARRAYCC 2  3  3  5 ENDARRAYRIGHTLEFTBEGINARRAYCC C1  C2 ENDARRAYRIGHT LEFTBEGINARRAYC 3  4 ENDARRAYRIGHTTHIS CAN BE SOLVED TO GIVE C1   3 QUADQUAD C2  1 THEN THE APPROXIMATION VECTOR IS XBFHAT  C1 PBF1  C2 PBF2  3110T  210T 120T NOTE THAT THE APPROXIMATION FBFHAT IS THE SAME AS FBF IN THEFIRST TWO COEFFICIENTS  THE VECTOR HAS BEEN BF PROJECTEDINDEXPROJECTION ONTO THE PLANE FORMED BY THE VECTORS PBF1 ANDPBF2  THE ERROR IN THIS CASE HAS LENGTH 3ENDEXAMPLEJUMPING NOW TO HIGHER NUMBERS OF VECTORS WHAT WE CAN DO FOR TWOVECTORS IN T WE CAN DO FOR M INGREDIENT VECTORS  WE APPROXIMATEXBF AS XBF  SUMI1M CI PBFI  EBF  XBFHAT  EBFTO MINIMIZE  EBF   XBF  XBFHAT  IF THE NORM USED ISTHE L2 OR L2 NORM THIS IS THE EM LINEAR LEASTSQUARESINDEXLINEAR LEASTSQUARES PROBLEM  WHENEVER THE NORM MEASURING THEAPPROXIMATION ERROR  EBF IS INDUCED FROM AN INNER PRODUCT WECAN EXPRESS THE MINIMIZATION IN TERMS OF AN ORTHOGONALITY CONDITIONTHE MINIMUMNORM ERROR MUST BE ORTHOGONAL TO EACH VECTOR PJ LA XBF SUMI1M CI PBFI PJ RA  0 QQUADJ12LDOTS MTHIS GIVES US M EQUATIONS IN THE M UNKNOWNS WHICH MAY BE WRITTENASBEGINEQUATIONBEGINBMATRIXLA PBF1PBF1 RA  LA PBF2PBF1 RA  CDOTS  LAPBFMPBF1 RA  LA PBF1PBF2 RA  LA PBF2PBF2 RA  CDOTS  LAPBFMPBF2 RA  VDOTS  VDOTS LA PBF1PBFM RA  LA PBF2PBFM RA  CDOTS  LA PBFMPBFM RA ENDBMATRIXBEGINBMATRIXC1  C2  VDOTS  CM  ENDBMATRIX BEGINBMATRIXLA XBFPBF1 RA  LA XBFPBF2 RA VDOTS  LA XBFPBFM RA  ENDBMATRIXLABELEQPROJ2ENDEQUATIONWE DEFINE THE VECTORBEGINEQUATION PBF  BEGINBMATRIX LA XBF PBF1 RA LA XBF PBF2 RA VDOTS  LA XBF PBFM RA ENDBMATRIXLABELEQPBFENDEQUATIONAS THE EM CROSSCORRELATION VECTOR INDEXCROSSCORRELATION ANDBEGINEQUATIONCBF  BEGINBMATRIXC1  C2  VDOTS  CM ENDBMATRIX    LABELEQCBF  ENDEQUATIONAS THE VECTOR OF COEFFICIENTS  THEN REFEQPROJ2 CAN BE WRITTEN AS RCBF  PBFWHERE R IS THE MATRIX OF INNER PRODUCTS IN REFEQPROJ2EQUATIONS OF THIS FORM ARE KNOWN AS THE BF NORMAL EQUATIONSINDEXNORMAL EQUATIONS SINCE THE SOLUTION MINIMIZES THE SQUARE OFTHE ERROR IT IS KNOWN AS A EM LEASTSQUARE INDEXLEASTSQUARESOR EM MINIMUM MEANSQUARE INDEXMINIMUM MEANSQUARE SOLUTIONDEPENDING ON THE PARTICULAR INNER PRODUCT USEDSUBSECTIONTHE GRAMMIAN MATRIXLABELSECGRAMMIANTHE MATSIZEMM MATRIX BEGINEQUATIONR  BEGINBMATRIXLA PBF1PBF1 RA  LA PBF2PBF1 RA  CDOTS  LAPBFMPBF1 RA  LA PBF1PBF2 RA  LA PBF2PBF2 RA  CDOTS  LAPBFMPBF2 RA  VDOTS  VDOTS LA PBF1PBFM RA  LA PBF2PBFM RA  CDOTS  LAPBFMPBFM RA ENDBMATRIXLABELEQGRAMDEFENDEQUATIONIN THE LEFTHAND SIDE OF REFEQPROJ2 IS SAID TO BE THE BF  GRAMMIAN INDEXGRAMMIAN OF THE SET T  SINCE THE IJTHELEMENT OF THE MATRIX IS RIJ  LA PBFJPBFIRAIT FOLLOWS THAT THE GRAMMIAN IS A HERMITIAN SYMMETRIC MATRIX THAT IS RH  RWHERE H INDICATES CONJUGATETRANSPOSE  SOME IMPLICATIONS OF THEHERMITIAN STRUCTURE ARE EXAMINED IN SECTION REFSECDIAGONALSOLUTION OF REFEQPROJ2 REQUIRES THAT R BE INVERTIBLE  THEFOLLOWING THEOREM DETERMINES CONDITIONS UNDER WHICH R ISINVERTIBLE  RECALL THAT A MATRIX R FOR WHICH  XBFH R XBF  0FOR ANY NONZERO VECTOR XBF IS SAID TO BE POSITIVEDEFINITE SEEBOX REFBOXPD  INDEXPOSITIVEDEFINITE AN IMPORTANT ASPECT OFPOSITIVEDEFINITE MATRICES IS THAT THEY ARE ALWAYS INVERTIBLE  IF RIS SUCH THAT XBFH R XBF GEQ 0FOR ANY NONZERO VECTOR XBF THEN R IS SAID TO BE EM  POSITIVESEMIDEFINITE INDEXPOSITIVESEMIDEFINITE INPUTLINALGDIRPOSDEFTEXBEGINTHEOREM LABELTHMGRAMMPD  A GRAMMIAN MATRIX R IS ALWAYS POSITIVE SEMIDEFINITE THAT IS  XBFH R XBF GEQ 0 FOR ANY XBF IN CBBM  IT IS  POSITIVE DEFINITE IF AND ONLY IF THE VECTORS  PBF1PBF2LDOTSPBFM   ARE LINEARLY INDEPENDENT INDEXLINEARLY INDEPENDENTENDTHEOREMBEGINPROOF  LET YBF  Y1Y2LDOTSYMT BE AN ARBITRARY VECTOR  THENBEGINEQUATIONBEGINALIGNEDYBFH R YBF  SUMI1MSUMJ1M YBARI YJ LAPBFJPBFIRA  SUMI1MSUMJ1M LA YJ PBFJYI PBFI RALABELEQPR1  LEFTLANGLESUMJ1M YJ PBFJSUMI1M YI PBFIRIGHTRANGLE  LEFT SUMJ1M YJ PBFJRIGHT2 GEQ 0ENDALIGNEDENDEQUATIONHENCE R IS POSITIVE SEMIDEFINITEIF R IS NOT POSITIVE DEFINITE THEN THERE IS A NONZERO VECTOR YBF SUCHTHAT YBFH R YBF  0SO THAT BY REFEQPR1 SUMI1M YI PBFI  0THUS THE PBFI ARE LINEARLY DEPENDENT  CONVERSELY IF R IS POSITIVE DEFINITE THEN  YBFH R YBF  0FOR ALL NONZERO YBF AND BY REFEQPR1 SUMI1M YI PBFI NEQ 0THIS MEANS THAT THE  PBFI ARE LINEARLYINDEPENDENT INDEXLINEARLY INDEPENDENTENDPROOFAS A COROLLARY TO THIS THEOREM WE GET ANOTHER PROOF OF THECAUCHYSCHWARZ INEQUALITY  THE MATSIZE22 GRAMMIAN R  BEGINBMATRIXLA XXRA  LA XYRA  LA YXRA  LA YY RA ENDBMATRIXIS POSITIVE SEMIDEFINITE WHICH MEANS THAT ITS DETERMINANT ISNONNEGATIVE LA XXRA LA YY RA  LA XYRALA YXRA GEQ 0WHICH IS EQUIVALENT TO REFEQSW1  THE CONCEPT OF USING ORTHOGONALITY FOR THE EUCLIDEAN INNER PRODUCT TOFIND THE MINIMUM NORM SOLUTION GENERALIZES TO EM ANY INDUCED NORMAND ITS ASSOCIATED INNER PRODUCTIF THE SET OF VECTORS PBF1PBF2LDOTSPBFM ARE ORTHOGONALINDEXORTHOGONAL THEN THE GRAMMIAN IN REFEQGRAMDEF ISDIAGONAL SIGNIFICANTLY REDUCING THE AMOUNT OF COMPUTATION REQUIRED TOFIND THE COEFFICIENTS OF THE VECTOR REPRESENTATION  IN THIS CASE THECOEFFICIENTS ARE OBTAINED SIMPLY BYBEGINEQUATION CJ  FRAC LA XBFPBFJRA LA PBFJPBFJ RA LABELEQPROJ4ENDEQUATIONEACH COEFFICIENT USES THE SAME PROJECTION FORMULA THAT WAS USED INREFEQPROJ1 FOR A SINGLE DIMENSION  THE COEFFICIENTS CAN ALSO BEREADILY INTERPRETED FOR ORTHOGONAL VECTORS THE COEFFICIENT OF EACHVECTOR INDICATES THE STRENGTH OF THE VECTOR COMPONENT IN THE SIGNALREPRESENTATIONBEGINEXERCISESITEM LABELEXGRAMDET THERE IS A CONNECTION BETWEEN GRAMMIANS  INDEXGRAMMIAN AND LINEAR INDEPENDENCE INDEXLINEAR INDEPENDENCE  AS DEMONSTRATED IN THEOREM REFTHMGRAMMPD  WE WILL EXPLORE THIS  CONNECTION FURTHER IN THIS PROBLEM  LET PBF1PBF2LDOTSPBFN BE A SET OF VECTORS AND LET  US SUPPOSE THAT THE FIRST K1 VECTORS OF THIS SET HAVE PASSED A  TEST FOR LINEAR INDEPENDENCE  WE FORM EBFK  CK1K PBF1  CK2K PBF2  CDOTS C1KPBFK1  PBFK AND WANT TO KNOW IF EBFK IS EQUAL TO ZERO FOR ANY SET OFCOEFFICIENTS CBFK  CK1K CK2KLDOTSC1K1TIF SO THEN PBFK IS LINEARLY DEPENDENT  LET AK  PBF1PBF2LDOTSPBFKBE A DATA MATRIX AND LET RK  AKHAK BE THE CORRESPONDINGGRAMMIANBEGINENUMERATEITEM SHOW THAT THE SQUARED ERROR CAN BE WRITTEN ASBEGINEQUATION EBFKH EBFK  SIGMAK2  CBFHBEGINBMATRIX RK1   HBFK  HBFKH  RKK ENDBMATRIXCBFKLABELEQGRAMDET1ENDEQUATIONFOR SOME HBFK AND RKK  IDENTIFY HBFK AND RKKITEM DETERMINE THE MINIMUM VALUE OF SIGMAK2 BY MINIMIZING  REFEQGRAMDET1 WITH RESPECT TO CBFK SUBJECT TO THE  CONSTRAINT THAT THE LAST ELEMENT OF CBFK IS EQUAL TO 1  HINT  TAKE THE GRADIENT OF CBFHBEGINBMATRIX RK1   HBFK  HBFKH  RKK ENDBMATRIXCBFK  LAMBDACBFHDBF  1WHERE LAMBDA IS A LAGRANGE MULTIPLIER AND DBF 00LDOTS01T  SHOW THAT WE CAN WRITE THE CORRESPONDINGEQUATIONS ASBEGINEQUATIONBEGINBMATRIX RK1  HBFK  HBFKH  RKKENDBMATRIXCBFK  SIGMAK2 DBFLABELEQGRAMDET2ENDEQUATIONITEM SHOW THAT REFEQGRAMDET2 CAN BE MANIPULATED TO BECOME SIGMAK2  RKK  HBFKH RK11 HBFKTHE QUANTITY SIGMAK2 IS CALLED THE EM SCHUR COMPLEMENTINDEXSCHUR COMPLEMENT OFRK  IF SIGMAK20 THEN PBFK IS LINEARLY DEPENDENTENDENUMERATEENDEXERCISESSECTIONTHE ORTHOGONALITY PRINCIPLETHE BF ORTHOGONALITY PRINCIPLE INDEXORTHOGONALITY PRINCIPLE FORLEASTSQUARES LS OPTIMIZATION INTRODUCED IN SECTIONREFSECHILBAPPROX IS NOW FORMALIZEDBEGINTHEOREM  THE ORTHOGONALITY PRINCIPLE LET PBF1PBF2LDOTSPBFM BE DATA  VECTORS IN A VECTOR SPACE S  LET XBF BE ANY VECTOR IN  S  IN THE REPRESENTATION XBF  SUMI1M CI PBFI  EBF   XBFHAT  EBFTHE INDUCED NORM INDEXINDUCED NORM OF THE ERROR VECTOR EBFIS MINIMIZED WHEN THE ERROR EBF  XBFXBFHAT IS ORTHOGONAL TOEACH OF THE DATA VECTORS LA XBF  SUMI1M CI PBFIPBFJRA  0QQUADJ12LDOTSMENDTHEOREMBEGINPROOF ONE PROOF RELIES ON THE PROJECTION THEOREM THEOREM  REFTHMPROJ WITH THE OBSERVATION THAT V   LSPANPBF1PBF2LDOTSPBFM IS A SUBSPACE OF S  WE PRESENT A  MORE DIRECT PROOF USING THE CAUCHYSCHWARZ INEQUALITY    IN THE CASE THAT XBF IN LSPANPBF1PBF2LDOTSPBFM  THE ERROR IS ZERO AND HENCE IS ORTHOGONAL TO THE DATA VECTORS  THIS  CASE IS THEREFORE TRIVIAL AND IS EXCLUDED FROM WHAT FOLLOWS    IF XBF NOTIN LSPANPBF1PBF2LDOTSPBFM LET YBF  BE A FIXED VECTOR THAT IS ORTHOGONAL TO ALL OF THE DATA VECTORS LA YBFPBFIRA  0QQUAD I12LDOTSMSUCH THAT  XBF  SUMI1M AI PBFI  YBFFOR SOME SET OF COEFFICIENTS  A1A2LDOTSAM  LET EBF BE AVECTOR SATISFYINGBEGINEQUATION X  SUMI1M CI PBFI  EBFLABELEQPROVEORTHOG1ENDEQUATIONFOR SOME SET OF COEFFICIENTS  C1 C2LDOTS CM  THEN BY THECAUCHYSCHWARZ INEQUALITY INDEXCAUCHYSCHWARZ INEQUALITYBEGINALIGNAT2 EBF 2 YBF 2 GEQ LA EBFYBFRA2  QQUADTEXTCAUCHYSCHWARZ NOTAG  LA XBFYBF RA  LA SUMI1M CI PBFI YBFRA2  LA XBFYBFRA2  QQUAD TEXTORTHOGONALITY OF YBF NOTAGENDALIGNATTHE LOWER BOUND IS INDEPENDENT OF THE COEFFICIENTS CI ANDHENCE NO SET OF COEFFICIENTS CAN MAKE THE BOUND SMALLER  BY THEEQUALITY CONDITION FOR THE CAUCHYSCHWARZ INEQUALITY THE LOWER BOUNDIS ACHIEVED   IMPLYING THE MINIMUM EBF   WHEN EBF  ALPHA YBFFOR SOME SCALAR ALPHA  SINCE EBF MUST SATISFYREFEQPROVEORTHOG1 IT MUST BE THE CASE THAT EBF  YBF HENCE THEERROR IS ORTHOGONAL TO THE DATAENDPROOFWHEN CBF IS OBTAINED VIA THE PRINCIPLE OF ORTHOGONALITY THEOPTIMAL ESTIMATE XBFHAT  SUMI1M CI PBFIIS ALSO ORTHOGONAL TO THE ERROR EBF  XBF  XBFHAT SINCE IT IS A LINEARCOMBINATION OF THE DATA VECTORS PBFI  THUSBEGINEQUATIONLA XBFHAT EBF RA  0LABELEQXHATORTHOENDEQUATIONSUBSECTIONREPRESENTATIONS IN INFINITEDIMENSIONAL SPACEIF THERE ARE AN INFINITE NUMBER OF VECTORS IN T  PBF1PBF2LDOTS THEN THE REPRESENTATION XBFHAT  SUMI1INFTY CI PBFIMUST BE REGARDED WITH SOME DEGREE OF SUSPICION BECAUSE A LINEARCOMBINATION IS DEFINED TECHNICALLY ONLY IN TERMS OF A FINITE SUMTHE CONVERGENCE OF THIS INFINITE SUM MUST THEREFORE BE EXAMINEDCAREFULLY  HOWEVER IF T IS AN ORTHONORMAL SET THEN THEREPRESENTATION CAN BE SHOWN TO CONVERGESECTIONERROR MINIMIZATION VIA GRADIENTSLABELSECGRADMININDEXGRADIENT WHILE THE ORTHOGONALITY THEOREM IS USED PRINCIPALLYTHROUGHOUT THIS CHAPTER AS THE GEOMETRICAL BASIS FOR FINDING A MINIMUMERROR APPROXIMATION UNDER AN INDUCED NORM IT IS PEDAGOGICALLYWORTHWHILE TO CONSIDER ANOTHER APPROACH BASED ON GRADIENTS WHICHREAFFIRMS WHAT WE ALREADY KNOW BUT DEMONSTRATES THE USE OF SOME NEWTOOLSMINIMIZING  EBF 2 FOR THE INDUCED NORM IN XBF  SUMI1M CI PBFI  EBFREQUIRES MINIMIZING BEGINALIGNJCBF  LA XBF  SUMJ1M CJ PBFJ XBF  SUMI1M CIPBFI RA NONUMBER   LA XBFXBF RA  2 REAL LEFTSUMI1M CBARI LA   XBFPBFIRARIGHT  SUMI1M SUMJ1M CJ CBARI LA PBFJ PBF I RA LABELEQGRADMIN1BENDALIGNUSING THE VECTOR NOTATIONS DEFINED IN REFEQPBF REFEQCBFAND REFEQGRAMDEF WE CAN WRITE REFEQGRADMIN1B ASBEGINEQUATIONJCBF   XBF2  2 REALLEFTCBFH PBFRIGHT  CBFH RT CBFLABELEQGRADMIN2ENDEQUATIONSOME GRADIENT FORMULAS ARE DERIVED IN SECTION REFSECIMPGRAD INPARTICULAR THE FOLLOWING GRADIENT FORMULAS ARE DERIVED PARTIALDCBFBAR DBFH CBF  ZEROBF QQUADPARTIALDCBFBAR CBFH DBF  DBF QQUADPARTIALDCBFBAR REALCBFH DBF  FRAC12 DBF QQUADPARTIALDCBFBAR CBFH R CBF  R CBFTAKING THE GRADIENT OF REFEQGRADMIN2 USING THE LAST TWO OF THESEWE OBTAINBEGINEQUATION PARTIALDCBFBAR  LEFT XBF2  2 REALCBFH PBF  CBFH RCBFRIGHT  PBF  R CBFLABELEQGRADMIN1AENDEQUATIONEQUATING THIS RESULT TO ZERO WE OBTAIN RCBF  PBFGIVING US AGAIN THE NORMAL EQUATIONS INDEXNORMAL EQUATIONSTO DETERMINE WHETHER THE EXTREMUM WE HAVE OBTAINED BY THE GRADIENT ISIN FACT A MINIMUM WE COMPUTE THE GRADIENT A SECOND TIME  WE HAVE THEHESSIAN MATRIX INDEXHESSIAN MATRIX PARTIALDCBFBAR  R CBF  RWHICH IS A POSITIVESEMIDEFINITE MATRIX SO THE EXTREMUM MUST BE AMINIMUMRESTRICTING ATTENTION FOR THE MOMENT TO REAL VARIABLES CONSIDER THEPLOT OF THE NORM OF THE ERROR JCBF AS A FUNCTION OF THE VARIABLESC1 C2 LDOTS CM  SUCH A PLOT IS CALLED AN EM ERROR  SURFACE INDEXERROR SURFACE BECAUSE JCBF IS QUADRATIC INCBF AND R IS POSITIVE SEMIDEFINITE THE ERROR SURFACE IS APARABOLIC BOWL  FIGURE REFFIGERRORSURF1 ILLUSTRATES SUCH AN ERRORSURFACE FOR TWO VARIABLES C1 AND C2  BECAUSE OF ITS PARABOLICSHAPE ANY EXTREMUM MUST BE A MINIMUM AND IS IN FACT A GLOBALMINIMUMBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE PLOTJSURF    EPSFIGFILEPICTUREDIRQUADERREPS    CAPTIONAN ERROR SURFACE FOR TWO VARIABLES    LABELFIGERRORSURF1  ENDCENTERENDFIGURESECTIONMATRIX REPRESENTATIONS OF LEASTSQUARES PROBLEMSLABELSECMATLSWHILE VECTOR SPACE METHODS APPLY TO BOTH INFINITE ANDFINITEDIMENSIONAL VECTORS SIGNALS THE NOTATIONAL POWER OF MATRICESCAN BE APPLIED WHEN THE BASIS VECTORS ARE FINITEDIMENSIONAL  THELINEAR COMBINATION OF THE FINITE SET OF VECTORSPBF1PBF2LDOTSPBFM CAN BE WRITTEN AS XBFHAT  SUMI1M CI PBFI   PBF1 PBF2 CDOTSPBFMBEGINBMATRIX C1  C2   VDOTS  CM ENDBMATRIXTHIS IS THE LINEAR COMBINATION OF THE COLUMNS OF THE MATRIX ADEFINED BY  A  BEGINBMATRIX PBF1  PBF2  CDOTS  PBFM ENDBMATRIXWHICH WE COMPUTE BY XBFHAT  ACBFTHE APPROXIMATION PROBLEM CAN BE STATED AS FOLLOWS   FBOXPARBOX09TEXTWIDTH BEGINQUOTE DETERMINE CBF TO MINIMIZE  EBF22 IN THE PROBLEM BEGINEQUATION XBF  A CBF  EBF  XBFHAT  EBF LABELEQMATLS ENDEQUATION ENDQUOTE BOXEDBEGINEQUATIONBOXEDRULE8EM0EM2EM   ADD A LITTLE SIZE TO THE BOXTEXT DETERMINE CBF TO MINIMIZE  EBF22 IN THE EQUATION XBF  A CBF  EBF  XBFHAT  EBF  LABELEQMATLSENDEQUATIONNOINDENT THE MINIMUM  EBF22  XBF  ACBF2 OCCURS WHEN EBFIS ORTHOGONAL TO EACH OF THE VECTORS LA XBF  ACBF PBFJRA  0QQUAD J12LDOTSMSTACKING THESE ORTHOGONALITY CONDITIONS WE OBTAIN BEGINBMATRIX PBF1H  PBF2H  VDOTS  PBFMHENDBMATRIXXBF  A CBF  ZEROBFRECOGNIZING THAT THE STACK OF VECTORS IS SIMPLY AH WE OBTAINBEGINEQUATION AHA CBF  AH XBFLABELEQLMAT9ENDEQUATIONTHE MATRIX AH A IS THE GRAMMIAN R AND THE VECTOR AH XBF ISTHE CROSSCORRELATION PBF  WE CAN WRITE REFEQLMAT9 ASBEGINEQUATIONR CBF  AH XBF  PBFLABELEQGAZENDEQUATIONTHESE EQUATIONS ARE THE NORMAL EQUATIONS INDEXNORMAL EQUATIONSTHEN THE OPTIMAL LEASTSQUARES COEFFICIENTS AREBEGINEQUATIONBOXEDCBF  AHA1 AH XBF  R1 PBFLABELEQLMAT0ENDEQUATIONBY THEOREM REFTHMGRAMMPD AHA IS POSITIVE DEFINITE IF THEPBF1LDOTSPBFM ARE LINEARLY INDEPENDENT  THE MATRIX AHA1 AH IS CALLED A EM PSEUDOINVERSE INDEXPSEUDOINVERSEOF A AND IS OFTEN DENOTED ADAGGER  MORE IS SAID ABOUTPSEUDOINVERSES IN SECTION REFSECPSINV  WHILEREFEQLMAT0 PROVIDES AN ANALYTICAL PRESCRIPTION FOR THE OPTIMALCOEFFICIENTS IT SHOULD RARELY BE COMPUTED EXPLICITLY AS SHOWN SINCEMANY PROBLEMS ARE NUMERICALLY UNSTABLE SUBJECT TO AMPLIFICATION OFROUNDOFF ERRORS  NUMERICAL STABILITY IS DISCUSSED IN SECTIONREFSECMATCOND  STABLE METHODS FOR COMPUTING PSEUDOINVERSES AREDISCUSSED IN SECTIONS REFSECQR AND REFSECPSEUDOINVERSESVDIN SC MATLAB THE PSEUDOINVERSE MAY BE COMPUTED USING THE COMMANDTT PINVUSING REFEQLMAT0 THE APPROXIMATION ISBEGINEQUATIONBOXED XBFHAT  A CBF  AAHA1 AH XBFLABELEQLSMAT1ENDEQUATIONTHE MATRIX P  AAHA1AH IS A EM PROJECTION MATRIXINDEXPROJECTION MATRIX WHICH WE ENCOUNTERED IN SECTIONREFSECPROJECTIONS  THE MATRIX P PROJECTS ONTO THE RANGE OF ACONSIDER GEOMETRICALLY WHAT IS TAKING PLACE WE WISH TO SOLVE THEEQUATION A CBF  XBF BUT THERE IS NO EXACT SOLUTION SINCE XBFIS NOT IN THE RANGE OF A  SO WE PROJECT XBF ORTHOGONALLY DOWNONTO THE RANGE OF A AND FIND THE BEST SOLUTION IN THAT RANGE SPACETHE IDEA IS SHOWN IN FIGURE REFFIGPSOLBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRORTHOGPROJ6    CAPTIONPROJECTION SOLUTION    LABELFIGPSOL  ENDCENTERENDFIGUREA USEFUL REPRESENTATION OF THE GRAMMIAN R  AHA CAN BE OBTAINED BYCONSIDERING A AS A STACK OF ROWSBEGINEQUATIONA  BEGINBMATRIX QBF1H  QBF2H  VDOTS  QBFNHENDBMATRIXLABELEQXSTACKROWENDEQUATIONSO THAT AH  QBF1 QBF2LDOTSQBFN ANDBEGINEQUATIONAHA  BEGINBMATRIXQBF1  QBF2  CDOTS  QBFN ENDBMATRIXBEGINBMATRIXQBF1H  QBF2H  VDOTS  QBFNHENDBMATRIX  SUMI1N QBFI QBFIHLABELEQXSTACKROWENDEQUATIONSUBSECTIONWEIGHTED LEASTSQUARESLABELSECWLSINDEXWEIGHTED LEASTSQUARESA WEIGHT CAN ALSO BE APPLIED TO THE DATA POINTS REFLECTING THECONFIDENCE IN THE DATA AS ILLUSTRATED BY THE NEXT EXAMPLE  THIS ISNATURALLY INCORPORATED INTO THE INNER PRODUCTDEFINE A WEIGHTED INNER PRODUCT AS LA XBFYBFRAW  XBFH W YBFTHEN MINIMIZING EBFW2  ACBF  XBFW2 LEADS TO THEWEIGHTED NORMAL EQUATIONSBEGINEQUATIONAH WACBF  AH WXBFLABELEQWLS1ENDEQUATIONSO THE COEFFICIENTS WHICH MINIMIZE THE WEIGHTED SQUARED ERROR AREBEGINEQUATION  LABELEQWLS2  CBF  AH W A1 AH W YBFENDEQUATIONANOTHER APPROACH TO REFEQWLS2 IS TO PRESUME THAT WE  HAVE AFACTORIZATION OF THE WEIGHT W  SHS SEE SECTIONREFSECCHOLESKY  THEN WE WEIGHT THE EQUATION  SA CBF APPROX SYBFMULTIPLYING THROUGH BY SAH AND SOLVING FOR CBF WE OBTAIN CBF  SAHSA1SAH SYBFWHICH IS EQUIVALENT TO REFEQWLS2SUBSECTIONSTATISTICAL PROPERTIES OF THE LEASTSQUARES ESTIMATELABELSECLSPROPSUPPOSE THAT THE SIGNAL XBF HAS THE TRUE MODEL ACCORDING TO THEEQUATIONBEGINEQUATIONXBF  A CBF0  EBFLABELEQGRAMSTATENDEQUATIONFOR SOME TRUE MODEL PARAMETER VECTOR CBF0 AND THAT WEASSUME A STATISTICAL MODEL FOR THE MODEL ERROR EBF ASSUME THATEACH COMPONENT OF EBF IS A ZEROMEAN IID RANDOM VARIABLE WITHVARIANCE SIGMAE2  THE ESTIMATED PARAMETER VECTOR ISBEGINEQUATIONCBF  AHA1 AH XBFLABELEQCESTENDEQUATIONTHIS LEASTSQUARES ESTIMATE BEING A FUNCTION OF THE RANDOM VECTORXBF IS ITSELF A RANDOM VECTOR  WE WILL DETERMINE THE MEAN ANDCOVARIANCE MATRIX FOR THIS RANDOM VECTORBEGINDESCRIPTIONITEMMEAN OF CBF SUBSTITUTING THE TRUE MODEL OF REFEQGRAMSTAT INTOREFEQCEST WE OBTAINBEGINALIGNEDCBF   AHA1 AHA CBF0  AHA1AH  EBF  CBF0  AHA1AH  EBFENDALIGNEDIF WE NOW TAKE THE EXPECTED VALUE OF OUR ESTIMATED PARAMETER VECTOR WEOBTAIN ECBF  ECBF0  AHA1AH  EBF  CBF0SINCE EACH COMPONENT OF EBF HAS ZERO MEAN  THUS THE EXPECTEDVALUE OF THE ESTIMATE IS EQUAL TO THE TRUE VALUE  SUCH AN ESTIMATE ISSAID TO BE BF UNBIASED INDEXUNBIASEDITEMCOVARIANCE OF CBF THE COVARIANCE CAN BE WRITTEN ASBEGINALIGNEDCOVCBF  ECBF  CBF0CBF  CBF0H  AHA1 AH EEBF EBFH AAHA1ENDALIGNEDSINCE THE COMPONENTS OF EBF ARE IID IT FOLLOWS THATEEBF EBFH  SIGMAE2 I SO THAT COVCBF  SIGMAE2AHA1  SIGMAE2 R1ITEM SMALLEST COVARIANCE ANOTHER INTERESTING FACT OF ALL POSSIBLE  UNBIASED LINEAR ESTIMATES THE ESTIMATOR REFEQLMAT0 HAS THE  SMALLEST COVARIANCE  SUPPOSE WE HAVE ANOTHER UNBIASED LINEAR  ESTIMATOR CBFTILDE GIVEN BY CBFTILDE  L XBFWHERE ECBFTILDE  CBF0  USING OUR STATISTICAL MODEL REFEQGRAMSTAT WE OBTAIN CBFTILDE  LA CBF0  L EBFIN ORDER FOR THE ESTIMATE CBFTILDE TO BE UNBIASED WE MUST HAVEECBFTILDE  CBF0 SO LA  IWE THEREFORE OBTAIN CBFTILDE  CBF0  L EBF  THE COVARIANCE OFCBFTILDE IS COVCBFTILDE  ECBFTILDE  CBF0CBFTILDE  CBF0H SIGMAE2 LLHWE WILL SHOW THAT LLH  R1 IN THE SENSE THAT THE MATRIX LLH R1 IS POSITIVE SEMIDEFINITE  INDEXPOSITIVESEMIDEFINITE LET Z  L  R1AHTHEN FOR ANY ZBF  0 LEQ  ZH ZBF 2  LA ZH ZBF ZH ZBF RA  ZBFH Z ZHZBFBUT  ZZH  LLH  R1WHERE WE HAVE USED THE FACT THAT LA  I  THUS FOR ANY ZBF  ZBFH LLH  R1 ZBF GEQ 0SO LLH  R1 IS POSITIVE SEMIDEFINITE OR R1 IS A SMALLERCOVARIANCE MATRIX  THE ESTIMATOR CBF IS SAID TO BE A BEST LINEARUNBIASED ESTIMATOR BLUE INDEXBEST LINEAR UNBIASED ESTIMATE  BLUE INDEXMINIMUM VARIANCE ESTIMATEIT WILL BE SHOWN IN CHAPTER REFCHAPEST THAT UNDER THECONDITION THAT THE NOISE EBF IS GAUSSIAN THE COVARIANCE OFCBF IS IN FACT THE SMALLEST COVARIANCE AMONG ALL POSSIBLEUNBIASED ESTIMATORS  WE SHALL SEE IN SECTION REFSECCRLB THATTHERE IS A LOWER BOUND ON THE VARIANCE OF UNBIASED ESTIMATORS INDEXCRAMERRAO LOWER BOUND CRLB ENDDESCRIPTIONSECTIONMINIMUM ERROR IN VECTOR SPACE APPROXIMATIONSLABELSECMINERRINDEXMINIMUM ERRORIN THIS SECTION WE EXAMINE HOW MUCH ERROR IS LEFT WHEN AN OPTIMALMINIMALNORM SOLUTION IS OBTAINED  UNDER THE MODEL THAT XBF  SUMI1M CI PBFI  EBFWHEN THE COEFFICIENTS ARE FOUND SO THAT THE ESTIMATION ERROR ISORTHOGONAL TO THE DATA WE HAVE XBF  XBFHAT  EBFMINWHERE EBFMIN DENOTES THE MINIMUM ACHIEVABLE ERRORTAKING THE SQUARED NORM OF BOTH SIDES WE OBTAINBEGINEQUATION  XBF 2   XBFHAT 2   EBFMIN 2LABELEQEMIN1ENDEQUATIONTHIS RESULT SOMETIMES CALLED THE STATISTICIANS PYTHAGOREAN THEOREMINDEXPYTHAGOREAN THEOREMSTATISTICIANSFOLLOWS BECAUSE XBFHAT IS ORTHOGONAL TO THE MINIMUMNORM ERROR LA XBFHAT EBFMINRA  0THE STATISTICIANS PYTHAGOREAN THEOREM IS ILLUSTRATED IN FIGUREREFFIGPYTHAG1BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRPYTHAG1    CAPTIONSTATISTICIANS PYTHAGOREAN THEOREM    LABELFIGPYTHAG1  ENDCENTERENDFIGURESEE ALSO LEMMA REFLEMPYTH  THE SQUARED NORM OF THE MINIMUMERROR IS  EBFMIN2   XBF2   XBFHAT2WHEN WE USE THE MATRIX FORMULATION WE CAN OBTAIN A MORE EXPLICITREPRESENTATION FOR THE MINIMUM ERROR  THEN XBFHAT  ACBF SOBEGINEQUATION  XBFHAT 2  CBFH AH A CBF  CBFH R CBF CBFH PBFLABELEQEMIN2ENDEQUATIONWHERE PBF FROM REFEQGAZ HAS BEEN EMPLOYED  THIS GIVES  EBFMIN2  XBFH XBF  CBFH PBFANOTHER FORM FOR XBFHAT2 IS OBTAINED FROMREFEQLSMAT1BEGINEQUATION XBFHAT2  ACBFH ACBF  XBFH AAHA1 AH XBFLABELEQEMIN3ENDEQUATIONTHENBEGINALIGNEDEBFMIN2   XBFH XBF  XBFH AAHA1 AH XBF  XBFHI  AAHA1AH XBFENDALIGNEDIT CAN BE SHOWN SEE EXERCISE REFEXREDUCEERR THAT BEGINEQUATION I  AAHA1AHLABELEQREDUCERRENDEQUATIONIS A POSITIVESEMIDEFINITE MATRIX INDEXPOSITIVESEMIDEFINITE MATRIXFROM WHICH WE CAN CONCLUDE THAT EBFMIN2 IS SMALLER THANXBF2BEGINEXERCISESITEM SHOW THAT REFEQXSTACKROW IS TRUE  ITEM SHOW THAT I  AAHA1AHIS POSITIVESEMIDEFINITE AND HENCE THAT THE MINIMUM ERROREBFMIN HAS SMALLER NORM THAN THE ORIGINAL VECTOR XBF  HINTCONSIDER 0 LEQ  BXBF2 WHERE B  I  AAHA1AHENDEXERCISESCHAPTERPARTAPPLICATIONS OF THE ORTHOGONALITY THEOREMLABELSECOTAPP1BECAUSE A NUMBER OF VECTOR SPACES AND INNER PRODUCTS CAN BEFORMULATED THE ORTHOGONALITY PRINCIPLE IS USED IN A VARIETY OFAPPLICATIONS  THE ORTHOGONALITY THEOREM PROVIDES THE FOUNDATION FOR AGOOD PART OF SIGNAL PROCESSING THEORY SINCE IT PROVIDES APRESCRIPTION FOR AN OPTIMUM ESTIMATOR BF IN THE OPTIMUM  LEASTSQUARES ESTIMATOR THE ERROR IS ORTHOGONAL TO THE DATA THETHEOREM IS APPLIED BY DEFINING AN INNER PRODUCT AND HENCE THE INDUCEDNORM TO MATCH THE NEEDS OF THE PROBLEM  UNDER VARIOUS DEFINITIONS OFINNER PRODUCTS MUCH OF APPROXIMATION THEORY ESTIMATION THEORY ANDPREDICTION THEORY CAN BE ACCOMMODATED  EXAMPLES ARE GIVEN IN THE NEXTSEVERAL SECTIONSSECTIONAPPROXIMATION BY CONTINUOUS POLYNOMIALSLABELSECPOLYAPPROX1INDEXPOLYNOMIAL APPROXIMATIONCONTINUOUS POLYNOMIALSSUPPOSE WE WANT TO FIND THE BEST POLYNOMIAL APPROXIMATION OF A REALCONTINUOUS FUNCTION FT OVER AN INTERVAL T IN AB IN THESENSE THAT INTAB FT PT2 DTIS MINIMIZED FOR A POLYNOMIAL PT OF DEGREE M1  THE VECTOR SPACEUNDERLYING THE PROBLEM IS S  CAB  WE WILL NAIVELY TAKE ASBASIS VECTORS THE FUNCTIONS 1TT2LDOTSTM1 SO THAT PT  C0  C1 T  C2 T2  CDOTS  CM1 TM1THE OPTIMAL COEFFICIENTS CAN BE DETERMINED FOR EXAMPLEDIRECTLY BY CALCULUS BUT THE ORTHOGONALITY THEOREM APPLIES USING THEINNER PRODUCT LA FG RA  INTAB FTGTDTTHEN USING REFEQPROJ2 WE OBTAINBEGINEQUATIONBEGINBMATRIXLA 11 RA  LA 1T RA  CDOTS  LA 1TM1 RA  LA T1 RA  LA TT RA  CDOTS  LA TTM1 RA VDOTS LA TM11 RA  LA TM1TRA  CDOTS  LA TM1TM1 RAENDBMATRIXBEGINBMATRIXC0  C1  VDOTS  CM ENDBMATRIX BEGINBMATRIX LA F1 RA  LA FT RA  VDOTS  LA FTM1  RA ENDBMATRIXLABELEQPOLYAPPROX1ENDEQUATIONIF WE TAKE THE SPECIFIC CASE THAT THE FUNCTION IS TO BE APPROXIMATEDOVER THE INTERVAL 01 THEN THE GRAMMIAN MATRIX INREFEQPOLYAPPROX1 CAN BE COMPUTED EXPLICITLY AS LA TITJ RA  INT01 TIJDT  FRAC1IJ1QQUADIJ01LDOTSM1 SO THATBEGINEQUATIONR  BEGINBMATRIX1  FRAC12  FRAC13  CDOTS   FRAC1M  FRAC12  FRAC13  FRAC14  CDOTS  FRAC1M1 VDOTS FRAC1M  FRAC1M1  FRAC1M2  CDOTS  FRAC12M ENDBMATRIXLABELEQHILBERTGENDEQUATIONA MATRIX OF THIS PARTICULAR FORM IS KNOWN AS A BF HILBERT MATRIXINDEXHILBERT MATRIX THE HILBERT MATRIX IS FAMOUS AS A CLASSICEXAMPLE OF A MATRIX THAT IS ILLCONDITIONED AS M INCREASES THEMATRIX BECOMES ILLCONDITIONED INDEXILLCONDITIONED EXPONENTIALLYFAST WHICH MEANS AS DISCUSSED IN SECTION REFSECMATCOND THAT ITWILL SUFFER FROM SEVERE NUMERICAL PROBLEMS IF M IS EVEN MODERATELYLARGE NO MATTER HOW IT IS INVERTED  BECAUSE OF THIS THE PARTICULARSET OF BASIS FUNCTIONS CHOSEN IS NOT RECOMMENDED  THE USE OF THELEGENDRE POLYNOMIALS DESCRIBED IN EXAMPLE REFEXMLEGENDREPOLY OROTHER ORTHOGONAL POLYNOMIALS IS PREFERRED FOR POLYNOMIALAPPROXIMATION BEGINEXAMPLE  LET FT  ET AND M3  FOR ONLY THREE PARAMETERS THE HILBERT  MATRIX REFEQHILBERTG IS STILL WELLCONDITIONED  THE VECTOR  ON THE RIGHTHAND OF REFEQPOLYAPPROX1 IS BBF  BEGINBMATRIX E1  1  E2 ENDBMATRIXAND THE COEFFICIENTS IN REFEQPOLYAPPROX1 ARE COMPUTED AS BEGINBMATRIXC0  C1  C2 ENDBMATRIX  R1 BBF BEGINBMATRIX 10130     08511     08392 ENDBMATRIXTHE APPROXIMATING POLYNOMIAL IS ET APPROX 10130  8511 T  8392 T2FIGURE REFFIGHILB1 SHOWS THE ABSOLUTE ERROR ET  PT FOR THISPOLYNOMIAL FOR T IN01  FOR COMPARISON THE ERROR WE WOULD GETBY APPROXIMATING ET BY THE FIRST THREE TERMS OF THE TAYLOR SERIESEXPANSION  ET APPROX 1  T  T22IS ALSO SHOWN AS IS THE WEIGHTED LEASTSQUARES WLS APPROXIMATIONDISCUSSED SUBSEQUENTLY  THE ERROR IN THE TAYLOR SERIES STARTS SMALLBUT INCREASES TO A LARGER VALUE THAN DOES THE LEASTSQUARESAPPROXIMATION  HOW WOULD THE TAYLOR SERIES HAVE COMPARED IF THESERIES HAD BEEN EXPANDED ABOUT THE MIDPOINT OF THE REGION ATT0FRAC12 USE THE FILE PROGSHILB1M THEN MOVE THE LEGEND AND SAVEBEGINFIGUREHTBP  CENTERLINEPSFIGFILEPICTUREDIRHILB1EPS  CAPTIONCOMPARISON OF LS WLS AND    TAYLOR SERIES APPROXIMATIONS TO ET   LABELFIGHILB1ENDFIGUREENDEXAMPLETHE BASIS FUNCTIONS OF THE PREVIOUS EXAMPLE GIVE RISE TO THE HILBERTMATRIX AS THE GRAMMIAN  HOWEVER A SET OF EM ORTHOGONALPOLYNOMIALS CAN BE USED THAT HAS A DIAGONAL AND HENCEWELLCONDITIONED GRAMMIAN  NOW SUPPOSE THAT FOR SOME REASON IT IS MORE IMPORTANT TO GET THEAPPROXIMATION MORE CORRECT ON THE EXTREMES OF THE INTERVAL OFAPPROXIMATION  WE WILL DENOTE THE APPROXIMATING POLYNOMIAL IN THISCASE BY PWT  TO ATTEMPT TO MAKE THE APPROXIMATION MORE EXACT ONTHE EXTREMES OF THE INTERVAL OF APPROXIMATION WE USE A WEIGHTED NORM INTAB WTFT  PWT2 DTWHICH IS INDUCED FROM THE INNER PRODUCT LA FG RA  INTAB SQRTWT FTGTDTBEGINEXAMPLE  CONTINUING THE EXAMPLE ABOVE WITH FT  ET OVER 01 TAKE  THE WEIGHTING FUNCTION AS WT  10T052THEN THE GRAMMIAN MATRIX IS R  BEGINPMATRIXFRAC12SQRT52  FRAC14SQRT52  FRAC316SQRT52FRAC14SQRT52  FRAC316SQRT52 FRAC532SQRT52FRAC316SQRT52 FRAC532SQRT52FRAC1396SQRT52ENDPMATRIXAND THE RIGHTHAND VECTOR COMPUTED NUMERICALLY IS BBF   138603    0860513   0690724THE APPROXIMATING POLYNOMIAL IS NOW PWT  10109  8535 T  8415 T2FIGURE REFFIGHILB1 SHOWS THE ERROR ET  PWT AND ET PT  AS EXPECTED THE ERROR IS SMALLER THOUGH ONLY SLIGHTLY FORPWT NEAR THE ENDPOINTS BUT LARGER IN BETWEENENDEXAMPLEAS VARIOUS WEIGHTINGS ARE IMPOSED THE ERROR AT SOME VALUES OF T ISREDUCED WHILE ERROR FOR OTHER VALUES OF T MAY INCREASE  THISRAISES THE FOLLOWING INTERESTING AND IMPORTANT QUESTION IS THERESOME WAY TO DESIGN THE APPROXIMATION SO THAT THE MAXIMUM ERROR ISMINIMIZED  THIS IS WHAT LINFTY APPROXIMATION IS ALL ABOUT  THEAPPROXIMATION IS FOUND SO THAT THE MAXIMUM ERROR IS MINIMIZED MORE WILL BE SAID ABOUT THIS IN CHAPTER REFCHAPAPPROXSECTIONAPPROXIMATION BY DISCRETE POLYNOMIALSLABELSECPOLYAPPROX2INDEXPOLYNOMIAL APPROXIMATIONDISCRETE POLYNOMIALSWE CAN APPROXIMATE DISCRETE SAMPLED DATA USING POLYNOMIALS IN AMANNER SIMILAR TO THE CONTINUOUS POLYNOMIAL APPROXIMATIONS OF SECTIONREFSECPOLYAPPROX1 USING A SET OF DISCRETETIME BASIS FUNCTIONS1KLDOTSKM1  WE DESIRE TO FIT AN M1ST ORDERPOLYNOMIAL THROUGH THE DATA POINTS X1X2LDOTSXN SO THATXK APPROX PKQQUAD K12LDOTSNWHERE PK  C0  C1 K  C2 K2  CDOTS  CM1 KM1THE POLYNOMIAL PK CAN BE WRITTEN AS PK  1 K K2 CDOTS KM1BEGINBMATRIXC0  C1  C2   VDOTS  CM1 ENDBMATRIXIF MN AND THE XK ARE DISTINCT THEN THERE EXISTS A POLYNOMIALTHE EM INTERPOLATING POLYNOMIAL INDEXINTERPOLATING POLYNOMIALPASSING EXACTLY THROUGH ALL N POINTS  IF M N THEN THERE ISPROBABLY NOT A POLYNOMIAL THAT WILL PASS THROUGH ALL N POINTS INWHICH CASE WE DESIRE TO FIND THE POLYNOMIAL TO MINIMIZE THE SQUAREDERROR SUMK1N XK  PK2THIS CAN BE EXPRESSED AS A VECTOR NORM  XBF  PBF 2WHICH IS INDUCED FROM THE EUCLIDEAN INNER PRODUCT LA XBF YBF RA XBFH YBF WHERE XBF  BEGINBMATRIX X1  X2  VDOTS XNENDBMATRIXQQUADTEXTANDQQUADPBF  BEGINBMATRIX P1  P2  VDOTS  PN ENDBMATRIXWE CAN WRITE PBF IN TERMS OF THE COEFFICIENTS OF THE POLYNOMIAL AS PBF  BEGINBMATRIX1  1  1 CDOTS  1 1  2  4  CDOTS  2M1 1  3  9  CDOTS  3M1 VDOTS 1  N  N2  CDOTS  NM1 ENDBMATRIXBEGINBMATRIXC0  C1  C2  VDOTS  CM1 ENDBMATRIX PBF1  PBF2 PBF3  CDOTS PBFMBEGINBMATRIXC0  C1   C2  VDOTS  CM1 ENDBMATRIX  P ABFTHE VECTORS PBFI I 12LDOTS M REPRESENT THE DATA IN THISAPPROXIMATION PROBLEM  IF P IS SQUARE IT IS CALLED A EM  VANDERMONDE MATRIX INDEXVANDERMONDE MATRIX ABOUT WHICH MORE ISPRESENTED IN SECTION REFSECVANDERMONDE  AS WITH THECONTINUOUSTIME POLYNOMIAL APPROXIMATION THERE MAY BE BETTER BASISFUNCTIONS FOR THIS PROBLEM FROM A NUMERICAL POINT OF VIEWUSING THIS NOTATION THE APPROXIMATION PROBLEM BECOMES XBF  P CBF  EBFWHICH IS A PROBLEM IN THE SAME FORM AS REFEQNORM3 FROM WHICHOBSERVE THAT THE CBF WHICH MINIMIZES  EBF2 IS CBF  PT P1 PT XBFTHE APPROXIMATED VECTOR PBF IS THUS PBF  P CBF  PPTP1PT XBFBEGINEXAMPLE WE DESIRE TO APPROXIMATE THE FUNCTION XK  SINKPI7USING A QUADRATIC POLYNOMIAL M3 TO OBTAIN THE BEST MATCH FORK1MC 7  THE P MATRIX ISBEGINBMATRIX1  1  1 1  2  4 1  3  9 1  4  16 1  5  25 1  6  36 1  7  49 ENDBMATRIXAND THE COEFFICIENTS ARE COMPUTED AS CBFT 006120588500833  FIGURE REFFIGDISCAPPROXA SHOWSXK AND FIGURE REFFIGDISCAPPROXB SHOWS THE ERROR PK XKBEGINFIGUREHTBP  CENTERLINEMBOXSUBFIGUREMBOXXKEPSFIGFILEPICTUREDIRDISCAPPROX1EPS           WIDTH045TEXTWIDTHQUAD     SUBFIGUREMBOXXK PKEPSFIGFILEPICTUREDIRDISCAPPROX2EPS           WIDTH045TEXTWIDTH  CAPTIONA DISCRETE FUNCTION AND THE ERROR IN ITS APPROXIMATION  LABELFIGDISCAPPROX USE DISCAPPROXMENDFIGUREENDEXAMPLESECTIONLINEAR REGRESSIONLABELSECLINREGFROM THE  DATA IN FIGURE REFFIGREGRESS1A WHERE THERE ARE NPOINTS XBFI I12LDOTSN WITH EACH XBFI  XIYIT ITWOULD APPEAR THAT WE CAN APPROXIMATELY FIT A LINEOF THE FORMBEGINEQUATIONYI APPROX A XI  B QQUAD I12LDOTSNLABELEQREGRESS1ENDEQUATIONFOR SUITABLY CHOSEN SLOPE A AND INTERCEPT B  AS STATED THIS IS AEM LINEAR REGRESSION INDEXREGRESSION PROBLEM THAT IS A PROBLEMOF DETERMINING A FUNCTIONAL RELATION BETWEEN THE MEASURED VARIABLESXI AND YI  NONLINEAR REGRESSIONS ARE ALSO USED SUCH AS THEQUADRATIC REGRESSIONBEGINEQUATION YI APPROX A0  A1 XI  A2 XI2LABELEQREGRESS2ENDEQUATIONOR WE MAY HAVE DATA VECTORS XBFI IN RBB3 WITH XBFI XIYIZIT AND WE MAY REGRESS AMONG THE POINTS ASBEGINEQUATIONZI APPROX A XI  B YI  C LABELEQREGRESS3ENDEQUATIONIN ALL SUCH REGRESSION PROBLEMS WE DESIRE TO CHOOSE THE REGRESSIONPARAMETERS SO THAT THE RIGHTHAND SIDE OF THE REGRESSION EQUATIONSPROVIDES A GOOD REPRESENTATION OF THE LEFTHAND SIDEBEGINFIGURET  CENTERLINEMBOXSUBFIGUREORIGINAL DATAEPSFIGFILEPICTUREDIRREGRESS1EPS                WIDTH045TEXTWIDTHQUADSUBFIGUREINTERPOLATED LINE AND                  ERRORSEPSFIGFILEPICTUREDIRREGRESS2EPS                WIDTH045TEXTWIDTH  CAPTIONDATA FOR REGRESSION  LABELFIGREGRESS1 TEST2REGRESSMENDFIGUREWE WILL CONSIDER IN DETAIL THE LINEAR REGRESSION PROBLEMREFEQREGRESS1  WE CAN STACK THE EQUATIONS TO OBTAINBEGINEQUATION BEGINBMATRIX Y1  Y2  VDOTS  YN ENDBMATRIX BEGINBMATRIX A X1  B  A X2  B  VDOTS  A XN  BENDBMATRIX  BEGINBMATRIX  E1  E2  VDOTS  ENENDBMATRIXLABELEQLINREGRESSENDEQUATIONFOR SOME ERROR TERMS EI  LET YBF  Y1 Y2 LDOTS YNTQQUAD EBF  E1E2 LDOTS ENT QQUADCBF  BEGINBMATRIX  A B ENDBMATRIXAND A  BEGINBMATRIX X1  1  X2  1  VDOTS  XN  1ENDBMATRIXTHEN REFEQLINREGRESS IS OF THE FORMBEGINEQUATIONYBF  A CBF  EBFLABELEQREGRESS4ENDEQUATIONWHICH AGAIN IS IN THE FORM REFEQMATLS SO THE BEST IN THELEASTSQUARES SENSE ESTIMATE OF CBF ISBEGINEQUATION CBF  AHA1AH YBFLABELEQ2REGRESSENDEQUATIONTHE LINE FOUND BY REFEQ2REGRESS MINIMIZES THE SUMS OF THESQUARES OF THE EM VERTICALDISTANCES BETWEEN THE DATA ABSCISSAS AND THE LINE AS SHOWN IN FIGUREREFFIGREGRESS1B  TO MINIMIZE EM SHORTEST DISTANCES OF THEDATA TO THE INTERPOLATING LINE THE METHOD OF EM TOTAL LEAST  SQUARES DISCUSSED IN SECTION REFSECTLS MUST BE USEDSINCE AHA IN REFEQ2REGRESS IS A MATSIZE22 MATRIXEXPLICIT CLOSEDFORM EXPRESSIONS FOR M AND B IN CBF CAN BEFOUND  THE SLOPE AND INTERCEPT FOR REAL DATA AREBEGINEQUATIONBEGINSPLITA  FRAC N SUMI1N XBARI YI  LEFTSUMI1N XIRIGHT  LEFTSUMJ1N  YIRIGHTN SUMI1N XI2     LEFTSUMI1N XIRIGHTLEFTSUMI1N XBARIRIGHT B  FRACLEFTSUMI1N XI2RIGHTLEFTSUMJ1N  YIRIGHT  LEFTSUMI1N XIRIGHTLEFTSUMI1N XBARI  YIRIGHT N SUMI1N XI2  LEFTSUMI1N XI  RIGHTLEFTSUMI1N XBARI RIGHTENDSPLITLABELEQLINREGRESSAENDEQUATIONBEGINEXAMPLE WEIGHTED LEASTSQUARES INDEXWEIGHTED LEASTSQUARES  FIVE MEASUREMENTS XIYII12LDOTS5 ARE MADE IN A SYSTEM  OF WHICH THE FIRST THREE ARE BELIEVED TO BE FAIRLY ACCURATE AND TWO  ARE KNOWN TO BE SOMEWHAT CORRUPTED BY MEASUREMENT NOISE  THE  MEASUREMENTS ARE 125  335  65  53  34FROM THESE FIVE MEASUREMENTS  THE DATA ARE TO BE FITTED TO A LINE ACCORDING TO THE MODEL Y  AX   B  THE MEASUREMENTS STACK UP IN THE MODEL EQUATION AS BEGINBMATRIX1  1  3  1  6  1  5  1  3  1 ENDBMATRIXBEGINBMATRIXA  B ENDBMATRIX  BEGINBMATRIX25  35  5  3  4ENDBMATRIX  EBFOR ACBF  YBF  EBFIN FINDING THE BEST MINIMUM SQUAREDERROR SOLUTION TO THIS PROBLEMIT IS APPROPRIATE TO WEIGHT MOST HEAVILY THOSE EQUATIONS WHICH AREBELIEVED TO BE THE MOST ACCURATE  LET W  DIAG10101011THEN USING REFEQWLS2 WE CAN DETERMINE THE OPTIMAL UNDER THEWEIGHTED INNER PRODUCT SET OF COEFFICIENTS  FIGUREREFFIGREGRESS2 ILLUSTRATES THE DATA AND THE LEASTSQUARES LINESFITTED TO THEM  THE ACCURATE DATA ARE PLOTTED WITH TIMES AND THEINACCURATE DATA ARE PLOTTED WITH CIRC  THE WEIGHTED LEASTSQUARESLINE FITS MORE CLOSELY ON AVERAGE TO THE MORE ACCURATE DATA WHILETHE UNWEIGHTED LEASTSQUARES LINE IS PULLED OFF SIGNIFICANTLY BY THEINACCURATE DATA AT X5BEGINFIGUREHTBP  CENTERLINE MBOXEPSFIGFILEPICTUREDIRREGRESS3EPS  CAPTIONILLUSTRATION OF LEASTSQUARES AND WEIGHTED LEASTSQUARES    APPROXIMATIONS  LABELFIGREGRESS2 TEST2REGRESS2MENDFIGUREENDEXAMPLEBEGINEXERCISES  ITEM GIVEN THE SET OF DATA X   225359 QQUAD Y  42 5 2 1243BEGINENUMERATEITEM MAKE A PLOT OF THE DATAITEM DETERMINE THE BEST LEASTSQUARES LINE THAT FITS THIS DATA AND  PLOT THE LINEITEM ASSUMING THAT THE FIRST AND LAST POINTS ARE BELIEVED TO BE THE  MOST ACCURATE FORMULATE A WEIGHTING MATRIX AND COMPUTE A WEIGHTED  LEASTSQUARES LINE THAT FITS THE DATA  PLOT THIS LINEENDENUMERATEITEM FORMULATE THE REGRESSION PROBLEM REFEQREGRESS2 IN A   LINEAR FORM AS IN REFEQREGRESS4ITEM FORMULATE THE REGRESSION PROBLEM REFEQREGRESS3 IN A   LINEAR FORM AS IN REFEQREGRESS4ITEM FORMULATE THE REGRESSION Y APPROX CEAX AS A LINEAR  REGRESSION PROBLEM WITH REGRESSION PARAMETERS C AND AITEM FORMULATE THE REGRESSION Y APPROX AXBT AS A LINEAR  REGRESSION PROBLEMITEM PERFORM THE COMPUTATIONS TO VERIFY THE SLOPE AND INTERCEPT OF  THE LINEAR REGRESSION IN REFEQLINREGRESSAITEM AS A MEASURE OF FIT IN A CORRELATION PROBLEM THE CORRELATION  COEFFICIENT ANALOGOUS TO REFEQCORRCOEFF CAN BE OBTAINED AS RHO  FRACLA XBF YBFRA  LA XBFONEBFRA LA YBFONEBFRA XBF  LA XBFONEBFRA YBF  LA YBFONEBFRATHE CORRELATION COEFFICIENT RHO  PM 1 IF X AND Y ARE EXACTLYFUNCTIONALLY RELATED AND RHO  0 IF THERE IS THEY ARE INDEPENDENTFOR THE LINEAR REGRESSION IN REFEQ2REGRESS DETERMINE ANEXPLICIT EXPRESSION FOR RHOITEM GIVEN A SET OF DATA XBFI I12LDOTS M WHERE EACH VECTOR  XBFI IN RBBD FORMULATE THE LEASTSQUARES SOLUTION TO FIND  THE BEST DDIMENSIONAL PLANE FITTING THIS DATAITEM LET US DEFINE AN INNER PRODUCT BETWEEN MATRICES X AND Y AS LA XY RA  TRACEXYHWHERE TRACECDOT IS THE SUM OF THE DIAGONAL ELEMENTS SEE SECTIONREFSECTPOSETRACE  WE WANT TO APPROXIMATE THE MATRIX Y BY THE SCALARLINEAR COMBINATION OF MATRICES X1X2LDOTSXM AS Y  C1 X1  C2 X2  CDOTS  CM XM  EUSING THE ORTHOGONALITY PRINCIPLE DETERMINE A SET OF NORMAL EQUATIONSTHAT CAN BE USED TO FIND C1C2LDOTSCM THAT MINIMIZE THEINDUCED NORM OF EITEM FOR THE ARMA INPUTOUTPUT RELATIONSHIP OF REFEQARMA  DETERMINE A SET OF LINEAR EQUATIONS FOR DETERMINING THE ARMA MODEL  PARAMETERS A1A2LDOTSAPB0B1LDOTSBQ ASSUMING  THAT THE MODEL OR PQ IS KNOWN AND THAT THE INPUT IS KNOWNENDEXERCISESSECTIONLEASTSQUARES FILTERINGLABELSECLSFILTIN THE LEASTSQUARES FILTER PROBLEM WE DESIRE TO FILTER A SEQUENCE OFINPUT DATA FT USING A FILTER WITH IMPULSE RESPONSE HT OFLENGTH M TO PRODUCE AN OUTPUT THAT MATCHES A DESIRED SEQUENCEDT AS CLOSELY AS POSSIBLE  EXAMPLES IN WHICH SUCH ACIRCUMSTANCE ARISES ARE GIVEN IN SECTION REFSECADFILT IN THECONTEXT OF ADAPTIVE FILTERING  IF WE CALL THE OUTPUT OF THE FILTERYT WE HAVE THE FILTER EXPRESSION YT   SUMI0M1 HI FTIWE CAN WRITE DT  YT  ET WHERE ET IS THE ERROR BETWEENTHE FILTER OUTPUT AND THE DESIRED FILTER OUTPUT DT  SUMI0M1 HI FTI  ETWE WANT TO CHOOSE THE FILTER COEFFICIENTS HI IN SUCH A WAYTHAT THE ERROR BETWEEN THE FILTER OUTPUT AND THE DESIRED SIGNAL SHOULDBE AS SMALL AS POSSIBLE THAT IS WE WANT TO MAKE ET  DT  YTSMALL FOR EACH TWHEN DOING EM LEASTSQUARES FILTERING INDEXLEASTSQUARES FILTERING THE CRITERION OF MINIMAL ERROR IS THATTHE SUM OF THE SQUARED ERRORS IS AS SMALL AS POSSIBLEBEGINEQUATION MIN SUMII1I2 EI2LABELEQLSNORMEENDEQUATIONWHERE I1 IS THE STARTING INDEX AND I2 THE ENDING INDEX OVERWHICH WE DESIRE TO MINIMIZE  THE SQUARED NORM IN REFEQLSNORMEIS INDUCED FROM THE INNER PRODUCT DEFINED BYBEGINEQUATION LA XBF YBF RA  SUMII1I2 XI YBARILABELEQLSIP01ENDEQUATIONLETTING YBF  BEGINBMATRIX YI1  YI11  VDOTS   YI2 ENDBMATRIX QQUAD HBF  BEGINBMATRIX H0  H1 VDOTS  HM1 ENDBMATRIX QQUAD XBF  BEGINBMATRIX XI1 XI11  VDOTS  XI2 ENDBMATRIXTHE INNER PRODUCT REFEQLSIP01 CAN BE WRITTEN AS LA XBFYBF RA  YBFH XBFAND THE FILTERED OUTPUTS CAN BE WRITTEN AS YBF  A HBFWHERE A IS A MATRIX OF THE INPUT DATA FT  THE MATRIX A TAKESVARIOUS FORMS DEPENDING ON THE ASSUMPTIONS MADE ON THE DATA ASDESCRIBED IN THE FOLLOWING  LET DBF  BEGINBMATRIX DI1  DI11  VDOTS  DI2ENDBMATRIXBE A VECTOR OF DESIRED OUTPUTS  THEN WE WANT DBF APPROX YBFWE CAN REPRESENT OUR APPROXIMATION PROBLEM AS DBF  A HBF  EBFWHERE EBF IS THE DIFFERENCE BETWEEN THE OUTPUT YBF AND THEDESIRED OUTPUT DBF  WE DESIRE TO FIND THE FILTER COEFFICIENTSHBF TO MINIMIZE  EBF   BY COMPARISON WITH REFEQMATLSOBSERVE THAT THE SOLUTION ISBEGINEQUATION HBF  AH A1 AH YBFLABELEQLSFILTSOLENDEQUATIONWE NOW EXAMINE THE FORM OF THE A MATRIX UNDER VARIOUS ASSUMPTIONSABOUT THE INPUTS  ASSUME THAT WE HAVE AVAILABLE TO US FOR THEPURPOSE OF FINDING THE COEFFICIENTS THE DATA F1F2LDOTSFNWITH A TOTAL OF N DATA POINTSBEGINDESCRIPTIONITEMTHE COVARIANCE METHOD INDEXCOVARIANCE  METHODCOVARIANCE METHOD IN THIS METHOD WE USE ONLY DATA THAT  IS EXPLICITLY AVAILABLE NOT MAKING ANY ASSUMPTIONS ABOUT DATA  OUTSIDE THIS SEGMENT OF OBSERVED DATA  THE DATA MATRIX A IN THIS  CASE IS THE MATSIZENM1M MATRIX A  BEGINBMATRIX FM  FM1  FM2  CDOTS  F1 FM1  FM  FM1  CDOTS  F2 VDOTS FN  FN1  FN2  CDOTS  FNM1 ENDBMATRIXLET QBFI BE THE MATSIZEM1 DATA VECTOR CORRESPONDING TO ACONJUGATED ROW OF A AS IN REFEQXSTACKROW THENBEGINEQUATION QBFI  BEGINBMATRIX FBARI   FBARI1   CDOTS   FBARIM1ENDBMATRIXLABELEQGRAMMXENDEQUATIONWITH THE NOTATION THAT FI  0 WHERE I IS OUTSIDE THE RANGE 1TO N AND WE CAN REPRESENT THE DATA MATRIX AS A  BEGINBMATRIX QBFMH  QBFM1H  VDOTS  QBFNHENDBMATRIXTHE GRAMMIAN CAN BE WRITTEN ASBEGINEQUATIONR  AH A  SUMIMN QBFI QBFHILABELEQGRAMM3ENDEQUATIONTHE GRAMMIAN R IS A HERMITIAN MATRIXITEMTHE AUTOCORRELATION METHOD INDEXAUTOCORRELATION    METHODAUTOCORRELATION METHOD IN THIS CASE WE ASSUME THAT  DATA PRIOR TO F1 AND AFTER FN ARE ALL ZERO  THE OUTPUT IS  TAKEN FROM I1  1 UP THROUGH I2  NM1 PRODUCING THE  MATSIZENM1M DATA MATRIX A  BEGINBMATRIX F1  0  0  CDOTS  0 F2  F1  0  CDOTS  0 F3  F2  F1  CDOTS  0 VDOTS FM  FM1  FM2  CDOTS  F1 FM1  FM  FM2  CDOTS  F2 VDOTS FN  FN1  FN2  CDOTS  FNM1 0  FN  FN1  CDOTS  FNM2 VDOTS 000 CDOTS  FN ENDBMATRIXTHE TERMS COVARIANCE METHOD AND AUTOCORRELATION METHOD DO NOTPRODUCE RESPECTIVELY A COVARIANCE MATRIX AND AN AUTOCORRELATIONMATRIX IN THE USUAL SENSE  RATHER THESE ARE THE TERMS FOR THESEMETHODS COMMONLY EMPLOYED IN THE SPEECH PROCESSING LITERATURE SEEEG CITEMAKHOUL1975  USING THE NOTATION OF REFEQGRAMMXWE CAN WRITE THE DATA MATRIX AS A  BEGINBMATRIX QBFH1 QBFT2  VDOTS  QBFTNM1 ENDBMATRIXIN A MANNER SIMILAR TO REFEQGRAMM3 WE CAN WRITE R  AH A  SUMI1NM1 QBFI QBFHITHIS IS A TOEPLITZ MATRIX INDEXTOEPLITZ MATRIXITEMPREWINDOWING METHOD INDEXPREWINDOWING METHOD IN THIS METHOD WE ASSUME THAT FT0 FOR  T 1 AND USE DATA UP TO FN SO THAT I1  1 AND I2  N  THEN THE DATA MATRIX IS THE MATSIZENM MATRIXBEGINEQUATION A  BEGINBMATRIXF1  0  0  CDOTS  0 F2  F1  0  CDOTS  0 F3  F2  F1  CDOTS  0 VDOTS FM  FM1  FM2  CDOTS  F1 FM1  FM  FM2  CDOTS  F2 VDOTS FN  FN1  FN2  CDOTS  FNM1 ENDBMATRIX  BEGINBMATRIX QBFH1  QBFH2  VDOTS   QBFHN ENDBMATRIXLABELEQPREWINDOW1ENDEQUATIONAND R  SUMI1N QBFIQBFHIITEMPOSTWINDOWING METHOD WE BEGIN WITH I1M AND ASSUME THAT  DATA AFTER N ARE EQUAL TO ZERO  THEN A IS THE MATSIZENM MATRIX A  BEGINBMATRIXFM  FM1  FM2  CDOTS  F1 FM1  FM  FM2  CDOTS  F2 VDOTS FN  FN1  FN2  CDOTS  FNM1 0  FN  FN1  CDOTS  FNM2 VDOTS 000 CDOTS  FN ENDBMATRIXAND R  SUMIMMN QBFIQBFHIENDDESCRIPTIONBEGINEXAMPLE  SUPPOSE WE OBSERVE THE DATA SEQUENCE  F1 LDOTS F5   12345WHICH WE WANT TO FILTER WITH A FILTER OF LENGTH M3  THE DATAMATRICES CORRESPONDING TO EACH INTERPRETATION LABELED RESPECTIVELYATEXT COV ATEXT AC ATEXT PRE AND ATEXT  POST WITH THEIR CORRESPONDING GRAMMIANS ARE SHOWN HEREBEGINALIGNED ATEXT COV  BEGINBMATRIXHFILL 3  HFILL 2  HFILL 1 HFILL 4  HFILL 3  HFILL 2 HFILL 5  HFILL 4  HFILL 3 ENDBMATRIXQQUADATEXT AC  BEGINBMATRIXHFILL 1  HFILL 0  HFILL 0 HFILL 2  HFILL 1  HFILL 0 HFILL 3  HFILL 2  HFILL 1 HFILL 4  HFILL 3  HFILL 2 HFILL 5  HFILL 4  HFILL 3 HFILL 0  HFILL 5  HFILL 4 HFILL 0  HFILL 0  HFILL 5 ENDBMATRIX  EQNSKIPATEXTPRE  BEGINBMATRIXHFILL 1  HFILL 0  HFILL 0 HFILL 2  HFILL 1  HFILL 0 HFILL 3  HFILL 2  HFILL 1 HFILL 4  HFILL 3  HFILL 2 HFILL 5  HFILL 4  HFILL 3 ENDBMATRIX QQUADATEXTPOST  BEGINBMATRIXHFILL 3  HFILL 2  HFILL 1 HFILL 4  HFILL 3  HFILL 2 HFILL 5  HFILL 4  HFILL 3 HFILL 0  HFILL 5  HFILL 4 HFILL 0  HFILL 0  HFILL 5 ENDBMATRIXENDALIGNEDBEGINALIGNEDRTEXT COV  BEGINBMATRIXHFILL 50  HFILL 38  HFILL 26 HFILL 38  HFILL 29  HFILL 20 HFILL 26  HFILL 20  HFILL 14 ENDBMATRIX QQUADRTEXT AC  BEGINBMATRIXHFILL 55  HFILL 40  HFILL 26 HFILL 40  HFILL 55  HFILL 40 HFILL 26  HFILL 40  HFILL 55 ENDBMATRIX  RTEXT PRE  BEGINBMATRIXHFILL 55  HFILL 40  HFILL 26 HFILL 40  HFILL 30  HFILL 20 HFILL 26  HFILL 20  HFILL 14 ENDBMATRIX QQUADRTEXT POST  BEGINBMATRIXHFILL 50  HFILL 38  HFILL 26 HFILL 38  HFILL 54  HFILL 40 HFILL 26  HFILL 40  HFILL 55 ENDBMATRIXENDALIGNEDENDEXAMPLEOBSERVE THAT WHILE ALL OF THE DATA MATRICES ARE TOEPLITZ CONSTANTALONG THE DIAGONALS THE ONLY GRAMMIAN WHICH IS TOEPLITZ IS THEONE WHICH ARISES FROM THE AUTOCOVARIANCE FORM OF THE DATA MATRIXSC MATLAB CODE TO COMPUTE THE LEASTSQUARES FILTER COEFFICIENTS ISGIVEN IN ALGORITHM REFALGLSFILT  BEGINNEWPROGENVLEASTSQUARES FILTER COMPUTATIONLSFILTM   LSFILTLEASTSQUARES FILTER ENDNEWPROGENV BEGINEXAMPLE   FOR THE INPUT DATA OF THE PREVIOUS EXAMPLE THE FOLLOWING DESIRED   DATA ARE KNOWN DBF  2 5 1117 2317 15TWE WANT TO FIND A FILTER OF LENGTH M3 THAT PRODUCES THIS DATAUSING THE FOUR DIFFERENT DATA SETS IN THE EXAMPLE WITH SELECTIONS OFDBF CORRESPONDING TO THE DATA USED WE OBTAIN FROM THE SC MATLABCOMMANDS SMALLSKIPBEGINALLTTINDENT HCV  LSFILTFD3531INDENT HAC  LSFILTFD32INDENT HPRE  LSFILTFD1533INDENT HPOST  LSTILFFD3734SMALLSKIPENDALLTTTHE FILTER COEFFICIENTS HBFTEXTCOV  BEGINBMATRIX15225 ENDBMATRIXT QQUADHBFTEXTAUTO  BEGINBMATRIX 213 ENDBMATRIXT HBFTEXTPRE  BEGINBMATRIX 213 ENDBMATRIXT QQUADHBFTEXTPOST  BEGINBMATRIX 213 ENDBMATRIXTRESPECTIVELYENDEXAMPLEBEGINEXAMPLE  AN APPLICATION OF LEASTSQUARES FILTERING IS ILLUSTRATED IN FIGURE  REFFIGLSEQ IN A CHANNEL EQUALIZER  INDEXEQUALIZERLEASTSQUARES APPLICATION  A SEQUENCE OF BITS  BT IS PASSED THROUGH A DISCRETETIME CHANNEL WITH UNKNOWN  IMPULSE RESPONSE THE OUTPUT OF WHICH IS CORRUPTED BY NOISE  TO  COUNTERACT THE EFFECT OF THE CHANNEL THE SIGNAL IS PASSED THROUGH  AN EQUALIZER WHICH IN THIS CASE IS AN FIR FILTER WHOSE COEFFICIENTS  HAVE BEEN DETERMINED USING A LEASTSQUARES CRITERION  IN ORDER TO  DETERMINE WHAT THE COEFFICIENTS ARE SOME SET OF KNOWN DATA  A  EM TRAINING SEQUENCE  IS USED AT THE BEGINNING OF THE  TRANSMISSION  THIS SEQUENCE IS DELAYED AND USED AS THE DESIRED  SIGNAL DT  USING THIS TRAINING SEQUENCE THE FILTER  COEFFICIENTS HK ARE COMPUTED BY USING REFEQLSFILTSOL  AFTER WHICH THE COEFFICIENTS ARE LOADED INTO THE EQUALIZER FILTER    THIS EXAMPLE IS MORE A DEMONSTRATION OF A CONCEPT THAN A PRACTICAL  REALITY  WHILE EQUALIZERS ARE COMMON ON MODERN MODEM TECHNOLOGY  THEY ARE MORE COMMONLY IMPLEMENTED USING ADAPTIVE FILTERS  ADAPTIVE  EQUALIZERS ARE EXAMINED IN SECTION REFSECRLSEX RLS ADAPTIVE  EQUALIZER AND SECTION REFSECLMS LMS ADAPTIVE EQUALIZERENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIREQUALIZER1    CAPTIONLEASTSQUARES EQUALIZER EXAMPLE    LABELFIGLSEQ  ENDCENTERENDFIGUREBEGINEXERCISES  ITEM VERIFY REFEQGRAMM3ITEM FOR THE DATA SEQUENCE 11235813  BEGINENUMERATE  ITEM WRITE DOWN THE DATA MATRIX A AND THE GRAMMIAN AH A USING    I THE COVARIANCE AND II THE AUTOCORRELATION METHODS  ITEM WE DESIRE TO USE THIS SEQUENCE TO TRAIN A SIMPLE LINEAR    PREDICTOR  THE DESIRED SIGNAL DT IS THE VALUE OF XT    AND THE DATA USED ARE THE TWO PRIOR SAMPLES  THAT IS  XT  A1 XT1  A2 XT2  ET WHERE ET THE PREDICTION ERRORDETERMINE THE LEASTSQUARES COEFFICIENTS FOR THE PREDICTOR USING THECOVARIANCE AND AUTOCORRELATION METHODSITEM DETERMINE THE MINIMUM LEASTSQUARES ERROR FOR BOTH METHODS  ENDENUMERATEITEM BF COMPUTER EXERCISE GENERATE A SEQUENCE OF N RANDOM PM  1 BITS AND PASS THEM THROUGH A CHANNEL WITH TRANSFER FUNCTION HZ  FRAC119Z1THEN ADD NOISE WITH VARIANCE SIGMAN2  01  DETERMINE ALEASTSQUARES FILTER FOR THIS CHANNEL FOR VARIOUS VALUES OF THEDELAYENDEXERCISESSUBSECTIONLEASTSQUARES PREDICTION AND AR SPECTRUM ESTIMATIONLABELSECLSSPECTESTINDEXSPECTRUM ESTIMATION CONSIDER NOW THE ESTIMATION PROBLEM INWHICH WE DESIRE TO PREDICT XT USING A LINEAR PREDICTORINDEXLINEAR PREDICTOR BASED UPON XT1 XT2 LDOTS XTMWE THEN HAVEBEGINEQUATION XT  SUMI1M AI XTI  FTLABELEQLSARSEENDEQUATIONUSING AI  HI AS THE COEFFICIENTS WHERE FT IS NOW USED TODENOTE THE FORWARD PREDICTOR INDEXLINEAR PREDICTORFORWARD ERRORTHE PREDICTOR OF REFEQLSARSE IS CALLED A EM FORWARD  PREDICTOR  THIS IS ESSENTIALLY THE PROBLEM SOLVED IN THE LASTSECTION IN WHICH THE DESIRED SIGNAL IS THE SAMPLE DT  XT ANDTHE DATA USED ARE THE EM PREVIOUS DATA SAMPLES  WE CAN MODEL THESIGNAL XT AS BEING THE OUTPUT OF A SIGNAL WITH INPUT FT WHERETHE SYSTEM FUNCTION IS HZ  FRACXZFZ  FRAC11  SUMI1N AI ZI FRAC1AZIF FT IS A RANDOM SIGNAL WITH POWER SPECTRAL DENSITY PSD SFZTHEN THE PSD OF XT ISBEGINEQUATION SXZ  FRAC11SUMI1M AI ZI1 SUMI1M  ABARI ZIPFZ   FRAC1AZABAR1ZLABELEQFORWARDPSDENDEQUATIONIF FT IS ASSUMED TO BE A WHITENOISE SEQUENCE WITHVARIANCE SIGMAF2 THEN THE RANDOM PROCESS XT HAS THE PSD SXZ  FRACSIGMAF2AZABAR1ZEVALUATING THIS ON THE UNIT CIRCLE ZEJOMEGA WE OBTAINBEGINEQUATION SXOMEGA DEFEQ BIGL SXZBIGRZEJOMEGA    FRACSIGMAF21    SUMI1M AI EJOMEGA     I2  FRACSIGMAF2AOMEGA2LABELEQPSD2ENDEQUATIONTHUS BY FINDING THE COEFFICIENTS OF THE LINEAR PREDICTOR WE CAN DETERMINEAN ESTIMATE OF THE SPECTRUM UNDER THE ASSUMPTION THAT THE SIGNAL ISPRODUCED BY THE AR MODEL REFEQLSARSE  WE CAN OBTAIN MORE DATA TO PUT IN OUR DATA MATRIX AND USUALLYDECREASE THE VARIANCE OF THE ESTIMATE BY USING A EM BACKWARD  PREDICTOR IN ADDITION TO A FORWARD PREDICTOR  INDEXLINEAR  PREDICTORBACKWARD  IN THE BACKWARD PREDICTOR THE M DATA POINTS XTXT1 LDOTS XTM1 ARE USED TO ESTIMATE XTM XTM  SUMI1M AI XTMI  BTWHERE BT IS THE BACKWARD PREDICTION ERROR  AS BEFORE IF WE VIEWXTM AS THE OUTPUT OF A SYSTEM DRIVEN BY AN INPUT BT WEOBTAIN A SYSTEM FUNCTION HBZ  FRACXZBZ  FRAC1ZM1  SUMI1M  ABARI ZI  FRAC1ZMABAR1ZIF BT IS A WHITENOISE SEQUENCE WITH VARIANCE SIGMAB2 SIGMAF2 THEN THE PSD OF THE SIGNAL XTM ISBEGINEQUATIONSXZ  SIGMAB2 FRAC1ABAR1ZAZLABELEQBACKWARDPSDENDEQUATIONTHE SAME AS IN REFEQFORWARDPSD  SINCE BOTH THE FORWARDPREDICTOR AND THE BACKWARD PREDICTOR USE THE SAME PREDICTORCOEFFICIENTS JUST CONJUGATED AND IN A DIFFERENT ORDER WE CAN USETHE BACKWARD PREDICTOR INFORMATION TO IMPROVE OUR ESTIMATE OF THECOEFFICIENTS  IF WE HAVE MEASURED DATA X1 X2 LDOTS XN WEWRITE OUR PREDICTION EQUATIONS AS FOLLOWS USING THE COVARIANCE METHODEMPLOYING ONLY MEASURED DATA BEGINBMATRIX XM  XM1  CDOTS  X1 XM1  XM  CDOTS  X2 VDOTS XN1  XN2  CDOTS  XNM XBF2  XBF3  CDOTS  XBFM1 XBAR3  XBAR4  CDOTS  XBARM2 VDOTS XBARNM1  XBARNM2  CDOTS  XBARN ENDBMATRIXBEGINBMATRIX A1  A2  VDOTS  AM ENDBMATRIXBEGINBMATRIX XM1   XM2   VDOTS XN  XBAR1  XBAR2  VDOTS XBARNMENDBMATRIX BEGINBMATRIX FM1  FM2  VDOTS  FN  BBARNM1  BBARNM2  VDOTS  BBARNM ENDBMATRIXLET US WRITE THIS AS XBF  A HBF  EBFWHERE XBF AND EBF NOW ARE MATSIZE2NM1 AND A ISMATSIZE2NMN  IN THE DATA MATRIX THE FIRST NM ROWSCORRESPOND TO THE FORWARD PREDICTOR AND THE SECOND NM ROWSCORRESPOND TO THE BACKWARD PREDICTOR  OUR OPTIMIZATION CRITERION ISTO MINIMIZE SUMIN1N FI2  BI2AS BEFORE A LEASTSQUARES SOLUTION IS STRAIGHTFORWARD  THISTECHNIQUE OF SPECTRUM ESTIMATION IS KNOWN AS THE FORWARDBACKWARDLINEAR PREDICTION FBLP INDEXLINEAR PREDICTORFORWARDBACKWARDTECHNIQUE OR THE MODIFIED COVARIANCE TECHNIQUE  INDEXMODIFIED  COVARIANCE METHOD AN ESTIMATE OF THE VARIANCE IS SIGMAHATF2  SIGMAHATB2   EBFMIN22A SC MATLAB FUNCTION THAT COMPUTES THE AR PARAMETERS USING THEMODIFIED COVARIANCE TECHNIQUE IS SHOWN IN ALGORITHM REFALGFBLPBEGINNEWPROGENVFORWARDBACKWARD LINEAR PREDICTOR    ESTIMATEFBLPMFBLPFORWARDBACKWARD LINEAR PREDICTORENDNEWPROGENVSECTIONMINIMUM MEANSQUARE ESTIMATIONLABELSECMMSINDEXMINIMUM MEANSQUAREIN THE LEASTSQUARES ESTIMATION OF THE PRECEDING SECTIONS WE HAVE NOTEMPLOYED NOR ASSUMED THE EXISTENCE OF ANY PROBABILISTIC MODEL  THEOPTIMIZATION CRITERION HAS BEEN TO MINIMIZE THE SUM OF SQUARED ERRORIN THIS SECTION WE CHANGE OUR VIEWPOINT SOMEWHAT BY INTRODUCING APROBABILISTIC MODEL FOR THE DATALET P1 P2 LDOTS PM BE RANDOM VARIABLES  WE DESIRE TO FINDCOEFFICIENTS CI TO ESTIMATE THE RANDOM VARIABLE X USING X  C1 P1  C2 P2  CDOTS  CM PM  EIN SUCH A WAY THAT THE NORM OF THE SQUARED ERROR IS MINIMIZEDUSING THE INNER PRODUCTBEGINEQUATIONLA X Y RA  EX YBARLABELEQMMSE0ENDEQUATIONTHE MINIMUM MEANSQUARE ESTIMATE OF CBF IS GIVEN BY  RCBF  PBFWHEREBEGINEQUATION R  BEGINBMATRIXEP1PBAR1  EP2PBAR1  CDOTS  EPM PBAR1 EP1PBAR2  EP2PBAR2  CDOTS  EPM PBAR2 VDOTS EP1PBARM  EP2PBARM  CDOTS  EPM PBARM  ENDBMATRIXQUAD TEXTANDQUADPBF  BEGINBMATRIXEXPBAR1  EXPBAR2  VDOTS  EXPBARM ENDBMATRIXLABELEQMMSERDENDEQUATIONTHE MINIMUM MEANSQUARED ERROR IN THIS CASE IS GIVEN USINGREFEQEMIN3 ASBEGINEQUATIONBEGINSPLITEMIN  SIGMAX2  PBFH R1 PBF  SIGMAX2  PBFH CBFENDSPLITLABELEQEMINMMSENDEQUATIONBEGINEXAMPLE LABELEXMMMSEPRED  SUPPOSE THAT  ZBF  X1X2X3TIS A REAL GAUSSIAN RANDOM VECTOR WITH MEAN ZERO AND COVARIANCE RZZ  COVZBF  EZBFZBFT  BEGINBMATRIX 1  2  1 2  2  3 1  3  4 ENDBMATRIXGIVEN MEASUREMENTS OF X1 AND X2 WE WISH TO ESTIMATE X3 USINGA LINEAR ESTIMATOR XHAT3  C1 X1  C2 X2THE NECESSARY CORRELATION VALUES IN REFEQMMSERD CAN BE OBTAINEDFROM THE COVARIANCE RZZ R  BEGINBMATRIX EX1X1  EX1X2  EX2X1  EX2  X2 ENDBMATRIX  BEGINBMATRIX 1  2  2  2 ENDBMATRIX QQUADTEXTAND QQUAD PBF  BEGINBMATRIXEX3 X1  EX3 X2ENDBMATRIX BEGINBMATRIX 1  3ENDBMATRIXFROM WHICH THE OPTIMAL COEFFICIENTS ARE CBF  BEGINBMATRIX 00714  01429 ENDBMATRIXTHE MINIMUM MEANSQUARED ERROR IS  EMIN  4  PBFT R1PBF  395ENDEXAMPLESECTIONMINIMUM MEANSQUARED ERROR MMSE FILTERINGLABELSECMMSSEFILTINDEXMINIMUM MEANSQUAREFILTERING A MINIMUM MEANSQUARE MMSFILTER INDEXWIENER FILTER IS MATHEMATICALLY SIMILAR TO TO ALEASTSQUARES FILTER EXCEPT THAT THE EXPECTATION OPERATOR IS USED ASTHE INNER PRODUCT  GIVEN A SEQUENCE OF DATA  FT  WE DESIRETO DESIGN A FILTER IN SUCH A WAY THAT WE GET AS CLOSE AS POSSIBLE TOSOME DESIRED SEQUENCE DT  IN THE INTEREST OF GENERALITY WEASSUME THE POSSIBILITY OF AN IIR FILTERBEGINEQUATIONYT   SUML0INFTY HL FTLLABELEQMMSE1ENDEQUATIONIN ADOPTING A STATISTICAL MODEL WE ASSUME THAT THE SIGNALS INVOLVEDARE WIDESENSE STATIONARY SO THAT FOR EXAMPLE EXT  EXTLQQUAD TEXTFOR ALL  LAND EXTXBARTLDEPENDS ONLY UPON THE TIME DIFFERENCE L AND NOT UPON THE SAMPLEINSTANT TUSINGBEGINEQUATION ET  DT  YTLABELEQMMSE2ENDEQUATIONAS THE ESTIMATOR ERROR BY THE ORTHOGONALITY PRINCIPLE THE SQUAREDNORM OF ERROR WHICH IN THIS CASE IS TERMED THE EM MEANSQUARED  ERROR  ET2  EET EBARTIS MINIMIZED WHEN THE ERROR IS ORTHOGONAL TO THE DATA  THAT IS THEOPTIMAL ESTIMATOR SATISFIESLA DT  SUML0INFTY HLFTLFTI RA  0FOR I012LDOTS ORBEGINEQUATION  LABELEQMMSE3  LA DTFTI RA  SUML0INFTY HL LA  FTLFTIRA  0 ENDEQUATIONUSING THE INNER PRODUCT REFEQMMSE0 WE OBTAINBEGINEQUATION  LABELEQMMSE4  SUML0INFTY HL EFTLFBARTI  EFBARTIDTENDEQUATIONEQUATION REFEQMMSE4 IS AN INFINITE SET OF NORMAL EQUATIONSFOR THIS CASE IN WHICH THE INNER PRODUCT ISDEFINED USING THE EXPECTATION THE NORMAL EQUATIONS ARE REFERRED TO ASTHE EM WIENERHOPF EQUATIONS   INDEXWIENERHOPF EQUATIONSWE CAN PLACE THE NORMAL EQUATIONS INTO A MORE STANDARD FORM BYEXPRESSING THE GRAMMIAN IN THE FORM OF AN AUTOCORRELATION MATRIXDEFINE RIL  EFTLFBARTI  LA FTLFTIRAAND PI  EFBARTIDT  LA DTFTIRAAND OBSERVE THAT RK  RBARK  THEN REFEQMMSE4 CAN BEWRITTEN ASBEGINEQUATIONSUML0INFTY HL RIL  PIQQUAD I01LDOTSLABELEQMMSE4AENDEQUATIONSOLUTION OF THIS PROBLEM FOR AN IIR FILTER IS REEXAMINED IN SECTIONREFSECFREQFILT  FOR NOW WE FOCUS ON THE SOLUTION WHEN H IS AN FIR FILTER WITHM COEFFICIENTS  THEN THE FILTER OUTPUT CAN BE WRITTEN AS YT  FBFTH HBFWHEREBEGINEQUATION FBFT  BEGINBMATRIXFBART FBART1 LDOTS FBARTM1ENDBMATRIXTLABELEQDEFFENDEQUATIONNOTE THE CONJUGATES IN THIS DEFINITION AND HBF  BEGINBMATRIX H0  H1 LDOTS  HM1ENDBMATRIXTUNDER THE ASSUMPTION OF AN FIR FILTER REFEQMMSE4A CAN BE WRITTENASBEGINEQUATIONSUML0M1 HL RIL  PIQQUAD I01LDOTSLABELEQMMSE5ENDEQUATIONWHICH WE CAN EXPRESS IN MATRIX FORM WITH RIL  RILBEGINEQUATION  LABELEQMMSE6  R HBF  PBFENDEQUATIONWHEREBEGINEQUATIONBEGINSPLITR  BEGINBMATRIX R0  RBAR1  RBAR2 CDOTS  RBARM1 R1  R0  RBAR1  CDOTS  RBARM2 R2  R1  R0  CDOTS  RBARM3 VDOTS RM1  RM2  RM3  CDOTS  R0 ENDBMATRIX  EFBFTFBFHTENDSPLITLABELEQREFFENDEQUATIONANDBEGINEQUATIONLABELEQPEFDBEGINSPLITPBF  BEGINBMATRIX P0   P1  P2  CDOTS PM1ENDBMATRIX  EFBFTDTENDSPLITENDEQUATIONTHE OPTIMAL WEIGHTS FROM REFEQMMSE6 ARE HBF  R1 PBFTHE MATRIX R IS THE GRAMMIAN MATRIX AND HAS THE SPECIAL FORM OF ATOEPLITZ MATRIX INDEXTOEPLITZ MATRIX BEING CONSTANT ON THEDIAGONALS  BECAUSE OF THIS SPECIAL FORM FAST ALGORITHMS EXIST FORINVERTING THE MATRIX AND SOLVING FOR THE OPTIMUM FILTER COEFFICIENTSTOEPLITZ MATRICES ARE DISCUSSED FURTHER IN SECTION REFSECTOEPLITZWE HAVE ALREADY SEEN ONE EXAMPLE OF THE SOLUTION OF TOEPLITZEQUATIONS WITH A SPECIAL RIGHTHAND SIDE IN MASSEYS ALGORITHM INSECTION REFSECLFSR1THE MINIMUM MEANSQUARED ERROR CAN BE DETERMINED USING REFEQEMINMMSTO BE   E2MIN  EE2MIN   D2   Y2USING THE NOTATION E2  SIGMAE2 AND D2  SIGMAD2AND NOTING THATBEGINALIGNED YT2  EYT YBART  EHBFHT XBFT  XBFHT HBF  HBFH R HBF  PBFH HBFENDALIGNEDWE OBTAIN BEGINEQUATION SIGMAE2MIN  SIGMAD2  PBFH HBFLABELEQEMINWIENENDEQUATIONBEGINEXAMPLE LABELEXMEQ1INDEXEQUALIZERMINIMUM MEANSQUARE  IN THIS EXAMPLE WE EXPLORE A SIMPLE EQUALIZER  SUPPOSE WE HAVE A  CHANNEL WITH TRANSFER FUNCTION  HCZ  FRAC116 Z1PASSING INTO THE CHANNEL IS A DESIRED SIGNAL DT  THE OUTPUT OFTHE CHANNEL IS UT SO THAT WE HAVEBEGINEQUATION  LABELEQWFEX1  UT  06 UT1  DTENDEQUATION  HOWEVER WE OBSERVE ONLY A NOISECORRUPTEDVERSION OF THE CHANNEL OUTPUT FT  UT  NTWHERE NT IS A ZEROMEAN WHITENOISE SEQUENCE WITH VARIANCESIGMAN2  016 WHICH IS UNCORRELATED WITH NUT  SUPPOSEFURTHERMORE THAT WE HAVE A STATISTICAL MODEL FOR THE DESIRED SIGNALIN WHICH WE KNOW THAT DT IS A FIRSTORDER AR SIGNAL GENERATED BY DT  5 DT1  NUTWHERE NUT IS A ZEROMEAN WHITENOISE SIGNAL WITH VARIANCESIGMANU2  01  BASED ON THIS INFORMATION WE DESIRE TO FIND ANOPTIMAL WIENER FILTER TO ESTIMATE DT USING THE OBSERVED SEQUENCEFT  THE DIAGRAM IS SHOWN IN FIGURE REFFIGWFEX1BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIREQUALIZER2    CAPTIONAN EQUALIZER PROBLEM    LABELFIGWFEX1  ENDCENTERENDFIGURETHE CASCADE OF THE AR PROCESS AND THE CHANNEL GIVES THE COMBINEDTRANSFER FUNCTION FROM NUT TO UT AS HZ  FRAC115Z116Z1  FRAC11  1Z1   3 Z2  FRAC11  A1 Z1  A2 Z2SO THAT UT 1 UT1 3 UT2  NUTIN THIS EXAMPLE SINCE THE CHANNEL OUTPUT IS AN AR2 PROCESS THEEQUALIZER USED IS A TWOTAP FIR FILTER WE NEED THE MATRIX R CONTAINING AUTOCORRELATIONS OF THE SIGNALFT AND THE CROSSCORRELATION VECTOR PBF  SINCE FT  UT NT AND SINCE NUT AND NT ARE UNCORRELATED WE HAVE R  RFF  RUU  RNNWHERE RUU IS THE AUTOCORRELATION MATRIX FOR THE SIGNAL UT ANDRNN IS THE AUTOCORRELATION MATRIX FOR THE SIGNAL NT  SINCENT IS A WHITENOISE SEQUENCE RNN  SIGMAN2 I WHERE IIS THE MATSIZE22 IDENTITY MATRIX  TO FIND RUU  BEGINBMATRIX RU0  RU1  RU1  RU0 ENDBMATRIXWE USE THE RESULTS FROM  SECTION REFSECARPROCESS  SPECIFICALLYFROM REFEQYW7 AND REFEQYW6 WE FIND BEGINALIGNEDSIGMAU2  RU0 LEFTFRAC1A21A2RIGHTFRACSIGMANU21A22    A12  01122 RU1  FRACA11A2SIGMAU2  00160ENDALIGNEDTHUS R  BEGINBMATRIX 16  0  0  16 ENDBMATRIX BEGINBMATRIX1122  0160  0160  1122  ENDBMATRIX  BEGINBMATRIX 2722   0160  0160  HFILL 2722 ENDBMATRIXFOR THE CROSSCORRELATION VECTORBEGINALIGNEDPBF  EBEGINBMATRIX FBARTDT  FBART1DTENDBMATRIX  EBEGINBMATRIXUBARTNBARTDT UBART1NBART1DT ENDBMATRIX  EBEGINBMATRIX UBARTDT  UBART1DT ENDBMATRIXENDALIGNEDSINCE DT IS UNCORRELATED WITH NTN  MULTIPLYINGREFEQWFEX1 THROUGH BY UBARTK AND TAKING EXPECTATIONS WEOBTAIN PK  EUBARTKDT  RUK  06 RUK1FROM WHICH WE CAN DETERMINE PBF BEGINBMATRIX HFILL 01206  HFILL 00513ENDBMATRIXTHE OPTIMAL FILTER COEFFICIENTS ARE HBF  R1 PBF  BEGINBMATRIXHFILL 03893 HFILL 02113 ENDBMATRIXTO COMPUTE THE MINIMUM MEANSQUARED ERROR FROM REFEQEMINWIEN WENEED SIGMAD2  THIS IS FOUND USING REFEQFIRSTARVAR AS SIGMAD2  FRACSIGMANU2152THEN SIGMAE2  00826THE ERROR SURFACE IS OBTAINED BY PLOTTING SEE REFEQGRADMIN2 JHBF  SIGMAD2  2 PBFTBEGINBMATRIXH0 H1ENDBMATRIX  H0 H1R BEGINBMATRIX H0  H1ENDBMATRIXAS A FUNCTION OF H0H1  FIGURE REFFIGWFTESTCONTSHOWS A CONTOUR PLOT OF THE ERROR SURFACEBEGINFIGURET USE WFTESTCONT AFTER RUNNING WFTESTCENTERINGEPSFIGFILEPICTUREDIRWFTESTCONTEPS  CAPTIONCONTOUR PLOT OF AN ERROR SURFACE  LABELFIGWFTESTCONTENDFIGURE WFTESTM WFTESTCONTMALGORITHM REFALGWIENFILT1 IS SC MATLAB CODE DEMONSTRATING THESECOMPUTATIONSBEGINNEWPROGENVTWOTAP CHANNEL    EQUALIZERWFTESTMWIENFILT1TWOTAP CHANNEL EQUALIZERENDNEWPROGENVENDEXAMPLEANOTHER EXAMPLE OF MMSE FILTER DESIGN IS GIVEN IN CONJUNCTION WITH THERLS FILTER IN REFSECRLSEXBEGINEXERCISESITEM IMPLEMENTATION OF EQUALIZER IN EXAMPLE REFEXMEQ1 GENERATE  THE SIGNAL FT IN EXAMPLE REFEXMEQ1 BY CREATING AN AR1  SIGNAL WHICH IS PASSED THROUGH HCZ THEN CORRUPTED BY ADDITIVE  NOISE  THEN PASS THIS SIGNAL THROUGH THE EQUALIZER DESIGNED IN THAT  EXAMPLE  COMPARE THE EQUALIZED SIGNAL WITH THE SIGNAL DT  ITEM FOR A DATA SEQUENCE  XT THE CORRELATION MATRIX R IS R  BEGINBMATRIX 5  3  3  5 ENDBMATRIXAND THE CROSS CORRELATION VECTOR PBF WITH A DESIRED SIGNAL IS PBF  BEGINBMATRIX 2  5 ENDBMATRIXBEGINENUMERATEITEM DETERMINE THE OPTIMAL WEIGHT VECTORITEM DETERMINE THE MINIMUM MEANSQUARED ERRORENDENUMERATEITEM FOR A ZEROMEAN RANDOM VECTOR XBF  X1X2X3 WITH  COVARIANCE COVXBF  EXBFXBFT  BEGINBMATRIX 1  7  5  7  4  2 5  2  3 ENDBMATRIXBEGINENUMERATEITEM DETERMINE THE OPTIMAL COEFFICIENTS OF THE PREDICTOR OF X1 IN TERMSOF X2 AND X3 XHAT1  C1 X2  C2 X3ITEM DETERMINE THE MINIMUM MEANSQUARED ERRORITEM HOW IS THIS ESTIMATOR MODIFIED IF THE MEAN OF XBF IS E XBF   123TENDENUMERATEITEM CITEHAYKIN1996 A RADAR SIGNAL IS TRANSMITTED AS  ST  A0 EJOMEGA0 TTHE SAMPLED RECEIVED SIGNALS ARE REPRESENTED AS XT  A1 EJOMEGA1 T  NUTWHERE OMEGA1 IS THE RECEIVED SIGNAL FREQUENCY IN GENERALDIFFERENT FROM OMEGA0 BECAUSE OF DOPPLER SHIFT AND NUT IS AWHITE NOISE SIGNAL WITH VARIANCE SIGMAN2  LET XBFT  X0X1LDOTS XM1TBE A VECTOR OF RECEIVED SIGNAL SAMPLESBEGINENUMERATEITEM SHOW THAT R  EXBFT XBFHT  SIGMANU2 I  SIGMA1 SBFOMEGA1SBFHOMEGA1WHERE SBFOMEGA1  1EJOMEGA1EJ2OMEGA1LDOTSEM1  J OMEGA1TITEM THE TIME SERIES XT IS APPLIED TO AN FIR WIENER FILTER WITH  M COEFFICIENTS IN WHICH THE CROSSCORRELATION BETWEEN XT AND  THE DESIRED SIGNAL DT IS PRESET TO PBF  SBFOMEGA0DETERMINE AN EXPRESSION FOR THE TAPWEIGHT VECTOR OF THE WIENER FILTERENDENUMERATEITEM A CHANNEL WITH TRANSFER FUNCTION H2Z  FRAC112Z1AND OUTPUT UT IS DRIVEN BY AN AR1 SIGNAL DT GENERATED BY DT  4 DT1  NUTWHERE NUT IS A ZEROMEAN WHITE NOISE SIGNAL WITH SIGMANU2 2  THE CHANNEL OUTPUT IS CORRUPTED BY NOISE NT WITH VARIANCESIGMAN2  1 TO PRODUCE THE SIGNAL FT  UT  NTDESIGN A SECONDORDER WIENER EQUALIZER TO MINIMIZE THE AVERAGE SQUAREDERROR BETWEEN FT AND DTITEM LABELEXLINPRED BF LINEAR PREDICTION A COMMON APPLICATION  OF WIENER FILTERING IS IN THE CONTEXT OF LINEAR PREDICTION  LET  DT  XT BE THE DESIRED VALUE AND LET XHATT  SUMI1M WFI XTIBE THE PREDICTED VALUE OF XT USING AN MTH ORDER PREDICTOR BASEDUPON THE MEASUREMENTS XT1XT2LDOTSXTM ANDLET FMT  XT  XHATTBE THE EM FORWARD PREDICTION ERROR  INDEXFORWARD  PREDICTORSEELINEAR PREDICTOR INDEXLINEAR PREDICTORFORWARD THEN FMT  SUMI0M AFI XTIWHERE AF0  1 AND AFI  WFI I12LDOTSMASSUME THAT XT IS A ZEROMEAN RANDOM SEQUENCE  WE DESIRE TODETERMINE THE OPTIMAL SET OF COEFFICIENTS WFII12LDOTSMTO MINIMIZE EFMT2BEGINENUMERATEITEM USING THE ORTHOGONALITY PRINCIPLE WRITE DOWN THE NORMAL  EQUATIONS CORRESPONDING TO THIS MINIMIZATION PROBLEM  USE THE  NOTATION RJL  EXTL XBARTJ TO OBTAIN THE WIENERHOPF  EQUATION R WBFF  RBFWHERE R  EXBFT1XBFHT1 RBF  EXBFT1XT ANDXBFT1  XBART1XBART2LDOTSXBARTMTITEM DETERMINE AN EXPRESSION FOR THE MINIMUM MEANSQUARED ERROR PM   MIN EFMT2ITEM SHOW THAT THE EQUATIONS FOR THE OPTIMAL WEIGHTS AND THE MINIMUM  MEANSQUARED ERROR CAN BE COMBINED INTO EM AUGMENTED WIENERHOPF  INDEXWIENERHOPF   EQUATIONS AS BEGINBMATRIX R0  RBFH  RBF  R ENDBMATRIXBEGINBMATRIX 1  WBFF ENDBMATRIX  BEGINBMATRIXPM   ZEROBF ENDBMATRIXITEM IF XT HAPPENS TO BE GENERATED BY AN ARM PROCESS DRIVEN  BY WHITE NOISE SUCH THAT IT IS THE OUTPUT OF A SYSTEM WITH TRANSFER  FUNCTION HZ  FRAC11SUMK1M AK ZKSHOW THAT THE PREDICTION COEFFICIENTS ARE WFK  AK AND HENCETHE COEFFICIENTS OF THE PREDICTION ERROR FILTER FMT ARE AFI  AIHENCE CONCLUDE THAT IN THIS CASE THE FORWARD PREDICTION ERRORFMT IS A EM WHITENOISE SEQUENCE  THE PREDICTIONERRORFILTER CAN THUS BE VIEWED AS A EM WHITENING FILTER FOR THE SIGNALXT ITEM NOW LET XHATTM  SUMI1M WBI XTI1BE THE EM BACKWARD PREDICTOR OF XTM USING THE DATA XTM1XTM2 LDOTS XT AND LET BMT  XTM  XHATTMBE THE BACKWARD PREDICTION ERROR  INDEXBACKWARD  PREDICTORSEELINEAR PREDICTOR INDEXLINEAR PREDICTORBACKWARDSHOW THAT THE WIENERHOPF EQUATIONS FOR THE OPTIMAL BACKWARD PREDICTOR CAN BE WRITTEN ASBEGINEQUATION R WBF  OVERLINERBFBLABELEQBACKPRED1ENDEQUATIONWHERE RBFB IS THE BACKWARD ORDERING OF RBF  DEFINED ABOVEITEM FROM REFEQBACKPRED1 SHOW THAT RH OVERLINEWBFBB  RBFWHERE WBFBB IS THE BACKWARD ORDERING OF WBFB  HENCE CONCLUDETHAT OVERLINEWBFBB  WBFFTHAT IS THE OPTIMAL BACKWARD PREDICTION COEFFICIENTS ARE THE REVERSEDCONJUGATED OPTIMAL FORWARD PREDICTION COEFFICIENTSENDENUMERATEITEM LET XT  08 XT1   NUTWHERE NUT IS A WHITENOISE ZEROMEAN UNIT VARIANCE NOISEPROCESS  WE WANT TO DETERMINE AN OPTIMAL PREDICTORBEGINENUMERATEITEM IF THE ORDER OF THE PREDICTOR IS 2 DETERMINE THE OPTIMAL  PREDICTOR XHATTITEM IF THE ORDER OF THE PREDICTOR IS 1 DETERMINE THE OPTIMAL  PREDICTOR XHATTENDENUMERATEITEM BF RANDOM VECTORS THE MEANSQUARED METHODS TO THIS POINT HAVE  BEEN FOR RANDOM SCALARS  SUPPOSE WE HAVE THE RANDOM VECTOR  APPROXIMATION PROBLEM YBF  C1 PBF1  C2 PBF2  CDOTS  CM XBFM  EBFIN WHICH WE DESIRE TO FIND AN APPROXIMATION YBF IN SUCH A WAY THATTHE NORM OF EBF IS MINIMIZED  LET US DEFINE AN INNER PRODUCTBETWEEN RANDOM VECTORS AS LA XBFYBF RA  TRACE EXBF YBFHBEGINENUMERATEITEM BASED UPON THIS INNER PRODUCT AND ITS INDUCED NORM DETERMINE A  SET OF NORMAL EQUATIONS FOR FINDING C1C2LDOTS CMITEM AS AN EXERCISE IN COMPUTING GRADIENTS USE THE FORMULA FOR THE  GRADIENT OF THE TRACE SEE APPENDIX REFAPPDXGRADIENT TO ARRIVE AT  THE SAME SET OF NORMAL EQUATIONSENDENUMERATEITEM MULTIPLE GAINSCALED VECTOR QUANTIZATION  INDEXVECTOR QUANTIZATION LET X AND Y BE VECTOR SPACES OF THE SAME  DIMENSIONALITY  SUPPOSE THAT THERE ARE TWO SETS OF VECTORS XC1  XC2 SUBSET X  LET YC BE THE SET OF VECTORS EM POOLED FROM  XC1 AND XC2 BY THE INVERTIBLE MATRICES T1 AND T2  RESPECTIVELY  THAT IS IF XBF IN XCI THEN YBF  TI XBF  IS A VECTOR IN YC  INDICATE THAT A VECTOR YBF IN YC CAME  FROM A VECTOR IN XCI BY A SUPERSCRIPT I SO YBFI IN YC  MEANS THAT THERE IS A VECTOR XBF IN XC SUCH THAT YBFI  TI  XBF  DISTANCES RELATIVE TO A VECTOR YBFI IN YC ARE BASED  UPON THE L2 NORM OF THE VECTORS OBTAINED BY MAPPING BACK TO  XCI SO THAT DYBFIYBF  YBFI  YBF  YBFI  YBFI T1YBFI  YBF  YBFI  YBFT WIYBFI  YBFWHERE WI  TITTI1  THIS IS A WEIGHTED NORM WITH THE WEIGHTINGDEPENDENT UPON THE VECTOR IN YC  WE DESIRE TO FIND A EM SINGLE VECTOR YBF0 IN Y THAT IS THE  BEST REPRESENTATION OF THE DATA POOLED FROM ALL THREE DATA SETS IN  THE SENSE THAT SUMYBF IN YC YBF  YBF0  SUMYBF1 IN YC  YBF1  YBF01  SUMYBF2 IN YC YBF2  YBF02IS MINIMIZED  SHOW THAT YBF0  Z1 RBFWHERE Z  SUMYBF1 IN YC W1  SUMYBF2 IN YC W2 QQUADTEXTANDQQUADRBF  SUMYBF1 IN YC W1  YBF1  SUMYBF2 IN YC W2YBF2HINT THIS IS PROBABLY EASIER USING GRADIENTS THAN TRYING TO IDENTIFYTHE APPROPRIATE INNER PRODUCTENDEXERCISESSECTIONCOMPARISON OF LEASTSQUARES AND MINIMUM MEANSQUARESLABELSECCMPIT IS INTERESTING TO CONTRAST THE METHOD OF LEASTSQUARES AND THEMETHOD OF MINIMUM MEANSQUARES BOTH OF WHICH ARE WIDELY USED INSIGNAL PROCESSING  FOR THE METHOD OF LEASTSQUARES WE MAKE THEFOLLOWING OBSERVATIONSBEGINENUMERATEITEM ONLY THE SEQUENCE OF DATA OBSERVED AT THE TIME OF THE ESTIMATE  IS USED IN FORMING THE ESTIMATEITEM DEPENDING UPON ASSUMPTIONS MADE ABOUT THE DATA BEFORE AND AFTER  THE OBSERVATION INTERVAL THE GRAMMIAN MATRIX MAY NOT BE TOEPLITZITEM NO STATISTICAL MODEL IS NECESSARILY ASSUMEDENDENUMERATEFOR THE METHOD OF MINIMUM MEANSQUARES WE MAKE THE FOLLOWINGOBSERVATIONSBEGINENUMERATEITEM A STATISTICAL MODEL FOR THE CORRELATIONS AND CROSSCORRELATIONS  IS NECESSARY  THIS MUST BE OBTAINED EITHER FROM EXPLICIT KNOWLEDGE  OF THE CHANNEL AND SIGNAL AS WAS SEEN IN EXAMPLE REFEXMEQ1 OR  ON THE BASIS OF THE MULTIVARIABLE DISTRIBUTION OF THE DATA AS WAS  SEEN IN EXAMPLE REFEXMMMSEPRED  IN THE ABSENCE OF SUCH  KNOWLEDGE IT IS COMMON TO ESTIMATE THE NECESSARY AUTOCORRELATION  AND CROSS CORRELATION VALUES  AN EXAMPLE OF AN ESTIMATE OF THE AUTOCORRELATION RN   EXKXBARKN USING THE DATA X1X2LDOTSXN ISBEGINEQUATION RHATN  FRAC1N SUMK1NN XK XBARKNLABELEQRHATNENDEQUATIONTHIS IS ACTUALLY A BIASED ESTIMATE OF RN SEE EXERCISEREFEXBIASCORR BUT IT HAS BEEN FOUND SEE EG CITEBOXJENKINS TOPRODUCE A LOWER VARIANCE WHEN THE LAG N IS CLOSE TO NIN ORDER FOR REFEQRHATN TO BE A REASONABLE ESTIMATE OF RNTHE RANDOM PROCESS XK MUST BE EM ERGODIC INDEXERGODIC SOTHAT THE TIME AVERAGE ASYMPTOTICALLY APPROACHES THE ENSEMBLE AVERAGETHIS ASSUMPTION OF ERGODICITY IS USUALLY MADE TACITLY BUT IT IS VITALWHEN THE DATA SEQUENCE USED TO COMPUTE THE ESTIMATE OF THE CORRELATIONPARAMETERS IS THE SAME AS THE DATA SEQUENCE FOR WHICH THE FILTERCOEFFICIENTS ARE COMPUTED THE MINIMUM MEANSQUARED ERROR TECHNIQUE ISESSENTIALLY THE SAME AS THE LEASTSQUARES TECHNIQUEITEM COMMONLY THE COEFFICIENTS OF THE MMS TECHNIQUE ARE COMPUTED  USING A SEPARATE SET OF DATA WHOSE STATISTICS ARE EM ASSUMED TO  BE THE SAME AS THOSE OF THE REAL DATA SET OF INTEREST  THIS SET OF  DATA IS USED AS A EM TRAINING SET INDEXTRAINING SET TO FIND  THE AUTOCORRELATION FUNCTIONS AND THE FILTER COEFFICIENTS  PROVIDED  THAT THE TRAINING DATA DOES HAVE THE SAME OR VERY SIMILAR  STATISTICS AS THE DATA SET OF INTEREST THIS WORKS WELL  HOWEVER  IF THE TRAINING DATA IS SIGNIFICANTLY DIFFERENT FROM THE DATA SET OF  INTEREST FINDING THE OPTIMUM FILTER COEFFICIENTS CAN ACTUALLY LEAD  TO POOR PERFORMANCE BECAUSE THE BEST SOLUTION TO THE  WRONG PROBLEM IS USEDITEM WE ALSO NOTE THAT THE TRUE GRAMMIAN MATRIX R USED IN  PREDICTION AND OPTIMAL FIR FILTERING PROBLEMS IS ALWAYS A TOEPLITZ  MATRIX AND HENCE FAST ALGORITHMS APPLY TO FINDING THE  COEFFICIENTSENDENUMERATEIN SECTION REFSECRLS WE EXAMINE HOW THE COEFFICIENTS OF THE LSFILTER CAN BE UPDATED ADAPTIVELY SO THAT THE COEFFICIENTS AREMODIFIED AS NEW DATA ARRIVES  IN SECTION REFSECLMS WE DEVELOP ANALGORITHM SO THAT THE COEFFICIENTS OF THE MMS FILTER CAN BE UPDATEDADAPTIVELY BY APPROXIMATING THE EXPECTATION  THESE TWO CONCEPTS FORMTHE HEART OF ADAPTIVE FILTERING THEORYBEGINEXERCISES  ITEM LABELEXBIASCORR FOR THE ESTIMATED AUTOCORRELATION RHATN  FRAC1N SUMK1NN XK XBARKNBEGINENUMERATEITEM TAKE THE EXPECTATION ERHATN AND SHOW THAT IT IS NOT  EQUAL TO RN THE TRUE VALUE OF THE AUTOCORRELATIONITEM DETERMINE A SCALING FACTOR TO MAKE THE RHATN AN UNBIASED  ESTIMATEITEM WRITE A SC MATLAB FUNCTION THAT COMPUTES RHATN FROM  REFEQRHATNENDENUMERATEENDEXERCISESINPUTDETESTDIRFREQFILTSECTIONA DUAL APPROXIMATION PROBLEMLABELSECDUALAPPROXINDEXDUAL APPROXIMATION THE APPROXIMATION PROBLEMS WE HAVE SEEN UPTILL NOW HAVE SELECTED A POINT FROM A FINITEDIMENSIONAL SUBSPACE OFTHE HILBERT SPACE OF THE PROBLEM  IN EACH CASE BECAUSE THE SOLUTIONWAS IN A FINITEDIMENSIONAL SUBSPACE SOLVING AN MATSIZEMMSYSTEM OF EQUATIONS WAS SUFFICIENT  IN SOME APPROXIMATION PROBLEMSTHE SUBSPACE IN WHICH THE SOLUTION LIES IS NOT FINITE DIMENSIONAL SOA SIMPLE FINITE SET OF EQUATIONS CANNOT BE SOLVED TO OBTAIN THESOLUTION  THERE ARE SOME PROBLEMS HOWEVER IN WHICH A FINITE SET OFCONSTRAINTS PROVIDES US WITH SUFFICIENT INFORMATION TO SOLVE THEPROBLEM FROM A FINITE SET OF EQUATIONSWE BEGIN WITH A DEFINITIONBEGINDEFINITION  LET M BE A SUBSPACE OF A LINEAR SPACE S AND LET X0 IN S  THE SET V  X0  M IS SAID TO BE A EM TRANSLATION OF M BY  X0  THIS TRANSLATION IS CALLED A BF LINEAR  VARIETY INDEXLINEAR VARIETYENDDEFINITIONA LINEAR VARIETY IS NOT IN GENERAL A SUBSPACEBEGINEXAMPLE  LET M  000010 IN THE VECTOR SPACE GF23  INTRODUCED IN EXAMPLE REFEXMVS1 AND LET XBF0  111 IN  S  THEN XBFM  111101IS A LINEAR VARIETYENDEXAMPLEA VERSION OF THE ORTHOGONALITY THEOREM APPROPRIATE FOR LINEARVARIETIES IS ILLUSTRATED IN FIGURE REFFIGLINVAR  LET V  XBF0 M BE A CLOSED LINEAR VARIETY IN A HILBERT SPACE H  THEN THERE ISA EM UNIQUE VECTOR VBF0 IN V OF MINIMUM NORM  THE MINIMIZINGVECTOR VBF0 IS ORTHOGONAL TO M  THIS RESULT IS AN IMMEDIATECONSEQUENCE OF THE PROJECTION THEOREM FOR HILBERT SPACES SIMPLYTRANSLATE THE VARIETY AND THE ORIGIN BY XBF0BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRLINVAR1    CAPTIONMINIMUM NORM TO A LINEAR VARIETY    LABELFIGLINVAR  ENDCENTERENDFIGURELET S BE A HILBERT SPACE  GIVEN A SET OF LINEARLY INDEPENDENTVECTORS YBF1YBF2LDOTSYBFM IN S LET M LSPANYBF1YBF2LDOTSYBFM  THE SET OF XBF IN S SUCH THAT BEGINALIGNEDLA XBFYBF1RA  0 LA XBFYBF2RA  0 VDOTS LA XBFYBFMRA  0 ENDALIGNEDIS A SUBSPACE WHICH BECAUSE OF THESE INNERPRODUCT CONSTRAINTS MUSTBE MPERP  SUPPOSE NOW WE HAVE PROBLEM IN WHICHTHERE ARE INNERPRODUCT CONSTRAINTS OF THE FORMBEGINEQUATIONLABELEQDUAL1BEGINSPLITLA XBFYBF1 RA  A1 LA XBFYBF2 RA  A2   VDOTS LA XBFYBFM RA  AMENDSPLITENDEQUATIONIF WE CAN FIND ANY POINT XBFXBF0 THAT SATISFIES THE CONSTRAINTSIN REFEQDUAL1 THEN FOR ANY VBF IN MPERP XBF0 VBFALSO SATISFIES THE CONSTRAINTS  HENCE THE SPACE OF SOLUTIONS OFREFEQDUAL1 IS THE LINEAR VARIETY V  XBF0  MPERP  ALINEAR VARIETY V SATISFYING THE M CONSTRAINTS IN REFEQDUAL1IS SAID TO HAVE EM CODIMENSION M INDEXCODIMENSION SINCE THEORTHOGONAL COMPLEMENT OF THE SUBSPACE MPERP PRODUCING IT HASDIMENSION MBEGINEXAMPLE  IN RBB3 LET YBF1  100 AND YBF2  010  AND LET  M  LSPANYBF1YBF2  THE SET OF POINTS SUCH THAT LA XBFYBF1RA  0 QQUADQQUADLA XBFYBF2 RA  0IS LSPAN001  MPERPNOW FOR THE CONSTRAINTSLA XBFYBF1RA  3 QQUADQQUADLA XBFYBF2 RA  4OBSERVE THAT IF XBF  34S FOR ANY S IN RBB THEN THECONSTRAINTS ARE SATISFIED  THE SET V  340  MPERP IS ALINEAR VARIETY OF CODIMENSION 2ENDEXAMPLEWE ARE NOW IN A POSITION TO STATE THE MINIMIZATION PROBLEM  BEGINTHEOREM LABELTHMDUALAPPROX DUAL APPROXIMATION LET  YBF1YBF2LDOTSYBFM BE   LINEARLY INDEPENDENT IN A HILBERT SPACE S AND LET M   LSPANYBF1LDOTSYBFM  THE ELEMENT XBFIN S SATISFYINGBEGINEQUATIONBEGINSPLITLA XBFYBF1 RA  A1 LA XBFYBF2 RA  A2  VDOTS  LA XBFYBFM RA  AMENDSPLITLABELEQDUAL2ENDEQUATIONWITH MINIMUM NORM LIES IN M SPECIFICALLY XBF  SUMI1M CI YBFIWHERE THE COEFFICIENTS IN THIS LINEAR COMBINATION SATISFYBEGINEQUATIONBEGINBMATRIX LA YBF1YBF1RA  LA YBF2YBF1RA  CDOTS  LA YBFMYBF1RA LA YBF1YBF2RA  LA YBF2YBF2RA  CDOTS  LA YBFMYBF2RA VDOTS LA YBF1YBFMRA  LA YBF2YBFMRA  CDOTS  LA YBFMYBFMRA ENDBMATRIXBEGINBMATRIX C1  C2  VDOTS  CM ENDBMATRIX BEGINBMATRIX A1 A2  VDOTS  AM ENDBMATRIXLABELEQDUAL3ENDEQUATIONENDTHEOREMBEGINPROOF  BY THE DISCUSSION ABOVE THE SOLUTION LIES IN THE LINEAR VARIETY V   XBF0  MPERP FOR SOME XBF0  FURTHERMORE THE OPTIMAL SOLUTION  XBF0 IS ORTHOGONAL TO MPERP SO THAT XBF0 IN MPERPPERP   M  THUS XBF0 IS OF THE FORM XBF0  SUMI1M CI YBFITAKING INNER PRODUCTS OF THIS EQUATION WITH YBF1YBF2LDOTSYBFM AND RECOGNIZING THAT FOR THE SOLUTION LA XBF0 YBFIRA AI WE OBTAIN THE SET OF EQUATIONS IN REFEQDUAL3ENDPROOFBEGINEXAMPLE  FOR THE LINEAR VARIETY OF THE PREVIOUS PROBLEM LET US FIND THE  SOLUTION OF MINIMUM NORM  USING REFEQDUAL3 WE FINDX  340TO BE THE MINIMUM NORM SOLUTION SATISFYING THE CONSTRAINTSENDEXAMPLEBEGINEXAMPLE  WE EXAMINE HERE A PROBLEM IN WHICH THE SOLUTION SPACE IS INFINITE  DIMENSIONAL  SUPPOSE WE HAVE AN LTI SYSTEM WITH CAUSAL IMPULSE  RESPONSE HT  E2T  3E4T IN WHICH THE INITIAL CONDITIONS  ARE Y0  0 AND YDOT0  0  WE DESIRE TO DETERMINE AN INPUT  SIGNAL XT SO THAT THE OUTPUT YT  XTHT SATISFIES THE  CONSTRAINTSY1 1 QQUADQQUADINT01 YT DT  0IN SUCH A WAY THAT THE INPUT ENERGY INT01 XT2 DTIS MINIMIZED  WRITING THE CONVOLUTION INTEGRAL FOR THE FIRST OUTPUTTHE FIRST CONSTRAINT CAN BE WRITTEN INT01 E21TAU  3E41TAUXTAU DTAU  1USING THE INNER PRODUCT LA FGRA  INT01 FTAUGTAU DTTHE FIRST CONSTRAINT CAN BE WRITTEN AS LA XY1RA  1WHERE Y1TAU  E21TAU  3E41TAU  H1TAUTHE SECOND CONSTRAINT CAN BE WRITTEN USING THE INTEGRAL OF THE IMPULSERESPONSE SEE EXERCISE REFEXINTSYSRESP KT  INT0T HTAUDT  FRAC54  FRAC34E4T FRAC12 E2TTHEN THE SECOND CONSTRAINT IS LA XY2RA  0WHERE  Y2TAU  FRAC54  FRAC34 E41TAU  FRAC12E21TAU  K1TAUTHE SOLUTION X0T MUST LIE IN THE SPACE SPANNED BY Y1 ANDY2 X0  C1 Y1T  C2 Y2TTHEN THE EQUATION REFEQDUAL3 BECOMES BEGINBMATRIXLA Y1Y1RA  LA Y1Y2RA  LA Y1Y2 RA  LA Y2Y2 RA ENDBMATRIXBEGINBMATRIXA1  A2 ENDBMATRIX BEGINBMATRIX 236756 0682808 0682808   0818254ENDBMATRIXBEGINBMATRIX C1  C2  ENDBMATRIX  BEGINBMATRIX 0  1ENDBMATRIXWHICH HAS SOLUTION BEGINBMATRIXA1A2 ENDBMATRIXT  BEGINBMATRIX0464166  160945 ENDBMATRIX DUALMINMMAENDEXAMPLESECTIONMINIMUMNORM SOLUTION OF UNDERDETERMINED EQUATIONSLABELSECLS2THE SOLUTION TO THE DUAL APPROXIMATION PROBLEM PROVIDES A METHOD OFFINDING A LEASTSQUARES SOLUTION TO AN UNDERDETERMINED SET OFEQUATIONSBEGINEXAMPLE  SUPPOSE THAT WE ARE TO SOLVE THE SET OF EQUATIONSBEGINEQUATION BEGINBMATRIX12 3  541 ENDBMATRIX BEGINBMATRIXX1X2  X3ENDBMATRIX  BEGINBMATRIX 4  6 ENDBMATRIXLABELEQMINNORMSOL1ENDEQUATIONONE SOLUTION IS XBF  BEGINBMATRIX1  2  3 ENDBMATRIXHOWEVER OBSERVE THAT THE VECTOR VBF  111T IS IN THENULLSPACE OF A SO THAT A VBF  0 ANY VECTOR OF THE FORM BEGINBMATRIX1  2  3 ENDBMATRIX  T BEGINBMATRIX 1  1 1 ENDBMATRIXFOR T IN RBB IS ALSO A SOLUTION TO REFEQMINNORMSOL1ENDEXAMPLEWHEN SOLVING M EQUATIONS WITH N UNKNOWNS WITH M  N UNLESS THEEQUATIONS ARE INCONSISTENT AS IN THE EXAMPLE BEGINBMATRIX 123  2  4 6ENDBMATRIXBEGINBMATRIXX1X2  X3 ENDBMATRIX BEGINBMATRIX4  7 ENDBMATRIXTHERE WILL BE AN INFINITE NUMBER OF SOLUTIONSLET XBF BE A SOLUTION OF AXBF  BBF WHERE A IS ANMATSIZEMN MATRIX WITH MN AND LET N  NULLSPACEATHEN IF XBF0 IS A SOLUTION TO AXBF  BBF SO IS ANY VECTOR OFTHE FORM XBF0  NBF WHERE NBF IN N  IF THE NULLSPACE IS NOTTRIVIAL A VARIETY OF SOLUTIONS ARE POSSIBLE  IN ORDER TO HAVE AWELLDETERMINED ALGORITHM FOR UNIQUELY SOLVING THE PROBLEM SOMECRITERION MUST BE ESTABLISHED REGARDING WHICH SOLUTION IS DESIRED  AREASONABLE CRITERION IS TO FIND THE SOLUTION XBF OF SMALLEST NORMTHAT IS WE WANT TO BEGINALIGNEDTEXTMINIMIZE  XBF TEXTSUBJECT TO   A XBF  BBFENDALIGNEDTHE MINIMUM NORM SOLUTION IS APPEALING FROM A NUMERIC STANDPOINTBECAUSE REPRESENTATIONS OF SMALL NUMBERS ARE USUALLY EASIER THANREPRESENTATIONS OF LARGE NUMBERS  IT ALSO LEADS TO A UNIQUE SOLUTIONTHAT CAN BE COMPUTED USING THE FORMULATION OF THE DUAL PROBLEM OF THEPREVIOUS SECTIONLET US WRITE A IN TERMS OF ITS ROWS AS A  BEGINBMATRIX YBF1H  YBF2H   VDOTS   YBFMHENDBMATRIXTHEN WE OBSERVE THAT THE EQUATION AXBF  BBF IS EQUIVALENT TO BEGINALIGNEDYBFH1 XBF  B1 YBFH2 XBF  B2 VDOTS YBFHM XBF  BMENDALIGNEDOUR CONSTRAINT EQUATION THEREFORE CORRESPONDS TO M INNERPRODUCTCONSTRAINTS OF THE SORT SHOWN IN REFEQDUAL1  BY THEOREMREFTHMDUALAPPROX THE MINIMUMNORM SOLUTION MUST BE OF THE FORMBEGINEQUATION XBF  SUMI1M CI YBFILABELEQDUAL4ENDEQUATIONWHERE THE CI ARE THE SOLUTION TO REFEQDUAL3WE CAN WRITE REFEQDUAL4 ASBEGINEQUATION XBF  AH CBFLABELEQDUAL5ENDEQUATIONWHERE AH  BEGINBMATRIXYBF1  YBF2  CDOTS  YBFMENDBMATRIXFURTHERMORE IN MATRIX NOTATION WE CAN WRITE REFEQDUAL3 IN THEFORM AAHCBF  BBFPROVIDED THAT THE ROWS ARE LINEARLY INDEPENDENT THE MATRIX AAH ISINVERTIBLE AND WE CAN SOLVE FOR CBF AS CBF  AAH1BBFSUBSTITUTING THIS INTO REFEQDUAL5 WE OBTAIN THE MINIMUMNORMSOLUTION BEGINEQUATIONXBF  AHAAH1 BBFLABELEQPSEUDOINV2ENDEQUATIONBEGINEXAMPLE  THE MINIMUM NORM SOLUTION TO REFEQMINNORMSOL1 FOUND USING  REFEQPSEUDOINV2 IS XBF  BEGINBMATRIX1  0  1 ENDBMATRIXENDEXAMPLETHE MATRIX AHAAH1 IS A PSEUDOINVERSE INDEXPSEUDOINVERSEOF THE MATRIX ABEGINEXERCISESITEM LABELEXINTSYSRESP LET HT BE THE IMPULSE RESPONSE OF A  SYSTEM AND LET YT  XTHT  SHOW THAT INT0T YTDT  YTKTWHERE KT IS THE INTEGRAL OF THE IMPULSE RESPONSE KT  INT0T HTDTITEM A SYSTEM IS KNOWN TO HAVE IMPULSE RESPONSE HT  3E2T   4E5T AND IS INITIALLY RELAXED INITIAL CONDITIONS ARE ZERO  DETERMINE AN INPUT XT SO THAT THE OUTPUT SATISFIES THE  CONDITIONS Y2  2 QQUAD TEXTANDQQUAD INT02 YTDT  3IN SUCH A WAY THAT THE INPUT ENERGY XT2 IS MINIMIZED  ITEM LET HT  02T  304T FOR K GEQ 0 BE THE IMPULSE    RESPONSE OF A DISCRETETIME SYSTEM WITH ZERO INITIAL CONDITIONS    IT IS DESIRED TO DETERMINE A CAUSAL INPUT SEQUENCE XT SUCH    THAT THE OUTPUT YT  HTFT SATISFIES THE CONSTRAINTS BEGINALIGNEDY10  5SUMJ010 YJ  2ENDALIGNEDSUCH THAT THE INPUT ENERGY SUMK010 XT2 IS MINIMIZEDFORMULATE THIS AS A DUAL APPROXIMATION PROBLEM AND FIND THE MINIMIZINGSEQUENCE XTITEM CITELUENBERGER1969  USING THE PROJECTION THEOREM SOLVE THE  FINITE DIMENSIONAL PROBLEM BEGINALIGNEDTEXTMINIMIZE   XBFT Q XBF TEXTSUBJECT TO   A XBF  BBFENDALIGNEDWHERE XBF IN RBBN Q IS A POSITIVEDEFINITE SYMMETRIC MATRIXAND A IS A MATSIZEMN MATRIX WITH M  NITEM CITELUENBERGER1969  LET XBF BE A VECTOR IN A HILBERT SPACE  S AND LET  XBF1 XBF2LDOTSXBFN AND   YBF1YBF2LDOTS YBFM BE SETS OF LINEARLY INDEPENDENT  VECTORS IN S  WE DESIRE TO MINIMIZE  XBF  XBFHAT WHILE  SATISFYING XBF IN M  LSPAN XBF1XBF2LDOTS XBFMAND LA XBFHAT YBFIRA  CI I12LDOTS MBEGINENUMERATEITEM FIND EQUATIONS FOR THE SOLUTION WHICH ARE SIMILAR TO THE NORMAL  EQUATIONS ITEM GIVE A GEOMETRIC INTERPRETATION OF THE EQUATIONSENDENUMERATEITEM CITELUENBERGER1969 CONSIDER THE PROBLEM OF FINDING THE VECTOR  XBF OF MINIMUM NORM SATISFYING LA XBFYBFIRA GEQ CI FOR  I12LDOTS N WHERE THE YBFIS ARE LINEARLY INDEPENDENT  BEGINENUMERATE  ITEM SHOW THAT THIS PROBLEM HAS A UNIQUE SOLUTION  ITEM SHOW THAT A NECESSARY AND SUFFICIENT CONDITION THAT XBF  SUMI1N AI YBFIIS A SOLUTION IS THAT THE VECTOR ABF WITH COMPONENTS AI SATISFY RT ABF GEQ CBF QQUAD TEXTANDQQUAD ABF GEQ ZEROBFAND THAT AI  0 IF LA XBFYBFIRA  CI  R IS THE GRAMMIANMATRIX OF YBF1YBF2LDOTS YBFN  ENDENUMERATEENDEXERCISESINPUTLINALGDIRWLS   IRLS SECTIONSECTIONSIGNAL TRANSFORMATION AND GENERALIZED FOURIER SERIESLABELSECGFSMUCH OF THE TRANSFORM THEORY EMPLOYED IN SIGNAL PROCESSING ISENCOMPASSED BY REPRESENTATIONS IN AN APPROPRIATE LINEAR VECTOR SPACETHE SET OF BASIS FUNCTIONS FOR THE TRANSFORMATION IS CHOSEN SO THATTHE COEFFICIENTS CONVEY DESIRED INFORMATION ABOUT THE SIGNAL  BYDETERMINING THE BASIS FUNCTIONS APPROPRIATELY DIFFERENT INFORMATIONCAN BE EXTRACTED FROM A SIGNAL BY FINDING A REPRESENTATION OF THESIGNAL IN THE BASISIN THIS SECTION WE ARE LARGELY BUT NOT ENTIRELY INTERESTED INAPPROXIMATING CONTINUOUSTIME FUNCTIONS  THE METRIC SPACE IS L2AND WE DEAL WITH AN INFINITE NUMBER OF BASIS FUNCTIONS SO SOMEWHATMORE CARE IS NEEDED THAN IN THE PREVIOUS SECTIONS OF THIS CHAPTERFINDING THE BEST REPRESENTATION IN AN L2 NORM SENSE OF A FUNCTIONXT AS XT APPROX SUMI0M CI PITWHERE PIT IS A SET OF BASIS FUNCTIONS IS THE APPROXIMATIONPROBLEM WE HAVE SEEN ALREADY MANY TIMES  IF THE BASIS FUNCTIONS AREORTHONORMAL THE COEFFICIENTS WHICH MINIMIZE X  SUMI0M CIPI2 CAN BE FOUND AS CI  LA XPIRA THE SET OF COEFFICIENTSCII12LDOTSM PROVIDES THE BEST REPRESENTATION IN THELEASTSQUARES SENSE OF X  THE MINIMUM SQUARED ERROR OF THE SERIESREPRESENTATION IS  X  SUMI1M CI PI2  X2  SUMI1M LA XPIRA2SINCE THE ERROR IS NEVER NEGATIVE IT FOLLOWS THATBEGINEQUATION SUMI1M CI2  SUMI1M LA XPIRA2 LEQ X2LABELEQBESSELINEQENDEQUATIONTHIS INEQUALITY IS KNOWN AS EM BESSELS INEQUALITYTHE FUNCTION SUMI1M CI PI OBTAINED AS A BEST L2APPROXIMATION OF XT IS SAID TO BE THE EM PROJECTION OF XTONTO THE SPACE SPANNED BY P1P2LDOTSPM  THIS MAY BEWRITTEN AS  XPROJP1P2LDOTSPMTASSUME THAT X AND  PI ARE IN SOME HILBERT SPACE H  IF THESET OF BASIS FUNCTIONS  PI IS INFINITE WE CAN TAKE THE LIMITIN REFEQBESSELINEQ AS M RIGHTARROW INFTY  THEREPRESENTATION OF THIS LIMIT IS THE INFINITE SERIES YT  SUMI1INFTY CI PITSINCE YMT  SUMI1M CI PITIS A CAUCHY SEQUENCE AND THE HILBERT SPACE IS COMPLETE WE CONCLUDETHAT YT IS IN THE HILBERT SPACE  FOR ANY ORTHONORMAL SET PI THE BEST APPROXIMATION OF X IN THE L2 SENSE IS THEFUNCTION Y  WE NOW WANT TO ADDRESS THE QUESTION OF WHEN XY FORAN ARBITRARY X IN H  WE MUST FIRST POINT OUT THAT BY THEEQUALITY XY WHAT WE MEAN IS THAT  XY  0WHERE THE NORM IS THE L2 NORM SINCE WE ARE DEALING WITH A HILBERTSPACE  FUNCTIONS THAT DIFFER ON A SET OF MEASURE ZERO ARE EQUALIN THE SENSE OF THE L2 NORM  THUS EQUAL DOES NOT NECESSARILYMEAN POINTFORPOINT EQUAL AS DISCUSSED IN SECTION REFSECMETTECHWE NOW DEFINE A CONDITION UNDER WHICHIT IS POSSIBLE TO REPRESENT EVERY X USING THE BASIS SET PIBEGINDEFINITION  AN ORTHONORMAL SET PII12LDOTSINFTY IN A HILBERT SPACE  S IS BF COMPLETEFOOTNOTETHIS REFERS TO COMPLETENESS OF THE    SET OF FUNCTIONS WHICH CONCERNS THE REPRESENTATIONAL ABILITY    OF THE FUNCTIONS NOT THE COMPLETENESS OF THE SPACE WHICH IS USED    TO DESCRIBE THE FACT THAT ALL CAUCHY SEQUENCES CONVERGE  SOME    AUTHORS USE TOTAL IN PLACE OF COMPLETE HERE INDEXTOTAL      SETSEECOMPLETE SET INDEXCOMPLETE SET IF X  SUMI1INFTY LA XPIRA PIFOR EVERY X IN SENDDEFINITIONBEGINEXAMPLE  BY MEANS OF A SIMPLE COUNTEREXAMPLE IT IS STRAIGHTFORWARD TO SHOW  THAT SIMPLY HAVING AN INFINITE SET OF ORTHONORMAL FUNCTIONS IS NOT  SUFFICIENT TO ESTABLISH COMPLETENESS  IN L202PI CONSIDER  THE FUNCTION XT  COS T  AN INFINITE SET OF ORTHOGONAL  FUNCTIONS IS T   PNT  SINNT N12LDOTS  IN THE  GENERALIZED FOURIER SERIES REPRESENTATION XHATT  SUMI1INFTY CI PITWE FIND THAT THE COEFFICIENTS ARE PROPORTIONAL TO LA COS T SIN NTRA  INT02PI COST SINNTDT  0HENCE XHATT0 WHICH IS NOT A GOOD REPRESENTATION  WE CONCLUDETHAT THE SET IS NOT COMPLETEENDEXAMPLESOME RESULTS REGARDING COMPLETENESS ARE EXPRESSED IN THE FOLLOWINGTHEOREM WHICH WE STATE WITHOUT PROOFBEGINTHEOREM CITEKEENER  A SET OF ORTHONORMAL FUNCTIONS PI I12LDOTS IS COMPLETE  IN AN INNER PRODUCT SPACE S WITH INDUCED NORM IF ANY OF THE FOLLOWING EQUIVALENT STATEMENTS HOLDS  BEGINENUMERATE  ITEM FOR ANY X IN S X  SUMI1INFTY LA XPI RA PIITEM FOR ANY EPSILON  0 THERE IS AN N  INFTY SUCH THAT FOR  ALL N GEQ N  X  SUMI1N LA XPIRA PI EPSILONIN OTHER WORDS WE CAN APPROXIMATE ARBITRARILY CLOSELY ITEM PARSEVALS EQUALITY HOLDS X2  SUMI1INFTY LA  XPIRA2 FOR ALL X IN SITEM IF LA XPIRA  0 FOR ALL I THEN X0  THIS WAS SHOWN  TO FAIL IN THE LAST EXAMPLEITEM THERE IS NO NONZERO FUNCTION F IN S FOR WHICH THE SET  PII12LDOTS CUP F FORMS AN ORTHOGONAL SET  ENDENUMERATEENDTHEOREMFOR A FINITEDIMENSIONAL SPACE S OF DIMENSION M TO HAVE MLINEARLY INDEPENDENT FUNCTIONS PK K12LDOTSM IS SUFFICIENTFOR COMPLETENESSWHEN PI IS A COMPLETE BASIS SET THEN THE SEQUENCE C1C2LDOTS COMPLETELY DESCRIBES X THERE IS A ONETOONERELATIONSHIP BETWEEN X AND C1C2LDOTS  EXCEPT THAT XIS ONLY UNIQUE UP TO A SET OF MEASURE ZERO  WE SOMETIMES SAYTHAT THE SEQUENCE C1C2LDOTS IS THE BF TRANSFORM OR THEBF GENERALIZED FOURIER SERIES INDEXFOURIER SERIESGENERALIZED OFX  INDEXLEFTRIGHTARROWWRITING CBF  C1C2LDOTSWE CAN REPRESENT THE TRANSFORM RELATIONSHIP AS X LEFTRIGHTARROW CBFWE CAN DEFINE EM DIFFERENT TRANSFORMATIONS DEPENDING UPON THE SETOF ORTHONORMAL BASIS FUNCTIONS WE CHOOSE  SINCE EACH COEFFICIENT INTHE TRANSFORM IS A PROJECTION OF X ONTO THE BASIS FUNCTION THETRANSFORM COEFFICIENT PI DETERMINES HOW MUCH OF PI IS IN XIF WE WANT TO LOOK FOR PARTICULAR FEATURES OF A SIGNAL ONE WAY IS TODESIGN A SET OF ORTHOGONAL BASIS FUNCTIONS THAT HAVE THOSE FEATURESAND COMPUTE A TRANSFORM USING THOSE SIGNALSIF PII12LDOTS IS A COMPLETE SET THERE IS NO ERROR IN THEREPRESENTATION SO BESSELS INEQUALITY REFEQBESSELINEQINDEXBESSELS INEQUALITY INDEXINEQUALITIESBESSELSBECOMES AN EQUALITYBEGINEQUATIONX2  SUMI1INFTY CI2LABELEQPARSEVALENDEQUATIONTHIS RELATIONSHIP IS KNOWN AS EM PARSEVALS EQUALITYINDEXPARSEVALS EQUALITY IT SHOULD BEFAMILIAR IN VARIOUS SPECIAL CASES TO SIGNAL PROCESSORS  WE CAN WRITETHIS AS X  CBFWHERE THE NORM ON THE LEFT IS THE L2 NORM IF X IS A FUNCTIONAND THE NORM ON THE RIGHT IS THE L2 NORMFOR TRANSFORMATIONS USING ORTHONORMAL BASIS SETS THE ANGLES ARE ALSOPRESERVEDBEGINLEMMA  IF X AND Y HAVE A GENERALIZED FOURIER SERIES REPRESENTATION  USING SOME ORTHONORMAL BASIS SET PII12LDOTS IN A HILBERT  SPACE S WITH X LEFTRIGHTARROW CBF QQUADTEXTANDQQUADY LEFTRIGHTARROW BBFTHENBEGINEQUATION LA XY RA  LA CBF BBF RALABELEQPRESERANGLEIPENDEQUATIONENDLEMMABEGINPROOF  WE CAN WRITE X  SUMI1INFTY CI PI QQUAD TEXTANDQQUAD   Y  SUMI1INFTY BI PITHENBEGINALIGNLA XY RA  LA SUMI1INFTY CI PI SUMJ1INFTY BJ PJRA  LABELEQANGLEPROOF1  SUMI1INFTY CI BI  LA CBFBBFRA NONUMBERENDALIGNWHERE THE CROSS PRODUCTS IN THE INNER PRODUCT INREFEQANGLEPROOF1 ARE ZERO BECAUSE OF ORTHOGONALITYENDPROOFBEGINEXAMPLE LABELEXMFS BF FOURIER SERIES  INDEXFOURIER    SERIES THE SET OF  FUNCTIONS WHICH ARE PERIODIC ON 02PI CAN BE REPRESENTED USING  THE SERIES FT  SUMNINFTYINFTY CN FRAC1SQRT2PI EJN TTHE BASIS FUNCTIONS PNT  EJ N TSQRT2PI ARE ORTHONORMALSINCEINT02PI EJNT EJMT  DT  BEGINCASES0  N NEQ M 2PI  N  MENDCASESTHEN FROM REFEQPROJ4CN  FRAC1SQRT2PI INT02PI FT EJNTDTBY PARSEVALS RELATIONSHIP WE HAVE INT02PIFT2 DT   SUMN CN2MORE COMMONLY WE USE THE NONNORMALIZED BASIS FUNCTIONS YNT EJNT SO THE SERIES IS FT  SUMN BN EJNTABSORBING THE NORMALIZING CONSTANT INTO THE COEFFICIENT AS BN  FRAC12PI INT02PI FT EJNT DTIN THIS CASE PARSEVALS RELATIONSHIP MUST BE NORMALIZED AS INT02PI FT2  DT  FRAC12PI SUMI BI2MORE GENERALLY FOR A FUNCTION PERIODIC WITH PERIOD T0 WE HAVE THEFAMILIAR FORMULAS FT  SUMN BN EJNOMEGA0 TWHERE OMEGA0  2PIT0 AND BN  FRAC1T0INT0T0 FTEJNOMEGA0 T DTENDEXAMPLEBEGINEXAMPLE  DISCRETE FOURIER TRANSFORM DFT INDEXDISCRETE FOURIER TRANSFORM DFT A DISCRETETIME SEQUENCE XTT01LDOTSN1 IS TO BE  REPRESENTED AS A LINEAR COMBINATION OF THE FUNCTIONS PKT   1SQRTNEJ2PI TK N BY XT  FRAC1SQRTN SUMK0N1 CK EJ2PI TKNTHE INNER PRODUCT IN THIS CASE IS LA XTYT RA  SUMK0N1 XT YBARTIT CAN BE SHOWN SEE EXERCISE REFEXORTHOGDFT THAT THE SET OFBASIS FUNCTIONS PKT ARE ORTHOGONAL WITH LA PKTPLT RA BEGINCASES1  KBMOD L PMODN 0  TEXTOTHERWISEENDCASESTHE COEFFICIENTS ARE THEREFORE COMPUTED BY CK  FRAC1SQRTNSUMT0N1 XT EJ2PI TKNMORE COMMONLY WE USE THE EM NONNORMALIZED BASIS FUNCTIONS EJ2PI  TKN AND SHIFT ALL OF THE NORMALIZATION INTO THE RECONSTRUCTIONFORMULA  THEN WE HAVE XT  FRAC1N SUMK0N1 DK EJ2PI NKNAND DK  SUMT0N1 XT EJ2PI TKNWHICH IS THE USUAL FOURIER TRANSFORM PAIR PARSEVALSRELATIONSHIP UNDER THIS NORMALIZATION IS SUMT0N1 XT2  FRAC1N SUMK0N1 DK2ENDEXAMPLEBEGINEXERCISESITEM LABELEXORTHOGDFT SHOW THAT THE SET OF FUNCTIONS DEFINED BY PKT FRAC1SQRTN EJ2PI KTNARE ORTHONORMAL WITH RESPECT TO THE INNER PRODUCT LA XTYTRA  SUMK0N1 XTYBARTITEM LET  FT  ET2BE PERIODIC ON 0PIBEGINENUMERATEITEM FIND THE FOURIER SERIES COEFFICIENTS OF THIS FUNCTIONITEM FIND THE SUM OF THE SERIES SUMN LEFTFRAC2PI FRAC11 16N2RIGHT2HINT USE PARSEVALS THEOREMITEM IN THIS PROBLEM WE WILL SHOW THAT THE SET OF CONTINUOUS  FUNCTIONS IS COMPLETE WITH RESPECT TO THE UNIFORM SUP NORM  LET   FNT BE A CAUCHY SEQUENCE OF CONTINUOUS FUNCTIONS  LET  FT BE THE POINTWISE LIMIT OF FNT  FOR ANY EPSILON  0 LET N BE CHOSEN SO THAT MAX FNTFMT LEQ  EPSILON3  SINCE FKT IS CONTINUOUS THERE IS A DELTA SUCH  THAT FTDELTA  FT  EPSILON3  FROM THIS CONCLUDE THAT  FTDELTA  FT  EPSILONAND HENCE THAT FT IS CONTINUOUSENDENUMERATEENDEXERCISESSECTIONSETS OF COMPLETE ORTHOGONAL FUNCTIONSLABELSECCOFTHERE ARE SEVERAL SETS OF COMPLETE ORTHOGONAL FUNCTIONS THAT ARE USEDIN COMMON APPLICATIONS  WE WILL EXAMINE A FEW OF THE MORECOMMONLYUSED SETS  MOSTLY STATING RESULTS WITHOUT PROOFSSUBSECTIONTRIGONOMETRIC FUNCTIONSAS SEEN IN EXAMPLE REFEXMFS THE FAMILIAR TRIGONOMETRIC FUNCTIONSEMPLOYED IN FOURIER SERIES ARE ORTHOGONAL  THEY FORM A COMPLETE SETOF ORTHOGONAL FUNCTIONSINPUTFUNCTDIRORTHOGPOLYSUBSECTIONSINC FUNCTIONSINDEXSINC FUNCTIONTHE FUNCTION COMMONLY KNOWN AS A SINC FUNCTION SINCT  FRACSINPI TPI TCAN BE USED TO FORM A SET OF ORTHOGONAL FUNCTIONSBEGINEQUATION PKT  SINC2BTK2BLABELEQSINCSHIFTENDEQUATIONIT CAN BE SHOWN SEE EXERCISE REFEXSINCORTHOG FOR THE INNERPRODUCT LA FG RA INTINFTYINFTY FTGBARTDTTHAT LA PKTPLTRA  FRAC12BDELTAKL  IF FT ISA BANDLIMITED FUNCTION SUCH THAT ITS FOURIER TRANSFORM SATISFIES FOMEGA  0TEXT FOR  OMEGA NOT IN 2PI B2PI BTHEN IN THE SERIES REPRESENTATION FT  SUMK CK PKTTHE COEFFICIENTS ARE FOUND TO BEBEGINEQUATION CK  FRACLA FPKRALA PKPKRA  FK2BLABELEQSINCSAMPENDEQUATIONTHIS GIVES RISE TO THE FAMILIAR SAMPLING THEOREM REPRESENTATION OF ABANDLIMITED FUNCTION FT  SUMK FK2B FRACSIN2PI BTK2B2PI  BTK2BBEGINEXERCISES  ITEM PROVE PARSEVALS THEOREM FOR FOURIER TRANSFORMS IF Y1T    LEFTRIGHTARROW Y1OMEGA AND Y2T LEFTRIGHTARROW    Y2OMEGA THEN INTINFTYINFTY Y1T YBAR2TDT  FRAC12PIINTINFTYINFTY Y1OMEGA YBAR2OMEGADOMEGAITEM LABELEXSINCORTHOG SHOW FOR PKT DEFINED AS IN  REFEQSINCSHIFT THAT LA PK PLRA   FRAC12BDELTAKL    ALONG THE WAY SHOW THAT INTINFTYINFTY FRACSIN TT FRACSINTZTZ DT FRACPI SIN ZZHINT USE PARSEVALS THEOREM AND FOURIER TRANSFORMSITEM SHOW THAT REFEQSINCSAMP IS CORRECT FOR A BANDLIMITED  FUNCTION FTITEM SHOW THAT FZ  2BINTINFTYINFTY FT P0TZDTSO THAT FOR BANDLIMITED FUNCTIONS P0T BEHAVES LIKE A DELTAFUNCTIONENDEXERCISESINPUTLINALGDIRWAVELETSTEXINPUTLINALGDIRMATCHEDFTEXSETEXSECTREFSECGRADMINBEGINEXERCISESITEM LABELEXGRAMDET THERE IS A CONNECTION BETWEEN GRAMMIANS  INDEXGRAMMIAN AND LINEAR INDEPENDENCE INDEXLINEAR INDEPENDENCE  AS DEMONSTRATED IN THEOREM REFTHMGRAMMPD  WE EXPLORE THIS  CONNECTION FURTHER IN THIS PROBLEM  LET PBF1PBF2LDOTSPBFN BE A SET OF VECTORS AND LET  US SUPPOSE THAT THE FIRST K1 VECTORS OF THIS SET HAVE PASSED A  TEST FOR LINEAR INDEPENDENCE  WE FORM EBFK  CK1K PBF1  CK2K PBF2  CDOTS C1KPBFK1  PBFK AND WANT TO KNOW IF EBFK IS EQUAL TO ZERO FOR ANY SET OFCOEFFICIENTS CBFK  CK1K CK2KLDOTSC1K1TIF SO THEN PBFK IS LINEARLY DEPENDENT ON PBF1LDOTSPBFK1  LET AK  PBF1PBF2LDOTSPBFKBE A DATA MATRIX AND LET RK  AKHAK BE THE CORRESPONDINGGRAMMIANBEGINENUMERATEITEM SHOW THAT THE SQUARED ERROR CAN BE WRITTEN ASBEGINEQUATION EBFKH EBFK  SIGMAK2  CBFKHBEGINBMATRIX RK1   HBFK  HBFKH  RKK ENDBMATRIXCBFKLABELEQGRAMDET1ENDEQUATIONFOR SOME HBFK AND RKK  IDENTIFY HBFK AND RKKITEM DETERMINE THE MINIMUM VALUE OF SIGMAK2 BY MINIMIZING  REFEQGRAMDET1 WITH RESPECT TO CBFK SUBJECT TO THE  CONSTRAINT THAT THE LAST ELEMENT OF CBFK IS EQUAL TO 1  HINT  TAKE THE GRADIENT OF CBFKHBEGINBMATRIX RK1   HBFK  HBFKH  RKK ENDBMATRIXCBFK  LAMBDACBFKHDBF  1WHERE LAMBDA IS A LAGRANGE MULTIPLIER AND DBF 00LDOTS01T  SHOW THAT WE CAN WRITE THE CORRESPONDINGEQUATIONS ASBEGINEQUATIONBEGINBMATRIX RK1  HBFK  HBFKH  RKKENDBMATRIXCBFK  SIGMAK2 DBFLABELEQGRAMDET2ENDEQUATIONITEM SHOW THAT REFEQGRAMDET2 CAN BE MANIPULATED TO BECOME SIGMAK2  RKK  HBFKH RK11 HBFKTHE QUANTITY SIGMAK2 IS CALLED THE EM SCHUR COMPLEMENTINDEXSCHUR COMPLEMENT OFRK  IF SIGMAK20 THEN PBFK IS LINEARLY DEPENDENTENDENUMERATEEXSKIPSETEXSECTREFSECMINERR ITEM  SHOW THAT  REFEQXSTACKROW IS TRUE ITEM LABELEXREDUCEERR REFERRING TO REFEQREDUCERR SHOW THAT I  AAHA1AHIS POSITIVE SEMIDEFINITE AND HENCE THAT THE MINIMUM ERROREBFMIN HAS SMALLER NORM THAN THE ORIGINAL VECTOR XBF  HINTCONSIDER 0 LEQ  BXBF2 WHERE B  I  AAHA1AHEXSKIPSETEXSECTREFSECLINREG  ITEM CONSIDER THE SET OF DATA X   225359 QQUAD Y  42 5 2 1243BEGINENUMERATEITEM MAKE A PLOT OF THE DATAITEM DETERMINE THE BEST LEASTSQUARES LINE THAT FITS THIS DATA AND  PLOT THE LINEITEM ASSUMING THAT THE FIRST AND LAST POINTS ARE BELIEVED TO BE THE  MOST ACCURATE FORMULATE A WEIGHTING MATRIX AND COMPUTE A WEIGHTED  LEASTSQUARES LINE THAT FITS THE DATA  PLOT THIS LINEENDENUMERATEITEM FORMULATE THE REGRESSION PROBLEM REFEQREGRESS2 IN A   LINEAR FORM AS IN REFEQREGRESS4ITEM FORMULATE THE REGRESSION PROBLEM REFEQREGRESS3 IN A   LINEAR FORM AS IN REFEQREGRESS4ITEM FORMULATE THE REGRESSION Y APPROX CEAX AS A LINEAR  REGRESSION PROBLEM WITH REGRESSION PARAMETERS C AND AITEM FORMULATE THE REGRESSION Y APPROX AXB AS A LINEAR  REGRESSION PROBLEMITEM PERFORM THE COMPUTATIONS TO VERIFY THE SLOPE AND INTERCEPT OF  THE LINEAR REGRESSION IN REFEQLINREGRESSAITEM AS A MEASURE OF FIT IN A CORRELATION PROBLEM THE CORRELATION  COEFFICIENT ANALOGOUS TO REFEQCORRCOEFF CAN BE OBTAINED AS RHO  FRACLA XBF YBFRA  LA XBFONEBFRA LA YBFONEBFRA XBF  LA XBFONEBFRA YBF  LA YBFONEBFRATHE CORRELATION COEFFICIENT RHO  PM 1 IF X AND Y ARE EXACTLYFUNCTIONALLY RELATED AND RHO  0 IF THEY ARE INDEPENDENT  FOR THELINEAR REGRESSION IN REFEQ2REGRESS DETERMINE AN EXPLICITEXPRESSION FOR RHO ITEM GIVEN A SET OF DATA XBFI I12LDOTS M WHERE EACH VECTOR   XBFI IN RBBD FORMULATE THE LEASTSQUARES SOLUTION TO FIND   THE BEST DDIMENSIONAL PLANE FITTING THIS DATAITEM DEFINE AN INNER PRODUCT BETWEEN MATRICES X AND Y AS LA XY RA  TRACEXYHWHERE TRACECDOT IS THE SUM OF THE DIAGONAL ELEMENTS SEE SECTIONREFSECTPOSETRACE  WE WANT TO APPROXIMATE THE MATRIX Y BY THE SCALARLINEAR COMBINATION OF MATRICES X1X2LDOTSXM AS Y  C1 X1  C2 X2  CDOTS  CM XM  EUSING THE ORTHOGONALITY PRINCIPLE DETERMINE A SET OF NORMAL EQUATIONSTHAT CAN BE USED TO FIND C1C2LDOTSCM THAT MINIMIZE THEINDUCED NORM OF EITEM FOR THE ARMA INPUTOUTPUT RELATIONSHIP OF REFEQARMA  DETERMINE A SET OF LINEAR EQUATIONS FOR DETERMINING THE ARMA MODEL  PARAMETERS A1A2LDOTSAPB0B1LDOTSBQ ASSUMING  THAT THE MODEL ORDER PQ IS KNOWN AND THAT THE INPUT AND OUTPUT  ARE KNOWN  EXSKIP SETEXSECTREFSECLSFILT  ITEM VERIFY REFEQGRAMM3ITEM FOR THE DATA SEQUENCE 11235813  BEGINENUMERATE  ITEM WRITE DOWN THE DATA MATRIX A AND THE GRAMMIAN AH A USING    I THE COVARIANCE AND II THE AUTOCORRELATION METHODS  ITEM WE DESIRE TO USE THIS SEQUENCE TO TRAIN A SIMPLE LINEAR    PREDICTOR  THE DESIRED SIGNAL DT IS THE VALUE OF XT    AND THE DATA USED ARE THE TWO PRIOR SAMPLES  THAT IS  XT  A1 XT1  A2 XT2  ET WHERE ET IS THE PREDICTION ERRORDETERMINE THE LEASTSQUARES COEFFICIENTS FOR THE PREDICTOR USING THECOVARIANCE AND AUTOCORRELATION METHODSITEM DETERMINE THE MINIMUM LEASTSQUARES ERROR FOR BOTH METHODS  ENDENUMERATE ITEM BF COMPUTER EXERCISE GENERATE A SEQUENCE OF N RANDOM PM   1 BITS AND PASS THEM THROUGH A CHANNEL WITH TRANSFER FUNCTION  HZ  FRAC119Z1  THEN ADD NOISE WITH VARIANCE SIGMAN2  01  DETERMINE A LEASTSQUARES FILTER FOR THIS CHANNEL FOR VARIOUS VALUES OF THE DELAYEXSKIPSETEXSECTREFSECMMSSEFILT ITEM IMPLEMENTATION OF EQUALIZER IN EXAMPLE REFEXMEQ1 GENERATE   THE SIGNAL FT IN EXAMPLE REFEXMEQ1 BY CREATING AN AR1   SIGNAL WHICH IS PASSED THROUGH HCZ THEN CORRUPTED BY ADDITIVE   NOISE  THEN PASS THIS SIGNAL THROUGH THE EQUALIZER DESIGNED IN THAT   EXAMPLE  COMPARE THE EQUALIZED SIGNAL WITH THE SIGNAL DTITEM FOR A DATA SEQUENCE  XT THE CORRELATION MATRIX R IS R  BEGINBMATRIX 5  3  3  5 ENDBMATRIXAND THE CROSSCORRELATION VECTOR PBF WITH A DESIRED SIGNAL IS PBF  BEGINBMATRIX 2  5 ENDBMATRIXBEGINENUMERATEITEM DETERMINE THE OPTIMAL WEIGHT VECTOR ITEM DETERMINE THE MINIMUM MEANSQUARED ERROR ENDENUMERATEITEM CONSIDER A ZEROMEAN RANDOM VECTOR XBF  X1X2X3 WITH  COVARIANCE COVXBF  EXBFXBFT  BEGINBMATRIX 1  7  5  7  4  2 5  2  3 ENDBMATRIXBEGINENUMERATEITEM DETERMINE THE OPTIMAL COEFFICIENTS OF THE PREDICTOR OF X1 IN TERMSOF X2 AND X3 XHAT1  C1 X2  C2 X3ITEM DETERMINE THE MINIMUM MEANSQUARED ERRORITEM HOW IS THIS ESTIMATOR MODIFIED IF THE MEAN OF XBF IS E XBF   123TENDENUMERATEITEM CITEHAYKIN1996 A DISCRETETIME RADAR SIGNAL IS TRANSMITTED AS ST  A0 EJOMEGA0 TTHE SAMPLED NOISY RECEIVED SIGNALS ARE REPRESENTED AS XT  A1 EJOMEGA1 T  NUTWHERE OMEGA1 IS THE RECEIVED SIGNAL FREQUENCY IN GENERALDIFFERENT FROM OMEGA0 BECAUSE OF DOPPLER SHIFT AND NUT IS AWHITENOISE SIGNAL WITH VARIANCE SIGMAN2  LET XBFT  X0X1LDOTS XM1TBE A VECTOR OF RECEIVED SIGNAL SAMPLESBEGINENUMERATEITEM SHOW THAT R  EXBFT XBFHT  SIGMANU2 I  SIGMA12 SBFOMEGA1SBFHOMEGA1WHERE SBFOMEGA1  1EJOMEGA1EJ2OMEGA1LDOTSEM1  J OMEGA1TQQUAD TEXTAND QQUADSIGMA12  EA12ITEM THE TIME SERIES XT IS APPLIED TO AN FIR WIENER FILTER WITH  M COEFFICIENTS IN WHICH THE CROSSCORRELATION BETWEEN XT AND  THE DESIRED SIGNAL DT IS PRESET TO PBF  SBFOMEGA0DETERMINE AN EXPRESSION FOR THE TAPWEIGHT VECTOR OF THE WIENER FILTERENDENUMERATEITEM A CHANNEL WITH TRANSFER FUNCTION HCZ  FRAC112Z1AND OUTPUT UT IS DRIVEN BY AN AR1 SIGNAL DT GENERATED BY DT  4 DT1  NUTWHERE NUT IS A ZEROMEAN WHITENOISE SIGNAL WITH SIGMANU2 2  THE CHANNEL OUTPUT IS CORRUPTED BY NOISE NT WITH VARIANCESIGMAN2  15 TO PRODUCE THE SIGNAL FT  UT  NTDESIGN A SECONDORDER WIENER EQUALIZER TO MINIMIZE THE AVERAGE SQUAREDERROR BETWEEN FT AND DTITEM LABELEXLINPRED BF LINEAR PREDICTION INDEXLINEAR    PREDICTOR A COMMON APPLICATION OF WIENER FILTERING IS IN THE  CONTEXT OF LINEAR PREDICTION  LET DT  XT BE THE DESIRED  VALUE AND LET XHATT  SUMI1M WFI XTIBE THE PREDICTED VALUE OF XT USING AN MTH ORDER PREDICTOR BASEDUPON THE MEASUREMENTS XT1XT2LDOTSXTM ANDLET FMT  XT  XHATTBE THE EM FORWARD PREDICTION ERROR  INDEXFORWARD  PREDICTORSEELINEAR PREDICTOR INDEXLINEAR PREDICTORFORWARD THEN FMT  SUMI0M AFI XTIWHERE AF0  1 AND AFI  WFI I12LDOTSMASSUME THAT XT IS A ZEROMEAN RANDOM SEQUENCE  WE DESIRE TODETERMINE THE OPTIMAL SET OF COEFFICIENTS WFII12LDOTSMTO MINIMIZE EFMT2BEGINENUMERATEITEM USING THE ORTHOGONALITY PRINCIPLE WRITE DOWN THE NORMAL  EQUATIONS CORRESPONDING TO THIS MINIMIZATION PROBLEM  USE THE  NOTATION RJL  EXTL XBARTJ TO OBTAIN THE WIENERHOPF  EQUATION R WBFF  RBFWHERE R  EXBFT1XBFHT1 RBF  EXBFT1XT ANDXBFT1  XBART1XBART2LDOTSXBARTMTITEM DETERMINE AN EXPRESSION FOR THE MINIMUM MEANSQUARED ERROR PM   MIN EFMT2ITEM SHOW THAT THE EQUATIONS FOR THE OPTIMAL WEIGHTS AND THE MINIMUM  MEANSQUARED ERROR CAN BE COMBINED INTO EM AUGMENTED WIENERHOPF  INDEXWIENERHOPF   EQUATIONS AS BEGINBMATRIX R0  RBFH  RBF  R ENDBMATRIXBEGINBMATRIX 1  WBFF ENDBMATRIX  BEGINBMATRIXPM   ZEROBF ENDBMATRIXITEM SUPPOSE THAT XT HAPPENS TO BE AN ARM PROCESS DRIVEN BY WHITE NOISE  NUT SUCH THAT IT IS THE OUTPUT OF A SYSTEM WITH TRANSFER  FUNCTION HZ  FRAC11SUMK1M AK ZKSHOW THAT THE PREDICTION COEFFICIENTS ARE WFK  AK AND HENCETHE COEFFICIENTS OF THE PREDICTION ERROR FILTER FMT ARE AFI  AIHINT SEE SECTION REFSECARPROCESS WRITE DOWN THE YULEWALKEREQUATIONS INDEXYULEWALKER EQUATIONSHENCE CONCLUDE THAT IN THIS CASE THE FORWARD PREDICTION ERRORFMT IS A EM WHITENOISE SEQUENCE  THE PREDICTIONERRORFILTER CAN THUS BE VIEWED AS A EM WHITENING FILTER FOR THE SIGNALXT  INDEXWHITENING FILTERITEM NOW LET XHATTM  SUMI1M WBI XTI1BE THE EM BACKWARD PREDICTOR OF XTM USING THE DATA XTM1XTM2 LDOTS XT AND LET BMT  XTM  XHATTMBE THE BACKWARD PREDICTION ERROR  A BACKWARD PREDICTOR SEEMS STRANGE AFTER ALL WHY PREDICT WHAT WE SHOULD HAVE ALREADY SEEN  BUTTHE CONCEPT WILL HAVE USEFUL APPLICATIONS IN FAST ALGORITHMS FORINVERTING THE AUTOCORRELATION MATRIX  INDEXBACKWARD PREDICTORSEELINEAR PREDICTOR INDEXLINEAR PREDICTORBACKWARDSHOW THAT THE WIENERHOPF EQUATIONS FOR THE OPTIMAL BACKWARD PREDICTORCAN BE WRITTEN ASBEGINEQUATION R WBFB  OVERLINERBFBLABELEQBACKPRED1ENDEQUATIONWHERE RBFB IS THE BACKWARD ORDERING OF RBF  DEFINED ABOVEITEM FROM REFEQBACKPRED1 SHOW THAT RH OVERLINEWBFBB  RBFWHERE WBFBB IS THE BACKWARD ORDERING OF WBFB  HENCE CONCLUDETHAT OVERLINEWBFBB  WBFFTHAT IS THE OPTIMAL BACKWARD PREDICTION COEFFICIENTS ARE THE REVERSEDCONJUGATED OPTIMAL FORWARD PREDICTION COEFFICIENTSENDENUMERATEITEM LET XT  08 XT1   NUTWHERE NUT IS A REAL WHITENOISE ZEROMEAN UNITVARIANCE NOISEPROCESS  WE WANT TO DETERMINE AN OPTIMAL PREDICTORBEGINENUMERATEITEM IF THE ORDER OF THE PREDICTOR IS 2 DETERMINE THE OPTIMAL  PREDICTOR XHATTITEM IF THE ORDER OF THE PREDICTOR IS 1 DETERMINE THE OPTIMAL  PREDICTOR XHATTENDENUMERATEITEM BF RANDOM VECTORS THE MEANSQUARED METHODS TO THIS POINT HAVE  BEEN FOR RANDOM SCALARS  SUPPOSE WE HAVE THE RANDOM VECTOR  APPROXIMATION PROBLEM YBF  C1 PBF1  C2 PBF2  CDOTS  CM PBFM  EBFIN WHICH WE DESIRE TO FIND AN APPROXIMATION YBF IN SUCH A WAY THATTHE NORM OF EBF IS MINIMIZED  LET US DEFINE AN INNER PRODUCTBETWEEN RANDOM VECTORS AS LA XBFYBF RA  TRACE EXBF YBFHBEGINENUMERATEITEM BASED UPON THIS INNER PRODUCT AND ITS INDUCED NORM DETERMINE A  SET OF NORMAL EQUATIONS FOR FINDING C1C2LDOTS CMITEM AS AN EXERCISE IN COMPUTING GRADIENTS USE THE FORMULA FOR THE  GRADIENT OF THE TRACE SEE APPENDIX REFAPPDXGRADIENT TO ARRIVE AT  THE SAME SET OF NORMAL EQUATIONSENDENUMERATEITEM BF MULTIPLE GAINSCALED VECTOR QUANTIZATION INDEXVECTOR QUANTIZATION LET X AND Y BE VECTOR SPACES OF THE SAME  DIMENSIONALITY  SUPPOSE THAT THERE ARE TWO SETS OF VECTORS XC1  XC2 SUBSET X  LET YC BE THE SET OF VECTORS EM POOLED FROM  XC1 AND XC2 BY THE INVERTIBLE MATRICES T1 AND T2  RESPECTIVELY  THAT IS IF XBF IN XCI THEN YBF  TI XBF  IS A VECTOR IN YC  INDICATE THAT A VECTOR YBF IN YC CAME  FROM A VECTOR IN XCI BY A SUPERSCRIPT I SO YBFI IN YC  MEANS THAT THERE IS A VECTOR XBF IN XC SUCH THAT YBFI  TI  XBF  DISTANCES RELATIVE TO A VECTOR YBFI IN YC ARE BASED  UPON THE L2 NORM OF THE VECTORS OBTAINED BY MAPPING BACK TO  XCI SO THATFOOTNOTETHE NOTATION DEFEQ MEANS IS DEFINED AS D2YBFIYBF  YBFI  YBF2 DEFEQ YBFI  YBFI TI1YBFI  YBF2  YBFI  YBFT WIYBFI  YBFWHERE WI  TITTI1  THIS IS A WEIGHTED NORM WITH THE WEIGHTINGDEPENDENT UPON THE VECTOR IN YC  NOTE IN THIS PROBLEM  CDOT1 AND  CDOT 2 REFER TO THE WEIGHTED NORM FOR EACH DATASET NOT THE L1 AND L2 NORMS RESPECTIVELY  WE DESIRE TO FIND A EM SINGLE VECTOR YBF0 IN Y THAT IS THEBEST REPRESENTATION OF THE DATA POOLED FROM BOTH DATA SETS IN THESENSE THAT SUMYBF IN YC YBF  YBF02  SUMYBF1 IN YC  YBF1  YBF012  SUMYBF2 IN YC YBF2  YBF022IS MINIMIZED  SHOW THAT YBF0  Z1 RBFWHERE Z  SUMYBF1 IN YC W1  SUMYBF2 IN YC W2 QQUADTEXTANDQQUADRBF  SUMYBF1 IN YC W1  YBF1  SUMYBF2 IN YC W2YBF2HINT THIS IS PROBABLY EASIER USING GRADIENTS THAN TRYING TO IDENTIFYTHE APPROPRIATE INNER PRODUCTEXSKIPSETEXSECTREFSECCMP  ITEM LABELEXBIASCORR LET X1X2LDOTSXN BE    SEQUENCE OF MEASURED DATAAN ESTIMATE OF THE CORRELATION FUNCTION OF THIS DATA ASSUMING THESEQUENCE IS ERGODIC IS INDEXAUTOCORRELATIONESTIMATE FROM DATA RHATK  FRAC1N SUMIK1N XI XBARIKBEGINENUMERATEITEM TAKE THE EXPECTATION ERHATK AND SHOW THAT IT IS NOT  EQUAL TO RK THE TRUE VALUE OF THE AUTOCORRELATIONITEM DETERMINE A SCALING FACTOR TO MAKE THE RHATK AN UNBIASED  ESTIMATEITEM WRITE A SC MATLAB FUNCTION THAT COMPUTES RHATK FROM  REFEQRHATNENDENUMERATESETEXSECTREFSECFREQFILTEXSKIPITEM LET XT YT AND VT BE CONTINUOUSTIME RANDOM PROCESSES  WITH YT  XT  VT AND SVS  1  DETERMINE AN OPTIMAL  CAUSAL FILTER HT  TO DETERMINE XT WHEN  BEGINENUMERATE  ITEM THE PSD OF XT IS SXS  FRACS2 16S4  53 S2  196ITEM THE PSD OF XT IS SXS  FRACS4  10 S2 9 S4  53 S2  196  ENDENUMERATEITEM SPECTRAL FACTORIZATION THE FEJERRIESZ THEOREM BECAUSE OFINDEXSPECTRAL FACTORIZATIONINDEXFEJERRIESZ THEOREMFEJERRIESZ THEOREMINDEXSQUARE ROOTOF A TRANSFER FUNCTION  THE IMPORTANCE OF THE CANONICAL FACTORIZATION IN SIGNAL PROCESSING  IT IS OF INTEREST TO DETERMINE WHEN A SQUARE ROOT OF A FUNCTION  EXISTS  IN THIS PROBLEM YOU WILL PROVE THE FOLLOWING IF WEJOMEGA   SUMNMM WN EJOMEGA N IS REAL AND WEJOMEGA GEQ 0 FOR  ALL OMEGA THEN THERE IS A FUNCTION YZ  SUMN0M YN ZNSUCH THAT WEJOMEGA  YEJOMEGA2BEGINENUMERATEITEM SHOW THAT WN  WBARNITEM SHOW THAT WBARZ  W1ZBARITEM SHOW THAT IF ZI IS A ROOT OF WZ THEN 1ZBARI IS A  ROOT OF WZITEM ARGUE THAT IF ZI  EJTHETAI IS A ROOT ON THE UNIT  CIRCLE THEN IT MUST HAVE EVEN MULTIPLICITY HINT USE THE FACT  THAT WEJOMEGAGEQ 0ITEM LET ZC  ZIMC  WZI  0 ZI LEQ 1 TEXT ONLY HALF    THE ROOTS ON Z1 BE THE SET OF ROOTS INSIDE AND HALF    THOSE ON THE UNIT CIRCLE  THEN ZC HAS M ELEMENTS AND WZ  A ZM PRODI1M ZZIPRODI1M ZZBARI  1FROM THIS FORM FIND YZENDENUMERATEEXSKIPSETEXSECTREFSECDTFFITEM LABELADDITIVEWHITEBF FILTERING IN WHITE NOISE LET XT YT AND VT BEDISCRETETIME RANDOM PROCESSES WITHYT  XT  VTAND BEGINALIGNEDSVZ    1 SXZ    FRACBZAZENDALIGNEDWHERE BZ AND AZ ARE POLYNOMIALS IN Z WITH THE DEGREE OFBZ STRICTLY BF LOWER THAN THE DEGREE OF AZ  FURTHERMOREASSUME RXVT EQUIV 0SHOW THAT REFADDITIVE HOLDS IN THE DISCRETETIME CASE THAT ISSHOW THAT THE OPTIMAL CAUSAL FILTER ISHZ  1  FRAC1SYZITEM LETYT  XT  VTWHERE BEGINALIGNEDRVT   FRAC23DELTAT RXT   FRAC1027LEFTFRAC12RIGHTTENDALIGNEDWITH EXT  EVT  EXTVT  0SHOW THATBEGINENUMERATEITEM SYZ  FRAC1FRACZ31FRAC13Z1ZOVER 211OVER 2ZAND THUS OBTAIN SYZ AND SYZITEM LEFTFRACSXYZSYZRIGHT     FRACFRAC131  FRAC12Z AND THUS THAT THE WIENER  FILTER IS HZ  FRACFRAC131  FRAC13Z ITEM CONFIRM THAT THIS RESULT AGREES WITH THE RESULTS OFEXERCISE REFADDITIVEWHITEENDENUMERATEITEM LET XT YT AND VT BE DISCRETETIME RANDOM PROCESSES  WITH YT  XT  VT SVZ  1 AND SXZ  FRACZ4  90067 Z3  2804Z2 90067Z 1Z4 20111Z3  30446Z2 20111Z1DETERMINE THE FILTER HZ TO OPTIMALLY PREDICT XT2EXSKIPSETEXSECTREFSECDUALAPPROXITEM LABELEXINTSYSRESP LET HT BE THE IMPULSE RESPONSE OF A  SYSTEM AND LET YT  XTHT  SHOW THAT INT0T YTDT  XTKTWHERE KT IS THE INTEGRAL OF THE IMPULSE RESPONSE KT  INT0T HSDSITEM A SYSTEM IS KNOWN TO HAVE IMPULSE RESPONSE HT  3E2T   4E5T AND IS INITIALLY RELAXED INITIAL CONDITIONS ARE ZERO  DETERMINE AN INPUT XT SO THAT THE OUTPUT SATISFIES THE  CONDITIONS Y2  2 QQUAD TEXTANDQQUAD INT02 YTDT  3IN SUCH A WAY THAT THE INPUT ENERGY XT2  INT02XT2DT IS MINIMIZED  ITEM LET HT  02T  304T FOR T GEQ 0 BE THE IMPULSE    RESPONSE OF A DISCRETETIME SYSTEM WITH ZERO INITIAL CONDITIONS    IT IS DESIRED TO DETERMINE A CAUSAL INPUT SEQUENCE XT SUCH    THAT THE OUTPUT YT  HTXT SATISFIES THE CONSTRAINTS BEGINALIGNEDY10  5 SUMJ010 YJ  2ENDALIGNEDAND SUCH THAT THE INPUT ENERGY SUMK010 XT2 ISMINIMIZED  FORMULATE THIS AS A DUAL APPROXIMATION PROBLEM AND FINDTHE MINIMIZING SEQUENCE XT  EXSKIP SETEXSECTREFSECLS2ITEM CITELUENBERGER1969  USING THE PROJECTION THEOREM SOLVE THE  FINITE DIMENSIONAL PROBLEM BEGINALIGNEDTEXTMINIMIZE   XBFH Q XBF TEXTSUBJECT TO   A XBF  BBFENDALIGNEDWHERE XBF IN CBBN Q IS A POSITIVE DEFINITE SYMMETRIC MATRIXAND A IS AN MATSIZEMN MATRIX WITH M  NITEM CITELUENBERGER1969 LET XBF BE A VECTOR IN A HILBERT SPACE  S AND LET  XBF1 XBF2LDOTSXBFN AND   YBF1YBF2LDOTS YBFM BE SETS OF LINEARLY INDEPENDENT  VECTORS IN S  WE DESIRE TO MINIMIZE  XBF  XBFHAT WHERE  THE NORM IS THE INDUCED NORM WHILE SATISFYING XBFHAT IN M  LSPAN XBF1XBF2LDOTS XBFMAND LA XBFHAT YBFIRA  CI I12LDOTS MBEGINENUMERATEITEM FIND EQUATIONS FOR THE SOLUTION WHICH ARE SIMILAR TO THE NORMAL  EQUATIONS  ITEM GIVE A GEOMETRIC INTERPRETATION OF THE EQUATIONS ENDENUMERATE ITEM CITELUENBERGER1969 CONSIDER THE PROBLEM OF FINDING THE VECTOR   XBF OF MINIMUM NORM SATISFYING LA XBFYBFIRA GEQ CI FOR   I12LDOTS N WHERE THE YBFI ARE LINEARLY INDEPENDENT   BEGINENUMERATE   ITEM SHOW THAT THIS PROBLEM HAS A UNIQUE SOLUTION   ITEM SHOW THAT A NECESSARY AND SUFFICIENT CONDITION THAT  XBF  SUMI1N AI YBFI  IS A SOLUTION IS THAT THE VECTOR ABF WITH COMPONENTS AI SATISFY  RT ABF GEQ CBF QQUAD TEXTANDQQUAD ABF GEQ ZEROBF  AND THAT AI  0 IF LA XBFYBFIRA  CI  R IS THE GRAMMIAN MATRIX OF YBF1YBF2LDOTS YBFN   ENDENUMERATEEXSKIPSETEXSECTREFSECGFSITEM LABELEXORTHOGDFT SHOW THAT THE FUNCTIONS DEFINED BY PKT FRAC1SQRTN EJ2PI KTNARE ORTHONORMAL WITH RESPECT TO THE INNER PRODUCT LA XTYTRA  SUMK0N1 XTYBARTITEM LET GT  ET2 FOR 0 LEQ T LEQ PI AND LET FT BE  THE PIPERIODIC EXTENSION OF GT FT  SUMK GTKPIBEGINENUMERATEITEM FIND THE FOURIER SERIES COEFFICIENTS OF FTITEM FIND THE SUM OF THE SERIES SUMN LEFTFRAC22PI2 FRAC11 16N2RIGHTHINT USE PARSEVALS THEOREMENDENUMERATEEXSKIPSETEXSECTREFSECCOFITEM PROPERTIES OF THE BERNSTEIN POLYNOMIALS AND RELATED FORMULAS  PROVE THE PROPERTIES REFEQBPROP1 REFEQBPROP2 AND  REFEQBPROP3   HINT USING THE BINOMIAL THEOREM SUMJ0N N CHOOSE J XJ YNJ  XYNSHOW THAT  SUMJ0N JNN CHOOSE J XJ YNJ     XXYN1AND SUMJ0N JN2 N CHOOSE J XJ YNJ  11N    X2 XYN2  XNXYN1 ITEM SHOW THAT THE LEGENDRE POLYNOMIAL PNT IS A SOLUTION TO THE   DIFFERENTIAL EQUATION  1T2Y  NN1Y  0   ITEM LABELEXCHEBPOLY1 SHOW THAT THE ORTHOGONALITY RELATION FOR     CHEBYSHEV POLYNOMIALS IN REFEQCHEBORTHOG IS TRUEITEM USING REFEQCHEB1 DETERMINE T2T AND T3TITEM SHOW THAT THE DEFINITION OF CHEBYSHEV POLYNOMIALS  REFEQCHEB1 SATISFIES THE RECURRENCE IN REFEQCHEBRECURR  FOR T 1 SHOW FOR T1 THAT TNT  COSHNCOSH1T SATISFIES  THE RECURSION REFEQCHEBRECURRITEM BF THE CHRISTOFFELDARBOUX FORMULA  BEGINENUMERATE  ITEM USING REFEQPOLYRECURR SHOW THAT THE POLYNOMIALS    PKT ORTHOGONAL WITH RESPECT TO THE INNER PRODUCT LA FGRAW     INTAB FTGTDT SATISFY INTAB PNT PN1TWTDT  ANALSO SHOW THAT CN  AN1ITEM CONSIDER THE PARTIAL SUM SNT  SUMK0N LA FPKRAW PKTSHOW THAT THE SUM CAN BE WRITTEN AS SNT INTAB FY KNXY WYDYWHERE KNXY  FRACANPN1XPNY  PNXPN1YXYAND WHERE AN COMES FROM REFEQPOLYRECURR  WALTER P 78THIS FORMULA FOR KNXY IS KNOWN AS THE EM CHRISTOFFELDARBOUXFORMULA AND IS ANALOGOUS TO THE DIRICHLET KERNEL OF FOURIERSERIES  HINT FORM XYKXY AND USE THE RESULTS FROM PART AINDEXDIRICHLET KERNEL INDEXCHRISTOFFELDARBOUX FORMULA SEE SECTION REFSECDIRICHLETKERNEL  ENDENUMERATEITEM IT IS ALSO POSSIBLE TO DEFINE ORTHOGONAL POLYNOMIALS OF A  DISCRETE VARIABLE USING THE INNER PRODUCT LA FGRA  SUMI WXI FXIGXIWHERE THE XI ARE INTEGERS IN THE INTERVAL A LEQ XI LEQ B AND WXI 0  THIS AMOUNTS TO DEFINING THE INNER PRODUCT USINGDELTA FUNCTIONS IN THE INTEGRAL  ITEM SHOW THAT THE EACH OF THE POLYNOMIALS PRODUCED BY  ORTHOGONALIZING 1TT2LDOTS USING THE GRAMSCHMIDT  PROCEDURE OVER THE INTERVAL AB HAS ZEROS WHICH ARE REAL  SIMPLE AND LOCATED IN AB ITEM RECURRENCE FOR ORTHOGONAL POLYNOMIALS   BEGINENUMERATE   ITEM SHOW THAT THE LEGENDRE POLYNOMIALS SATISFY THE RECURSION  PN1T  FRAC2N1N1 T PNT  FRACNN1 PN1T  USE THIS RECURRENCE TO COMPUTE P3T P4T AND P5T ITEM THE CHEBYSHEV POLYNOMIALS SATISFY THE RECURSION   TN1T  2T TNT  TN1  USE THIS RECURRENCE TO FIND T3T T4T AND T5T   ENDENUMERATEITEM IN THIS EXERCISE WE INTRODUCE THE IDEA OF EM GAUSSIAN    QUADRATURE INDEXGAUSSIAN QUADRATURE A FAST AND IMPORTANT  METHOD OF NUMERICAL INTEGRATION  INDEXNUMERICAL INTEGRATIONSEEGAUSSIAN QUADRATURE THE IDEA IS TO APPROXIMATE THE  INTEGRAL AS A SUMMATION INTAB FT DT APPROX SUMI1M AI FTIUNLIKE MANY CONVENTIONAL NUMERICAL INTEGRATION FORMULAS IN GAUSSIANQUADRATURE THE ABSCISSAS ARE NOT EVENLY SPACED  THE PROBLEM IS TOFIND THE TI ABSCISSAS AND AI WEIGHTS SO THATTHE INTEGRAL IS AS ACCURATE AS POSSIBLE  IN THE GAUSSIAN QUADRATUREMETHOD OF NUMERIC INTEGRATION FOR POLYNOMIALS UP TO DEGREE 2M1THE RESULT OF THE INTEGRATION IS EM EXACT  FOR SUFFICIENTLY SMOOTHNONPOLYNOMIAL FUNCTIONS THE METHOD IS OFTEN VERY ACCURATE  THESOLUTION MAKES SIGNIFICANT USE OF ORTHOGONAL POLYNOMIALS   FORPURPOSES OF THIS EXERCISE WE WILL ASSUME THE INNER PRODUCT LAFGRA  INT11 FTGTDTBEGINENUMERATEITEM  AS THIS FIRST PART SHOWS WITHOUT  LOSS OF GENERALITY WE MAY  RESTRICT ATTENTION TO THE INTERVAL A1 B1SHOW THAT FOR THE INTEGRAL INTAB GXDXTHE SUBSTITUTION  T  FRAC1BA2X  ABLEADS TO AN INTEGRAL OF THE FORM INT11 FTDTHENCE THE LIMITS OF A AND B CAN BE CONVERTED TO LIMITS OF 1 TO 1ITEM IF PNT IS A SET OF POLYNOMIALS ORTHOGONAL OVER  11 WHERE PNT IS A POLYNOMIAL OF DEGREE N SHOW THAT LA PTPMTRA  0FOR ALL POLYNOMIALS PT OF DEGREE LEQ M1ITEM LET FT BE A POLYNOMIAL OF DEGREE 2M1  SHOW THAT FT  CAN BE WRITTEN AS FT  QTPNT  RTWHERE QT AND RT ARE OF DEGREE LEQ M1  HINT DIVIDEITEM SHOW THAT THERE ARE SERIES EXPANSIONS QT  SUMK0M1 ALPHAK PKT QQUADTEXTAND QQUAD   RT  SUMK0M1 BETAK PKTITEM SHOW THATBEGINEQUATION INT11 FTDT  BETA0 INT11 P0TDTLABELEQGINT3ENDEQUATIONITEM LET T1 T2LDOTS TM BE THE ROOTS OF PMT  SHOW THATBEGINEQUATION SUMI1M AI FTI  SUMK0M1 BETAK SUMI1M AIPKTILABELEQGINT4ENDEQUATION BEGINALIGNED   SUMI1M AI FTI  SUMI1M AI QTIPMTI  RTI   SUMI1M AI RTI  SUMI1M AI SUMK0M1 BETAK PKTI   SUMK0M1 BETAK SUMI1M AI PKTI ENDALIGNED ITEM SHOW THAT IF THE WEIGHTS AI ARE CHOSEN SO THAT SUMI1M AI PKTI  BEGINCASES  INT11 P0TDT  K0   0  K12LDOTS N1ENDCASESTHEN REFEQGINT4 CAN BE WRITTEN ASBEGINEQUATION SUMI1M AI FTI  B0 INT11 P0TDTLABELEQGINT5ENDEQUATIONITEM WRITE REFEQGINT5 AS A MATRIX EQUATION FOR THE WEIGHTS  AIITEM HENCE EQUATING REFEQGINT3 AND REFEQGINT5 WRITE  DOWN THE FORMULA FOR GAUSSIAN QUADRATUREITEM GENERALIZE THIS TO FINDING INT11 WTFTDT WHERE THE  POLYNOMIALS PKT ARE ORTHOGONAL WITH RESPECT TO THE INNER  PRODUCT LA FGRA  INT11 FTGTWTDTENDENUMERATE  ITEM PROVE PARSEVALS THEOREM FOR FOURIER TRANSFORMS IF Y1T    LEFTRIGHTARROW Y1OMEGA AND Y2T LEFTRIGHTARROW    Y2OMEGA THEN INTINFTYINFTY Y1T YBAR2TDT  FRAC12PIINTINFTYINFTY Y1OMEGA YBAR2OMEGADOMEGAITEM LABELEXSINCORTHOG SAMPLING THEOREM REPRESENTATIONS  BEGINENUMERATE  ITEM SHOW FOR PKT DEFINED AS IN  REFEQSINCSHIFT THAT LA PK PLRA   FRAC12BDELTAKL    ALONG THE WAY SHOW THAT INTINFTYINFTY FRACSIN TT FRACSINTZTZ DT FRACPI SIN ZZHINT USE PARSEVALS THEOREM AND FOURIER TRANSFORMSITEM SHOW THAT REFEQSINCSAMP IS CORRECT  FOR A BANDLIMITED FUNCTION FTITEM SHOW THAT IF FT IS BANDLIMITED TO B HZ  FZ  2BINTINFTYINFTY FT P0TZDTTHUS FOR BANDLIMITED FUNCTIONS P0T BEHAVES LIKE A DELTAFUNCTION  ENDENUMERATEEXSKIPITEM SHOW THAT IF PHIT IS NORMALIZED THEN 2J2PHI2J  T IS NORMALIZEDITEM IN REFEQTWOSCALE2 SHOW THAT THE COEFFICIENTS CN MUST  SATISFY SUMN CN  2 ITEM USING REFEQWAVEORTHOG1 SHOW THAT   BEGINENUMERATE   ITEM THE SET OF FUNCTIONS   2J2PHI2J T  N N IN ZBB  FORMS AN ORTHOGONAL SET FOR EACH   FIXED J ITEM THE SET OF FUNCTIONS  2J2 PSI2JT  N N IN ZBB FORMS AN   ORTHOGONAL SET FOR EACH FIXED J     ENDENUMERATEITEM SHOW THAT THERE IS NO ORTHOGONAL SCALING FUNCTION DEFINED BY A  TWOSCALE EQUATION REFEQTWOSCALE2 WITH EXACTLY THREE NONZERO  COEFFICIENTS C0 C1 AND C2  ITEM FOR THE MULTIRESOLUTION ANALYSIS    BEGINENUMERATE    ITEM SHOW THAT WJ PERP WJ    ITEM SHOW THAT FOR J  J VJ   VJ OPLUS BIGOPLUSK0JJ1 WJK    ENDENUMERATE  ITEM SHOW THAT IF PHIT OBEYS THE TWOSCALE RELATIONSHIP IN    REFEQTWOSCALE1 AND IF PHIHATOMEGA REPRESENTS THE    FOURIER TRANSFORM OF PHIT THEN PHIHATOMEGA  M0OMEGA2PHIHATOMEGA2WHERE BEGINEQUATIONM0OMEGA  FRAC1SQRT2 SUMN HN EJNOMEGALABELEQM0ENDEQUATIONIS THE SCALED DISCRETETIME FOURIER TRANSFORM OF THE COEFFICIENTSEQUENCEITEM BF DECIMATION INDEXDECIMATION INDEXMULTIRATE PROCESSINGBECAUSE OF THE CONNECTION OF  WAVELET TRANSFORMS WITH MULTIRATE SIGNALING IT IS WORTHWHILE TO  EXAMINE THE TRANSFORM OF DECIMATED SIGNALS  YOU WILL SHOW THAT IF  YN IS A DECIMATION OF XN YN  XNDTHENBEGINEQUATIONYZ  FRAC1D SUMK0D1 XEJ2PI KDZ1DLABELEQDECIMATEENDEQUATIONBEGINENUMERATEITEM LET PN BE THE PERIODIC SAMPLING SEQUENCE PN  BEGINCASES  1  N  0 PM D PM 2D LDOTS   0  TEXTOTHERWISEENDCASESSHOW THAT PN  FRAC1DSUMK0D1 EJ2PI KNDITEM LET ZN  XNPN  THEN YN  ZND  SHOW THAT YZ  SUMM YM ZM  SUMM ZMITEM FINALLY SHOW THAT REFEQDECIMATE IS TRUEENDENUMERATEITEM SHOW THAT THE ORTHOGONALITY CONDITION REFEQWAVEORTHOG1  IS EQUIVALENT TO  M0OMEGA22  M0OMEGA2PI2  1HINT RECOGNIZE THAT REFEQWAVEORTHOG1 IS A DECIMATEDCONVOLUTION AND USE THE FACT THAT IF THE FOURIER TRANSFORM OF ASEQUENCE ZN IS ZEJOMEGA THEN THE FOURIER TRANSFORM OF Z2NIS FRAC12ZEJOMEGA2  ZEJOMEGA2PI ITEM COMPUTER EXERCISE  IN THIS EXAMPLE YOU WILL BE INTRODUCED TO A   RUDIMENTARY APPROACH TO DATA COMPRESSION USING WAVELETS  WRITE A   PROGRAM WHICH WAVELET TRANSFORMS DATA THEN TRUNCATES THE DATA USING   A PRESET THRESHOLD THEN INVERSE TRANSFORMS THE DATA  USING   SAMPLED SPEECH OR MUSIC DATA EXPLORE THE QUALITY OF THE   INVERSETRANSFORMED DATA AS A FUNCTION OF THE THRESHOLD  DETERMINE   HOW MANY COEFFICIENTS ARE SET TO ZERO AS A FUNCTION OF THE THRESHOLDEXSKIPITEM LABELEXMF1 LET PHIT BE A ONEDIMENSIONAL BASIS FUNCTION  FOR DIGITAL TRANSMISSION OF THE FORM PHIT  UT  UTTA UNIT PULSE  ASSUME THAT ST  PHIT IS TRANSMITTED  LETRT  ST NOISEFREE RECEPTION  SHOW THE OUTPUT OF THECORRELATOR Y1T  INT0T RUPHIUDUAND THE OUTPUT OF THE MATCHED FILTER WITH IMPULSE RESPONSE HT PHITT  Y2T  RTHTSHOW THAT AT THE SAMPLE INSTANT T  T Y1T  Y2TITEM FOR THE BASIS SIGNALS SHOWN IN FIGURE REFFIGMFEX2 DRAW A  SIGNAL CONSTELLATION  SUCH A SIGNALING TECHNIQUE IS CALLED EM    PULSEPOSITION MODULATIONITEM LET  PHIMT  BEGINCASESCOS2PI FC   2PI M DELTA FT  0 LEQ T LEQ T 0  TEXTOTHERWISEENDCASESFOR M01LDOTSM1 BE A SET OF BASIS FUNCTIONS  DETERMINE THE  MINIMUM FREQUENCY SEPARATION DELTA F SUCH THAT INT0T PHIMT PHIKTDT  0FOR K NEQ M  ASSUME THAT FC T  N FOR SOME INTEGER NDIGITAL TRANSMISSION WITH SUCH SIGNALS IS CALLED FREQUENCYSHIFTKEYING INDEXFREQUENCYSHIFT KEYINGITEM SPREADSPECTRUM MULTIPLE ACCESS  IN THIS EXERCISE WE EXAMINE  MATCHED FILTERS FOR A MORE COMPLICATED SCENARIO SPREAD SPECTRUM  MULTIPLE ACCESS INDEXSPREAD SPECTRUM MULTIPLE ACCESS  IN THIS MODEL  K USERS ARE TRANSMITTING  SIMULTANEOUSLY WITH THE KTH USER TRANSMITTING A SIGNAL SKT  SUMN BKN SQRT2 WK PHIKTNTWHERE PHIKT IS THE KTH USERS UNIQUE WAVEFORM A SIGNAL WITHSUPPORT OVER 0T  THE RECEIVED SIGNAL CONSISTS OF THE SUM OF EACHUSERS DELAYED SIGNAL APPEARING IN ADDITIVE NOISE RT  SUMK1K SUMN BKN WK PHIKTNT TAUK  ZTTHE USERS BASIS FUNCTIONS ARE EM NOT NECESSARILY ORTHOGONALASSUME THAT THE USERS ARE ORDERED SO THAT TAU1 LEQ TAU2 LEQCDOTS LEQ TAUK  T  A MATCHEDFILTER OR CORRELATOR OUTPUT ISOBTAINED FOR EACH USER OVER THE NTH BIT INTERVAL AS YKN  INTINFTYINFTY RT PHIKTNT TAUKLET YBFN  Y1NY2NLDOTSYKNT BE THE VECTOR OFMATCHED FILTER OUTPUTS FOR ALL USERS AT INTERVAL N  BEGINENUMERATEITEM SHOW THAT YBFN  H1BN1  H0BN  H1BN1WBF  ZBFNWHERE HM IS A CORRELATION MATRIX WITH ELEMENTS HIJM  INTINFTYINFTY PHIITTAUIPHIJTMTTAUJDTB IS A DIAGONAL MATRIX OF BITS BN  DIAGB1NB2NLDOTSBKN WBF  W1W2LDOTSWKT AND ZBFN Z1NZ2NCDOTS ZKNT WHERE ZKN  INT ZT PHIKT  NT  TAUK DTITEM IF ZT IS WHITE WITH EZTZTS  SIGMAZ2 DELTATS  SHOW THAT ZBFN SATISFIES EZBFN ZBFTM  BEGINCASES  SIGMAZ2 H0  NM   SIGMAZ2 H1  N  M1   SIGMAZ2 H1  N  M1   0  TEXTOTHERWISEENDCASESENDENUMERATEENDEXERCISESSECTIONREFERENCESTHE HILBERT APPROXIMATION THEORY PRESENTED HERE IS SUMMARIZED FROMCITELUENBERGER1969 AND CITEKEENER  SOME OF THE DISCUSSION ABOUTTHE GRAMMIAN MATRIX WAS DRAWN FROM CITESCHARFL1991THE VARIOUS WINDOWING METHODS ARE DESCRIBED IN CITECHAPTER11HAYKIN1996  A DISCUSSION OF LEASTSQUARES ANDMINIMUM MEANSQUARES FILTERING IS INCITEHAYKIN1996PROAKISRADERSCHARFL1991 OUR DISCUSSION OF WIENER FILTERING IS DRAWN FROMCITEKAILATHFILTBOOK AND CITESOLODOVNIKOV  A THOROUGH DISCUSSIONOF THE SPECTRAL FACTORIZATION PROBLEM APPEARS IN CITEPAPOULIS1977THE GRAMSCHMIDT PROCEDURE IS DISCUSSED IN MOST BOOKS ON LINEARALGEBRA  SPECIFIC RESULTS ON NUMERIC ACCURACY OF THE METHOD CAN BEFOUND IN CITEGVLSEVERAL VARIANTS ON LEASTSQUARES AND CONSTRAINED LEASTSQUARESINCLUDING PSEUDOCODE FOR SEVERAL USEFUL ALGORITHMS ARE INCITELAWSONHANSENORTHOGONAL FUNCTIONS ARE WIDELY DISCUSSED IN CITEABRAMOWITZINCLUDING AN EXTENSIVE TABLE OF POLYNOMIALS ORTHOGONAL WITH RESPECT TOMANY WEIGHTING FUNCTIONS AND THEIR PROPERTIES  IN ADDITION TOORTHOGONAL POLYNOMIALS IN CONTINUOUS TIME THERE ARE ALSO ORTHOGONALPOLYNOMIALS IN DISCRETE VARIABLES  THESE ARE SUMMARIZED INCITEABRAMOWITZ AND EXAMINED MORE THOROUGHLY INCITEERDELYI1953 AND CITESZEGO1967  A RECENT BOOK DESCRIBING AVARIETY OF ORTHOGONAL FUNCTIONS AND THEIR SMOOTHNESS PROPERTIES ISCITEWALTER1994THE USE OF THE FUNCTION SINXX THE SINC FUNCTION AS ANORTHOGONAL BASIS IS INTRODUCED IN CITEKEENER  AN EXTENSIVEDISCUSSION OCCURS IN CITESTENGERBOOK AND CITESTENGERPAPERTHERE HAS BEEN AN EXPLOSION OF LITERATURE ON WAVELETS AND WAVELETTRANSFORMS  THE DEFINITIVE REFERENCE IS PROBABLYCITEDAUBECHIES1992 SEE ALSO CITEDAUBECHIES3  OTHER BOOKS WITHBROAD COVERAGE CITECHUI1992MALLAT1998  AMONG THE GENERALIZATIONSDISCUSSED IN THESE BOOKS ARE BIORTHOGONAL WAVELETS IN WHICH DIFFERENTFILTERS ARE USED TO RECONSTRUCT THE SIGNAL THAN TO ANALYZE ITWAVELET PACKETS CHOOSING DIFFERENT TREES OF COEFFICIENTS ANDSEVERAL OTHER FAMILIES OF WAVELETS  A RECENT TUTORIAL ISCITEBURRUSGOPINATH  A THOROUGH DISCUSSION OF IMPLEMENTATION OFWAVELET TRANSFORMS AND A VARIETY OF OTHER USEFUL TRANSFORMS AS WELLIS PROVIDED IN CITEWICKERHAUSER1994  A DEFINITIVE REFERENCE ONMULTIRATE SIGNAL PROCESSING IS CITEVAIDYANATHAN1993 FOR A SOLIDINTRODUCTION TO THIS AREA SEE CITEVAIDYANATHAN1990IRLS IS DISCUSSED IN CITEBURRUS1994 AND REFERENCES THEREIN WHERETHE NUMBER OF ITERATIONS REQUIRED TO DESIGN A FILTER IS CLOSELYEXAMINED  AN ALTERNATIVE VIEWPOINT ON ESTIMATION USING THE L1NORM INDEXL1NORM ESTIMATIONL1NORM ESTIMATION FOR SPECTRALESTIMATION IS INVESTIGATED INCITESCHROEDER1989SCHROEDER1990DENOEL1985  WARD1984  A MORE THOROUGH TREATMENT IS PRESENTED INCITEBLOOMFIELD1983THE VECTOR SPACE VIEWPOINT SIGNAL CONSTELLATIONS AND MATCHED FILTERSAND ARE PRESENTED IN EVERY TEXT ON DIGITAL COMMUNICATIONS  SEE FOREXAMPLE CITEWOZENCRAFT CITEPROAKIS3RDED OR CITEBLAHUTCOMMA HISTORICAL TREATMENT OF ORTHOGONAL FUNCTIONS USED IN SIGNALING ISGIVEN IN CITEHARMUTH WHICH ALSO PRESENTS OTHER USEFUL ORTHOGONALFUNCTIONS OTHER THAN THOSE PRESENTED HERETHERE IS A TREMENDOUS LITERATURE ON ORTHOGONAL POLYNOMIALS  A RECENTSURVEY IS CITEWALTER1994  A CLASSIC REFERENCE IS CITESZEGO1967ADDITIONAL INFORMATION IS FOUND IN CITEABRAMOWITZ LOCAL VARIABLES TEXMASTER TEST ENDCHAPTERSOME IMPORTANT MATRIX FACTORIZATIONSLABELCHAPMATFACTTHERE ARE SOME MATRIX FACTORIZATIONS THAT ARISE COMMONLY ENOUGH INMATRIX ANALYSIS IN GENERAL AND IN SIGNAL PROCESSING IN PARTICULARTHAT THEY WARRANT SOME SPECIFIC ATTENTION  IN THIS CHAPTERFACTORIZATIONS ARE DISCUSSED WHICH FORM THE HEART OF MANY SIGNALPROCESSING ROUTINES  THE FACTORIZATIONS PRESENTED IN THIS CHAPTER AREAS FOLLOWSBEGINDESCRIPTIONITEMLU A SQUARE MATRIX A CAN BE FACTORED AS A  LU WHERE L  IS A LOWER TRIANGULAR MATRIX WITH ONES ON THE MAIN DIAGONAL AND U  IS UPPER TRIANGULAR  ITS MAIN APPLICATION IS IN THE NUMERICAL  SOLUTION OF THE PROBLEM AXBF  BBF INDEXMATRIX FACTORIZATIONSLUINDEXLU FACTORIZATIONINDEXMATRIX FACTORIZATIONSSEESVDITEMCHOLESKY A HERMITIAN SYMMETRIC POSITIVE DEFINITE MATRIX A  CAN BE FACTORED AS   A  LLH  INDEXSQUARE ROOTOF A MATRIX  WHERE L IS LOWER TRIANGULAR  THE CHOLESKY FACTORS OF A MATRIX MAY  BE REGARDED AS THE SQUARE ROOT OF THE MATRIX A  CLOSELY  RELATED IS THE FACTORIZATION A  LDLH WHERE D IS DIAGONAL OR  A  UDUH WHERE UU IS UPPER TRIANGULAR  THE CHOLESKY  FACTORIZATION IS USED IN SIMULATION TO COMPUTE A VECTOR NOISE OF  DESIRED COVARIANCE AND IN SOME ESTIMATION AND KALMAN FILTERING  ROUTINESINDEXMATRIX FACTORIZATIONSCHOLESKYINDEXCHOLESKY FACTORIZATIONITEMQR A GENERAL MATRIX A CAN BE FACTORED AS   A  QR    WHERE Q IS A UNITARY MATRIX QQH  I AND R IS UPPER  TRIANGULAR  THE QR FACTORIZATION IS USED IN THE SOLUTION OF  LEASTSQUARES PROBLEMSINDEXMATRIX FACTORIZATIONSQRINDEXQR FACTORIZATIONENDDESCRIPTIONA FACTORIZATION IMPORTANT ENOUGH TO WARRANT ITS OWN CHAPTER IS THESINGULAR VALUE DECOMPOSITION SVD IN WHICH A IS FACTORED AS A  USIGMA VHWHERE U AND V ARE UNITARY AND SIGMA IS DIAGONAL  THE SVD ANDITS APPLICATIONS IS PRESENTED IN CHAPTER REFCHAPSVD INDEXMATRIX FACTORIZATIONSSVD INDEXSVDINPUTLINALGDIRLUFACTSECTIONTHE CHOLESKY FACTORIZATIONLABELSECCHOLESKYINDEXMATRIX FACTORIZATIONSCHOLESKY INDEXCHOLESKY FACTORIZATIONTHE CHOLESKY FACTORIZATION IS USED TO COMPUTE A SQUARE ROOTINDEXMATRIX SQUARE ROOT OF A POSITIVEDEFINITE MATSIZEMMHERMITIAN MATRIX AS B  LLHWHERE L IS LOWER TRIANGULAR  OCCASIONALLY THE L MATRIX ISNORMALIZED TO PRODUCE A MATRIX LTILDE THAT IT HAS ONES ALONG THEMAIN DIAGONAL AND THE SCALING FACTOR IS INCORPORATED IN A DIAGONALMATRIX FACTOR AS LH  BEGINBMATRIXL11   L22   DDOTS  LMMENDBMATRIXLTILDEH  SQRTD UTHEN WE CAN WRITE B  UHD UWHERE D  DIAGL112 L222 LDOTS LMM2 BEGINEXAMPLE FOR THE B SHOWN WE HAVEBEGINALIGNEDB  BEGINBMATRIX 4  8  12 8 20   20 12  20  41 ENDBMATRIX  BEGINBMATRIX2  0  0 4  2  0 6  2  1 ENDBMATRIXBEGINBMATRIX2  4  6 0  2  2 0  0  1 ENDBMATRIX  LLT  BEGINBMATRIX1  0  0 2  1  0 3  1  1 ENDBMATRIXBEGINBMATRIX4  0  0 0  4  0 0  0  1 ENDBMATRIXBEGINBMATRIX1  2  3 0  1  1 0  0  1 ENDBMATRIX  UT D UENDALIGNED ENDEXAMPLEIF THE CHOLESKY FACTORIZATION DOES NOTEXIST SAY AS DETERMINED BY THE ALGORITHM BELOW THEN TO THEPRECISION AVAILABLE THE MATRIX B IS NOT POSITIVE DEFINITEBEGINEXAMPLE INDEXGAUSSIAN RANDOM NUMBER  IN A SIMULATION OF A SIGNAL PROCESSING ALGORITHM IT IS NECESSARY TO  GENERATE GAUSSIAN RANDOM VECTORS WITH COVARIANCE R  SYSTEM  LIBRARIES OFTEN PROVIDE GENERATORS WHICH SIMULATE INDEPENDENT  NC01 RANDOM VARIABLES  THESE CAN BE USED TO GENERATE  NC0R RANDOM VECTORS AS FOLLOWS  FIRST FACTOR R AS R  LLT WHERE L IS LOWER TRIANGULAR  FOR EACH RANDOM VECTOR DESIRED CREATEA VECTOR XBF OF NC01 INDEPENDENT RANDOM VARIABLES USING THEGAUSSIAN RANDOM NUMBER GENERATOR AND LET ZBF  L XBFTHEN SINCE EXBFXBFT  I EZBFZBFT  LEXBFXBFTLT  LLT  RSO ZBF HAS THE DESIRED COVARIANCEENDEXAMPLEBEGINEXAMPLE  THE CHOLESKY FACTORIZATION CAN BE USED TO SOLVE SYSTEMS OF  EQUATIONS  FOR THE EQUATION A XBF BBFWHERE A IS HERMITIAN AND POSITIVE DEFINITE WRITE A  LLH SOLUTION THEN REQUIRES SOLVING THE TWO SETS OF TRIANGULAR SYSTEMS BEGINALIGNEDLYBF  BBF LH XBF   YBFENDALIGNEDMUCH AS WAS DONE FOR THE LU DECOMPOSITIONENDEXAMPLEBEGINEXAMPLE APPLICATION OF CHOLESKY FACTORIZATION TO NORMAL  EQUATIONS  THE LEASTSQUARES SOLUTION REFEQLSMAT1 AH A XBF  AH BBFCAN BE SOLVED USING THE CHOLESKY FACTORIZATION WHERE AHA  LLHLET AHBBF  PBF  THEN FIRST SOLVE BY SUBSTITUTION LYBF  PBFTHEN SOLVE BY BACKSUBSTITUTION LH XBF  YBFSOLVING THE NORMAL EQUATIONS USING THE CHOLESKY FACTORIZATION ISSOMETIMES CALLED THE NORMAL EQUATION APPROACH INDEXLEASTSQUARESNORMAL  EQUATION APPROACHENDEXAMPLEWE WILL SEE IN SECTION REFSECQRFACT THAT THE QR DECOMPOSITION CANBE USED TO SOLVE LEASTSQUARES PROBLEMS  WHY THEN WOULD WE CONSIDERUSING THE CHOLESKY FACTORIZATION  IN FAVOR OF USING THE QR COMPUTINGAHA REQUIRED TO USE THE CHOLESKY FACTORIZATION REQUIRES A GOODDYNAMIC RANGE CAPABILITY ESSENTIALLY DOUBLE THE WORD SIZE FOR AFIXEDPOINT REPRESENTATION IN ORDER TO NOT BE HURT BY AN INCREASE INCONDITION NUMBER  ON THE OTHER HAND FOR AN MATSIZEMN MATRIXA IF M GG N THEN AHA AND ITS FACTORIZATIONS WILL REQUIRELESS STORAGE AND APPROXIMATELY HALF THE COMPUTATION OF THE QRREPRESENTATION  IN THIS CASE IF IT CAN BE DETERMINED THAT THE SYSTEMOF EQUATIONS IS SUFFICIENTLY WELL CONDITIONED SOLUTION USING CHOLESKYFACTORIZATION MAY BE JUSTIFIEDTHE CHOLESKY FACTORIZATION IS ALSO USED IN SQUARE ROOT KALMANINDEXSQUARE ROOT KALMAN FILTER FILTERING APPLICATIONS WHICH ARENUMERICALLY STABLE METHODS OF COMPUTING KALMAN FILTER UPDATES  SEEEG CITEVERHAEGEN SUBSECTIONALGORITHMS FOR COMPUTING THE CHOLESKY FACTORIZATIONTHERE ARE SEVERAL ALGORITHMS WHICH CAN BE USED TO COMPUTE THE CHOLESKYFACTORIZATION WHICH ARE MENTIONED FOR EXAMPLE IN CITEGVL  THEALGORITHM PRESENTED REQUIRES M33 FLOATING OPERATIONS AND REQUIRESNO ADDITIONAL STORAGE  THE ALGORITHM IS DEVELOPED RECURSIVELY  WRITE B  BEGINBMATRIX ALPHA  VBFH  VBF  B1 ENDBMATRIXAND NOTE THAT IT CAN BE FACTORED ASBEGINEQUATION B  BEGINBMATRIX SQRTALPHA  0  VBFSQRTALPHA   IN1 ENDBMATRIX BEGINBMATRIX1  0  0  B1  VBF VBFHALPHA ENDBMATRIXBEGINBMATRIX SQRTALPHA  VBFHSQRTALPHA  0  IN1ENDBMATRIXLABELEQBCHOLENDEQUATIONIF WE COULD FIND THE CHOLESKY FACTORIZATION OF B1  VBFVBFHALPHAAS G1G1H WE WOULD HAVE B  BEGINBMATRIX SQRTALPHA  0  VBFSQRTALPHA  G1ENDBMATRIX BEGINBMATRIX SQRTALPHA  VBFHSQRTALPHA   0  G1H ENDBMATRIX  GGHWE THEREFORE PROCEED RECURSIVELY DECOMPOSING B INTO SUCCESSIVELYSMALLER BLOCKS  THE SC MATLAB CODE IS DEMONSTRATED IN ALGORITHMREFALGCHOLESKY FOR DEMONSTRATION PURPOSES SINCE SC MATLAB HASA BUILTIN CHOLESKY FACTORIZATION VIA THE FUNCTION TT CHOLBEGINNEWPROGENVCHOLESKY    FACTORIZATIONCHOLESKYMCHOLESKYCHOLESKY    FACTORIZATIONENDNEWPROGENVBEGINEXERCISESITEM COMPUTE THE CHOLESKY FACTORIZATION OF  A  BEGINBMATRIX 464  62518  41822 ENDBMATRIXAS ALLT  THEN WRITE THIS AS A  UTDU WHERE U IS AN UPPER TRIANGULAR MATRIXWITH 1 ALONG THE DIAGONALITEM SHOW THAT REFEQBCHOL IS TRUEITEM GIVEN A ZEROMEAN DISCRETETIME INPUT SIGNAL FT WHICH WE  FORM INTO A VECTOR QBFT  BEGINBMATRIX FBART  FBART1  CDOTS   FBARTMENDBMATRIXTWE DESIRE TO FORM A SET OF OUTPUTS BBFT  BEGINBMATRIX B0T  B1T  CDOTS  BMTENDBMATRIXTBY BBFT  HQBFT THAT ARE UNCORRELATED THAT IS EBIT BBARJT  0 TEXT IF INEQ JLET R  EQBFT QBFBART BE THE CORRELATION MATRIX OF THE INPUTDATA  BEGINENUMERATEITEM DETERMINE THE MATRIX H WHICH DECORRELATES THE INPUT DATAITEM INTERPRET THE RESULTS AS A BANK OF  BACKWARD PREDICTORSENDENUMERATEITEM WRITE A SC MATLAB ROUTINE TT BACKCHOL WHICH FACTORS A  SYMMETRIC POSITIVE DEFINITE MATRIX AS A  UUH WHERE U IS AN  UPPER TRIANGULAR MATRIXITEM WRITE SC MATLAB ROUTINES TT FORSUBLYB AND TT    BACKSUBUYB TO SOLVE LYBF  BBF FOR A LOWER TRIANGULAR MATRIX    L AND U YBF  BBF FOR AN UPPER TRIANGULAR MATRIX U  ITEM DEVELOP A MEANS OF COMPUTING THE SOLUTION TO THE WEIGHTED    LEASTSQUARES PROBLEM REFEQWLS2 USING THE CHOLESKY    FACTORIZATION  ITEM LET X  XBF1 XBF2 LDOTS XBFN BE A SET OF    REALVALUED ZEROMEAN DATA WITH CORRELATION MATRIX RXX  FRAC1N XXTDETERMINE A TRANSFORMATION ON X THAT PRODUCES A DATA SET Y  SUCHTHAT RYY  FRAC1N YYTIS EQUAL TO AN IDENTITYENDEXERCISESSECTIONUNITARY MATRICES AND THE QR FACTORIZATIONLABELSECQRWE BEGIN WITH A DESCRIPTION OF THE Q IN THE QR FACTORIZATIONSUBSECTIONUNITARY MATRICESINDEXUNITARY MATRIX INDEXORTHOGONAL MATRIXBEGINDEFINITION  A MATSIZEMM MATRIX Q WITH COMPLEX ELEMENTS IS SAID TO BE  BF UNITARY IF  QHQ  IIF Q HAS REAL ELEMENTS AND QTQ  I THEN Q IS SAID TO BE AN BF  ORTHOGONAL MATRIXENDDEFINITIONFOR A UNITARY OR ORTHOGONAL MATRIX WE ALSO HAVE QQH  IBEGINLEMMA LABELLEMQRSAMENORMIF YBF  Q XBF FOR AN MATSIZEMM MATRIX Q  THEN YBF   XBF  FOR ALL XBF IN RBB IF AND ONLY IF  Q IS UNITARY WHERE THE NORM IS THE USUAL EUCLIDEAN NORMENDLEMMAA TRANSFORMATION WHICH DOES NOT CHANGE THE LENGTH OF A VECTOR IS SAIDTO BE BF ISOMETRIC OR LENGTHPRESERVING INDEXISOMETRIC THEPROOF OF THE LEMMA IS STRAIGHTFORWARD AND IS GIVEN AS AN EXERCISETHIS LEMMA ALLOWS US TO MAKE TRANSFORMATIONS ON VARIABLES EM  WITHOUT CHANGING THEIR LENGTH  THE LEMMA PROVIDES THE BASIS FORPARSEVALS THEOREM FOR FINITEDIMENSIONAL VECTORSBEGINLEMMA  LABELLEMQRSAMEFNORM  IF Y  QX FOR AN MATSIZEMM UNITARY  MATRIX Q THEN   YF  XFWHERE  CDOT F IS THE FROBENIUS NORMENDLEMMATHERE IS A USEFUL ANALOGY THAT CAN NOW BE INTRODUCEDBEGINDESCRIPTIONITEMHERMITIAN MATRICES SATISFYING AH  A ARE ANALOGOUS TO  REAL  NUMBERS NUMBERS WHOSE COMPLEX CONJUGATE IS EQUAL TO ITSELFITEMUNITARY MATRICES SATISFYING UHUI ARE ANALOGOUS TO COMPLEX  NUMBERS Z ON THE UNIT CIRCLE SATISFYING Z2  1ITEMORTHOGONAL MATRICES SATISFYING QTQ1 ARE ANALOGOUS TO THE  REAL NUMBERS Z  PM 1 SUCH THAT Z21ENDDESCRIPTIONTHE BILINEAR TRANSFORMATION INDEXBILINEAR TRANSFORMATIONBEGINEQUATION Z  FRAC1JR1JRLABELEQCAYLEYPREENDEQUATIONTAKES REAL NUMBERS R INTO THE UNIT CIRCLE Z1 MAPPING THENUMBER RINFTY TO Z1  ANALOGOUSLY BY EM CAYLEYS FORMULABEGINEQUATIONU  I  JRIJR1LABELEQCAYLEYFORMENDEQUATIONA HERMITIAN MATRIX R IS MAPPED TO A UNITARY MATRIX THAT DOES NOTHAVE AN EIGENVALUE OF 1INDEXCAYLEY TRANSFORMATIONSUBSECTIONTHE QR FACTORIZATIONLABELSECQRFACTIN THE QR FACTORIZATION AN MATSIZEMN MATRIX A IS WRITTEN AS A  QRWHERE Q IS AN MATSIZEMM UNITARY MATRIX AND R IS UPPERTRIANGULAR MATSIZEMN  AS DISCUSSED BELOW THERE ARE SEVERALWAYS IN WHICH THE QR FACTORIZATION CAN BE COMPUTED  IN THIS SECTIONWE FOCUS ON SOME OF THE USES OF THE FACTORIZATIONTHE MOST IMPORTANT APPLICATION OF QR IS TO FULLRANK LEASTSQUARESPROBLEMS  CONSIDER AXBF APPROX BBFWHERE M  N AND THE COLUMNS OF A ARE LINEARLY INDEPENDENT  INTHIS CASE THE PROBLEM IS SAID TO BE A FULLRANK LEASTSQUARESPROBLEM  THE SOLUTION XBFHAT WHICH MINIMIZES  A XBFHAT  BBF2 IS XBFHAT  AHA1AH BBFINDEXLEASTSQUARESQR SOLUTION HOWEVER THE CONDITION NUMBER OFAHA IS THE SQUARE OF THE CONDITION OF A SO DIRECT COMPUTATION ISNOT ADVISED  THIS POOR CONDITIONING CAN BE MITIGATED USING THE QRDECOMPOSITION  WHEN MN THE QR DECOMPOSITION CAN BE WRITTEN AS A  QR  QBEGINBMATRIXR1  ZEROBF ENDBMATRIXWHERE R1 IS MATSIZENN AND THE ZEROBF DENOTES AMATSIZEMNN BLOCK OF ZEROS  ALSO LETBEGINEQUATION QH BBF  BEGINBMATRIX CBF  DBF ENDBMATRIXLABELEQQBF1ENDEQUATIONWHERE CBF IS MATSIZEN1 AND DBF IS MATSIZEMN1  THENBEGINALIGNAXBF  BBF 22   QR XBF  BBF22 NONUMBER     QRXBF  QH BBF22LABELEQSAMENORMUSE  LEFT BEGINBMATRIX R1  0 ENDBMATRIXXBF   BEGINBMATRIXCBF  DBF ENDBMATRIXRIGHT22 LABELEQSN2   R1 XBF  CBF22   DBF22 NONUMBERENDALIGNWHERE REFEQSAMENORMUSE FOLLOWS SINCE BBF  QQH BBF ANDREFEQSN2 FOLLOWS FROM LEMMA REFLEMQRSAMENORM  SUCH PULLINGOF ORTHOGONAL MATRICES OUT OF THIN AIR TO SUIT SOME ANALYTICALPURPOSE IS QUITE COMMON  THE VALUE XBFHAT THAT MINIMIZESREFEQSN2 SATISFIES  R1 XBFHAT  CBFWHICH CAN BE READILY COMPUTED SINCE R1 IS A TRIANGULAR MATRIX  PUTANOTHER WAY SOLVING AHA XBF  AH BBF WITH A  QR LEADS TOBEGINEQUATION RH R XBF  RH QH BBF  FBFLABELEQQR1ENDEQUATIONWHERE FBF  RH QH BBF  EQUATION REFEQQR1 LEADS TO A PAIROF TRIANGULAR EQUATIONS WHICH CAN BE SOLVED AS WAS DONE FOR THE LUDECOMPOSITIOIN SOLVE RH YBF  FBFTHEN SOLVE RXBF  YBFIF A DOES NOT HAVE FULL COLUMN RANK OR IF M N COMPUTING THE QRDECOMPOSITION AND SOLVING LEASTSQUARES PROBLEMS THEREBY IS MOREDIFFICULT  THERE ARE ALGORITHMS TO COMPUTE THE QR DECOMPOSITION INTHIS CASE WHICH INVOLVE COLUMN PIVOTING  HOWEVER IN THISCIRCUMSTANCE IT IS RECOMMENDED TO USE THE SVD AND HENCE THESETECHNIQUES ARE NOT DISCUSSED HERE  SUBSECTIONQR FACTOR AND LEASTSQUARES FILTERSAS AN EXAMPLE OF THE USE OF THE QR FACTORIZATION CONSIDER THELEASTSQUARES PROBLEMBEGINEQUATION DBFK  AK HBFK  EBFKLABELEQQRLSFILT1ENDEQUATIONWHERE WE WISH TO MINIMIZE  EBFK 22 IN WHICH DBFK  BEGINBMATRIX D1  D2  CDOTS   DKENDBMATRIXT A  BEGINBMATRIX QBF1H  QBF2H  VDOTS  QBFKHENDBMATRIXSEE SECTION REFSECLSFILT  THE LEAST SQUARESSOLUTION CAN BE OBTAINED BY FINDING THE QR FACTORIZATION OF XK AK  QK BEGINBMATRIXR1K  ZEROBF ENDBMATRIXTHEN R1K HBF  LEFTQK DBFKRIGHTMATSIZEM1THUS WE CAN FIND HBF BY BACK SUBSTITUTIONSUBSECTIONCOMPUTING THE QR FACTORIZATIONLABELSECQRCOMPAT LEAST FOUR MAJOR WAYS OF COMPUTING THE QR FACTORIZATION ARE WIDELYREPORTED  THESE AREBEGINENUMERATEITEM THE GRAMSCHMIDT ALGORITHM  INDEXGRAMSCHMIDT PROCESSITEM THE MODIFIED GRAMSCHMIDT ALGORITHM  INDEXMODIFIED GRAMSCHMIDT THE GRAMSCHMIDT ALGORITHMS ARE DISCUSSED IN SECTION  REFSECGRAMSCHMITITEM HOUSEHOLDER TRANSFORMATIONSITEM GIVENS TRANSFORMATIONSENDENUMERATETHE GRAMSCHMIDT METHODS PROVIDE AN ORTHONORMAL BASIS SPANNING THECOLUMN SPACE OF A  THE QR FACTORIZATION USING THE HOUSEHOLDERTRANSFORMATION AND THE GIVENS ROTATIONS RELY ON SIMPLE INVERTIBLE ANDORTHOGONAL GEOMETRIC TRANSFORMATIONS  THE HOUSEHOLDER TRANSFORMATIONIS SIMPLY A REFLECTION OPERATION WHICH IS USED TO ZERO MOST OF ACOLUMN OF A MATRIX WHILE THE GIVENS ROTATION IS A SIMPLETWODIMENSIONAL ROTATION WHICH IS USED TO ZERO A PARTICULAR SINGLEELEMENT OF A MATRIX  THESE OPERATIONS MAY BE APPLIED IN SUCCESSION TOOBTAIN AN UPPER TRIANGULAR MATRIX IN THE QR FACTORIZATION  THEY MAYALSO BE USED IN OTHER CIRCUMSTANCES WHERE ZEROING PARTICULAR ELEMENTSOF A MATRIX WHILE PRESERVING THE EIGENVALUES OF THE MATRIX ISNECESSARY  OF THESE THE GRAMSCHMIDT METHODS ARE THE LEAST COMPLEXCOMPUTATIONALLY BUT ARE ALSO THE MOST POORLY CONDITIONEDSUBSECTIONHOUSEHOLDER TRANSFORMATIONSINDEXHOUSEHOLDER TRANSFORMATIONS RECALL FROM SECTIONREFSECPROJMAT THAT THE MATRIX WHICH PROJECTS ORTHOGONALLY ONTOLSPANVBF IS SEE REFEQPROJMAT1 PV  FRACVBF VBFHVBFHVBFAND THE ORTHOGONAL PROJECTION MATRIX IS PVPERP  IPVTHESE ARE SIMILAR TO THE HOUSEHOLDER TRANSFORMATION WITH RESPECT TO ANONZERO VECTOR VBF WHICH IS A TRANSFORMATION OF THE FORMBEGINEQUATIONBEGINSPLITHV  I  2 FRACVBF VBFHVBFH VBF  I  2PVENDSPLITLABELEQHVDEFENDEQUATIONIT IS STRAIGHTFORWARD TO SHOW THAT HV IS UNITARY AND HVHHVSEE EXERCISE REFEXHOUSE1  THE VECTOR VBF IS CALLED A BFHOUSEHOLDER VECTOR  OBSERVE HVVBF  VBF AND IF ZBF PERP VBFWITH RESPECT TO THE EUCLIDEAN INNER PRODUCT THAT HV ZBF  ZBFWRITE XBF AS XBF  PVBF XBF  PVBFPERP XBFTHEN HV XBF  PVBFPERP XBF  PVBF XBFWHICH CORRESPONDS TO A EM REFLECTION INDEXREFLECTION OF THEVECTOR XBF ACROSS THE SPACE PERPENDICULAR TO VBF AS SHOWN INFIGURE REFFIGHOUSEREFLECT  REFLECTING TWICE RETURNS THE ORIGINALPOINT HV2 XBF  XBF  AS AN OPERATOR WE CAN WRITE HV  PVBFPERP  PVBFBEGINFIGUREHTBPBEGINCENTERINPUTPICTUREDIRHOUSEHOLDER1ENDCENTERCAPTIONTHE HOUSEHOLDER TRANSFORMATION OF A VECTOR  LABELFIGHOUSEREFLECTENDFIGURETHE HOUSEHOLDER TRANSFORMATION CAN BE USED TO ZERO OUT ALL THEELEMENTS OF A VECTOR EXCEPT FOR ONE COMPONENT  THAT IS FOR A VECTORXBF  X1 X2 LDOTS XNT THERE IS A VECTOR VBF IN THEHOUSEHOLDER TRANSFORMATION HV SUCH THAT HV XBF  BEGINBMATRIXALPHA  0  VDOTS  0 ENDBMATRIXFOR SOME SCALAR ALPHA  SINCE HV IS UNITARY XBF2 HVXBF2 HENCE ALPHA  PM XBF2  ONE WAY OF VIEWINGTHE HOUSEHOLDER TRANSFORMATION IS AS A UNITARY TRANSFORMATION WHICHCOMPRESSES ALL OF THE ENERGY IN A VECTOR INTO A SINGLE COMPONENTZEROING OUT THE OTHER COMPONENTS OF THE VECTOR  TO FIND THE VECTORVBF IN THE TRANSFORMATION HV WRITE HV XBF  ALPHA BEGINBMATRIX 1  0  VDOTS  0 ENDBMATRIX ALPHA EBF1THENBEGINALIGNEDHVXBF  I  2FRACVBF VBFHVBFH VBF XBF  XBF  2FRACVBFH XBFVBF VBFHVBF  ALPHA EBF1ENDALIGNEDSO THAT LEFT2FRACVBFH XBFVBF VBFHRIGHTVBF  XBF  ALPHAEBF1THIS MEANS THAT VBF IS A SCALAR MULTIPLE OF XBF  ALPHAEBF1SINCE WE KNOW THAT ALPHA  PM  XBF2 AND SINCE SCALINGVBF BY A NONZERO SCALAR DOES NOT CHANGE THE HOUSEHOLDERTRANSFORMATION WE WILL TAKE VBF  XBF PM  XBF2 EBF1ALTHOUGH EITHER SIGN MAY BE TAKEN NUMERICAL CONSIDERATIONS SUGGEST APREFERRED VALUE  FOR REAL VECTORS IF XBF IS CLOSE TO A MULTIPLEOF EBF1 THEN VBF  XBF  SIGNX1  XBF2 EBF1 HAS ASMALL NORM WHICH COULD LEAD TO A LARGE RELATIVE ERROR IN THECOMPUTATION OF THE FACTOR 2VBFT VBF  THIS DIFFICULTY CAN BEAVOIDED BY CHOOSING THE SIGN BY VBF  XBF  SIGNX1  XBF2 EBF1BY THIS SELECTION  VBF  GEQ XBF  FOR COMPLEX VECTORSCHOOSING ACCORDING TO THE SIGN OF THE REAL PART IS APPROPRIATETHE OPERATION OF HV ON XBF CAN BE UNDERSTOOD GEOMETRICALLY USINGFIGURE REFFIGHOUSE1 WHERE THE SIGN HERE IS TAKEN SO THAT VBF XBF   XBF2 EBF1  SINCE VBF IS THE SUM OF TWOEQUALLENGTH VECTORS IT IS THE DIAGONAL OF AN EQUILATERALPARALLELOGRAM  THE OTHER DIAGONAL ORTHOGONAL TO THE FIRST  SEEEXERCISE REFEXPOTH RUNS FROMTHE VECTOR XBF TO TO THE VECTOR XBF2 EBF1  FROM THEFIGURE IT IS CLEAR THAT PVBF XBF  VBF2 AND PVBFPERPXBF  XBF  VBF2BEGINFIGUREHTBPBEGINCENTER  INPUTPICTUREDIRHOUSEHOLDERENDCENTERCAPTIONZEROING ELEMENTS OF A VECTOR BY A HOUSEHOLDER TRANSFORMATION  LABELFIGHOUSE1ENDFIGUREIN THE QR FACTORIZATION WE WANT TO CONVERT A TO AN UPPER TRIANGULARFORM USING A SEQUENCE OF ORTHOGONAL TRANSFORMATIONS  TO USE THEHOUSEHOLDER TRANSFORMATION TO COMPUTE THE QR FACTORIZATION OF AMATRIX FIRST CHOOSE A HOUSEHOLDER TRANSFORMATION H1 TO ZERO OUTALL BUT THE FIRST ELEMENT OF THE FIRST COLUMN OF A USING THE VECTORVBF1  FOR THE SAKE OF ILLUSTRATION LET A BE MATSIZE43THEN H1 A  BEGINBMATRIX ALPHA1  TIMES  TIMES 0  TIMES  TIMES 0  TIMES  TIMES 0  TIMES  TIMES ENDBMATRIXWHERE TIMES INDICATES ELEMENTS OF THE MATRIX WHICH ARE NOT ZERO INGENERAL  LET Q1  H1  TO CONTINUE THE PROCESS FOR THE MATSIZE32 MATRIX ON THE LOWERRIGHT CHOOSE A HOUSEHOLDER TRANSFORMATION MATRIX H2 TO ZERO OUTTHE LAST 2 ELEMENTS USING THE VECTOR VBF2  COMBINING WITHTHE FIRST TRANSFORMATION IS DONE BY BEGINBMATRIX1  ZEROBF  0  H2 ENDBMATRIXBEGINBMATRIX ALPHA1  TIMES  TIMES 0  TIMES  TIMES 0  TIMES  TIMES 0  TIMES  TIMES ENDBMATRIX Q2 Q1 A  BEGINBMATRIX ALPHA1  TIMES  TIMES 0  ALPHA2  TIMES 0  0  TIMES 0  0  TIMES ENDBMATRIXWHERE Q2  BEGINBMATRIX1  ZEROBF  0  H2 ENDBMATRIXFOR THE SAKE OF IMPLEMENTATION DESCRIBEDBELOW NOTE THAT Q2 CAN BE FORMED AS A HOUSEHOLDER MATRIX ASBEGINEQUATIONQ2  I  2FRACVBFTILDE2 VBFTILDE2HVBFTILDE2H VBFTILDE2 LABELEQHOUSE3ENDEQUATIONWHERE VBFTILDE2  BEGINBMATRIX 0  VBF2 ENDBMATRIXTHE LAST TWO ELEMENTS IN THE THIRD COLUMN CAN BE REDUCED WITH A THIRDHOUSEHOLDER TRANSFORMATION H3  IN CONJUNCTION WITH THE OTHERELEMENTS OF THE MATRIX THIS CAN BE WRITTEN AS BEGINBMATRIX1  0  ZEROBF  0  1  0  0  0  H3 ENDBMATRIXBEGINBMATRIX ALPHA1  TIMES  TIMES 0  ALPHA2  TIMES 0  0  TIMES 0  0  TIMES ENDBMATRIX Q3 Q2 Q1 A  BEGINBMATRIX ALPHA1  TIMES  TIMES 0  ALPHA2  TIMES 0  0  ALPHA3 0  0  TIMES ENDBMATRIXWHERE Q3  BEGINBMATRIX1  0  ZEROBF  0  1  0  0  0  H3ENDBMATRIX  I  2 FRACVBFTILDE3 VBFTILDE3HVBFTILDE3H  VBFTILDE3 AND VBFTILDE3  BEGINBMATRIX 0  0  VBF3 ENDBMATRIXSINCE H2 AND H3 ARE ORTHOGONAL SO ARE Q2 AND Q3 SEEEXERCISE REFEXSTACKORTHOG AND SO IS QH  Q3 Q2 Q1  THUS AHAS BEEN REDUCED TO THE PRODUCT OF AN ORTHOGONAL MATRIX TIMES ANUPPER TRIANGULAR MATRIX A  QR  Q1 Q2 Q3 RFOR A GENERAL MATSIZEMN MATRIX COMPUTATION OF THE QR ALGORITHMINVOLVES FORMING N ORTHOGONAL MATRICES QJ J12LDOTSNTHEN Q  Q1 Q2 CDOTS QNWHERE  QJ  I  2 FRACVBFTILDEJ VBFTILDEJHVBFTILDEJH  VBFTILDEJAND VBFTILDEJ  UNDERBRACE00LDOTS0J1VBFJTTSUBSECTIONALGORITHMS FOR HOUSEHOLDER TRANSFORMATIONSIN THIS SECTION SOME SAMPLE SC MATLAB CODE IS DEVELOPED TO COMPUTETHE QR DECOMPOSITION USING HOUSEHOLDER TRANSFORMATIONS  THE CODE ISFOR DEMONSTRATION PURPOSES ONLY SINCE SC MATLAB HAS THE FUNCTIONTT QR BUILTININ THE INTEREST OF EFFICIENCY THE HOUSEHOLDER TRANSFORMATION MATRIXQ IS NOT EXPLICITLY FORMED  RATHER THAN EXPLICITLY FORMING HVAND THEN MULTIPLYING HV A WE NOTE THATBEGINEQUATION HV A  LEFTI  2 FRACVBF VBFHVBFH VBFRIGHT A  A BETA VBF WBFHLABELEQHOUSELEFTENDEQUATIONWHERE BETA  2VBFH VBF AND WBF  AH VBF  IT IS OFTEN THECASE THAT ONLY THE R MATRIX IS EXPLICITLY NEEDED SO THE Q ISREPRESENTED IMPLICITLY BY THE SEQUENCE OF VBFJ VECTORS FROM WHICHQ CAN BE COMPUTED AS DESIRED  ALGORITHM REFALGHOUSELEFTILLUSTRATES A FUNCTION WHICH APPLIES A HOUSEHOLDER TRANSFORMATIONHV REPRESENTED ONLY BY THE HOUSEHOLDER VECTOR VBF ON THE LEFTOF A AS HV A AND ALSO SHOWS THE FUNCTION TT MAKEHOUSE WHICHCOMPUTES THE HOUSEHOLDER VECTOR VBF  ALSO SHOWN IS THE FUNCTIONTT HOUSERIGHT WHICH APPLIES THE HOUSEHOLDER TRANSFORMATION ON THERIGHT TO ZERO OUT ROWS OF ABEGINNEWPROGENVHOUSEHOLDER TRANSFORMATION FUNCTIONS 1 COMPUTE    VBF 2 HV A  GIVEN  VBF AND 3 COMPUTE A HV GIVEN VBF  HOUSELEFTCOMPUTE PROTECTHPROTECTVPROTECT GIVEN VBFMAKEHOUSEMHOUSELEFTMHOUSERIGHTMENDNEWPROGENVBEGINEXAMPLE  LET  A  BEGINBMATRIX1  2  3 4  5  6 6  7  8 ENDBMATRIXTHEN THE SC MATLAB FUNCTION CALLS TT VL  MAKEHOUSEA1 ANDTT VR  MAKEHOUSEA1 RETURN THE VECTORS VL  BEGINBMATRIX828011  4  6 ENDBMATRIXT QQUADVR  BEGINBMATRIX474166  2  3 ENDBMATRIXTTHEN HVA CAN BE COMPUTED USING TT HOUSELEFTAVL AND A HVCAN BE COMPUTED FROM TT HOUSERIGHTAVR  THE RESULTS AREBEGINALIGNED TT HOUSELEFTAVL  BEGINBMATRIX728011  879108  10302 0  0213011  0426022 0  0819517  163903 ENDBMATRIX TT HOUSERIGHTAVR  BEGINBMATRIX374166  0  0 855236  0294503  194175 117595  0490838  323626 ENDBMATRIXENDALIGNEDENDEXAMPLEALGORITHM REFALGHOUSE1 COMPUTES THE QR FACTORIZATION USING THESIMPLIFICATIONS NOTED HERE  THE RETURN VALUES ARE THE MATRIX R ANDTHE VECTOR OF VBF VECTORS  THE COMPLEXITY OF THE ALGORITHM ISAPPROXIMATELY 2N2MN3 FLOATING OPERATIONSBEGINNEWPROGENVQR FACTORIZATION VIA HOUSEHOLDER TRANSFORMATIONSQRHOUSEMHOUSE1QR FACTORIZATION VIA HOUSEHOLDER  TRANSFORMATIONSENDNEWPROGENVIN ORDER TO SOLVE THE LEASTSQUARES EQUATION AS DESCRIBED ABOVE WE MUST BEABLE TO COMPUTE QH BBF  SINCE Q  Q1 Q2 CDOTS QN AND EACHQ IS HERMITIAN SYMMETRIC QH BBF  QNH QN1H CDOTS Q1H BBFWHICH MAY BE ACCOMPLISHED CONCEPTUALLY USING THE FOLLOWINGALGORITHM WHICH OVERWRITES BBF WITH QH BBFBEGINPROGTABSFOR J1N    BBF  QJ BBF ENDENDPROGTABSTHE MULTIPLICATION CAN BE ACCOMPLISHED WITHOUT EXPLICITLY FORMING THEQJ MATRICES USING THE IDEA SHOWN IN REFEQHOUSELEFTCOMPUTATION OF QH BBF IS THUS ACCOMPLISHED AS SHOWN IN ALGORITHMREFALGQRQTB BEGINNEWPROGENVCOMPUTATION OF QH BBFQRQTBMQRQTBCOMPUTATION OF QH BBFENDNEWPROGENVBEGINEXAMPLE  SUPPOSE IT IS DESIRED TO FIND THE LEASTSQUARES SOLUTION TO BEGINBMATRIX7 8  8 8 6  2 1 7  3 0 7  3 6 9  5ENDBMATRIXXBF  BEGINBMATRIX47  26 24 23  39  ENDBMATRIXUSING TT VR  QRHOUSEA WE OBTAIN V  BEGINBMATRIXHFILL 192474 HFILL 0  HFILL 0 HFILL 8  HFILL 127989  HFILL  0 HFILL 1  HFILL 58844  HFILL 61096 HFILL 0  HFILL 7    HFILL 23046 HFILL 6  HFILL 23065  HFILL 19142 ENDBMATRIXQQUADQQUADR  BEGINBMATRIXHFILL 122474  HFILL 134722   HFILL 85732 HFILL 0   HFILL  98742    HFILL 48105HFILL 0  HFILL 0  HFILL    37893 HFILL  0HFILL 0HFILL  0 HFILL 0HFILL 0 HFILL 0ENDBMATRIXUSING TT QRQTBBV WE OBTAINQH BBF  BEGINBMATRIXHFILL 649115  HFILL 341800  HFILL  113680  0  0 ENDBMATRIX THE LEASTSQUARES SOLUTION COMES FROM SOLVING THE MATSIZE33UPPERTRIANGULAR SYSTEM OF EQUATIONS USING BACKSUBSTITUTION BEGINBMATRIX HFILL 122474  HFILL 134722 HFILL   85732 HFILL 0HFILL 98742 HFILL    48105 HFILL 0HFILL 0   37893ENDBMATRIXXBFHAT  BEGINBMATRIXHFILL 649115  HFILL 341800  HFILL  113680ENDBMATRIXWHICH LEADS TO XBFHAT  BEGINBMATRIX1  2  3 ENDBMATRIXENDEXAMPLEWHERE THE Q MATRIX IS EXPLICITLY DESIRED FROM V IT CAN BE COMPUTED BYBACKWARD ACCUMULATION  TO COMPUTE Q  Q1Q2LDOTS QR WE ITERATEAS FOLLOWS BEGINALIGNEDQ0  I Q1  QR Q0  Q2  QR1 Q1 VDOTS Q  QR  Q1 QR1ENDALIGNEDAN ALGORITHM IMPLEMENTING THIS IS SHOWN IN ALGORITHM REFALGMAKEHOUSEQBEGINNEWPROGENVCOMPUTATION OF Q FROM VQRMAKEQMMAKEHOUSEQCOMPUTATION OF Q FROM VENDNEWPROGENVSUBSECTIONQR FACTORIZATION USING GIVENS ROTATIONSLABELSECGIVENSINDEXGIVENS ROTATIONSUNLIKE THE HOUSEHOLDER TRANSFORMATION WHICH ZEROS OUT ENTIRE COLUMNSAT A STROKE THE GIVENS ROTATION MORE SELECTIVELY ZEROS ONE ELEMENT ATA TIME USING A ROTATIONA TWODIMENSIONAL ROTATION BY AN ANGLE THETA IS ILLUSTRATED INFIGURE REFFIGGIVROT1A  THE FIGURE DEMONSTRATES THAT THE POINT10 IS ROTATED INTO THE POINT COS THETASINTHETA AND THE POINT 01 IS ROTATED INTO THE POINT SIN THETA COSTHETA  BY THESE POINTS WE IDENTITY THAT A MATRIX GTHETA WHICHROTATES XYT ISBEGINEQUATION GTHETA  BEGINBMATRIX  COS THETA  SIN THETA  SIN THETA   COS THETA  ENDBMATRIX LABELEQGIVENS1ENDEQUATIONTHE ROTATION MATRIX IS ORTHOGONAL GTHETATGTHETA  I  ITSHOULD BE CLEAR THAT ANY POINT XY IN TWO DIMENSIONS CAN BEROTATED BY SOME ROTATION MATRIX G SO THAT ITS SECOND COORDINATE ISZERO  THIS IS ILLUSTRATED IN FIGURE REFFIGGIVROT1B  FOR AVECTOR XBF  X YT ITS SECOND COORDINATE CAN BE ZEROED BYMULTIPLICATION BY THE ORTHOGONAL MATRIX GTHETA WHEREINDEXROTATION MATRIXBEGINEQUATIONTHETAXY  TAN1 FRACYXLABELEQGIVETHETAENDEQUATIONBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODESUBFIGUREA GENERAL ROTATIONINPUTPICTUREDIRROT1QQUADQQUADSUBFIGUREROTATE THE SECOND COORDINATE TO ZEROINPUTPICTUREDIRROT2    CAPTIONTWODIMENSIONAL ROTATION    LABELFIGGIVROT1  ENDCENTERENDFIGUREIN THE GIVENS ROTATION APPROACH TO THE QR FACTORIZATION A MATRIX AIS ZEROED OUT ONE ELEMENT AT A TIME STARTING AT THE BOTTOM OF THEFIRST COLUMN AND WORKING UP THE COLUMNS  TO ZERO AIK WE USE X AJK AND YAIK APPLYING THE MATSIZE22 ROTATION MATRIXACROSS THE JTH AND ITH ROWS OF A  SUCH A ROTATION MATRIX ISCALLED A EM GIVENS ROTATION  WE WILL DENOTE BY GTHETAIKJTHE ROTATION MATRIX WHICH ZEROS AIK  FOR BREVITY WE WILL ALSOWRITE GIKJ  IN THE QR FACTORIZATION A SEQUENCE OF THESEROTATION MATRICES ARE USED  A SEQUENCE OF MATRICES PRODUCED BYSUCCESSIVE OPERATION OF GIVENS ROTATIONS MIGHT HAVE THE FOLLOWINGFORM WHERE THE CONVENTION OF TAKING JI1 IS USED  THE ROTATION ISSHOWN ABOVE THE ARROW AND THE ROWS AFFECTED BY THE PRECEDINGTRANSFORMATION ARE SHOWN IN BOLDBEGINEQUATIONBEGINALIGNED BEGINBMATRIX TIMESTIMESTIMES  TIMES  TIMES  TIMES TIMES  TIMES  TIMES  TIMES  TIMES  TIMES ENDBMATRIXSTACKRELG413LONGRIGHTARROWBEGINBMATRIX TIMESTIMESTIMES  TIMES  TIMES  TIMES TIMESBF  TIMESBF  TIMESBF  ZEROBF  TIMESBF  TIMESBF ENDBMATRIXSTACKRELG312LONGRIGHTARROWBEGINBMATRIX TIMESTIMESTIMES  TIMESBF  TIMESBF  TIMESBF ZEROBF  TIMESBF  TIMESBF  0  TIMES  TIMES ENDBMATRIX STACKRELG211LONGRIGHTARROWBEGINBMATRIX TIMESBF TIMESBF TIMESBF  ZEROBF  TIMESBF  TIMESBF 0  TIMES  TIMES  0  TIMES  TIMES ENDBMATRIX STACKRELG423LONGRIGHTARROWBEGINBMATRIX TIMES TIMESTIMES  0  TIMES  TIMES 0  TIMESBF  TIMESBF  0  ZEROBF  TIMESBF ENDBMATRIX STACKRELG322LONGRIGHTARROWBEGINBMATRIX TIMES TIMESTIMES  0  TIMESBF  TIMESBF 0  ZEROBF  TIMESBF  0  0  TIMES ENDBMATRIX STACKRELG433LONGRIGHTARROWBEGINBMATRIX TIMES TIMESTIMES 0  TIMES  TIMES 0  0  TIMESBF 0  0  ZEROBF ENDBMATRIXENDALIGNEDLABELEQGIV2ENDEQUATIONTHE TWODIMENSIONAL ROTATION REFEQGIVENS1 CAN BE MODIFIED TOFORM GTHETAIKJ BY DEFININGBEGINEQUATIONBEGINALIGNEDGTHETAIKJ  BEGINBMATRIX1 CDOTS  0  CDOTS  0  CDOTS  0 VDOTS  DDOTS  VDOTS   VDOTS  VDOTS 0  CDOTS   C CDOTS  S CDOTS  0 VDOTS  VDOTS DDOTS  VDOTS  VDOTS 0  CDOTS  S  CDOTS  C  CDOTS 0 VDOTS  VDOTS  VDOTS  DDOTS VDOTS 0  CDOTS  SMASHLOWER15EMVBOXHBOX0HBOXRULE0MM15EMJ  CDOTS  SMASHLOWER15EMVBOXHBOX0HBOXRULE0MM15EMI CDOTS 1ENDBMATRIX BEGINMATRIX   J   I   ENDMATRIX  BEGINMATRIX  ENDMATRIXENDALIGNEDLABELEQGIVENGENDEQUATIONWHERE C  COS THETA AND S  SIN THETA  AS IS APPARENT FROMTHE FORM OF GTHETAIKJ THE OPERATION GTHETAIKJA SETSTHE IKTH ELEMENT TO ZERO AND MODIFIES ITH AND JTH ROWS OFA LEAVING THE OTHER ROWS OF A UNMODIFIED  THE VALUE OF THETAIN GTHETAIKK IS DETERMINED FROM XY  AJKAIK INREFEQGIVETHETA  AS IS APPARENT BY STUDYING REFEQGIV2TAKING JI1 MAKES IT SO THAT THE DIAGONALIZATION ALREADYACCOMPLISHED IN PRIOR COLUMNS IS NOT AFFECTED BY GIVENS ROTATIONS ONLATER COLUMN  SINCE THIS IS THE MOST COMMON CASE WE WILL HENCEFORTHUSE THE ABBREVIATED NOTATION GIK OR GTHETAIK FORGTHETAIKJFOR THE MATSIZE43 EXAMPLE THE FACTORIZATION IS ACCOMPLISHED BYG41G31G21G42G32G43A RTHE Q MATRIX IS THUS OBTAINED ASBEGINALIGNEDQ  G41G31G21G42G32G43T  G43TG32TG42T G21T G31T G41TENDALIGNEDBEGINEXAMPLE  LET  A  BEGINBMATRIX1 2 3 4 1 3ENDBMATRIXA ROTATION MATRIX TO ZERO THE 31 ELEMENT THAT MODIFIES THE LASTTWO ROWS OF A IS G31  G312  BEGINBMATRIX1 0 0  0 09487 03162  0   03162  09487 ENDBMATRIXTHEN  GA  BEGINBMATRIX1 2 31623  47434 0     15811 ENDBMATRIXENDEXAMPLESUBSECTIONALGORITHMS FOR QR FACTORIZATION USING GIVENS ROTATIONSSEVERAL ASPECTS OF THE MATHEMATICS OUTLINED ABOVE FOR GIVENS ROTATIONSMAY BE STREAMLINED FOR A NUMERICAL IMPLEMENTATION  EXPLICITCOMPUTATION OF THETA IS NOT NECESSARY WHAT ARE NEEDED IS COSTHETA AND SIN THETA WHICH MAY BE DETERMINED FROM XYWITHOUT ANY TRIGONOMETRIC FUNCTIONS COS THETA  COS TAN LEFTFRACYXRIGHT FRACXSQRTX2  Y2 QQUAD SIN THETA FRACYSQRTX2Y2SEE ALGORITHM REFALGQRTHETA FOR A NUMERICALLY STABLE METHOD OFCOMPUTING THESE QUANTITIES BEGINNEWPROGENVFIND C AND S FOR A GIVENS ROTATION    QRTHETAMQRTHETAFIND C AND S FOR A GIVENS ROTATIONENDNEWPROGENVIN COMPUTING THE MULTIPLICATION GIK A IT IS CLEARLY MUCH MOREEFFICIENT TO ONLY MODIFY ROWS I AND K OF THE PRODUCT  AN EXPLICITQ MATRIX IS NEVER CONSTRUCTED  INSTEAD THE COS THETA AND SINTHETA INFORMATION IS SAVED  IT WOULD ALSO BE POSSIBLE TO REPRESENTBOTH NUMBERS USING A SINGLE QUANTITY AND STORE THE Q MATRIXINFORMATION IN THE LOWER TRIANGLE  HOWEVER IN THE INTEREST OF SPEEDTHIS IS NOT DONE  AN ALGORITHM TO COMPUTE THE QR FACTORIZATION ISSHOWN IN ALGORITHM REFALGQRGIVENS AND ALGORITHM REFALGQRQTBGIVCOMPUTES QH BBF FOR USE IN SOLVING LEASTSQUARES PROBLEMSFINALLY FOR THOSE INSTANCES IN WHICH IT IS NEEDED ALGORITHMREFALGMAKEGIVQ COMPUTES Q FROM THE THETA INFORMATION BYCOMPUTING Q  GM1TGM11TCDOTS G21TGM2TGM12TCDOTSGMNTWITH THE MULTIPLICATION DONE FROM LEFT TO RIGHTBEGINNEWPROGENVQR FACTORIZATION USING GIVENS ROTATIONS    QRGIVENSMQRGIVENSQR FACTORIZATION USING GIVENS      ROTATIONSENDNEWPROGENVBEGINNEWPROGENVCOMPUTATION OF QH BBF FOR THE GIVENS ROTATION    FACTORIZATIONQRQTBGIVMQRQTBGIVCOMPUTATION OF QH BBF FOR THE  GIVENS ROTATION FACTORIZATIONENDNEWPROGENVBEGINNEWPROGENVCOMPUTATION OF Q FROM THE THETAQRMAKEQGIVMMAKEGIVQCOMPUTATION OF Q FROM THETAENDNEWPROGENVSUBSECTIONSOLVING LEASTSQUARES PROBLEMS USING GIVENS ROTATIONSGIVENS ROTATIONS CAN BE USED TO SOLVE LEASTSQUARES PROBLEMS IN A WAYTHAT IS WELLSUITED FOR PIPELINED IMPLEMENTATION IN VLSICITEPROAKISRADER  INDEXLEASTSQUARESVLSIAPPROPRIATE  ALGORITHMS REWRITE THE EQUATION A XBF APPROX BBFAS A  BBFBEGINBMATRIX XBF  1 ENDBMATRIX APPROX 0LET THIS BE WRITTEN AS B HBF APPROX 0 WHERE B  ABBF AND HBFT  XBFT 1  THEN THELEASTSQUARES SOLUTION IS THE ONE WHICH MINIMIZES  BHBF22 HBFH BH B HBF  SINCE MULTIPLICATION BY AN ORTHOGONAL MATRIX DOESNOT CHANGE THE NORM  QBHBF 22   BHBF22 FOR ANORTHOGONAL MATRIX Q  THE MATRIX Q CAN BE SELECTED AS A GIVENSROTATION WHICH SELECTIVELY ZEROS OUT ELEMENTS OF THE MATRIX B  BYTHIS MEANS WE CAN TRANSFORM THE PROBLEM SUCCESSIVELY AS BEGINALIGNEDBHBF APPROX 0 Q1BHBF APPROX 0 Q2Q1BHBF APPROX 0 Q2Q1BHBF APPROX 0 LDOTSQP CDOTS Q2Q1B HBF APPROX 0ENDALIGNEDWITH AN APPROPRIATELYCHOSEN SEQUENCE OF QI MATRICES THE RESULT ISTHAT QP CDOTS Q2Q1B IS MOSTLY UPPER TRIANGULAR SO THAT WE OBTAINA SET OF EQUATION OF THE FOLLOWING FORM BEGINBMATRIX AHAT11  AHAT12 AHAT13  CDOTS   AHAT1N  BHAT1  AHAT22  AHAT23 CDOTS  AHAT2N  BHAT2  AHAT33 CDOTS  AHAT3N  BHAT3 VDOTS  AHATNN  BHATN TIMES  TIMES  TIMES  TIMES TIMES  BHATN1 TIMES  TIMES  TIMES  TIMES TIMES  BHATN2 VDOTS  ENDBMATRIXBEGINBMATRIX X1  X2  X3  VDOTS  XN  1 ENDBMATRIXAPPROX BEGINBMATRIX0  0  0  VDOTS  0  VDOTSENDBMATRIXPRACTICALLY SPEAKING MULTIPLICATION BY THE ORTHOGONAL MATRICES CANSTOP WHEN THE TOP N ROWS ARE MOSTLY TRIANGULARIZED AS SHOWN  WHILEIT WOULD BE POSSIBLE TO COMPLETE THE QR FACTORIZATION TO ZERO THELOWER PORTION OF THE MATRIX THE PART INDICATED WITH TIMESS THISIS NOT NECESSARY SINCE THE STRUCTURE ALLOWS THE SOLUTION TO BEOBTAINED  FROM THIS THE LEASTSQUARE SOLUTION ISBEGINALIGNEDXN  FRACBHATNAHATNN XN1  FRACBHATN1  AHATN1N XNAHATN1N1  VDOTS XI  FRACBHATI  SUMJI1N AIJXJAHATII VDOTS X1  FRACBHAT1  SUMJ2N A1JXJAHAT11ENDALIGNEDSUBSECTIONGIVENS ROTATIONS VIA CORDIC ROTATIONSINDEXCORDIC ROTATIONSFOR HIGHSPEED REALTIME APPLICATIONS IT MAY BE NECESSARY TO GO WITHPIPELINED AND PARALLEL ALGORITHMS FOR QR DECOMPOSITION  THE METHODKNOWN AS CORDIC ROTATIONS PROVIDES FOR PIPELINED IMPLEMENTATIONS OFTHE GIVENS ROTATIONS WITHOUT THE NEED TO COMPUTE TRIGONOMETRICFUNCTIONS OR SQUAREROOTS  CORDIC IS AN ACRONYM FOR COORDINATEROTATION DIGITAL COMPUTATION  CORDIC METHODS HAVE ALSO BEEN APPLIEDTO A VARIETY OF OTHER SIGNAL PROCESSING PROBLEMS INCLUDING DFTSFFTS DIGITAL FILTERING AND ARRAY PROCESSING  A SURVEY ARTICLE WITHA VARIETY OF REFERENCES IS CITEHU1992  A DETAILED APPLICATION OFCORDIC TECHNIQUES TO ARRAY PROCESSING USING A VLSI HARDWAREIMPLEMENTATION INCLUDING SOME VERY CLEVER DESIGNS FOR SOLUTION OFLINEAR EQUATIONS APPEARS IN CITERADER1996THE FUNDAMENTAL STEP IN GIVENS ROTATIONS IS THE TWO DIMENSIONALROTATIONBEGINEQUATION BEGINBMATRIX X  Y ENDBMATRIX  BEGINBMATRIX COS  THETA  SIN THETA  SIN THETA  COS THETAENDBMATRIXBEGINBMATRIXX  Y ENDBMATRIXLABELEQCORDIC1ENDEQUATIONWHERE THETA IS CHOSEN SO THAT Y0  THIS TRANSFORMATION ISAPPLIED SUCCESSIVELY TO APPROPRIATE PAIRS OF ROWS TO OBTAIN THE QRFACTORIZATION  SINCE IT IS REPEATEDLY USED IT IS IMPORTANT TO MAKETHE COMPUTATION AS EFFICIENT AS POSSIBLE  THE ROTATION INREFEQCORDIC1 CAN BE REWRITTEN ASBEGINEQUATION BEGINBMATRIXX  Y ENDBMATRIX  COS THETABEGINBMATRIX  1  TAN THETA  TAN THETA  1 ENDBMATRIXBEGINBMATRIX X  Y ENDBMATRIXLABELEQCORDIC0ENDEQUATIONWHICH STILL REQUIRES FOUR MULTIPLICATIONS  HOWEVER IF THE ANGLETHETA IS SUCH THAT TAN THETA IS A POWER OF TWO THEN THEMULTIPLICATION CAN BE ACCOMPLISHED USING ONLY BITSHIFT OPERATIONSA GENERAL ANGLE CAN BE CONSTRUCTED AS A SERIES OF ANGLES WHOSETANGENTS ARE POWERS OF TWO THETA  SUMI0INFTY RHOI THETAIWHERE RHOI  PM 1 AND THETAI IS CONSTRAINED SO THAT TANTHETAI  2I  IN PRACTICE THE SUM IS TRUNCATED AFTER A FEWTERMS USUALLY ABOUT FIVE OR SIX THETA APPROX SUMI0ITEXT MAX RHOI THETAITHE POWEROF TWO ANGLES FOR CORDIC ROTATIONS ARE SHOWN TABLEREFTABCORDIC UP TO THETA6  HIGHER ACCURACY IN THEREPRESENTATION CAN BE OBTAINED BY TAKING MORE TERMS ALTHOUGH FOR MOSTPRACTICAL PURPOSES UP TO FIVE TERMS IS OFTEN ADEQUATE  BEGINTABLEHTBP    BEGINCENTER      LEAVEVMODE      BEGINTABULARLCRR HLINEI  TAN THETAI  THETAI DEGREES MULTICOLUMN1CKAPPAI  HLINE 0  1  45                    070711 EXMATSP1  FRAC12   265605  063245EXMATSP2  FRAC14   140362  061357 EXMATSP3  FRAC18   712502  060883 EXMATSP4  FRAC116  357633  060764 EXMATSP5  FRAC132  178991  060728 EXMATSP6  FRAC164  089517  060726 EXMATSP HLINE      ENDTABULAR      CAPTIONPOWEROFTWO ANGLES FOR CORDIC COMPUTATIONS      LABELTABCORDIC    ENDCENTER  ENDTABLEBEGINEXAMPLE  AN ANGLE SUCH AS 37CIRC CAN BE REPRESENTED USING THE ANGLES IN  TABLE REFTABCORDIC AS 37 APPROX THETA0  THETA1  THETA2  THETA3  THETA4 THETA5  THETA6  3691832AN EFFICIENT REPRESENTATION IS SIMPLY THE SEQUENCE OF SIGNS 37SIM 1111111ENDEXAMPLETHE ROTATION BY THETA IN REFEQCORDIC1IS ACCOMPLISHEDSTAGEWISE BY A SERIES OF EM MICROROTATIONS  WHAT MAKES THIS MOREEFFICIENT IS THE FACT THAT THE FACTORS COS THETAI FROM EACHMICROROTATION CAN BE COMBINED TOGETHER INTO A PRECOMPUTED CONSTANT KAPPAITEXT MAX  PRODI0ITEXT MAX COS THETAITABLE REFTABCORDIC SHOWS THE VALUES OF KAPPA FOR THE FIRST FEWVALUES OF ITEXT MAX  THE MICROROTATIONS RESULT IN A SERIES OFINTERMEDIATE RESULTS  IN A CORDIC IMPLEMENTED IN IMAX STAGESTHE FOLLOWING RESULTS ARE OBTAINED BY SUCCESSIVE APPLICATION OFREFEQCORDIC0 BEGINALIGNEDBEGINBMATRIX X0  Y0 ENDBMATRIX  KAPPABEGINBMATRIXX  Y ENDBMATRIX BEGINBMATRIX X1  Y1 ENDBMATRIX  BEGINBMATRIX  X0  Y0 ENDBMATRIX  RHO0 20 BEGINBMATRIX  Y0  X0 ENDBMATRIX BEGINBMATRIX X2  Y2 ENDBMATRIX  BEGINBMATRIX  X1  Y1 ENDBMATRIX  RHO1 21 BEGINBMATRIX  Y1  X1 ENDBMATRIX BEGINBMATRIX X3  Y3 ENDBMATRIX  BEGINBMATRIX  X2  Y2 ENDBMATRIX  RHO2 22 BEGINBMATRIX  Y2  X2 ENDBMATRIX VDOTS BEGINBMATRIX X  Y ENDBMATRIX  BEGINBMATRIX  XIMAX  YIMAX ENDBMATRIX  RHOIMAX2IMAX BEGINBMATRIX   YIMAX  XIMAX ENDBMATRIX ENDALIGNEDTHE EFFECT OF MULTIPLICATION BY KAPPA IS TO NORMALIZE THE VECTOR SOTHAT THE FINAL VECTOR XYT HAS THE SAME LENGTH AS XYTIN CIRCUMSTANCES WHERE THE ANGLE OF THE VECTOR IS IMPORTANT BUT NOTITS LENGTH THE FIRST STEP MAY BE ELIMINATEDWHEN DOING ROTATION FOR THE QR ALGORITHM THE ANGLE THETA THROUGHWHICH TO ROTATE IS DETERMINED BY THE FIRST ELEMENT ON EACH OF THE TWOROWS BEING ROTATED  THESE ELEMENTS ARE REFERRED TO AS THE EM  LEADER OF THE PAIR OF ROWS  THE REST OF THE ELEMENTS ON THE ROWARE ROTATED AT AN ANGLE DETERMINED BY THE LEADER  FOR THE REGULARGIVENS ROTATION IT IS NECESSARY TO COMPUTE THE ANGLE WHICH AT AMINIMUM REQUIRES COMPUTATION OF A SQUARE ROOT  HOWEVER FOR THECORDIC IMPLEMENTATION IT IS POSSIBLE TO DETERMINE THE ANGLE TO ROTATETHROUGH IMPLICITLY USING THE MICROROTATIONS SIMPLY BY EXAMINING THESIGNS OF THE COMPONENTS OF THE LEADER  THE GOAL IS TO ROTATE A VECTORXBFT  XY TO A VECTOR X0  IF XBF IS IN QUADRANT I ORQUADRANT III THEN THE ROTATION IS NEGATIVE  IF XBF IS IN QUADRANTII OR QUADRANT IV THEN THE ROTATION IS POSITIVE  THE SIGN OF THEMICROROTATION IS DETERMINED BYBEGINEQUATION RHOI  SIGNXI1SIGNYI1LABELEQCORDIC2ENDEQUATIONIN A PIPELINES IMPLEMENTATION OF THE CORDIC ARCHITECTURE A SEQUENCEOF 2VECTORS FROM A PAIR OF ROWS OF THE MATRIX A ARE PASSED THROUGHTHE STRUCTURE SHOWN  AS THE FIRST VECTOR FROM EACH ROW  THE LEADER IS PASSED THE MICROROTATION ANGLE IS COMPUTED ACCORDING TOREFEQCORDIC2  THIS INFORMATION IS LATCHED AND USED FOR EACHSUCCEEDING VECTOR IN THE ROW  BECAUSE BUFFERING IS USED BETWEEN EACHSTAGE THE COMPUTATIONS MAY BE DONE IN A PIPELINED MANNER  AS AVECTOR PASSES THROUGH A STAGE ANOTHER VECTOR MAY IMMEDIATELY BEPASSED INTO THE STAGE THERE IS NO NEED TO WAIT FOR A SINGLE VECTOR TOPASS ALL THE WAY THROUGH  IT IS THE PIPELINED NATURE OF THEARCHITECTURE THAT LEADS TO ITS EFFICIENCYWHEN USING CORDIC FOR THE QR ALGORITHM SEVERAL ROWS MUST BE MODIFIEDIN SUCCESSION  THIS MAY BE ACCOMPLISHED BY CASCADING SEVERALPIPELINED CORDIC STRUCTURES IN SUCH A WAY THAT A MODIFIED ROW FROM ONESTAGE IS PASSED ON TO THE NEXT STAGE  THIS ALLOWS FOR MOREPARALLELISM IN THE COMPUTATION  ADDITIONAL DETAILS ARE PROVIDED INCITEPROAKISRADER AND CITERADER1996SUBSECTIONRECURSIVE UPDATES TO THE QR FACTORIZATIONLABELSECQRUPDATECONSIDER AGAIN THE LEASTSQUARES FILTERING PROBLEM OFREFEQQRLSFILT1 ONLY NOW CONSIDER THE PROBLEM OF UPDATING THEESTIMATE  THAT IS SUPPOSE THAT DATA QBF1QBF2LDOTSQBFKARE USED TO FORM AN ESTIMATE HBFK BY THE QR METHOD R1K HBFK  QK DBFKNOW A NEW DATA POINT BECOMES AVAILABLE AND WE DESIRE TO COMPUTEHBFK1 USING AS MUCH OF THE PREVIOUS WORK AS POSSIBLEIN THIS CASE IT IS MOST CONVENIENT TO REORDER THE DATA FROMLASTTOFIRST SO WE WILL LET AK  BEGINBMATRIX QBFKH  QBFK1H  VDOTS   QBF1 ENDBMATRIX YBFK   BEGINBMATRIX YK  YK1  CDOTS   Y1ENDBMATRIXTAND DBFK SIMILARLY  AS BEFORE LET  XK  QKRK  QK BEGINBMATRIX R1K  ZEROBFENDBMATRIX WHEN THE NEW DATA COMES THE A MATRIX IS UPDATED AS AK1  BEGINBMATRIX QBFK1H  AK ENDBMATRIXOBSERVE THAT  BEGINBMATRIX1    QHK  ENDBMATRIX AK1  BEGINBMATRIX QBFK1H  RKENDBMATRIX  BEGINBMATRIX QBFK1H  R1K   ZEROBFENDBMATRIX  HTHE MATRIX H HAS THE PROPERTY THAT HIJ  0 FOR I  J1  SUCHA MATRIX IS KNOWN AS AN UPPER EM HESSENBURG INDEXHESSENBURG  MATRIX MATRIX  BY FORCING A ZERO DOWN THE SUBDIAGONAL ELEMENTS OFH IT CAN BE CONVERTED TO AN UPPER TRIANGULAR MATRIX  THIS CAN BEACCOMPLISHED USING A SERIES OF GIVENS ROTATIONS ONE FOR EACHSUBDIAGONAL ELEMENT  LET THE GIVENS ROTATIONS BE INDICATED AS J1J2 LDOTS JM  WE THUS OBTAIN J1 J2 CDOTS JM BEGINBMATRIX1  QHK ENDBMATRIX A RK1  BEGINBMATRIXR1K1  ZEROBF ENDBMATRIXFROM WHICH QK1 CAN ALSO BE IDENTIFIEDBEGINEXERCISESITEM PROVE LEMMA REFLEMQRSAMENORMITEM SHOW THAT FOR A UNITARY MATRIX Q  INDEXUNITARYDETERMINANT BOXEDDETQ  1ITEM COLUMNSPACE PROJECTORS  LET X BE A RANKR MATRIX  SHOW  THAT THE MATRIX WHICH PROJECTS ORTHOGONALLY ONTO THE COLUMN SPACE OF  X IS PX  QMCSUBRANGE1RQMCSUBRANGE1RHWHERE X  QR IS THE QR FACTORIZATION OF X ANDQMCSUBRANGE1R IS THE SC MATLAB NOTATION FOR THE FIRSTR COLUMNS OF QITEM REGARDING THE CAYLEY FORMULA INDEXCAYLEY TRANSFORMATION  BEGINENUMERATE  ITEM SHOW THAT Z IN REFEQCAYLEYPRE HAS Z21  ITEM SHOW THAT U IN REFEQCAYLEYFORM SATISFIES UUHI  ITEM SOLVE REFEQCAYLEYFORM FOR R THUS FINDING A MAPPING    FROM UNITARY MATRICES TO HERMITIAN MATRICES  ITEM A MATRIX S IS EM SKEWSYMMETRIC IF ST  S    INDEXSKEWSYMMETRIC SHOW THAT    IF S IS SKEW SYMMETRIC THEN Q  ISIS1 IS ORTHOGONAL  ENDENUMERATEITEM LABELEXHOUSE1 FOR THE HOUSEHOLDER TRANSFORMATION  REFLECTION MATRIX H  I  2 VBF VBFHVBFHVBF VERIFY THE  FOLLOWING PROPERTIES AND PROVIDE A GEOMETRIC INTERPRETATION  BEGINENUMERATE  ITEM HVBF  VBF  ITEM IF ZBF PERP VBF THEN H ZBF  ZBF  ITEM HH H  HHH  I  ITEM HH  H  ITEM FOR VECTORS XBF AND YBF LA XBFYBF RA  LA HXBFHYBFRATHUS  XBF2   HXBF2  ENDENUMERATEITEM LABELEXSTACKORTHOG SHOW THAT IF Q IS AN ORTHOGONAL MATRIX THEN BEGINBMATRIX 1  ZEROBF  0  Q ENDBMATRIXIS ALSO ORTHOGONALITEM CITEPAGE 74GVL SHOW THAT IF Q  Q1  JQ2 IS UNITARY  WHERE QI IN RBBMATSIZEMM THEN THE MATSIZE2N2N  MATRIX Z  BEGINBMATRIX Q1  Q2  Q2  Q1 ENDBMATRIXIS ORTHOGONALITEM THE HOUSEHOLDER MATRIX DEFINED IN REFEQHVDEF USES A  REFLECTION WITH RESPECT TO AN ORTHOGONAL PROJECTION  IN THIS  PROBLEM WE WILL EXPLORE THE HOUSEHOLDER MATRIX USING A WEIGHTED  PROJECTION AND ITS ASSOCIATED INNER PRODUCT  LET W BE A HERMITIAN  MATRIX AND DEFINE SEE REFEQPRO2MAT2 HVW  I  2PVW  I  2FRACVBF VBFH WVBFH W VBFBEGINENUMERATEITEM SHOW THAT HVWXBFW  XBFW AND THAT  HVWH W HVW  WITEM SHOW THAT HV VBF VBFITEM SHOW THAT HVW HVW  I SO HV IS A REFLECTIONITEM DETERMINE A MEANS OF CHOOSING VBF SO THAT HVW XBF  BEGINBMATRIX1  0  VDOTS  0 ENDBMATRIXALPHAFOR SOME ALPHAENDENUMERATEITEM LABELEQHOUSEMAX CONSIDER THE PROBLEM YBFT  XBFT A  BEGINENUMERATE  ITEM DETERMINE XBF SUCH THAT THE FIRST COMPONENT OF YBF IS    MAXIMIZED SUBJECT TO THE CONSTRAINT THAT  XBF2  1  WHAT    IS THE MAXIMUM VALUE OF Y1 IN THIS CASE  ITEM LET H BE A HOUSEHOLDER MATRIX OPERATING ON THE FIRST COLUMN    OF A  COMMENT ON THE NONZERO VALUE OF HA COMPARED WITH     Y1 OBTAINED IN THE PREVIOUS PART  ENDENUMERATEITEM THE COMPUTATION IN REFEQHOUSELEFT APPLIES THE HOUSEHOLDER  MATRIX TO THE EM LEFT OF A MATRIX AS HV A  DEVELOP AN  EFFICIENT MEANS SUCH AS IN REFEQHOUSELEFT FOR COMPUTING  AHV WITH THE HOUSEHOLDER MATRIX ON THE RIGHTITEM IN THIS PROBLEM YOU WILL DEMONSTRATE THAT DIRECT COMPUTATION OF  THE PSEUDOINVERSE DIRECTLY IS NUMERICALLY INFERIOR TO COMPUTATION  USING A MATRIX FACTORIZATION SUCH AS THE QR OR CHOLESKY  FACTORIZATION  SUPPOSE IT IS DESIRED FIND THE LEASTSQUARES  SOLUTION TO A  BEGINBMATRIX  10000       1000110001   10002 10002   10003 10003   10004 10004   10005 ENDBMATRIXXBF  BEGINBMATRIX20001  20003  20005  20007   20009ENDBMATRIXTHE EXACT SOLUTION IS XBF  11TBEGINENUMERATEITEM DETERMINE THE CONDITION NUMBER OF A AND ATA IF POSSIBLEITEM COMPUTE THE LEASTSQUARES SOLUTION USING THE FORMULA  XBFHAT  ATA1AT BBF EXPLICITLYITEM COMPUTE THE SOLUTION USING THE QR DECOMPOSITION WHERE THE QR  DECOMPOSITION IS COMPUTED USING HOUSEHOLDER TRANSFORMATIONSITEM COMPUTE THE SOLUTION USING THE QR DECOMPOSITION WHERE THE QR  DECOMPOSITION IS COMPUTED USING GRAMSCHMIDTITEM COMPUTE THE SOLUTION USING THE CHOLESKY FACTORIZATIONITEM COMPARE THE ANSWERS AND COMMENTENDENUMERATEITEM ROTATION MATRICES INDEXROTATION MATRIX VERIFY THE FOLLOWING  PROPERTIES ABOUT ROTATION MATRICES OF THE FORM AND PROVIDE A  GEOMETRIC INTERPRETATION GTHETA  BEGINBMATRIX COS THETA  SIN THETA  SIN  THETA  COS THETA ENDBMATRIXBEGINENUMERATEITEM GTHETA GTHETA  IITEM GTHETA2  G2THETAITEM GTHETA GPHI  GTHETAPHIENDENUMERATETHUS GTHETA THETA IN RBB FORMS A GROUP  ITEM CITERADER1996 THE GRAMMIAN MATRIX FOR A LEASTSQUARES PROBLEM IS RK  AHAWHERE A  BEGINBMATRIXQBF1H  QBF2H  VDOTS  QBFKHENDBMATRIXLET Z  AH SO WE CAN WRITE R  ZZHBEGINENUMERATEITEM SHOW THAT IF Z1   Z Q1 WHERE Q1 IS A UNITARY MATRIX  THEN WE CAN WRITE R  Z1 Z1HITEM DESCRIBE HOW TO CONVERT Z TO A LOWER TRIANGULAR MATRIX L BY A  SERIES OF ORTHOGONAL TRANSFORMATIONS  THUS WE CAN WRITE R  LLHITEM DESCRIBE HOW TO SOLVE THE EQUATION R XBF  YBF FOR YBF  BASED UPON THIS REPRESENTATION OF R  ENDENUMERATENOTE THAT SINCE WE NEVER HAVE TO COMPUTE R EXPLICITLY THE NUMERICALPROBLEMS OF COMPUTING AHA NEVER ARISE  FOR FIXED POINT ARITHMETICTHE WORDLENGTH REQUIREMENTS ARE APPROXIMATELY HALF AS LONG ASCOMPUTING THE GRAMMIAN AND THEN FACTORING IT CITERADERSTEINHARDTITEM THE QR FACTORIZATION USING GIVENS ROTATIONS WAS GIVEN ONLY FOR  EM REAL MATRICES  DETERMINE A MODIFICATION TO THE ALGORITHM TO  HANDLE COMPLEX MATRICES  HINT ROTATE IN PHASEITEM WRITE AND TEST AN EFFICIENT ROUTINE TO COMPUTE QT B USING THE TT    THETA DATA RETURNED FROM TT QRGIVENS SIMILAR TO THE ROUTINE  TT QRTBT WRITTEN FOR THE HOUSEHOLDER METHOD  AN APPROPRIATE  CALLING SEQUENCE MIGHT BE TT QRTBTGIVBTHETAMN  QRQTBGIVMITEM IN THE CORDIC ALGORITHM THE MICROROTATION ANGLES ARE THETAI   TAN1 2I I01LDOTS  PROVE THAT IF THETA LEQ  THETAK THEN A REPRESENTATION OF THETA EXISTS OF THE FORM THETA  SUMIK1INFTY RHOI THETAIITEM DETERMINE A REPRESENTATION OF THETA  23CIRC USING THE  ANGLES IN THE CORDIC REPRESENTATIONITEM FOR THE LU FACTORIZATION IT IS POSSIBLE TO REPRESENT BOTH THE  L AND U FACTORS IN THE ORIGINAL MATRIX A WITH POSSIBLY SOME  PERMUTATION INFORMATION STORED SEPARATELY  DETERMINE A MEANS BY  WHICH Q AND R FACTORS CAN BE STORED IN THE ORIGINAL MATRIX A  FOR BOTH THE HOUSEHOLDER AND GIVENS METHODSITEM ONE WAY TO GENERALIZE THE GIVENS METHOD IS TO DEAL WITH A  ROTATION MATRIX WITH COMPLEX NUMBERS  FOR A VECTOR XBF IN  CBB2 FIND AN ALGORITHM FOR DETERMINING A UNITARY MATRIX OF THE  FORM  Q  BEGINBMATRIX C  SBAR  SC ENDBMATRIX SUCH THAT C IN RBB C2  S2 1 AND THE SECOND COMPONENT OF QXBF IS ZEROITEM SUPPOSE A  I  VBF VBFT  FIND THE CHOLESKY FACTORIZATION  OF AITEM FAST GIVENS TRANSFORMATIONS  LET D BE A DIAGONAL MATRIX LET  M BE A MATRIX SUCH THAT MH M  D AND LET Q  MD12  BEGINENUMERATE  ITEM SHOW THAT Q IS ORTHOGONAL  ITEM FOR A MATSIZE22 MATRIX M OF THE FORM M  BEGINBMATRIX BETA  1  1  ALPHA ENDBMATRIXSHOW HOW TO CHOOSE ALPHA AND BETA SO THAT FOR A 2VECTOR XBF M XBF  BEGINBMATRIX TIMES  0 ENDBMATRIXTHAT IS M SETS THE SECOND COMPONENT OF XBF TO ZERO AND MDMH  D1IS DIAGONAL  THUS MXBF ACTS LIKE A GIVENS ROTATION BUT WITHOUTTHE NEED TO COMPUTE A SQUARE ROOTITEM DESCRIBE HOW TO APPLY THE MATSIZE22 MATRIX TO PERFORM A  FAST QR DECOMPOSITION OF A MATRIX A ITEM WRITE AND TEST A SC MATLAB FUNCTION TO IMPLEMENT THE FAST QR  ENDENUMERATEFURTHER INFORMATION ON FAST GIVENS INCLUDING SOME IMPORTANT ISSUES OFSTABILIZING THE NUMERICAL COMPUTATIONS ARE GIVEN IN CITEGVLITEM LABELEXPOTH IN THE TEXT RELATED TO FIGURE REFFIGHOUSE1  IT WAS STATED THAT THE DIAGONALS OF A PARALLELOGRAM ARE ORTHOGONAL  PROVE THAT THIS IS TRUEITEM MATRIX SPACES FROM THE QR FACTORIZATION  INDEXFOUR    FUNDAMENTAL SUBSPACESIF A IN MMN  WHERE MN AND A HAS FULL COLUMN RANK THE QR FACTORIZATION CAN  BE WRITTEN AS A  Q1 Q2 BEGINBMATRIXR1  0 ENDBMATRIXWHERE Q1 IN MMN AND Q2 IN MMMN AND R1 INMNN  SHOW THATBEGINENUMERATEITEM AQ1R1  THIS IS KNOWN AS THE SKINNY QR FACTORIZATION   OBSERVE THAT THE COLUMNS OF Q1 ARE ORTHOGONALITEM RANGEA  RANGEQ1ITEM RANGEAPERP  RANGEQ2ENDENUMERATEENDEXERCISESSETEXSECTREFSECLUFACTBEGINEXERCISESITEM FOR THE MATRIX A  BEGINBMATRIX 2  5  9  1  4  7  3  2 1ENDBMATRIXDETERMINE THE LU FACTORIZATION BOTH WITH AND WITHOUT PIVOTING ITEM SHOW HOW TO OBTAIN THE L MATRIX FROM THE TT LU RETURNED   FROM TT NEWLUITEM WRITE A TT MATLAB ROUTINE TO SOLVE THE SYSTEM OF EQUATIONS  AXBF  BBF ASSUMING THAT THE LU FACTORIZATION IS OBTAINED USING  TT NEWLUITEM VERIFY THE FOLLOWING FACTS ABOUT TRIANGULAR MATRICES  BEGINENUMERATE  ITEM THE INVERSE OF AN UPPER TRIANGULAR MATRIX IS UPPER    TRIANGULAR  THE INVERSE OF A LOWER TRIANGULAR MATRIX IS LOWER    TRIANGULAR  ITEM THE PRODUCT OF TWO UPPER TRIANGULAR MATRICES IS UPPER TRIANGULAR  ENDENUMERATE ITEM SHOW THAT IF A MATRIX IS DIAGONALLY DOMINANT THEN NO PIVOTING   IS REQUIRED TO ENSURE THAT LIJ  1ITEM LABELEXNUMPOOR THIS EXERCISE ILLUSTRATES THE POTENTIAL  DIFFICULTY OF LU FACTORIZATION WITHOUT PIVOTING  SUPPOSE IT IS  DESIRED TO SOLVE THE SYSTEM OF EQUATIONS BEGINBMATRIX 2 45  612001  1  4  8  3ENDBMATRIXXBF  BEGINBMATRIX 5 33002 21ENDBMATRIXTHE TRUE SOLUTION TO THIS SYSTEM OF EQUATIONS IS XBF  1 2 3TAND THE MATRIX A IS VERY WELL CONDITIONED  COMPUTE THE SOLUTION TOTHIS PROBLEM USING THE LU DECOMPOSITION WITHOUT PIVOTING USINGARITHMETIC ROUNDED TO THREE SIGNIFICANT PLACES  THEN COMPUTE USINGPIVOTING AND COMPARE THE ANSWERS WITH THE EXACT RESULTEXSKIP SETEXSECTREFSECCHOLESKYITEM COMPUTE THE CHOLESKY FACTORIZATION OF  A  BEGINBMATRIX 464  62518  41822 ENDBMATRIXAS ALLT  THEN WRITE THIS AS A  UTDU WHERE U IS AN UPPER TRIANGULAR MATRIXWITH 1 ALONG THE DIAGONALITEM SHOW THAT REFEQBCHOL IS TRUEITEM GIVEN A ZEROMEAN DISCRETETIME INPUT SIGNAL FT WHICH WE  FORM INTO A VECTOR QBFT  BEGINBMATRIX FBART  FBART1  CDOTS   FBARTMENDBMATRIXTWE DESIRE TO FORM A SET OF OUTPUTS BBFT  BEGINBMATRIX B0T  B1T  CDOTS  BMTENDBMATRIXTBY BBFT  HQBFT THAT ARE UNCORRELATED THAT IS EBIT BBARJT  0 TEXT IF INEQ JLET R  EQBFT QBFHT BE THE CORRELATION MATRIX OF THE INPUTDATA  DETERMINE THE MATRIX H WHICH DECORRELATES THE INPUT DATA  ITEM LET X  XBF1 XBF2 LDOTS XBFN BE A SET OF    REALVALUED ZEROMEAN DATA WITH CORRELATION MATRIX RXX  FRAC1N XXTDETERMINE A TRANSFORMATION ON X THAT PRODUCES A DATA SET Y  Y  H XSUCH THAT RYY  FRAC1N YYTIS EQUAL TO AN IDENTITY ITEM WRITE A SC MATLAB ROUTINE TT BACKCHOL WHICH FACTORS A   SYMMETRIC POSITIVE DEFINITE MATRIX AS A  UUH WHERE U IS AN   UPPER TRIANGULAR MATRIXITEM WRITE SC MATLAB ROUTINES TT X FORSUBLB AND TT    BACKSUBUB TO SOLVE LXBF  BBF FOR A LOWER TRIANGULAR MATRIX    L AND U XBF  BBF FOR AN UPPER TRIANGULAR MATRIX U  ITEM DEVELOP A MEANS OF COMPUTING THE SOLUTION TO THE WEIGHTED    LEASTSQUARES PROBLEM XBF  AHWA1 AH W BBF USING THE    CHOLESKY FACTORIZATION ITEM SUPPOSE A  I  VBF VBFT  FIND THE CHOLESKY FACTORIZATION   OF AEXSKIPSETEXSECTREFSECQRCOMP ITEM PROVE LEMMA REFLEMQRSAMENORMITEM SHOW THAT FOR A UNITARY MATRIX Q  INDEXUNITARYDETERMINANT BOXEDDETQ  1ITEM COLUMNSPACE PROJECTORS  LET X BE A RANKR MATRIX  SHOW  THAT THE MATRIX WHICH PROJECTS ORTHOGONALLY ONTO THE COLUMN SPACE OF  X IS PX  QMCSUBRANGE1RQMCSUBRANGE1RHWHERE X  QR IS THE QR FACTORIZATION OF X ANDQMCSUBRANGE1R IS THE SC MATLAB NOTATION FOR THE FIRSTR COLUMNS OF QITEM REGARDING THE CAYLEY FORMULA INDEXCAYLEY TRANSFORMATION  BEGINENUMERATE  ITEM SHOW THAT Z IN REFEQCAYLEYPRE HAS Z21  ITEM SHOW THAT U IN REFEQCAYLEYFORM SATISFIES UUHI  ITEM SOLVE REFEQCAYLEYFORM FOR R THUS FINDING A MAPPING    FROM UNITARY MATRICES TO HERMITIAN MATRICES  ITEM A MATRIX S IS EM SKEWSYMMETRIC IF ST  S    INDEXSKEWSYMMETRIC SHOW THAT    IF S IS SKEW SYMMETRIC THEN Q  ISIS1 IS ORTHOGONAL  ENDENUMERATEITEM LABELEXHOUSE1 FOR THE HOUSEHOLDER TRANSFORMATION  REFLECTION MATRIX H  I  2 VBF VBFHVBFHVBF VERIFY THE  FOLLOWING PROPERTIES AND PROVIDE A GEOMETRIC INTERPRETATION  BEGINENUMERATE  ITEM HVBF  VBF  ITEM IF ZBF PERP VBF THEN H ZBF  ZBF  ITEM HH  H  ITEM HH H  HHH  I  ITEM FOR VECTORS XBF AND YBF LA XBFYBF RA  LA HXBFHYBFRATHUS  XBF2   HXBF2  ENDENUMERATEITEM DETERMINE A ROTATION THETA IN C  COS THETA AND S   SIN THETA SUCH THAT BEGINBMATRIX C  S  S  C ENDBMATRIXBEGINBMATRIX3  4ENDBMATRIX  BEGINBMATRIX5  0 ENDBMATRIXITEM LABELEXSTACKORTHOG SHOW THAT IF Q IS AN ORTHOGONAL MATRIX THEN BEGINBMATRIX 1  ZEROBF  0  Q ENDBMATRIXIS ALSO ORTHOGONALITEM CITEPAGE 74GVL SHOW THAT IF Q  Q1  JQ2 IS UNITARY  WHERE QI IN RBBMATSIZEMM THEN THE MATSIZE2M2M  MATRIX Z  BEGINBMATRIX Q1  Q2  Q2  Q1 ENDBMATRIXIS ORTHOGONALITEM THE HOUSEHOLDER MATRIX DEFINED IN REFEQHVDEF USES A  REFLECTION WITH RESPECT TO AN ORTHOGONAL PROJECTION  IN THIS  PROBLEM WE WILL EXPLORE THE HOUSEHOLDER MATRIX USING A WEIGHTED  PROJECTION AND ITS ASSOCIATED INNER PRODUCT  LET W BE A HERMITIAN  MATRIX AND DEFINE SEE REFEQPROJMAT2 HVW  I  2PVW  I  2FRACVBF VBFH WVBFH W VBFBEGINENUMERATEITEM SHOW THAT HVWH W HVW  W AND THAT HVWXBFW   XBFWWHERE XBFW  XBFH W XBFITEM SHOW THAT HVW VBF VBFITEM SHOW THAT HVW HVW  I SO HVW IS A REFLECTIONITEM DETERMINE A MEANS OF CHOOSING VBF SO THAT HVW XBF  BEGINBMATRIX1  0  VDOTS  0 ENDBMATRIXALPHAFOR SOME ALPHAENDENUMERATEITEM LABELEXHOUSEMAX CONSIDER THE PROBLEM YBFT  XBFT A  BEGINENUMERATE  ITEM DETERMINE XBF SUCH THAT THE FIRST COMPONENT OF YBF IS    MAXIMIZED SUBJECT TO THE CONSTRAINT THAT  XBF2  1  WHAT    IS THE MAXIMUM VALUE OF Y1 IN THIS CASE  ITEM LET H BE A HOUSEHOLDER MATRIX OPERATING ON THE FIRST COLUMN    OF A  COMMENT ON THE NONZERO VALUE OF THE FIRST COLUMN HA    COMPARED WITH Y1 OBTAINED IN THE PREVIOUS PART  ENDENUMERATEITEM LET XBF AND YBF BE NONZERO VECTORS IN RBBN  DETERMINE AHOUSEHOLDER MATRIX P SUCH THAT PXBF IS A MULTIPLE OF YBFGIVE A GEOMETRIC INTERPRETATION OF YOUR ANSWERITEM THE COMPUTATION IN REFEQHOUSELEFT APPLIES THE HOUSEHOLDER  MATRIX TO THE EM LEFT OF A MATRIX AS HV A  DEVELOP AN  EFFICIENT MEANS SUCH AS IN REFEQHOUSELEFT FOR COMPUTING  AHV WITH THE HOUSEHOLDER MATRIX ON THE RIGHTITEM IN THIS PROBLEM YOU WILL DEMONSTRATE THAT DIRECT COMPUTATION OF  THE PSEUDOINVERSE DIRECTLY IS NUMERICALLY INFERIOR TO COMPUTATION  USING A MATRIX FACTORIZATION SUCH AS THE QR OR CHOLESKY  FACTORIZATION  SUPPOSE IT IS DESIRED FIND THE LEASTSQUARES  SOLUTION TO A  BEGINBMATRIX  10000       1000110001   10002 10002   10003 10003   10004 10004   10005 ENDBMATRIXXBF  BEGINBMATRIX20001  20003  20005  20007   20009ENDBMATRIXTHE EXACT SOLUTION IS XBF  11TBEGINENUMERATEITEM DETERMINE THE CONDITION NUMBER OF A AND ATA IF POSSIBLEITEM COMPUTE THE LEASTSQUARES SOLUTION USING THE FORMULA  XBFHAT  ATA1AT BBF EXPLICITLYITEM COMPUTE THE SOLUTION USING THE QR DECOMPOSITION WHERE THE QR  DECOMPOSITION IS COMPUTED USING HOUSEHOLDER TRANSFORMATIONS ITEM COMPUTE THE SOLUTION USING THE QR DECOMPOSITION WHERE THE QR   DECOMPOSITION IS COMPUTED USING GRAMSCHMIDTITEM COMPUTE THE SOLUTION USING THE CHOLESKY FACTORIZATIONITEM COMPARE THE ANSWERS AND COMMENTENDENUMERATEITEM ROTATION MATRICES INDEXROTATION MATRIX VERIFY THE FOLLOWING  PROPERTIES ABOUT ROTATION MATRICES OF THE FORM AND PROVIDE A  GEOMETRIC INTERPRETATION GTHETA  BEGINBMATRIX COS THETA  SIN THETA  SIN  THETA  COS THETA ENDBMATRIXBEGINENUMERATEITEM GTHETA GTHETA  IITEM GTHETA GPHI  GTHETAPHIENDENUMERATENOTE THAT GTHETA THETA IN RBB FORMS A GROUP  ITEM CITERADER1996 THE GRAMMIAN MATRIX FOR A LEASTSQUARES PROBLEM IS RK  AHAWHERE A  BEGINBMATRIXQBF1H  QBF2H  VDOTS  QBFKHENDBMATRIXLET Z  AH SO WE CAN WRITE RK  ZZHBEGINENUMERATEITEM SHOW THAT IF Z1   Z Q1 WHERE Q1 IS A UNITARY MATRIX  THEN WE CAN WRITE RK  Z1 Z1HITEM DESCRIBE HOW TO CONVERT Z TO A LOWER TRIANGULAR MATRIX L BY A  SERIES OF ORTHOGONAL TRANSFORMATIONS  THUS WE CAN WRITE RK  LLHITEM DESCRIBE HOW TO SOLVE THE EQUATION RK XBF  YBF FOR YBF  BASED UPON THIS REPRESENTATION OF RK  ENDENUMERATENOTE THAT SINCE WE NEVER HAVE TO COMPUTE R EXPLICITLY THE NUMERICALPROBLEMS OF COMPUTING AHA NEVER ARISE  FOR FIXED POINT ARITHMETICTHE WORDLENGTH REQUIREMENTS ARE APPROXIMATELY HALF AS LONG ASCOMPUTING THE GRAMMIAN AND THEN FACTORING IT CITERADERSTEINHARDT ITEM THE QR FACTORIZATION USING GIVENS ROTATIONS WAS GIVEN ONLY FOR   EM REAL MATRICES  DETERMINE A MODIFICATION TO THE ALGORITHM TO   HANDLE COMPLEX MATRICES  HINT ROTATE IN PHASEITEM WRITE AND TEST AN EFFICIENT ROUTINE TO COMPUTE QT B USING THE TT    THETA DATA RETURNED FROM TT QRGIVENS SIMILAR TO THE ROUTINE  TT QRTBT WRITTEN FOR THE HOUSEHOLDER METHOD  AN APPROPRIATE  CALLING SEQUENCE MIGHT BE TT QRTBTGIVBTHETAMN  QRQTBGIVM ITEM IN THE CORDIC ALGORITHM THE MICROROTATION ANGLES ARE THETAI    TAN1 2I I01LDOTS  PROVE THAT IF THETA LEQ   THETAK THEN A REPRESENTATION OF THETA EXISTS OF THE FORM  THETA  SUMIK1INFTY RHOI THETAI ITEM DETERMINE A REPRESENTATION OF THETA  23CIRC USING THE  ANGLES IN THE CORDIC REPRESENTATIONITEM WE HAVE SEEN THAT IN THE LU FACTORIZATION IT IS POSSIBLE TO OVERWRITETHE ORIGINAL MATRIX A WITH INFORMATION ABOUT THE L AND UFACTORS WITH POSSIBLY SOME PERMUTATION INFORMATION STOREDSEPARATELY  IN THIS QUESTION WE WILL DETERMINE THAT THE SAMEOVERWRITING REPRESENTATION OF A ALSO WORKS FOR HOUSEHOLDER ANDGIVENS APPROACHES TO THE QR FACTORIZATION  BEGINENUMERATEITEM DETERMINE A MEANS BY WHICH THE Q AND R FACTORS COMPUTED  USING HOUSEHOLDER TRANSFORMATIONS CAN BE OVERWRITTEN IN THE ORIGINAL  A MATRIX  HINT LET VBF1  1ITEM DETERMINE HOW THE Q AND R FACTORS COMPUTED USING GIVENS  TRANSFORMATIONS CAN BE OVERWRITTEN IN THE ORIGINAL A MATRIXENDENUMERATE ITEM ONE WAY TO GENERALIZE THE GIVENS METHOD IS TO DEAL WITH A   ROTATION MATRIX WITH COMPLEX NUMBERS  FOR A VECTOR XBF IN   CBB2 FIND AN ALGORITHM FOR DETERMINING A UNITARY MATRIX OF THE   FORM   Q  BEGINBMATRIX C  SBAR  SC ENDBMATRIX   SUCH THAT C IN RBB C2  S2 1 AND THE SECOND COMPONENT OF Q XBF IS ZEROITEM FAST GIVENS TRANSFORMATIONSINDEXGIVENS TRANSFORMATIONSFAST  LET D BE A DIAGONAL MATRIX LET  M BE A MATRIX SUCH THAT MT M  D AND LET Q  MD12  BEGINENUMERATE  ITEM SHOW THAT Q IS ORTHOGONAL  ITEM FOR A MATSIZE22 MATRIX M1 OF THE FORM M1  BEGINBMATRIX BETA  1  1  ALPHA ENDBMATRIXSHOW HOW TO CHOOSE ALPHA AND BETA SO THAT FOR A 2VECTOR XBF M1 XBF  BEGINBMATRIX TIMES  0 ENDBMATRIXTHAT IS M SETS THE SECOND COMPONENT OF XBF TO ZERO AND M1DM1H  D1IS DIAGONAL  THUS M1XBF ACTS LIKE A GIVENS ROTATION BUT WITHOUTTHE NEED TO COMPUTE A SQUARE ROOTITEM DESCRIBE HOW TO APPLY THE MATSIZE22 MATRIX TO PERFORM A  FAST QR DECOMPOSITION OF A MATRIX A  ITEM WRITE AND TEST A SC MATLAB FUNCTION TO IMPLEMENT THE FAST QRENDENUMERATEFURTHER INFORMATION ON FAST GIVENS INCLUDING SOME IMPORTANT ISSUES OFSTABILIZING THE NUMERICAL COMPUTATIONS ARE GIVEN IN CITEGVLITEM LABELEXPOTH IN THE TEXT RELATED TO FIGURE REFFIGHOUSE1  IT WAS STATED THAT THE DIAGONALS OF AN EQUILATERAL PARALLELOGRAM ARE ORTHOGONAL  PROVE THAT THIS IS TRUEITEM MATRIX SPACES FROM THE QR FACTORIZATION  INDEXFOUR FUNDAMENTAL SUBSPACESIF A IN MMN  WHERE MN AND A HAS FULL COLUMN RANK THE QR FACTORIZATION CAN  BE WRITTEN AS A  Q1 Q2 BEGINBMATRIXR1  0 ENDBMATRIXWHERE Q1 IN MMN AND Q2 IN MMMN AND R1 INMNN  SHOW THATBEGINENUMERATEITEM AQ1R1  THIS IS KNOWN AS THE SKINNY QR FACTORIZATION   OBSERVE THAT THE COLUMNS OF Q1 ARE ORTHOGONALITEM RANGEA  RANGEQ1ITEM RANGEAPERP  RANGEQ2ENDENUMERATEENDEXERCISESSECTIONREFERENCESLABELSECREFFACTCOMPUTATION OF MATRIX FACTORIZATIONS IS WIDELY DISCUSSED IN A VARIETYOF NUMERICAL ANALYSIS TEXTS  THE CONNECTION OF THE LU WITH GAUSSIANELIMINATION IS DESCRIBED WELL IN CITESTRANG1988  MOST OF THEMATERIAL HERE ON THE QR FACTORIZATION HAS BEEN DRAWN FROMCITEGVL  IN ADDITION TO FACTORIZATIONS THIS SOURCE ALSO PROVIDESPERTURBATION ANALYSES OF THE ALGORITHMS AND COMPARISONS OF VARIANTSOF THE ALGORITHMS  A FAST GIVENS ROTATION ALGORITHM WHICH DOESNOT REQUIRE SQUARE ROOTS IS ALSO PRESENTED THERE  VARIANTS ON THECHOLESKY ALGORITHM PRESENTED HERE ARE PRESENTED IN CITEGVL  UPDATEALGORITHMS FOR THE QR FACTORIZATION IN ADDITION TO THE ONE FOR UPDATEBY ADDING A ROW ARE PRESENTED INCLUDING UPDATES FOR A RANKONEMODIFICATION AND COLUMN MODIFICATIONS ARE ALSO PRESENTED INCITEGVLTHE HOUSEHOLDER TRANSFORMATION APPEARED IN CITEHOUSEHOLDER1958APPLICATION OF HOUSEHOLDER TRANSFORMATIONS WITH WEIGHTED PROJECTIONSIS DISCUSSED IN CITERADERSTEINHARDTAPPLICATION OF QR FACTORIZATIONS TO LEASTSQUARES FILTERING ISEXTENSIVELY DISCUSSED IN CITEPROAKISRADER AND CITEHAYKIN1996CITEPROAKISRADER ALSO DEMONSTRATES APPLICATION OF GRAMSCHMIDT  ANDMODIFIED GRAMSCHMIDT TO LEASTSQUARES AND RECURSIVE UPDATES OFLEASTSQUARES  A DISCUSSION OF APPLICATIONS OF HOUSEHOLDER TRANSFORMSTO SIGNAL PROCESSING APPEARS IN CITESTEINHARDT1988 LOCAL VARIABLES TEXMASTER TEST ENDSECTIONOPERATOR NORMSLABELSECMATNORMAN OPERATOR NORM LIKE ANY NORM MUST SATISFY THE PROPERTIES DESCRIBEDIN SECTION REFSECNORMVS  THERE ARE SEVERAL DIFFERENT WAYS OFDEFINING THE NORM OF A TRANSFORMATION OPERATOR  ONE WAY IS TODEFINE THE NORM SO THAT IT PROVIDES INDICATION OF THE MAXIMAL AMOUNTOF CHANGE OF LENGTH OF A VECTOR THAT IT OPERATES ON  LET X AND YBE LP OR LP AND LET A BE A LINEAR OPERATOR AMC XRIGHTARROWY  THE P BF OPERATOR NORM OR PNORM OR LP NORM OF AIS  A P  SUPXIN X NEQ 0FRACAX PXP SUPX IN X X P  1AXPWHERE  CDOT P IS THE PNORM DEFINED IN SECTIONREFSECNORMVS  NOTE AX IN Y SO THE NORM AXP IS THENORM ON Y  WE COULD IN GENERAL USE DIFFERENT NORMS FOR X ANDAX BUT USUALLY THIS IS NOT DONE  THE NORM ON A SO OBTAINEDIS SAID TO BE EM SUBORDINATE TO THE NORM ON XINDEXSUBORDINATE NORM INDEXNORMSUBORDINATE FOR A SUBORDINATENORM IT IS STRAIGHTFORWARD TO VERIFY THAT I  1 WHERE I ISTHE IDENTITY OPERATOR  GEOMETRICALLY A SUBORDINATE NORM MEASURES THEMAXIMUM EXTENT THAT A TRANSFORMS THE UNIT CIRCLE  THE CONCEPT ISSHOWN IN FIGURE REFFIGOPNORMBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRNORM1    CAPTIONGEOMETRY OF THE OPERATOR NORM    LABELFIGOPNORM  ENDCENTERENDFIGURETHE PNORMS HAVE THE PROPERTY THAT AXBF P LEQ AP XBFPTHUS A BOUNDS THE AMPLIFYING POWER OF THE MATRIX AALSO THE PNORMS SATISFY THE BF SUBMULTIPLICATIVE PROPERTYINDEXSUBMULTIPLICATIVE PROPERTY ABP LEQ AP B PTHIS IS STRAIGHTFORWARD TO SHOW SINCE BY THE DEFINITION OF THE PNORMFOR ALL X IN X  AB X  LEQ  A   BX  LEQ A B X SUBSECTIONBOUNDED OPERATORSTHIS SECTION IS SOMEWHAT TECHNICAL AND MANY READERS MAY NEED ONLY THEFIRST DEFINITIONBEGINDEFINITIONIF THE NORM OF A TRANSFORMATION IS FINITE THE TRANSFORMATION IS SAIDTO BE EM BOUNDED INDEXBOUNDEDENDDEFINITIONTHE FOLLOWING THEOREM PRESENTS A RATHER REMARKABLE FACT ABOUTBOUNDED LINEAR OPERATORS BEGINTHEOREM LABELTHMCONTBOUND   A LINEAR OPERATOR AMC X RIGHTARROW Y IS BOUNDED IF AND ONLY IF   IT IS CONTINUOUS ENDTHEOREMSINCE A LINEAR FUNCTIONAL IS A LINEAR OPERATOR THE SAME THEOREMAPPLIES TO FUNCTIONALSBEGINPROOF  SUPPOSE THAT A IS BOUNDED WITH M SUCH THAT AX LEQ M  X FOR ALL X IN X  LET XN BE A SEQUENCE APPROACHING  ZERO XN RIGHTARROW 0  THEN AXN LEQ M  XNRIGHTARROW  0  BY THE PROPERTIES OF CONTINUITY CONTINUOUS FUNCTIONS PRESERVE  CONVERGENCE IT FOLLOWS THAT A IS CONTINUOUS    CONVERSELY ASSUME A IS CONTINUOUS  THEN THERE IS A DELTA  0  SUCH THAT AX1 FOR X   DELTA  THEN SINCE THE NORM OF  DELTA XX IS EQUAL TO DELTA A X   A XDELTA X X XDELTA XDELTATHE VALUE M1DELTA SERVES AS A BOUND FOR AENDPROOFTHE FOLLOWING THEOREM IS OF GREAT UTILITY BY SHOWING THAT LINEAROPERATORS FROM FINITEDIMENSIONAL SPACE ARE CONTINUOUS WE CANCONCLUDE FROM THE PREVIOUS THEOREM THAT THEY ARE ALSO BOUNDED  SINCE MANYOF THE RESULTS OF THIS CHAPTER RELY ON BOUNDED LINEAR OPERATORS THISTHEOREM REASSURES US THAT MATRICES  OPERATORS ON FINITE DIMENSIONALSPACES  WILL WORKBEGINTHEOREM LABELTHMFDBD  LET AMC X RIGHTARROW Y BE A LINEAR OPERATOR WHERE X AND Y  ARE NORMED LINEAR SPACES  IF X IS FINITE DIMENSIONAL THEN A IS  CONTINUOUSENDTHEOREMNOTE THAT THIS THEOREM DOES NOT ASSUME THAT Y IS FINITE DIMENSIONALPROOF OF THEOREM REFTHMFDBD MAKES USE OF THE FOLLOWING LEMMAWHICH IS THE MOST TECHNICAL PART OF THIS SECTIONBEGINLEMMA CITEPAGE 265NAYLORSELL LABELLEMLBD  LET X BE A FINITEDIMENSIONAL NORMED LINEAR SPACE AND LET  XBF1XBF2LDOTS XBFN BE A HAMEL BASIS FOR  X INDEXHAMEL BASIS  THEN FOR XBF IN X EACH COEFFICIENT  ALPHAI IN THE EXPANSION XBF  ALPHA1 XBF1  ALPHA2 XBF2  CDOTS  ALPHAN XBFNIS A CONTINUOUS LINEAR FUNCTION OF XBF  BEING CONTINUOUS IT ISBOUNDED SO THERE IS A CONSTANT M SUCH THAT ALPHAI LEQ M XBFENDLEMMABEGINPROOFSHOWING LINEARITY IS STRAIGHTFORWARD AND IS OMITTEDIT WILL SUFFICE TO SHOW THAT THERE IS AN M0 SUCH THATBEGINEQUATION MALPHA1  ALPHA2  CDOTS  ALPHAN LEQ XBFLABELEQLBD1ENDEQUATIONSINCE IT FOLLOWS THAT ALPHAI LEQ M1  XBF  WE WILLPROVE REFEQLBD1 FIRST FOR COEFFICIENTS  ALPHA1LDOTSALPHAN SATISFYING THE CONDITION ALPHA1 CDOTS  ALPHAN 1  LET A  ALPHA1LDOTSALPHAN ALPHA1 CDOTS ALPHAN1THIS SET IS CLOSED AND BOUNDED COMPACT  NOW DEFINE A FUNCTIONFMC A RIGHTARROW RBB BYBEGINEQUATION FALPHA1LDOTSALPHAN   ALPHA1 XBF1  CDOTS ALPHAN XBFNLABELEQLBD2ENDEQUATIONIT CAN BE SHOWN THAT F CONTINUOUS AND IT IS CLEAR THAT F0  LET M  MINALPHA1LDOTSALPHAN IN A FALPHA1LDOTSALPHANSINCE F IS CONTINUOUS ON A CLOSED BOUNDED SET THIS MINIMUM DOESEXIST FOR SOME POINT ALPHA1 LDOTS ALPHAN IN A  HENCEWE HAVE FOUND A POINT M THAT SATISFIES REFEQLBD1  IF M0THEN ALPHA1 XBF1  CDOTS  ALPHAN XBFN  0CONTRADICTING THE FACT THAT XBFI IS A BASIS LINEARLYINDEPENDENT  HENCE M0FOR GENERAL SETS OF COEFFICIENTS  ALPHAI SET BETA ALPHA1  CDOTS  ALPHAN  IF BETA0 THE RESULT ISTRIVIAL  IF BETA0 THEN WE WRITE BEGINALIGNED ALPHA1 XBF1  CDOTS  ALPHAN XBFN  BETAALPHA1BETA XBF1  CDOTS  ALPHANBETA XBFN  BETA FALPHA1BETALDOTSALPHANBETA  GEQ MBETA GEQMALPHA1  CDOTS ALPHANENDALIGNEDENDPROOFBEGINPROOF OF THEOREM REFTHMFDBD  LET XBF1XBF2LDOTSXBFN BE A HAMEL BASIS FOR X  LET XBF  IN X BE EXPRESSED IN TERMS OF THIS BASIS AS XBF  ALPHA1 XBF1  ALPHA2 XBF2  CDOTS  ALPHAN XBFNLET D  MAX1 LEQ I LEQ N A XBFI  THEN BEGINALIGNEDAXBF  AALPHA1 XBF1  ALPHA2 XBF2  CDOTS  ALPHANXBFN LEQ ALPHA1A XBF1  ALPHA2A XBF2 CDOTS  ALPHANA XBFN  LEQ DALPHA1  ALPHA2  CDOTS  ALPHANENDALIGNEDNOW BY THE LEMMA ABOVE THERE IS AN M SUCH THAT ALPHA1  CDOTSALPHAN LEQ M XBF SO THAT  A XBF  LEQ DM  XBFENDPROOFBEFORE CONSIDERING THE IMPORTANT SPECIAL CASE OF MATRIX TRANSFORMATIONSWE WILL CONSIDER SOME MORE GENERALIZED TRANSFORMATIONSBEGINEXAMPLELET X  C01 AND DEFINE AXRIGHTARROW X BY AXT  INT01 KTTAUXTAUDTAUWHERE T IN 01 AND K IS CONTINUOUS  WE WILL COMPUTE THELINFTY NORM OF THIS OPERATORBEGINALIGNED A X  MAXT IN 01 BIGLINT01 KTTAUXTAUDTAUBIGR  LEQ MAXT IN 01 INT01 KTTAUDTAU MAXT IN 01XT   MAXT IN 01 INT01 KTTAUDTAU XENDALIGNEDIT CAN BE SHOWN THAT THE INEQUALITY CAN BE ACHIEVED SO THAT A  MAXT IN 01 INT01 KTTAUDTAUSINCE KTTAU IS CONTINUOUS THEN A IS BOUNDEDENDEXAMPLEBEGINEXAMPLELET AC101 RIGHTARROW C01 BE THE OPERATOR AX  FRACDDTXTHE FUNCTION XT  SIN OMEGA0 T IN C101 HAS UNIFORM NORM 1FOR ANY VALUE OF OMEGA0 BUT AX  MAXTIN 01 OMEGA0 COS OMEGA0 TMAY HAVE NORM ARBITRARILY LARGE BY CHOOSING OMEGA0 TO BEARBITRARILY LARGE  THUS THE DIFFERENTIAL OPERATOR IS NOT BOUNDED ANDHENCE NOT CONTINUOUSENDEXAMPLESUBSECTIONTHE NEUMANN EXPANSIONLABELSECNEUMTHE NEUMANN EXPANSION PROVIDES A USEFUL EXPANSION FOR THE INVERSE OFTHE LINEAR OPERATOR IA1 INDEXNEUMANN EXPANSIONFOR A SCALAR X SUCH THAT X1 IT IS STRAIGHTFORWARD TO SHOW USINGTHE GEOMETRIC SERIES THAT 1XX2CDOTS  SUMI0INFTY XI  FRAC11X  1X1THERE IS A DIRECT EXTENSION TO LINEAR OPERATORSBEGINTHEOREM  SUPPOSE  CDOT  IS A NORM SATISFYING THE  SUBMULTIPLICATIVE PROPERTY AND A IS AN OPERATOR  WITH A  1  THEN IA1 EXISTS AND IA1  SUMI0INFTY AIENDTHEOREMBEGINPROOF  LET A1IF IA IS SINGULAR THEN THERE IS A VECTOR XBF SUCH THATIAXBF0  BUT THIS MEANS THAT XBF  AXBF LEQXBF A OR A GEQ 1  THIS IS A CONTRADICTIONBY MULTIPLICATION IT IS CLEAR THAT IAIAA2CDOTSAK1  IAKSINCE A  1 LIMKRIGHTARROW INFTY AK  0 SINCE AK LEQ AK RIGHTARROW 0 TEXT AS KRIGHTARROW INFTYTHUS IASUMI0INFTY AI  IHENCE THE QUANTITY SUMI0INFTY AIMUST BE THE INVERSE OF IAENDPROOFSUBSECTIONMATRIX NORMSINDEXMATRIX NORMSSEENORMWHEN SPECIALIZED TO MATRIX OPERATORS WE WILL CONSIDER THE CASESP1 P2 AND PINFTY WHICH ARE OF PARTICULAR INTERESTBEGINEQUATION BOXEDAINFTY  MAXXBFINFTY1 AXBFINFTY MAXI SUMJ AIJLABELEQLINFMATNORMENDEQUATIONTHAT IS IT IS THE LARGEST ROW SUMBEGINEQUATION BOXEDA1  MAXXBF11 AXBF1 MAXJ SUMI AIJLABELEQL1MATNORMENDEQUATIONTHAT IS IT IS THE LARGEST COLUMN SUMTO DEAL WITH THE L2 MATRIX NORM REQUIRES AN UNDERSTANDING OFEIGENVALUES AND CONSTRAINED OPTIMIZATION INDEXEIGENVALUEINDEXCONSTRAINED OPTIMIZATION  WE WANT TO MAXIMIZE  AXBF2 SUBJECT TO THE CONSTRAINT THAT  XBF2  1  THIS CANBE WRITTEN AS BEGINALIGNEDTEXTMAXIMIZE    AXBF22  XBFH AH A XBF TEXTSUBJECT TO   XBFH XBF  1ENDALIGNEDTHE CONSTRAINT CAN BE INCORPORATED USING A LAGRANGE MULTIPLIERINDEXLAGRANGE MULTIPLIER TOCREATE THE FUNCTIONAL J  XBFH AH A XBF  LAMBDA XBFH XBFTAKING THE GRADIENT WITH RESPECT TO XBF AND EQUATING THE RESULT TOZERO WE OBTAIN THE EQUATIONBEGINEQUATION AH A XBF  LAMBDA XBFLABELEQL2NORM1ENDEQUATIONTHE CORRESPONDING XBF MUST BE AN EIGENVECTOR OF AHAMULTIPLYING REFEQL2NORM1 BY XBFH AND RECALLING THECONSTRAINT THAT XBFHXBF  1 WE OBTAIN XBFH AH A XBF  LAMBDA XBFHXBF  LAMBDASINCE WE ARE MAXIMIZING THE QUANTITY ON THE LEFT LAMBDA MUST BETHE LARGEST EIGENVALUE OF AHA FOR AN MATSIZENN MATRIX AWITH EIGENVALUES LAMBDA1LAMBDA2LDOTSLAMBDAN THE BF  SPECTRAL RADIUS  RHOA INDEXSPECTRAL RADIUS IS DEFINED AS RHOA  MAXI LAMBDAITHE SPECTRAL RADIUS IS THE SMALLEST RADIUS OF A CIRCLE CENTERED AT THEORIGIN THAT CONTAINS ALL THE EIGENVALUES OF ATHEN THE L2 NORM IS DEFINED BY BOXED A2  SQRTRHOAHABECAUSE THE L2 NORM REQUIRES COMPUTATION OF EIGENVALUES IT IS MUCHMORE DIFFICULT TO COMPUTE THAN THE L1 OR LINFTY NORMSHOWEVER IT IS OF SIGNIFICANT THEORETICAL VALUE  WHEN A IS HERMITIAN A2  RHOATHE L2 NORM IS ALSO CALLED THE BF SPECTRAL NORMINDEXNORML2INDEXNORMSEEMETRICINDEXSPECTRAL NORMSEENORML2INDEXNORMSPECTRALSEENORML2  THE SPECTRAL NORM IS IN SOME SENSE A LOWER BOUND FOR ALL SUBMULTIPLICATIVE MATRIX NORMS BEGINTHEOREM LABELTHML2MINNORM   LET CDOT BE A NORM WHICH SATISFIES THE SUBMULTIPLICATIVE   PROPERTY  THEN RHOA LEQ A FOR AN MATSIZENN MATRIX A ENDTHEOREM THE PROOF IS LEFT AS AN EXERCISEFOR THE SUBORDINATE NORMS WE CAN ALSO SAY SOMETHING ABOUT THE NORM OFTHE INVERSE A1 WHEN IT EXISTS  FOR THE EQUATION AXBF BBF ASSUME THAT A1 EXISTS SO XBF  A1BBF  THENBEGINALIGNEDA1  MAXBBF NEQ 0FRAC A1BBF BBF  MAXXBF NEQ 0 FRACXBFAXBF  FRAC1MINX NEQ 0 FRACAXBFXBF  ENDALIGNEDFROM THIS WE CONCLUDE THATBEGINEQUATIONA11  MINXBF1 AXBFLABELEQINVMATNORMENDEQUATIONFOR EXAMPLE A121  SQRTLAMBDAMIN WHERELAMBDAMIN IS THE EM SMALLEST EIGENVALUE OF AHA A MATRIX NORM WHICH IS NOT A PNORM IS THE BF FROBENIUS NORMINDEXFROBENIUS NORM INDEXNORMFROBENIUS BOXED AF  LEFTSUMI1MSUMJ1N AIJ2RIGHT12THIS NORM IS ALSO CALLED THE BF EUCLIDEAN NORM  INDEXEUCLIDEAN  NORM INDEXNORMEUCLIDEAN IT SHOULD NOT BE CONFUSED WITH THEL2 NORM  THE FROBENIUS NORM IS OFTEN USED IN MATRIX ANALYSISSINCE IT IS RELATIVELY EASY TO COMPUTE  IT IS A NATURAL NORM FOREXAMPLE TO USE WHEN COMPARING HOW CLOSE TWO MATRICES A AND B AREUSING  A  BF  FOR THE FROBENIUS NORM I  SQRTNTHE FROBENIUS NORM CAN ALSO BE WRITTEN USINGBEGINEQUATION BOXED AF2  TRACEAHA LABELEQFROBTRACEENDEQUATIONTHE FOLLOWING RELATIONSHIPS EXIST BETWEEN THE NORMS FOR ANMATSIZEMN MATRIX ABEGINEQUATIONA2 LEQ AF LEQ SQRTNA2LABELEQ2F2ENDEQUATIONBEGINEQUATIONMAXIJAIJ LEQ A2 LEQ SQRTMNMAXIJAIJLABELEQM2MENDEQUATIONBEGINEQUATIONFRAC1SQRTNAINFTY LEQ A2 LEQSQRTMAINFTYLABELEQI2MENDEQUATIONBEGINEQUATIONFRAC1SQRTMA1 LEQA2 LEQ SQRTNA1LABELEQI2IENDEQUATIONA SEQUENCE OF MATRICES A0A1A2LDOTS IS SAID TOCONVERGE TO A MATRIX A IF LIMKRIGHTARROW INFTY AKA 0  IN THIS DEFINITION THE PARTICULAR NORM EMPLOYED IS IRRELEVANTSINCE ALL NORMS ARE EQUIVALENTBEGINEXERCISESITEM DETERMINE THE L1 L2 FROBENIUS AND LINFTY NORMS OF  THE FOLLOWING MATRICES A1  BEGINBMATRIX 4  3  3  6 ENDBMATRIX QQUAD   A2  BEGINBMATRIX 1  2  3  0  ENDBMATRIX QQUAD   A3  BEGINBMATRIX1  2  0  1 ENDBMATRIXITEM SHOW THAT THE FUNCTION F DEFINED IN REFEQLBD2 IS  CONTINUOUS  HINT  SHOW THAT FALPHA1LDOTSALPHAN   FBETA1LDOTS BETAN LEQ MALPHA1BETA1  CDOTS   ALPHAN  BETAN FOR SOME MITEM USING LEMMA REFLEMLBD SHOW THAT  BEGINENUMERATE  ITEM IF X IS A NORMED LINEAR SPACE IT IS COMPLETE  HINT LET    ZK BE A CAUCHY SEQUENCE IN X  WRITE ZK AS A LINEAR    COMBINATION OF THE BASIS VECTORS XBFI USING THE    COEFFICIENTS ALPHAKJ AND APPLY THE LEMMA TO SHOW THAT   ALPHAKJ IS A CAUCHY SEQUENCE OF REAL NUMBERS AND HENCE IS    CONVERGENT  ITEM IF X IS A NORMED LINEAR SPACE SHOW THAT EVERY FINITE    DIMENSIONAL SUBSPACE M OF X IS CLOSED  ENDENUMERATEITEM SHOW THAT REFEQLINFMATNORM IS TRUE THAT IS THAT THE  LINFTY MATRIX NORM IS THE LARGEST ROW SUMITEM SHOW THAT REFEQLINFMATNORM IS TRUE THAT IS THAT THE  L1 MATRIX NORM IS THE LARGEST COLUMN SUMITEM SHOW THAT NOT ALL NORMS SATISFY THE SUBMULTIPLICATIVE PROPERTY GVS P 57ITEM SHOW THAT FOR A SQUARE MATRIX F IF1 LEQ  11FITEM PROVE THEOREM REFTHML2MINNORMITEM USING THE NEUMANN FORMULA SHOW THAT IF F   1 THEN IF1 LEQ 1F1FOR A SUBORDINATE NORM CDOTITEM SHOW THAT FOR A SQUARE MATRIX F WITH  F1   I  IFLEQ FRACF1FHINT SHOW THAT IIF1  FIF1  STEWART P 188ITEM LET A BE NONSINGULAR AND LET E BE SUCH THAT A1E  1  SHOW THAT AE  IFA1WHERE F SATISFIES IF  IA1E1ALSO SHOW THAT F  LEQ FRACA1E1A1EITEM PROVIDE EXAMPLES DEMONSTRATING THAT THE INEQUALITIES IN  REFEQ2F2 REFEQM2M REFEQI2M AND REFEQI2I  CAN BE ACHIEVEDITEM CITEGVL FOR A MATSIZEMM MATRIX A AND A NONZERO   MATSIZEM1 VECTOR XBF SHOW THAT LEFTALEFTI  FRACXBFXBFHXBFHXBFRIGHTRIGHTF2   AF2  FRACAXBF22XBFHXBFITEM LET B BE A SUBMATRIX OF A  SHOW THAT BP LEQ  APITEM LET P BE A PROJECTION OPERATOR SEE SECTION  REFSECPROJECTIONS  SHOW THAT P1 ITEM LABELEXFROBINEQ  SHOW THAT FOR AN MATSIZEMM MATRIX  DBEGINEQUATIONFRAC1SQRTN LEFT  TRACED RIGHT LEQ  DFLABELEQFROBINEQENDEQUATIONHINT CAUCHYSCHWARZITEM LABELEQFROBINEQ2 SHOW THAT FOR SQUARE MATRICES A AND BBEGINEQUATION  ABF LEQ A2 BFLABELEQFORBINEQ2ENDEQUATION GRAY1972 P 725 24ITEM PROVE THAT IF RHOA  1 IF AND ONLY IF LIMKRIGHTARROW    INFTY AK XBF  0 FOR EVERY XBF  ITEM WEIGHTED NORMS  WE HAVE SEEN THAT WE CAN DEFINE A WEIGHTED NORM  BY  XBFW   WXBF  SHOW THAT USING THE WEIGHTED NORM   CDOT W THE CORRESPONDING SUBORDINATE MATRIX NORM IS   AW   WAW1ITEM A MATRIX A SUCH THAT A XBF  XBF IS CALLED EM    NORMPRESERVING OR EM ISOMETRIC INDEXISOMETRIC  SHOW THAT    A SQUARE MATRIX A IS ISOMETRIC IN THE SPECTRAL NORM IF AND ONLY    IF IT IS ORTHOGONAL OR UNITARY IF A IS COMPLEX    NOTE AN ORTHOGONAL MATRIX A SATISFIES ATA  I  A UNITARY    MATRIX A SATISFIES AHA  I INDEXORTHOGONAL MATRIX    INDEXUNITARY MATRIX  ORTEGA P 97ITEM CITEPAGE 74GVL SHOW THAT IF AIN RBBMATSIZEMN  HAS RANK N THEN AAH1 AT2  1ITEM CITEPAGE 74GVL SHOW THAT IF A IN MMN THEN AF  LEQ SQRTRANKAA2ENDEXERCISES LOCAL VARIABLES TEXMASTER TEST ENDSECTIONSOME RESULTS ON MATRIX RANKLABELSECRANKBEGINDEFINITION  AN MATSIZEMN MATRIX IS SAID TO BE BF FULL RANK IF THE RANK  IS AS LARGE AS POSSIBLE RANKA  MINMNAN MATSIZEMN MATRIX IS SAID TO BE BF RANK DEFICIENT IF IT ISNOT FULL RANK INDEXRANK INDEXRANKDEFICIENTENDDEFINITIONTHE FOLLOWING THEOREM PROVIDES A CHARACTERIZATION THE FOUR FUNDAMENTALSPACES OF THE PRODUCT OF MATRICES ABBEGINTHEOREM FOR MATRICES A AND B SUCH THAT AB EXISTS  BEGINENUMERATE  ITEM NULLSPACEB SUBSET NULLSPACEAB  ITEM RANGEAB SUBSET RANGEA  ITEM NULLSPACEA SUBSET NULLSPACEAB  ITEM RANGEAB SUBSET RANGEB  ENDENUMERATEENDTHEOREMBEGINPROOF  BEGINENUMERATE  ITEM IF BXBF  0 THEN ABXBF0 EVERY XBF IN NULLSPACEB    IS ALSO IN NULLSPACEAB  THUS  DIM NULLSPACEAB GEQ DIM NULLSPACEBITEM IF XBF IN RANGEAB THEN THERE IS SOME YBF SO THAT    XBF  ABYBF  ABYBF SO XBF IN RANGEA  ITEM IF YBF A  0 THEN YBF AB  0  ITEM IF XBF IN RANGEABT THEN THERE IS SOME YBF SO THATXBF  AB YBF  BTA YBF SO XBF IN RANGEBENDENUMERATEENDPROOFCOMBINING THE SECOND AND FOURTH OF ITEMS WE OBTAIN THEFOLLOWING FACTBEGINEQUATION BOXEDRANKAB LEQ RANKAQQUADQQUAD RANKAB LEQ RANKBLABELEQMATRANKPRODENDEQUATIONBEGINEXAMPLE  AN OBVIOUS BUT IMPORTANT EXAMPLE OF THIS FACT IS THAT THE MATRIX B   XBF YBFH WHERE XBF AND YBF ARE NONZERO VECTORS MUST  HAVE RANK LEQ 1 SINCE EACH VECTOR HAS RANK 1  INDEXRANKONE A  COMPUTATION OF THE FORM A  XBF YBFHIS SAID TO BE A EM RANKONE UPDATE TO THE MATRIX A  SIMILARLYIF X IS A MATSIZEM2 MATRIX AND Y IS A MATSIZE2NMATRIX THE UPDATE A  X YHIS TO BE A RANKTWO UPDATEA QUESTION EXPLORED IN SECTION REFSECSHERMAN IS HOW TO COMPUTE THEINVERSE OF A LOWRANK UPDATE OF A IF WE ALREADY KNOW THE INVERSE OFA  ENDEXAMPLEBEGINEXAMPLE  LET A BE A MATSIZE34 MATRIX OF ZEROS AND LET B BE A  MATSIZE43 MATRIX OF ZEROS  THEN THE NULLITY OF B IS 3  WHILE THE NULLITY OF AB IS 4ENDEXAMPLEBEGINTHEOREM LABELTHMCANCELLEFT CITECAMPBELLMEYER  SUPPOSE A IS  MATSIZEMN AND B AND C ARE  MATSIZENP  THEN AHAB  AHAC IF AND ONLY IF  ABACENDTHEOREMBEGINPROOF  THE RESULT CAN BE STATED EQUIVALENTLY AS AABC0 IF  ONLY IF ABC  THIS BECOMES NOW A QUESTION OF COMPARING THE  NULLSPACE OF AA AND A  WE NEED TO SHOW THAT NULLSPACEAA   NULLSPACEA  SINCE NULLSPACEA  RANGEAPERP IF  AA XBF  0 THEN AXBF  0 AND CONVERSELYENDPROOFBEGINDEFINITIONA BF SUBMATRIX OF A MATRIX A IS OBTAINED BY REMOVING ZERO OR MORECOLUMNS OF A AND ZERO OR MORE ROWS OF AENDDEFINITIONNOTATIONALLY WHEN THE ROWS AND COLUMNS OF A MATRIX RETAINED AREADJACENT THE  NOTATION CAN BE EMPLOYED AS DISCUSSED IN SECTIONREFSECMATRIXNOT  SINCE THE SUBMATRIX CANNOT BE LARGER THAN THEMATRIX WE HAVE THE FOLLOWING RESULTSBEGINFACTBOXFOR AN MATSIZEMN MATRIX A OF RANK R EVERY SUBMATRIX C ISOF RANK LEQ RENDFACTBOXBEGINFACTBOXFOR AN MATSIZEMN MATRIX A OF RANK R THERE IS AT LEAST ONEMATSIZERR MATRIX OF RANK EXACTLY RENDFACTBOXBASED UPON THE LATTER WE CAN GIVE AN EQUIVALENT DEFINITION OF THERANKBEGINFACTBOX  THE RANK OF A MATRIX IS THE SIZE OF THE LARGEST NONSINGULAR SQUARE  SUBMATRIX  THERE IS A MATSIZEKK SUBMATRIX WITH NONZERO  DETERMINANT BUT ALL MATSIZEK1K1 SUBMATRICES OF A HAVE  DETERMINANT 0ENDFACTBOXTHE FOLLOWING FACTS ARE ALSO TRUE ABOUT RANK CITEHORNJOHNSONBEGINITEMIZEITEM IF A IN MMK AND B IN MKN THEN RANKA  RANKB  K LEQ RANKAB LEQMINRANKARANKBITEM IF AB IN MMN THEN RANKAB LEQ RANKARANKBITEM FROBENIUS  IF AIN MMK B IN MKP AND  C IN MPN THEN RANKAB  RANKBC LEQ RANKB  RANKABCITEM RANK IS UNCHANGED UPON EITHER LEFT OR RIGHT MULTIPLICATION BY A  NONSINGULAR MATRIX  IF A IN MM AND C IN MN ARE  BOTH NONSINGULAR AND BIN MMN THEN RANKB  RANKAB  RANKBC  RANKABCITEM IF AB IN MMN THEN RANKA RANKB IF AND  ONLY IF THERE EXIST EM NONSINGULAR X IN MM AND Y IN  MN SUCH THAT B  XAYITEM LABELITRANKNOTE LABELPAGERANKPAGE IF A IN MMN HAS  RANKA  K THEN THERE IS A NONSINGULAR B IN MK AND X IN  MMK AND Y IN MKN SUCH THAT A  XBYITEM A MATRIX A IN MMNF OF RANK 1 CAN BE WRITTEN AS A  XBFT YBFFOR XBF IN FM AND YBF IN FNENDITEMIZESUBSECTIONNUMERIC RANKEVEN THOUGH THE RANK OF A MATRIX IS WELL DEFINED MATHEMATICALLY DUETO ROUNDOFF NUMERICAL DIFFICULTIES MAY ARISE WHEN ACTUALLY TRYING TOCOMPUTE THE RANK OF A MATRIX WITH REALVALUED ELEMENTSBEGINEXAMPLE  THE MATRIX A  BEGINBMATRIX 2  4  1  2  EPSILON ENDBMATRIXIS FULL RANK FOR ANY EPSILON NEQ 0  HOWEVER IT IS EM CLOSEUSING SOME MATRIX NORM TO A MATRIX THAT IS RANK DEFICIENTENDEXAMPLETHERE ARE A VARIETY OF WAYS OF NUMERICALLY COMPUTING THE RANK OF AMATRIX INCLUDING THE QR DECOMPOSITION WITH COLUMN PIVOTING SEE EGCITEGVL  HOWEVER ONE OF THE BEST WAYS IS TO USE THE SVD WHICHCAN PROVIDE INFORMATION NOT ONLY ON WHAT THE RANK IS NUMERICALLYBUT ALSO WHETHER THE MATRIX IS CLOSE TO ANOTHER MATRIX THAT IS RANKDEFICIENTBEGINEXERCISES  ITEM PROVE THEOREM REFTHMCANCELLEFT CAMPBELL AND MEYER P 3  ITEM SHOW THAT IF A2  A THEN RANKATRACEA  FROM CAMPBELL AND MEYER P 2ITEM LET A BE MATSIZEMN AND B BE MATSIZEMP AND LET V  RANGEAQQUAD TEXTAND W  RANGEBLET D  A BSHOW THATBEGINENUMERATEITEM DIMENSIONVW  RANKDITEM DIMENSIONVCAP W  RANKNULLSPACED  NULLITYDITEM HENCE ADDING THESE TWO RESULTS GET DIMENSIONVW  DIMENSIONVCAP W  DIMENSIONV  DIMENSIONWSTRANG P 200ENDENUMERATEITEM PROVE THAT THE SET OF FULL RANK MATRICES IS OPEN  GVL P 73ENDEXERCISES   LOCAL VARIABLES TEXMASTER BOOK ENDCHAPTEREIGENVALUES AND EIGENVECTORSLABELCHAPEIGENBEGINQUOTESOURCEMANFRED SCHROEDERNUMBER THEORY IN SCIENCE AND    COMMUNICATION  DOTS NEITHER HEISENBERG NOR BORN KNEW WHAT TO MAKE OF THE  APPEARANCE OF MATRICES IN THE CONTEXT OF THE ATOM  DAVID HILBERT  IS REPORTED TO HAVE TOLD THEM TO GO LOOK FOR A DIFFERENTIAL EQUATION  WITH THE SAME EIGENVALUES IF THAT WOULD MAKE THEM HAPPIER  THEY  DID NOT FOLLOW HILBERTS WELLMEANT ADVICE AND THEREBY MAY HAVE  MISSED DISCOVERING THE SCHRODINGER WAVE EQUATIONENDQUOTESOURCESECTIONEIGENVALUES AND LINEAR SYSTEMSTHE WORD EIGEN IS A GERMAN WORD THAT CAN BE TRANSLATED ASCHARACTERISTIC THE EIGENVALUES OF A LINEAR OPERATOR ARE THOSEVALUES WHICH CHARACTERIZE THE MODES OF THE OPERATOR  BEINGCHARACTERISTIC THE EIGENVALUES AND ASSOCIATED EIGENVECTORS OF ASYSTEM INDICATE SOMETHING THAT IS INTRINSIC AND INVARIANT IN THESYSTEMBEGINEXAMPLE  LABELEXMEIGEX1INDEXDIFFERENCE EQUATIONEIGENVECTORS ANDTO MOTIVATE THIS DESCRIPTION SOMEWHATCONSIDER THE FOLLOWING COUPLED DIFFERENCE EQUATIONSBEGINALIGN  Y1T1  Y1T 15Y2T   Y2T1  05Y1T  Y2TLABELEQEIGEX1ENDALIGNWHICH CAN BE WRITTEN IN MATRIX FORM AS YBFT1  BEGINBMATRIX1  15  05  1ENDBMATRIXYBFT  AYBFTWHERE YBFT  Y1TY2TT  IT IS DESIRED TO FIND A SOLUTIONTO THESE EQUATIONS  THE FORM OF THE EQUATIONS SUGGESTS THAT A GOODCANDIDATE SOLUTION IS Y1T  LAMBDAT X1QUAD Y2T  LAMBDAT X2 FOR SOME LAMBDA X1 AND X2 TO BE DETERMINED  SUBSTITUTION OFTHESE CANDIDATE SOLUTIONS INTO THE EQUATION GIVESBEGINALIGNLAMBDAT1X1  LAMBDAT X1  15LAMBDATX2 LAMBDAT1X2  05LAMBDAT X1  LAMBDATX2 ENDALIGNWHICH CAN BE WRITTEN MORE CONVENIENTLY AS ABEGINBMATRIXX1  X2 ENDBMATRIX LAMBDABEGINBMATRIXX1  X2 ENDBMATRIXORBEGINEQUATIONBOXEDAXBF  LAMBDA XBFLABELEQEIGEQENDEQUATIONENDEXAMPLEEQUATION REFEQEIGEQ IS THE EQUATION OF INTEREST IN EIGENVALUEPROBLEMS  THE DIFFERENCE EQUATION HAS BEEN REDUCED TO AN ALGEBRAICEQUATION WHERE WE WISH TO SOLVE FOR LAMBDA AND XBF  THE SCALARQUANTITY LAMBDA IS CALLED THE BF EIGENVALUE OF THE EQUATION ANDTHE VECTOR XBF IS CALLED THE BF EIGENVECTOR OF THE EQUATIONEQUATION REFEQEIGEQ MAY BE REGARDED AS AN OPERATOR EQUATIONTHE EIGENVECTORS OF A ARE THOSE VECTORS THAT ARE NOT CHANGED INDIRECTION BY THE OPERATION OF A THEY ARE SIMPLY SCALED BY THEAMOUNT LAMBDA  THIS IS ILLUSTRATED IN FIGURE REFFIGEIG1  AVECTOR IS AN EIGENVECTOR IF IT IS NOT MODIFIED IN DIRECTION ONLY INMAGNITUDE WHEN OPERATED ON BY A  THE VECTORS THUS FORM AN EM  INVARIANT OF THE OPERATOR A  THIS IS FULLY ANALOGOUS WITH THECONCEPT FROM THE THEORY OF LINEAR TIMEINVARIANT SYSTEMS EITHER INCONTINUOUS TIME OR DISCRETE TIME  THE STEADYSTATE OUTPUT OF AN LTISYSTEM TO A SINUSOIDAL INPUT IS A SINUSOIDAL SIGNAL AT THE EM SAME  FREQUENCY BUT WITH POSSIBLY DIFFERENT AMPLITUDE AND PHASE  THESYSTEM PRESERVES THE FREQUENCY OF THE SIGNAL ANALOGOUS TO PRESERVINGTHE DIRECTION OF A VECTOR WHILE MODIFYING ITS AMPLITUDE  SINUSOIDALSIGNALS ARE THEREFORE SOMETIMES REFERRED TO AS THE EM  EIGENFUNCTIONS OF AN LTI SYSTEM  IN THE STUDY OF LINEAR OPERATORSSEARCHING FOR THEIR EIGENFUNCTIONS IS AN IMPORTANT FIRST STEP TOUNDERSTANDING WHAT THE OPERATORS DOBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIREIGV1    CAPTIONTHE DIRECTION OF EIGENVECTORS IS NOT MODIFIED BY A    LABELFIGEIG1  ENDCENTERENDFIGUREBEGINDEFINITION  A EM NONZERO VECTOR XBF IS CALLED A BF RIGHT EIGENVECTOR FOR THE  EIGENVALUE LAMBDA IF AXBF  LAMBDA XBF AND A BF LEFT    EIGENVECTOR IF XBFH A  LAMBDA XBFH UNLESS OTHERWISE  STATED EIGENVECTOR MEANS RIGHT EIGENVECTORINDEXEIGENVALUEINDEXEIGENVECTORENDDEFINITIONEQUATION REFEQEIGEQ CAN BE WRITTEN IN THE FORMBEGINEQUATION  LABELEQEIGEQ1A  LAMBDA IXBF  0ENDEQUATIONONE SOLUTION OF REFEQEIGEQ1 IS THE SOLUTION XBF0  THIS ISKNOWN AS THE TRIVIAL SOLUTION AND IS NOT OF MUCH INTEREST  THE OTHERWAY THAT A SOLUTION MAY BE OBTAINED IS TO MAKE SURE THAT XBF IS INTHE NULLSPACE OF ALAMBDA I WHICH MEANS THAT WE MUST MAKE SURETHAT ALAMBDA I ACTUALLY HAS A NONTRIVIAL NULLSPACE  THEPARTICULAR VALUES OF LAMBDA THAT CAUSE ALAMBDA I TO HAVE ANONTRIVIAL NULLSPACE ARE THE EIGENVALUES OF A AND THE CORRESPONDINGVECTORS IN THE NULL SPACE ARE THE EIGENVECTORS  IN ORDER TO HAVE ANONTRIVIAL NULL SPACE THE MATRIX ALAMBDA I MUST BE SINGULAR  THEVALUES OF LAMBDA WHICH CAUSE ALAMBDA I TO BE SINGULAR AREPRECISELY THE EIGENVALUES OF A  AS DISCUSSED IN SECTIONREFSECTESTMATINV WE CAN DETERMINE IF A MATRIX IS SINGULAR BYEXAMINING ITS DETERMINANTBEGINDEFINITION LABELDEFCHARPOLY  THE POLYNOMIAL CHIALAMBDA  DETLAMBDA I I IS CALLED THE  BF CHARACTERISTIC POLYNOMIAL OF A  THE EQUATION DETLAMBDA  IA 0 IS CALLED THE CHARACTERISTIC EQUATION OF A  THE  EIGENVALUES OF A ARE THE ROOTS OF THE CHARACTERISTIC EQUATION  THE SET OF ROOTS OF THE CHARACTERISTIC POLYNOMIAL IS CALLED THE  BF SPECTRUM OF A AND IS DENOTED LAMBDAAINDEXSPECTRUM OF AN OPERATORINDEXCHARACTERISTIC POLYNOMIALINDEXCHARACTERISTIC EQUATIONENDDEFINITIONBEGINEXAMPLE LABELEXMEIGEX2  FOR THE MATRIX A OF EXAMPLE REFEXMEIGEX1 THE EIGENVALUES CAN  BE FOUND FROM DETA  LAMBDA I  DETBEGINBMATRIX1LAMBDA  15  05 1LAMBDAENDBMATRIX  1LAMBDA1LAMBDA  0515  0EXPANDING THE DETERMINANT WE OBTAIN LAMBDA2  25  0WHICH HAS ROOTS LAMBDA  05 OR LAMBDA  05ENDEXAMPLEIN THE STUDY OF LTI SYSTEMS THE CHARACTERISTIC POLYNOMIAL APPEARS INTHE DENOMINATORS OF TRANSFER FUNCTIONS  THE DYNAMICS OF THE SYSTEMARE THEREFORE GOVERNED BY THE ROOTS OF THE CHARACTERISTIC POLYNOMIAL THE EIGENVALUES  THIS IS ONE REASON WHY THE EIGENVALUES ARE OFINTEREST IN SIGNAL PROCESSINGBEGINEXAMPLE  THE LTI SYSTEM DESCRIBED BY THE DIFFERENCE EQUATIONBEGINALIGNXBFT1  AXBFT  BUBFT YBFT  CXBFTENDALIGNHAS THE ZTRANSFORM SEE SECTION REFSECLTI HZ  CZIA1B THE MATRIX INVERSE CAN BE WRITTEN AS HZ  CADJZIABFRAC1DETZIA THE NOTATION ADJZIA INDICATES THE EM ADJUGATE OF THE MATRIXINDEXADJUGATE ZIA NOT TO BE CONFUSED WITH THE ADJOINT  THEADJUGATE IS INTRODUCED IN SECTION REFSECDETINV  THE DENOMINATORIS THE CHARACTERISTIC EQUATION OF A AND THE POLES OF THE SYSTEM ARETHE EIGENVALUES OF THE MATRIX AENDEXAMPLEOFTEN THE EIGENVALUES ARE FOUND USING AN ITERATIVE NUMERICALPROCEDURE  ONCE THE EIGENVALUES ARE FOUND THE EIGENVECTORS AREDETERMINED BY FINDING VECTORS IN THE NULLSPACE OF ALAMBDA IBEGINEXAMPLE LABELEXMEIGEX3  FOR THE SYSTEM OF EXAMPLE REFEXMEIGEX1 WE HAVE FOUND THE  EIGENVALUES TO BE LAMBDA  PM 05  TO FIND THE EIGENVECTORS  SUBSTITUTE THE EIGENVALUES ONE AT A TIME INTO REFEQEIGEQ1 AND  FIND THE VECTORS IN THE NULLSPACE  WHEN LAMBDA05 WE GET BEGINBMATRIX15  15  05  05 ENDBMATRIXXBF1  0IT IS CLEAR THAT XBF1  11T WILL SATISFY THIS EQUATION ASWILL ANY MULTIPLE OF THIS  EM THE EIGENVECTORS ARE ONLY DETERMINED  UP TO A NONZERO SCALAR CONSTANT  THE EIGENVECTORS CAN BE SCALED TODIFFERENT MAGNITUDES  OFTEN IT IS CONVENIENT TO SCALE THE VECTORS SOTHEY HAVE UNIT NORM  THIS WOULD LEAD TO THE VECTOR XBF1 1SQRT21SQRT2TFOR THE OTHER EIGENVECTOR SUBSTITUTE LAMBDA  05 INTOREFEQEIGEQ1 BEGINBMATRIX05  15  05  15 ENDBMATRIXXBF2  0A SOLUTION IS XBF2  31T  SCALING TO HAVE UNIT NORM PROVIDESTHE SOLUTION XBF2  3SQRT101SQRT10T WE HAVE DETERMINED THE EIGENVALUES AND EIGENVECTORS OF THE SYSTEMDEFINED IN REFEQEIGEX1 AND HAVE ACTUALLY COME UP WITH TWOSOLUTIONS ONE FOR EACH EIGENVALUE  WHEN LAMBDA05 A SOLUTIONIS BEGINBMATRIXY1T  Y2T ENDBMATRIX 05T BEGINBMATRIX1  1 ENDBMATRIXFRAC1SQRT2 05T XBF1AND WHEN LAMBDA05 A SOLUTION IS BEGINBMATRIXY1T  Y2T ENDBMATRIX  05TBEGINBMATRIX 3  1 ENDBMATRIXFRAC1SQRT10  05T XBF2WHAT DO WE DO WITH THIS WEALTH OF SOLUTIONS  SINCE THE SYSTEM ISLINEAR THE RESPONSE DUE TO THE SUM OF SEVERAL INPUTS IS THE SUM OFTHE RESPONSES SO WE CAN TAKE LINEAR COMBINATIONS OF THESE SOLUTIONSFOR A TOTAL SOLUTION YBFT  C1 05T XBF1  C2 05T XBF2 THE CONSTANTS C1 AND C2 CAN BE FOUND TO MATCH AUXILIARYCONDITIONS ON THE SYSTEM SUCH AS INITIAL CONDITIONS  NOTE THAT INTHIS SOLUTION THE EM BEHAVIOR OF THE SYSTEM IS GOVERNED BY THE  EIGENVALUES THERE IS ONE MODE THAT GOES AS 05T ANDANOTHER MODE THAT GOES AS 05TENDEXAMPLESECTIONLINEAR DEPENDENCE OF EIGENVECTORSLABELSECEIG1THE EIGENVECTORS OF A MATRIX ARE OFTEN USED AS A SET OF BASIS VECTORSFOR SOME SPACE  IN ORDER TO ABLE TO SAY SOMETHING ABOUT THEDIMENSIONALITY OF THE SPACE SPANNED BY THE EIGENVECTORS IT ISIMPORTANT TO TELL WHEN THE EIGENVECTORS ARE LINEARLY INDEPENDENT  THEFIRST LEMMA PROVIDES PART OF THE STORYBEGINLEMMA  IF THE EIGENVALUES OF AN MATSIZEMM MATRIX A ARE ALL  DISTINCT THEN THE EIGENVECTORS OF A ARE LINEARLY INDEPENDENTENDLEMMABEGINPROOF START WITH M2 AND ASSUME TO THE CONTRARY THAT THE  EIGENVECTORS ARE LINEARLY DEPENDENT  THEN THERE EXIST CONSTANTS  C1 AND C2   SUCH THAT   BEGINEQUATION    LABELEQEVALP1   C1XBF1  C2 XBF2  0   ENDEQUATION  MULTIPLY BY A TO OBTAIN C1 AXBF1  C2 AXBF2  C1 LAMBDA1 XBF1  C2 LAMBDA2XBF2  0NOW TAKE LAMBDA2 TIMES EQUATION REFEQEVALP1 AND SUBTRACTIT  FROM THE LAST EQUATION TO OBTAIN C1LAMBDA1  LAMBDA2XBF1  0SINCE LAMBDA1 NEQ LAMBDA2 AND XBF1 NEQ 0 THIS MEANS THATC1  0  SIMILARLY IT CAN BE SHOWN THAT C2  0  THE TWO VECTORSMUST BE LINEARLY INDEPENDENTGENERALIZATION TO THE CASE FOR M2 PROCEEDS SIMILARLYENDPROOFIF THE EIGENVALUES ARE NOT DISTINCT THEN THE EIGENVECTORS MAY OR MAYNOT BE LINEARLY INDEPENDENT  THE MATRIX AI HAS M REPEATEDEIGENVALUES LAMBDA1 AND N LINEARLY INDEPENDENT EIGENVECTORS CANBE CHOSEN  ON THE OTHER HAND THE MATRIX A  BEGINBMATRIX4  2  0  4 ENDBMATRIXHAS REPEATED EIGENVALUES OF 4 4 AND BOTH EIGENVECTORS PROPORTIONAL TOXBF  10T THEY ARE LINEARLY DEPENDENTBEGINEXERCISESITEM EIGENFUNCTIONS AND EIGENVECTORS  BEGINENUMERATE  ITEM LET LC BE THE SECOND DERIVATIVE OPERATOR LC U     FRACD2DT2 U DEFINED FOR FUNCTIONS ON 01  SHOW THAT UNT  SINNPI TIS AN EIGENFUNCTION OF LC WITH EIGENVALUE LAMBDAN  NPI2ITEM IN MANY NUMERICAL PROBLEMS A DIFFERENTIATION OPERATOR IS  DISCRETIZED  SHOW THAT WE CAN APPROXIMATE THE SECOND DERIVATIVE  OPERATOR BY FRACD2DT2 APPROX FRACUTH  2UH  UTHH2WHERE H IS SOME SMALL NUMBER  ITEM DISCRETIZE THE INTERVAL 01 INTO 0T1T2LDOTSTN  WHERE TI  IN  LET UBF   UT0UT1LDOTSUTN1T AND SHOW THAT THE OPERATOR  LC U CAN BE APPROXIMATED BY THE OPERATOR FRAC1N2 LUBF WHERE  L  BEGINBMATRIX2  1 121 0121  DDOTS DDOTS  DDOTS 121 21 ENDBMATRIXITEM SHOW THAT THE EIGENVECTORS OF L ARE XBFN  BEGINBMATRIX SINNPIN SIN2NPIN  CDOTS   SINN1NPINENDBMATRIXQQUAD N12LDOTSNWHERE LAMBDAN  4 SIN2NPI2N  NOTE THAT XBFN IS SIMPLYA SAMPLED VERSION OF XNTENDENUMERATEITEM FIND THE EIGENVALUES OF THE  FOLLOWING MATRICES  BEGINENUMERATE  ITEM A DIAGONAL MATRIX A  BEGINBMATRIXA11   A22   DDOTS   ANNENDBMATRIXITEM A TRIANGULAR MATRIX EITHER UPPER OR LOWER UPPER IN THIS  EXERCISE A  BEGINBMATRIXA11  A12  A13  CDOTS  A1N  0  A22  A23  CDOTS A2N  VDOTS  DDOTS 0 0  0  CDOTS  ANN ENDBMATRIXITEM FROM THESE EXERCISES CONCLUDEBEGINFACTBOXTHE DIAGONAL ELEMENTS FORM THE EIGENVALUES OF A IF    A IS TRIANGULAR  ENDFACTBOXINDEXEIGENVALUESTRIANGULAR MATRIXINDEXTRIANGULAR MATRIXEIGENVALUESENDENUMERATEITEM FOR MATRIX T IN BLOCK TRIANGULAR FORM T  BEGINBMATRIXT11  T12  0  T22 ENDBMATRIXSHOW THAT LAMBDAT  LAMBDAT11 CUP LAMBDAT22ITEM SHOW THAT THE DETERMINANT OF A MATRIX IS THE PRODUCT OF THE    EIGENVALUESTHAT IS BOXED DETA  PRODI1N LAMBDAIINDEXDETERMINANTPRODUCT OF EIGENVALUESITEM SHOW THAT THE TRACE OF A MATRIX IS THE SUM OF THE EIGENVALUES BOXED TRACEA  SUMI1N LAMBDAIINDEXTRACESUM OF EIGENVALUESITEM SUPPOSE A IS A RANK1 MATRIX FORMED BY A  ABF BBFT  FIND THE EIGENVALUES OF A  ALSO SHOW THAT IF A IS RANK 1 THEN  DETIA  1TRACEAITEM LABELEXEIGSHIFTMAT SHOW THAT IF LAMBDA IS AN EIGENVALUE  OF A THEN LAMBDAR IS AN EIGENVALUE OF A RI AND THAT A  AND A  RI HAVE THE SAME EIGENVECTORSITEM SHOW THAT BEGINFACTBOXIF LAMBDA IS AN EIGENVALUE OF A THEN LAMBDAN  IS AN EIGENVALUE OF AN AND AN HAS THE SAME EIGENVECTORS AS  AENDFACTBOXITEM SHOW THAT LABELEXEIGINVBEGINSFACTBOXIF LAMBDA IS A NONZERO EIGENVALUE OF A THEN 1LAMBDA    IS AN EIGENVALUE OF A1  ENDSFACTBOXBEGINFACTBOXTHE EIGENVECTORS OF A CORRESPONDING TO NONZERO    EIGENVALUES ARE EIGENVECTORS OF A1  ENDFACTBOXITEM LABELEXPOLYEIG GENERALIZING THE PREVIOUS PROBLEMS SHOW THAT  IF LAMBDA1 LAMBDA2 LDOTS LAMBDAM ARE THE EIGENVALUES OF  A AND IF GX IS A SCALAR POLYNOMIAL THEN THE EIGENVALUES OF  GA ARE  GLAMBDA1 GLAMBDA2 LDOTS GLAMBDAM  GANTMACHER V1P84ITEM SHOW THAT THE EIGENVALUES OF A PROJECTION MATRIX P  ARE EITHER 1 OR 0ITEM IN THIS PROBLEM YOU WILL ESTABLISH SOME RESULTS ON EIGENVALUES  OF PRODUCTS OF MATRICES  BEGINENUMERATE  ITEM IF A AND B ARE BOTH SQUARE SHOW THAT THE EIGENVALUES OF    AB ARE THE SAME AS THE EIGENVALUES OF BA  ITEM SHOW THAT IF THE MATSIZENN MATRICES HAVE A COMMON SET    OF N LINEARLY INDEPENDENT EIGENVECTORS THEN ABBA  ITEM SHOW BY COUNTEREXAMPLE THAT THE CONVERSE TO THIS PROPERTY DOES    NOT APPLY  ENDENUMERATE ORGEGA P 249ITEM SHOW THAT A STOCHASTIC MATRIX HAS LAMBDA1 AS ITS EIGENVALUE  WITH LARGEST ABSOLUTE VALUE AND THAT XBF  11LDOTS1T IS  THE CORRESPONDING EIGENVECTORITEM LINEAR FIXEDPOINT PROBLEMS  SOME PROBLEMS ARE OF THE FORM A XBF  XBFIF A HAS AN EIGENVALUE EQUAL TO 1 THEN THIS PROBLEM HAS A SOLUTIONCONDITIONS GUARANTEEING THAT A HAS AN EIGENVALUE OF 1 ARE DESCRIBEDIN CITEMARCUSMINK  EXAMPLE PROBLEMS OF THIS SORT ARE THESTEADYSTATE PROBABILITIES FOR A MARKOV CHAIN AND DETERMININGVALUES FOR A COMPACTLYSUPPORTED WAVELET AT INTEGER VALUES OF THE ARGUMENTBEGINENUMERATEITEM LET  A BEGINBMATRIX 5  3  2  2  0  7  3  7  1ENDBMATRIXBE THE STATETRANSITION PROBABILITY MATRIX FOR A MARKOV MODELDETERMINE THE STEADYSTATE PROBABILITY PBF SUCH THAT A PBF PBFITEM THE TWOSCALE EQUATION FOR A SCALING FUNCTION INDEXSCALING    FUNCTION REFEQTWOSCALE3 IS PHIT  SUMK0N1 CK  PHI2TK  GIVEN THAT WE KNOW THAT THE PHIT IS ZERO FOR T  LEQ 0 AND FOR T GEQ N1 WRITE AN EQUATION OF THE FORM BEGINBMATRIX PHI1  PHI2  VDOTS  PHIN2  ENDBMATRIX A BEGINBMATRIX PHI1  PHI2  VDOTS  PHIN2  ENDBMATRIXWHERE A IS A MATRIX OF WAVELET COEFFICIENTS CK  GIVEN THECOEFFICIENTS DESCRIBE HOW TO SOLVE THIS EQUATION THEN HOW TO FINDPHIT AT ALL DYADIC RATIONAL NUMBERS NUMBERS OF THE FORM K2IFOR INTEGERS K AND IENDENUMERATEITEM CITEKAILATH80 LET ABBFCBFD REPRESEN A SYSTEM IN  STATESPACE FORM HAVING TRANSFER FUNCTION HS  CBFTSIA1  BBF  D  SHOW THAT THE ZEROS OF HS CAN BE COMPUTED AS THE  EIGENVALUES OF THE MATRIX A  BBF D1 CBFTENDEXERCISESSECTIONDIAGONALIZATION OF A MATRIXLABELSECDIAGONALINDEXDIAGONALIZATION OF A MATRIXIN THIS SECTION WE INTRODUCE A FACTORIZATION OF A MATRIX A AS A  SLAMBDA S1WHERE S IS DIAGONAL OR MOSTLY DIAGONAL MATRIX  WE WILL BEGIN BYASSUMING THAT THE MATSIZEMM MATRIX A HAS M LINEARLYINDEPENDENT EIGENVECTORS  LET THE EIGENVECTORS OF ABE XBF1 XBF2LDOTSXBFM SO THAT AXBFI  LAMBDAI XBFIQQUAD I12LDOTSM THESE EQUATIONS CAN BE STACKED SIDEBYSIDE TO OBTAIN AXBF1 AXBF2 CDOTS AXBFM  LAMBDA1 XBF1LAMBDA2XBF2 CDOTS  LAMBDAM XBFM  THE STACKED MATRIX ONTHE LEFT CAN BE WRITTEN ASBEGINEQUATIONAXBF1 AXBF2 CDOTS AXBFM  AXBF1 XBF2 CDOTSXBFMLABELEQEIGSTACK1ENDEQUATIONAND THE STACKED MATRIX ON THE RIGHT CAN BE WRITTEN AS LAMBDA1 XBF1 LAMBDA2XBF2 CDOTS  LAMBDAM XBFM XBF1 XBF2 CDOTS XBFMBEGINBMATRIXLAMBDA1  LAMBDA2   DDOTS   LAMBDAM ENDBMATRIXLET S BE THE SIDEBYSIDE STACKED MATRIX OF EIGENVECTORS AND LETLAMBDA BE THE DIAGONAL MATRIX FORMED FROM THE EIGENVALUES S  XBF1 XBF2 CDOTS XBFNQQUAD LAMBDA DIAGLAMBDA1 LAMBDA2 LDOTS LAMBDAN THEN REFEQEIGSTACK1 CAN BE WRITTEN AS BOXEDAS  SLAMBDA THIS EQUATION IS TRUE WHETHER OR NOT THE EIGENVECTORS ARE LINEARLYINDEPENDENT OR NOT  HOWEVER IF THE EIGENVECTORS EM ARE LINEARLYINDEPENDENT THEN S IS FULL RANK AND INVERTIBLE AND WE CAN WRITEBEGINEQUATIONBOXEDA  SLAMBDA S1LABELEQDIAG1ENDEQUATIONORBEGINEQUATIONBOXEDLAMBDA  S1ASLABELEQDIAG2ENDEQUATIONTHIS IS SAID TO BE A BF DIAGONALIZATION OF A AND A MATRIX WHICHHAS A DIAGONALIZATION IS SAID TO BE BF DIAGONALIZABLE  THE PARTICULAR FORM OF THE TRANSFORMATION FROM A TO LAMBDA ARISESIN A VARIETY OF CONTEXTS  MORE GENERALLY IF THERE ARE MATRICES AAND B WITH AN INVERTIBLE  MATRIX T SUCH THATBEGINEQUATIONBOXEDA  TBT1 LABELEQSIMILAR1ENDEQUATIONTHEN A AND B ARE SAID TO BE BF SIMILAR  INDEXSIMILAR MATRIXIT CAN BE SHOWN THATBEGINSFACTBOX IF A AND B ARE SIMILAR THEN  THEY HAVE THE SAME EIGENVALUESENDSFACTBOXTHE DIAGONALIZATION REFEQDIAG1 SHOW THAT A AND LAMBDA ARESIMILAR AND HENCE HAVE THE SAME EIGENVALUES  THIS IS CLEAR IN THISCASE SINCE THE EIGENVALUES OF A APPEAR ON THE DIAGONAL OFLAMBDA  OTHER TRANSFORMATIONS CAN BE USED TO FIND MATRICESSIMILAR TO A BUT THE SIMILAR MATRICES WILL NOT BE DIAGONAL UNLESSTHEY ARE FORMED FROM THE EIGENVECTORS OF ATHERE ARE A VARIETY OF USES FOR THE FACTORIZATION A  SLAMBDAS1  ONE SIMPLE ONE IS THAT POWERS OF A ARE EASY TO COMPUTEFOR EXAMPLE A2  SLAMBDA S1SLAMBDA S1  S LAMBDA2 S1 AND MORE GENERALLYBEGINEQUATION BOXEDAN  SLAMBDAN S1LABELEQAPOWERENDEQUATIONTHIS ALLOWS A MEANS FOR DEFINING FUNCTIONS OPERATING ON MATRICES  FOR A FUNCTION FX WITH THE POWER SERIES REPRESENTATION FT  SUMI FI TI THE FUNCTION OPERATING ON A DIAGONALIZABLE MATRIX CAN BE DEFINED AS FA  SUMI FI AI  SBIGLSUMI FI LAMBDAIBIGRS1SINCE LAMBDA IS DIAGONAL LAMBDAI IS COMPUTED SIMPLY BYCOMPUTING THE ELEMENTS ON THE DIAGONAL  AN IMPORTANT EXAMPLE OF THISIS INDEXMATRIX EXPONENTIAL EA  SUMI0INFTY FRACAII  SLEFTSUMI0INFTYFRACLAMBDAIIRIGHTS1  S ELAMBDA S1 WHERE  ELAMBDA  BEGINBMATRIX ELAMBDA1   ELAMBDA2    DDOTS  ELAMBDAM ENDBMATRIXBEGINEXAMPLE  LET  A  BEGINBMATRIX23  6 18  26 ENDBMATRIXTHEN A HAS THE EIGENDECOMPOSITION BEGINALIGNEDLAMBDA1  14  QQUAD XBF1  BEGINBMATRIX5547  8321ENDBMATRIX LAMBDA2  35  QQUAD XBF2  BEGINBMATRIX 4472  8944ENDBMATRIXENDALIGNEDTHEN  EA  BEGINBMATRIX05547  0447214 083205  0894427 ENDBMATRIXBEGINBMATRIXE14  E35 ENDBMATRIXBEGINBMATRIX05547  0447214 083205  0894427 ENDBMATRIX EXPEXMTHE SC MATLAB FUNCTION TT EXPM COMPUTES THE MATRIX EXPONENTIALTHIS COMPUTATION ARISES FREQUENTLY ENOUGH IN PRACTICE THATCONSIDERABLE EFFORT HAS BEEN DEDICATED TO EFFECTIVE NUMERICALSOLUTIONS  A TREATMENT OF THIS IS IN CITEMOLER19WAYS OF WHICH THEMETHOD PRESENTED HERE IS BUT ONE METHODENDEXAMPLETHE HOMOGENEOUS VECTOR DIFFERENTIAL EQUATION XBFDOTT  A XBFTHAS THE SOLUTION XBFT  EATXBF0 AND THE HOMOGENEOUS VECTORDIFFERENCE EQUATION XBFT1  A XBFT HAS THE SOLUTION XBFT ATXBF0  IN LIGHT OF THE DIAGONALIZATION DISCUSSED THEDIFFERENTIAL EQUATION IS STABLE IF THE EIGENVALUES OF A ARE IN THELEFTHALF PLANE AND THE DIFFERENCE EQUATION IS STABLE IF THEEIGENVALUES OF A ARE INSIDE THE UNIT CIRCLESUBSECTIONTHE JORDAN FORMLABELSECJORDANINDEXJORDAN FORM IF A HAS REPEATED EIGENVALUES THEN IT IS NOTALWAYS POSSIBLE TO DIAGONALIZE A  IF THE EIGENVECTORS ARE LINEARLYINDEPENDENT THEN EVEN WITH REPEATED EIGENVALUES A CAN BEDIAGONALIZED  IF SOME OF THE EIGENVECTORS ARE LINEARLY DEPENDENTTHEN A CANNOT BE EXACTLY DIAGONALIZED  INSTEAD A MATRIX WHICH ISNEARLY DIAGONAL IS FOUND TO WHICH A IS SIMILAR  THIS MATRIX ISKNOWN AS THE EM JORDAN FORM OF ABEGINTHEOREM JORDAN FORM  A MATSIZEMM MATRIX A WITH KLEQ M LINEARLY INDEPENDENT  EIGENVECTORS CAN BE WRITTEN AS A  TJT1WHERE J IS A BLOCKDIAGONAL MATRIX J  BEGINBMATRIXJ1   J2   DDOTS  JK ENDBMATRIXTHE BLOCKS JI ARE KNOWN AS EM JORDAN BLOCKS  EACH JORDAN BLOCKIS OF THE FORM JI  BEGINBMATRIX LAMBDAI  1   LAMBDAI  1   DDOTS  DDOTS   LAMBDAI  1  LAMBDAI ENDBMATRIXIF JI IS MATSIZELL THEN THE EIGENVALUE LAMBDAI IS REPEATEDL TIMES ALONG THE DIAGONAL AND 1 APPEARS L1 TIMES ABOVE THEDIAGONAL  TWO MATRICES ARE SIMILAR IF THEY HAVE THE SAME JORDAN FORMENDTHEOREMAN INDUCTIVE PROOF OF THIS THEOREM APPEARS IN CITEAPPENDIXBSTRANG1988  RATHER THAN REPRODUCE THE PROOF HERE WE CONSIDERSOME EXAMPLES AND APPLICATIONSBEGINEXAMPLE BEGINENUMERATEITEM  THE MATRIX A  BEGINBMATRIX4  1  3 0  4  1 0  0  4 ENDBMATRIXHAS A SINGLE EIGENVALUE LAMBDA4 AND ALL THREE EIGENVECTORS ARETHE SAME XBF1XBF2XBF3  100T  THERE IS THUS A SINGLEJORDAN BLOCK AND A IS SIMILAR TO J  BEGINBMATRIX4  1  0  0  41  0  04 ENDBMATRIXITEM THE MATRIX B  BEGINBMATRIX3  0  1 0  3  0 0  0  3 ENDBMATRIXHAS A SINGLE EIGENVALUE LAMBDA3 AND TWO EIGENVECTORS XBF1  100TQQUADTEXTANDQQUAD XBF2  010TTHE JORDAN FORM HAS TWO JORDAN BLOCKS  J1  BEGINBMATRIX 3  1  0  3 ENDBMATRIXQQUADTEXTANDQQUADJ2  3SO  J  BEGINBMATRIX 310  03  0  0 0  3 ENDBMATRIXENDENUMERATEENDEXAMPLEIF A HAS THE JORDAN FORM REPRESENTATION A  SJS1THEN AN  S JN S1AND EAT  S EJT S1BUT COMPUTING JN IS SOMEWHAT MORE COMPLICATED IF J IS NOT STRICTLY DIAGONAL  AS AN EXAMPLE FOR A MATSIZE33 JORDAN BLOCKBEGINEQUATION BEGINBMATRIX LAMBDA10  0LAMBDA1  00LAMBDAENDBMATRIXN   BEGINBMATRIX LAMBDAN  NLAMBDAN1  FRAC12NN1LAMBDAN2  0LAMBDAN  N LAMBDAN1 00LAMBDAN ENDBMATRIXLABELEQJORDANPOWENDEQUATIONTHE PRESENCE OF TERMS WHICH GROW AS A POLYNOMIAL FUNCTION OF N CANBE UNDERSTOOD BY COMPARISON WITH REPEATED ROOTS IN A DIFFERENTIALOR DIFFERENCE EQUATION THE REPEATED ROOTS GIVE RISE TO SOLUTIONS OFTHE FORM T ELAMBDA T FOR THE DIFFERENTIAL EQUATION AND TLAMBDAT FOR THE DIFFERENCE EQUATIONBEGINEXAMPLEA SIGNAL HAS TRANSFER FUNCTION YZ  FRAC3Z2  3 ZZ32WITH TIME FUNCTION YT  33TUT  4T3T UTPLACING THE SYSTEM INTO STATEVARIABLE FORM AS INREFEQASTATEMAT WE FIND A  BEGINBMATRIX01  009  6 ENDBMATRIXWHICH HAS REPEATED EIGENVALUES LAMBDA3 AND ONLY A SINGLEEIGENVECTOR  THE PRESENCE OF THE LINEARLY GROWING TERM 4T3T ISEQUIVALENT TO THE FACT THAT THE JORDAN FORM FOR A  IS NOT STRICTLYDIAGONALENDEXAMPLESUBSECTIONDIAGONALIZATION OF SELFADJOINT MATRICESLABELSECSYMMETRICHERMITIAN SYMMETRIC MATRICES ARISE IN A VARIETY OF CONTEXTS AS AMATHEMATICAL REPRESENTATION OF SYMMETRIC INTERACTIONS IF A AFFECTSB AS MUCH AS B AFFECTS A THEN A MATRIX DESCRIBING THEIRINTERACTIONS WILL BE SYMMETRIC  THROUGHOUT THIS SECTION WE EMPLOYINNER PRODUCT NOTATION INTERSPERSED WITH MORE TRADITIONAL MATRIXNOTATION TO REINFORCE THE ITS USE AND TO AVOID AS MUCH AS POSSIBLEHAVING TO SAY SYMMETRIC OR HERMITIAN  AS DISCUSSED IN SECTIONREFSECADJOINT SELFADJOINT MATRICES ARE MATRICES FOR WHICH INNERPAXBFXBF  INNERPXBFAXBFSELFADJOINT MATRICES ARE SYMMETRIC IF THE ELEMENTS ARE REAL ATAAND ARE HERMITIAN IF THE ELEMENTS ARE COMPLEX AH  A  THE FIRSTUSEFUL RESULT ABOUT SELFADJOINT MATRICES IS THEIR EIGENVALUES AREREALBEGINLEMMA LABELLEMREALEIG BEGINSFACTBOXTHE EIGENVALUES OF A SELFADJOINT MATRIX ARE REALENDSFACTBOXENDLEMMABEGINPROOF  LET LAMBDA AND XBF BE AN EIGENVALUE AND EIGENVECTOR OF A  SELFADJOINT MATRIX A  THENBEGINEQUATION LA AXBF XBF RA  LAMBDA LA XBFXBF RALABELEQREALEIG1ENDEQUATIONANDBEGINEQUATION  LABELEQREALEIG2  LA XBFA XBF RA  LAMBDABAR LA XBFXBF RAENDEQUATIONSINCE LA AXBFXBF RA  LA XBF A XBFRA WE MUST HAVELAMBDA  LAMBDABAR SO LAMBDA IS REALENDPROOFBEGINLEMMA LABELLEMORTHOGEIG BEGINFACTBOX FOR A SELFADJOINT MATRIX THE EIGENVECTORS CORRESPONDING TO  DISTINCT EIGENVALUES ARE ORTHOGONALENDFACTBOXENDLEMMABEGINPROOF  LET LAMBDA1 AND LAMBDA2 BE DISTINCT EIGENVALUES OF A  SELFADJOINT MATRIX A WITH CORRESPONDING EIGENVECTORS XBF1 AND  XBF2  THEN LA A XBF1 XBF2 RA  LA XBF1A XBF2RA  LA XBF1  LAMBDA2 XBF2RA  LAMBDA2 LAXBF1 XBF2RAWE ALSO HAVE LA A XBF1 XBF2 RA  LAMBDA1 LA XBF1XBF2RASO THAT LAMBDA1LAMBDA2LA XBF1XBF2 RA 0SINCE LAMBDA1 NEQ LAMBDA2 WE MUST HAVE XBF1 PERP XBF2ENDPROOFWE HAVE ALREADY OBSERVED THAT FOR HERMITIAN MATRICES WITH DISTINCTEIGENVALUES THAT DIAGONALIZATION IS POSSIBLE AND THE UNITARYDIAGONALIZING MATRIX U IS SIMPLY FORMED FROM THE EIGENVECTORS OFA  HOWEVER THIS THEOREM IS TRUE EM EVEN FOR MATRICES WITH  REPEATED EIGENVALUES  THIS THEOREM IS KNOWN AS THE EM SPECTRAL  THEOREM INDEXSPECTRAL THEOREM AND THE SET OF EIGENVALUES OF AHERMITIAN MATRIX IS KNOWN AS ITS EM SPECTRUMBEGINTHEOREM LABELTHMDIAGSYM  EVERY HERMITIAN MATSIZEMM MATRIX A CAN BE DIAGONALIZED BY A  UNITARY MATRIXBEGINEQUATION UHAU  LAMBDALABELEQDIAGSYMENDEQUATIONWHERE U IS UNITARY AND LAMBDA IS DIAGONALENDTHEOREMIT FOLLOWS THAT EVERY REALSYMMETRIC MATRIX A CAN BE DIAGONALIZED BY AN ORTHOGONAL MATRIXBEGINEQUATION QTAQ  LAMBDALABELEQDIAGSYM2ENDEQUATIONWHEN A HAS DISTINCT EIGENVALUES THEOREM REFEQDIAGSYM IS IMMEDIATE IN LIGHT OF THE DISCUSSION IN THE LAST SECTION  HOWEVER THE RESULT IS TRUE EVEN WHEN A HAS REPEATED EIGENVALUESWE CAN WRITE REFEQDIAGSYM ASBEGINEQUATIONA  U LAMBDA UH  SUMI1M LAMBDAI UBFI UBFIHLABELEQDIAGSYM3ENDEQUATIONTHE PROOF OF THEOREM REFTHMDIAGSYM IS AN EXERCISE SEE EXERCISEREFEXSPECTRH  THE PROOF FOLLOWS IN TWO SIMPLE STEPS FROM THEFOLLOWING KEY LEMMA WHICH IS INTERESTING IN ITS OWN RIGHT  IT SHOULDBE OBSERVED THAT THIS LEMMA APPLIES NOT ONLY TO HERMITIAN MATRICESBUT TO EM ANY SQUARE MATRIXBEGINLEMMA LABELLEMSCHUR SCHURS LEMMA  FOR ANY SQUARE MATRIX A THERE IS A UNITARY MATRIX U SUCH THAT UHA U  T AND T IS UPPER TRIANGULAR  EVERY MATRIX IS SIMILAR TO ANUPPER TRIANGULAR MATRIXENDLEMMAOBSERVE THAT SINCE THE EIGENVALUES OF A DIAGONAL MATRIX APPEAR ON THEDIAGONAL THIS LEMMA PROVIDES ONE METHOD OF COMPUTING THE EIGENVALUESOF ANY MATRIXBEGINPROOF  THE PROOF IS CONSTRUCTIVE    FOR TYPOGRAPHICAL CONVENIENCE THE LEMMA WILL BE DEMONSTRATED USING A  MATSIZE33 MATRIX EXTENSION TO AN ARBITRARY SQUARE MATRIX IS  STRAIGHTFORWARD  LET A BE A MATSIZE33 MATRIX  IT MUST  HAVE AT LEAST ONE EIGENVALUE LAMBDA1 WHICH MAY BE REPEATED BUT  THIS DOES NOT MATTER AND A CORRESPONDING EIGENVECTOR UBF1  WHICH WE ASSUME TO BE NORMALIZED TO A UNIT VECTOR  BY THE  GRAMSCHMIDT PROCESS IT IS POSSIBLE TO FIND TWO UNIT VECTORS  XBF12 XBF13 WHICH ARE ORTHOGONAL TO UBF1 AND FORM A  UNITARY MATRIX U1 WITH UBF1 IN THE FIRST COLUMN  THEN AU1  A BEGINBMATRIX UBF1  XBF12  XBF13 ENDBMATRIX   U1BEGINBMATRIX LAMBDA1  TIMES  TIMES   0  TIMES TIMES  0  TIMES  TIMES ENDBMATRIX  U1 BEGINBMATRIX LAMBDA1  TIMES  TIMES 0  0  MULTICOLUMN2CRAISEBOX15EX0CM0CMA2 ENDBMATRIXWHERE TIMES DENOTES AN ELEMENT WHICH TAKES ON AN ARBITRARY VALUENOW CONSIDER THE MATSIZE22 MATRIX A2 IN THE LOWER RIGHT OFTHE MATRIX ON THE RIGHT  IT ALSO HAS AT LEAST ONE EIGENVALUELAMBDA2 AND A CORRESPONDING EIGENVECTOR UBF22  AGAIN USINGGRAMSCHMIDT A MATSIZE22 UNITARY MATRIX M2 CAN BECONSTRUCTED M2  UBF22XBF23SO THAT A2 M2  BEGINBMATRIX LAMBDA2  TIMES  0  TIMESENDBMATRIXTHEN A MATSIZE33 UNITARY MATRIX CAN BECONSTRUCTED BY U2  BEGINBMATRIX 1  0  0 0  0  MULTICOLUMN2CRAISEBOX15EX0CM0CMM2 ENDBMATRIXTHEN AU1 U2  U2 U1 BEGINBMATRIX LAMBDA1  TIMES  TIMES 0  LAMBDA2  TIMES 0  0  TIMES ENDBMATRIXWHICH IS UPPER TRIANGULAR  THE MATRIX U  U1 U2 IS UNITARY SOTHE THEOREM IS PROVED FOR THE MATSIZE33 CASEENDPROOFBEGINLEMMA LABELLEMZEROEIG  LET A BE A MATSIZEMM MATRIX OF RANK RM  THEN AT LEAST  MR OF THE EIGENVALUES OF A ARE EQUAL TO ZEROENDLEMMATHE PROOF IS REQUIRED IN EXERCISE REFEXPROVEZEROEIGBEGINEXAMPLELET  A  BEGINBMATRIX 1  0  0  0  0  1  0  1  0ENDBMATRIXWHICH HAS EIGENVALUES LAMBDA1  LAMBDA2  1 AND LAMBDA3 1  FOLLOWING THE STEPS IN THE PROOF OF LEMMA REFLEMSCHUR WEFIRST FIND AN EIGENVECTOR OF A CORRESPONDING TO LAMBDA1  1  UBF1  BEGINBMATRIX 1  0  0 ENDBMATRIXTHEN TWO VECTORS WHICH ARE ORTHOGONAL TO THIS ARE XBF12 EBF2 AND XBF13  EBF3 GIVING U1  I THEN THEMATSIZE22 MATRIX A2 IN THE LOWERRIGHT CORNER OF AU1 IS A2  BEGINBMATRIX01  10 ENDBMATRIXWHICH HAS AN EIGENVALUE OF LAMBDA2  1 WITH A CORRESPONDINGEIGENVECTOR UBF22  FRAC1SQRT211T  THEN M2  FRAC1SQRT2BEGINBMATRIX11  11 ENDBMATRIXAND  U  U1U2  BEGINBMATRIX1 00 0FRAC1SQRT2   FRAC1SQRT2   0 FRAC1SQRT2  FRAC1SQRT2 ENDBMATRIX  UBF1UBF2UBF3THEN A HAS THE REPRESENTATIONBEGINALIGNA  ULAMBDA UT  SUMI13 LAMBDAI UBFI UBFIT  NONUMBER  LAMBDA1BEGINBMATRIX 1  0  0 0  0  0  0  0  0 ENDBMATRIX  LAMBDA2BEGINBMATRIX 0  0 0   0  FRAC12  FRAC12  0  FRAC12  FRAC12ENDBMATRIX  LAMBDA3BEGINBMATRIX 0  0  0  0  FRAC12 FRAC12  0  FRAC12  FRAC12 ENDBMATRIXNONUMBER  LAMBDA1 BEGINBMATRIX 1  0  0  0  FRAC12  FRAC12 0  FRAC12  FRAC12 ENDBMATRIX  LAMBDA2 BEGINBMATRIX0 0  0  0  FRAC12  FRAC12  0   FRAC12  FRAC12 ENDBMATRIX NONUMBER   LAMBDA1 P1  LAMBDA3 P2 LABELEQASPECTENDALIGNSINCE LAMBDA1  LAMBDA2  WE WILL SEE BELOW THAT THE MATRICES P1 AND P2 THAT APPEAR INREFEQASPECT ARE IN FACT PROJECTION MATRICES AND THAT P1 ANDP2 ARE ORTHOGONAL P1T P2  0  P1 PROJECTS ONTO THE SPACESPANNED BY THE VECTORS 100T 011T THE EIGENVECTORSCORRESPONDING TO THE EIGENVALUE LAMBDA1 AND P2 PROJECTS ONTOTHE SPACE SPANNED BY 011 THE EIGENVECTOR CORRESPONDING TO THEEIGENVALUE LAMBDA1ENDEXAMPLETHE DIAGONALIZATION A  ULAMBDA UH ILLUSTRATES AN IMPORTANTPRINCIPLE THAT OF FINDING AN APPROPRIATE COORDINATE SYSTEM IN WHICHTO SOLVE A PROBLEM  MANY PROBLEMS IN MATHEMATICS CAN BE SIMPLIFIED BYEXPRESSING THEM IN AN APPROPRIATE ORTHOGONAL COORDINATE SYSTEM WHERETHE GLOBAL PROBLEM CAN BE ADDRESSED AS A SERIES OF SCALAR PROBLEMSTHIS IS ONE REASON WHY EFFORTS ARE MADE TO FIND SETS ORTHOGONAL BASISFUNCTIONS AS DESCRIBED IN CHAPTER REFCHAPVECTAP   THECONVOLUTION THEOREM WHICH STATES THAT THE TRANSFORM OF A CONVOLUTIONIS THE PRODUCT OF THE TRANSFORMS IS ANOTHER EXAMPLE OF THEAPPLICATION OF THIS CONCEPT  RATHER THAN CONVOLVING TWO SIGNALSWHICH INVOLVES MOREORLESS GLOBAL INTERACTION OF THE SIGNALS THESIGNALS ARE REPRESENTED IN A TRANSFORM DOMAIN A NEW COORDINATESYSTEM WHERE THE CONVOLUTION CAN BE REPRESENTED AS MULTIPLICATIONTHE IMPORTANCE OF THIS IN REAL SIGNAL PROCESSING IS PROFOUND AS THISLEADS TO FAST CONVOLUTION USING THE FFT  EXERCISE REFEXCYCLICMATEXAMINES THIS TOPIC IN MORE DETAILSUBSUBSECTIONSYLVESTERS LAW OF INERTIAINDEXSYLVESTERS LAW OF INERTIAINDEXINERTIA OF A MATRIXBEGINDEFINITION  LET A BE A HERMITIAN MATRIX WITH LAMBDAA POSITIVE EIGENVALUES  LAMBDAA NEGATIVE EIGENVALUES AND LAMBDA0A ZERO EIGENVALUES  THE BF INERTIA OF A A IS THE TRIPLE LAMBDAALAMBDAALAMBDA0ATHE NUMBER OF POSITIVE NEGATIVE AND ZERO EIGENVALUES    THE BF SIGNATURE OF A IS LAMBDAA  LAMBDAA INDEXSIGNATURE OF A MATRIXENDDEFINITIONBEGINTHEOREM SYLVESTERS LAW OF INERTIA  LET A AND B BE MATSIZEMM HERMITIAN MATRICES  THEN THERE  IS A NONSINGULAR MATRIX S SUCH THAT A  SBSH IF AND ONLY IF A  AND B HAVE THE SAME INERTIAENDTHEOREMBEGINPROOF CITEHORNJOHNSON  THE CONVERSE IS PRESENTED AS AN EXERCISE  SUPPOSE THAT A  SBSH FOR SOME NONSINGULAR MATRIX S  THEN RANKA  RANKSBSH  RANKBSO LAMBDA0A  LAMBDA0B  IT REMAINS TO SHOW THATLAMBDAA  LAMBDAB  LET UBF1UBF2LDOTSUBFLAMBDAA BE THE ORTHONORMAL EIGENVECTORS OF ACORRESPONDING TO THE POSITIVE EIGENVALUES OF A WHICH WE DENOTE ASLAMBDA1A LAMBDA2ALDOTS LAMBDALAMBDAAA  LET SA  LSPANUBF1UBF2LDOTSUBFLAMBDAATHEN DIMENSIONSA  LAMBDAALET XBF  ALPHA1 UBF1  CDOTS  ALPHALAMBDAAUBFLAMBDAA NEQ 0  THEN XBFH A XBF  LAMBDA1AALPHA12  CDOTS LAMBDALAMBDAAALPHALAMBDAA20WE ALSO HAVE XBFH SBSH XBF  SH XBFH B SH XBF  0SO YBFH B YBF0 FOR ALL NONZERO VECTORS YBF IN LSPANSHVBF1 LDOTS SH VBFLAMBDAA WHICH HAS DIMENSIONLAMBDAA  THEN SEE EXERCISE REFEXSYLV2 B MUST HAVE ATLEAST LAMBDAA EIGENVALUES LAMBDAB GEQ LAMBDAAREVERSING THE ROLES OF A AND B IN THIS ARGUMENT WE SEE THATLAMBDAA  LAMBDABENDPROOFBEGINEXERCISESITEM PROVE IF A AND B ARE DIAGONALIZABLE THEY SHARE THE SAME  EIGENVECTOR MATRIX S IF AND ONLY IF ABBAITEM SHOW THAT IF A AND B ARE SIMILAR SO THAT B  T1AT  BEGINENUMERATE  ITEM  A AND B HAVE THE HAVE THE SAME EIGENVALUES AND THE SAME    CHARACTERISTIC EQUATION  ITEM IF XBF IS AN EIGENVECTOR OF A THEN ZBF  T1 XBF    IS AN EIGENVECTOR OF B  ITEM IF IN ADDITION C AND D ARE SIMILAR WITH D  T1CT    THEN AC IS SIMILAR TO BD  ENDENUMERATE  ITEM DETERMINE THE JORDAN FORM OF A1  BEGINBMATRIX 2  1  2  0  2  3  002 ENDBMATRIXAND A2  BEGINBMATRIX 202  023  0 02 ENDBMATRIXITEM SHOW THAT REFEQJORDANPOW IS TRUE FOR THE MATSIZE33  MATRIX SHOWN  THEN GENERALIZE BY INDUCTION TO AN MATSIZEMM  JORDAN BLOCKITEM SHOW THAT IF J IS A MATSIZE33 JORDAN BLOCK THAT EJT  BEGINBMATRIXELAMBDA T  TELAMBDA T   FRAC12 T2 ELAMBDA T 0ELAMBDA T  TELAMBDA T 00 ELAMBDA T ENDBMATRIXTHEN GENERALIZE BY INDUCTION TO A MATSIZEMM JORDAN BLOCKITEM SHOW THAT BEGINFACTBOX  A SELFADJOINT MATRIX IS POSITIVE SEMIDEFINITE IF AND  INDEXPOSITIVE SEMIDEFINITE ONLY IF ALL OF ITS EIGENVALUES ARE  GEQ 0ENDFACTBOXALSO SHOW THAT IF ALL THE EIGENVALUES ARE POSITIVE THEN THE MATRIX ISPOSITIVE DEFINITE INDEXPOSITIVE DEFINITEITEM SHOW THAT THE CONVERSE TO THE PREVIOUS PROBLEM IS NOT TRUE FIND  A MATRIX WITH POSITIVE EIGENVALUES WHICH IS NOT POSITIVE DEFINITEITEM SHOW THAT IS A IS POSITIVE DEFINITE THEN SO IS AK FOR K  IN ZBB POSITIVE AS WELL AS NEGATIVE POWERSITEM SHOW THAT IF A IS NONSINGULAR THEN A AH IS POSITIVE DEFINITEITEM LABELEXSPECTRH PROVE THEOREM REFTHMDIAGSYM BY ESTABLISHING  THE FOLLOWING   TWO STEPS  BEGINENUMERATE  ITEM SHOW THAT IF A IS SELFADJOINT AND U IS UNITARY THEN SO    IS U1 A U  ITEM SHOW THAT IF A SELFADJOINT MATRIX IS TRIANGULAR THEN IT MUST    BE DIAGONAL  ENDENUMERATEITEM LABELEXPROVEZEROEIG PROVE COROLLARY REFCORZEROEIGITEM A MATRIX N IS BF NORMAL IF IT COMMUTES WITH    NH NHN  NNH  BEGINENUMERATE  ITEM SHOW THAT SYMMETRIC HERMITIAN AND SKEW SYMMETRIC AND SKEW    HERMITIAN MATRICES ARE NORMAL  ITEM SHOW THAT FOR A NORMAL MATRIX THE TRIANGULAR MATRIX    DETERMINED BY THE SCHUR LEMMA IS DIAGONAL  ENDENUMERATEITEM LABELEXCYCLICMAT LETF  BEGINBMATRIX1  1  CDOTS  1 1  EJ2PIN  CDOTS  EJ2PI 1  EJ4PIN  CDOTS  EJ2NPI VDOTS  VDOTS 1  EJPI  EJ2PI  CDOTS  EJN2PIENDBMATRIXTHIS MATRIX COMPUTES AN NPOINT DFTBEGINENUMERATEITEM PROVE BY DIRECT MULTIPLICATION THAT THE MATRIX FSQRTN IS  UNITARY  HINT SHOW THAT BOXEDSUMN0N1 EJ2PI NKN  BEGINCASES N  K EQUIV0 BMOD N 0  K NOT EQUIV 0 BMOD N ENDCASESITEM A MATRIX C  BEGINBMATRIXC0  C1  C2  LDOTS  CN1 CN1  C0  C1  LDOTS  CN2 VDOTS C1C2   LDOTS    C0 ENDBMATRIXIS SAID TO BE A EM CIRCULANT MATRIX  SHOW THAT C IS DIAGONALIZEDBY F SO THAT FCFH IS DIAGONAL  COMMENT ON THE EIGENVALUESAND EIGENVECTORS OF A CIRCULANT MATRIX  THE FFTBASED APPROACH TOCYCLIC CONVOLUTION WORKS BY TRANSFORMING THE CYCLIC MATRIX TO ADIAGONAL MATRIX WHERE MULTIPLICATION POINTBYPOINT CAN OCCURFOLLOWED BY TRANSFORMATION BACK TO THE ORIGINAL SPACEENDENUMERATEENDEXERCISESSECTIONGEOMETRY OF INVARIANT SUBSPACESLABELSECGEOINVSUBBEGINDEFINITION LET A BE A MATRIX  IF S SUBSET RANGEA IS  SUCH THAT XBF IN S MEANS THAT AXBF IN S THEN S IS SAID TO  BE AN BF INVARIANT SUBSPACE FOR A INDEXINVARIANT SUBSPACEENDDEFINITIONSUBSPACES FORMED BY SETS OF EIGENVECTORS FORM THE INVARIANT SUBSPACESOF A MATRIX  FOR A MATSIZEMM MATRIX A WITH KLEQ MDISTINCT EIGENVALUES LET XBF1 XBF2LDOTSXBFM DENOTE THENORMALIZED EIGENVECTORS AND LET XII12LDOTSK DENOTE THE SETOF EIGENVECTORS ASSOCIATED WITH THE EIGENVALUE LAMBDAI  WE CANDENOTE THE ITH INVARIANT SUBSPACE OF A BY RI  LSPANXITHE MATRIX PI  SUMXBFJ IN XI XBFJ XBFJHINDEXPROJECTION MATRIXIS THE PROJECTION MATRIX WHICH PROJECTS ONTO RI  BY MEANS OF THEPROJECTORS ONTO INVARIANT SUBSPACES WE CAN DECOMPOSE AN OPERATOR AINTO SIMPLE PIECES SO THAT THE OPERATION OF A CAN BE EXPRESSED ASTHE SUM OF SIMPLE PROJECTION OPERATIONS  THIS IS WHAT THE FOLLOWINGTHEOREM DOES FOR USBEGINTHEOREM  LABELTHMADECOMP  LET A BE A MATSIZEMM SELFADJOINT  MATRIX WITH K LEQ M DISTINCT EIGENVALUES  THEN  BEGINENUMERATE  ITEM SPECTRAL DECOMPOSITION INDEXSPECTRAL DECOMPOSITIONBEGINEQUATION A  SUMI1K LAMBDAI PILABELEQSPECTDECOMPENDEQUATIONITEM RESOLUTION OF IDENTITY INDEXRESOLUTION OF IDENTITY  BEGINEQUATION    LABELEQRESOLVID    I  SUMI1K PI  ENDEQUATION  ENDENUMERATEENDTHEOREMTHE PROOF OF THIS THEOREM IS LEFT AS AN EXERCISE  BY THEOREM REFTHMADECOMP THE ACTION OF A ON THE VECTOR XBFCAN BE WRITTEN AS A XBF  SUMI1K LAMBDAI PI XBFTHIS CAN BE INTERPRETED AS FOLLOWSBEGINENUMERATEITEM FIRST FIND THE COMPONENTS OF XBF IN EACH OF THE INVARIANT  SUBSPACES R1R2LDOTS RK BY PROJECTING XBF INTO EACH OF  THESE SPACES XBF  P1 XBF  P2 XBF  CDOTS  PK XBFWHERE PI XBF IN RIITEM THEN STRETCH THESE COMPONENTS BY  LAMBDA1LAMBDA2LDOTSLAMBDAK RESPECTIVELYITEM THEN ADD ALL THE PIECES TOGETHERENDENUMERATETHEOREM REFTHMADECOMP ALSO PROVIDES A MEANS OF CONSTRUCTING ASELFADJOINT MATRIX WITH A GIVEN EIGENSTRUCTUREBEGINEXAMPLE  WE WANT TO CONSTRUCT A MATSIZE22 SELFADJOINT MATRIX WITH  EIGENVALUES LAMBDA1  5 AND LAMBDA2  10 WITH EIGENVECTORS  POINTING IN THE DIRECTIONS XBF1  BEGINBMATRIX 3  4 ENDBMATRIX QQUADXBF2  BEGINBMATRIX 4  3 ENDBMATRIXNOTE THAT THE EIGENVECTORS POINT IN ORTHOGONAL DIRECTIONS AS THEYMUST  SINCE THE VECTORS ARE NOT NORMALIZED WE MUST NORMALIZE THEMTHEN P1  FRAC125BEGINBMATRIX3  4ENDBMATRIXBEGINBMATRIX3   4 ENDBMATRIX  FRAC125BEGINBMATRIX 912 12   16ENDBMATRIX P2  FRAC125 BEGINBMATRIX4  3ENDBMATRIXBEGINBMATRIX4   3 ENDBMATRIX  FRAC125BEGINBMATRIX 1612 12   9ENDBMATRIXTHEN  A  5 P1  10 P2HAS THE DESIRED EIGENVALUES AND EIGENVECTORSENDEXAMPLEBEGINEXERCISESITEM PROVE THEOREM REFTHMADECOMPITEM CONSTRUCT MATSIZE33 MATRICES ACCORDING THE FOLLOWING SETS  OF SPECIFICATIONS SEE SPECEIGM  BEGINENUMERATE  ITEM LAMBDA1 LAMBDA21 LAMBDA3  2 WITH INVARIANT    SUBSPACES R1  LSPAN121T210TQQUADQQUAD R2     LSPAN125TIN THIS CASE DETERMINE THE EIGENVALUES AND EIGENVECTORS OF THE MATRIXYOU CONSTRUCT AND COMMENT ON THE RESULTSITEM LAMBDA1  1 LAMBDA2  4 LAMBDA3  9 WITH  CORRESPONDING EIGENVECTORS XBF1  FRAC1SQRT14BEGINBMATRIX1  23  ENDBMATRIX  QQUAD XBF2  FRAC1SQRT5BEGINBMATRIX 2  1  0  ENDBMATRIXQQUAD XBF3  FRAC1SQRT70BEGINBMATRIX365ENDBMATRIX  ENDENUMERATEITEM CITEPAGE 663KAILATH80 THE DIAGONALIZATION OF SELFADJOINT  MATRICES CAN BE EXTENDED TO MORE GENERAL MATRICES  LET A BE A  MATSIZEMM MATRIX WITH M LINEARLY INDEPENDENT EIGENVECTORS  XBF1XBF2LDOTS XBFM AND LET S  XBF1 XBF2  LDOTS XBFM  LET T  S1  THEN WE HAVE A  S LAMBDA  T WHERE LAMBDA IS THE DIAGONAL MATRIX OF EIGENVALUES  BEGINENUMERATE  ITEM LET TBFIT BE A ROW OF T  SHOW THAT TBFIT XBFJ     DELTAIJ  ITEM SHOW THAT A  SUMI1M LAMBDAI XBFI TBFIT  ITEM LET PI  XBFI TBFIT  SHOW THAT PI PJ  PI    DELTAIJ  ITEM SHOW THAT I  SUMI1M PI RESOLUTION OF IDENTITY  ITEM SHOW THAT  SIA1  SUMI1M FRACPISLAMBDAIENDENUMERATEENDEXERCISESSECTIONGEOMETRY OF QUADRATIC FORMSGEOMETRY OF QUADRATIC FORMS AND  THE MINIMAX PRINCIPALLABELSECGEOSYMBEGINDEFINITION  A EM QUADRATIC FORM INDEXQUADRATIC FORM OF A SELFADJOINT  MATRIX A IS A SCALAR OF THE FORM LA AYBFYBF RA  YBFH A  YBF  THIS WILL ALSO BE WRITTEN AS QAYBF  YBFH A YBFENDDEFINITIONQUADRATIC FORMS ARISE IN A VARIETY OF SIGNAL PROCESSING APPLICATIONSWHERE SQUAREDERROR TERMS OR GAUSSIAN DENSITIES ARE EMPLOYEDAN UNDERSTANDING OF THE GEOMETRY INDUCED BY QUADRATIC FORMS CAN ALSOAID IN UNDERSTANDING SOME ITERATIVE OPTIMIZATION AND FILTERINGOPERATIONSBEGINEXAMPLECONSIDER THE LEASTSQUARE FUNCTIONAL FROM REFEQGRADMIN2 WHEREWE ASSUME FOR CONVENIENCE THAT ALL VARIABLES ARE REALBEGINEQUATION JCBF    X2  2 CBFT PBF  CBFT R CBFLABELEQJCBF1ENDEQUATIONWE CAN WRITE THIS AS A QUADRATIC FORM WITH A SCALAR OFFSET BYCOMPLETING THE SQUARE  FROM SECTION REFAPPDXCTS WE SEE THAT WECAN WRITE JCBF  CBF  CBF0TRCBF  CBF0  DWHERE CBF0  R1PBF AND D   X   PBFTR1 PBFWE NOW MAKE A TRANSLATION OF THE COORDINATE SYSTEM BY YBF  CBF CBF0 AND WRITE WITH SOME ABUSE OF NOTATION JYBF  YBFT R YBF  DTHIS IS AN OFFSET QUADRATIC FORMENDEXAMPLEBEGINEXAMPLE LABELEXMGAUSCONTOUR  IT IS DESIRED TO MAKE A PLOT OF THE CONTOURS OF CONSTANT PROBABILITY  FOR THE 2DIMENSIONAL GAUSSIAN  VECTOR XBF SIM  NCMUBFR  THAT IS WE WANT TO PLOT FXBFXBF  FRAC12PIM2R12EXPFRAC12XBF MUBFT R1XBFMUBF  CFOR DIFFERENT VALUES OF THE CONSTANT C  AFTER SOME ALGEBRAICREDUCTION THIS REDUCES TO XBF  MUBFT R1XBF  MUBF  CWHERE C  2 LOG C2PIM2 R12  LETTING YBF  XBF  MUBFWE OBTAINBEGINEQUATIONYBFT R1 YBF  CLABELEQGAUSCONTOURENDEQUATIONTHIS IS AN EQUATION OF THE FORM QR1YBF  CENDEXAMPLEBY DIAGONALIZING THE MATRIX A IN THE QUADRATIC FORM QAYBF WETRANSFORM TO A NEW COORDINATE SYSTEM IN WHICH THE GEOMETRY BECOMESMORE APPARENT  FOR CONVENIENCE WE WILL ASSUME THAT REAL VECTORS ARE USEDUSING THE DECOMPOSITION  A  QLAMBDA QT WHERE Q  BEGINBMATRIXQBF1 QBF2  CDOTS  QBFM ENDBMATRIXWE CAN OBSERVETHATBEGINEQUATION QAYBF  YBFT QLAMBDA QT YBF  ZBFT LAMBDA ZBF SUMI1M LAMBDAI ZI2LABELEQEIGGEOM2ENDEQUATIONWHERE ZBF  QT YBFTHE NEW VARIABLE ZBF IS IN A COORDINATE SYSTEM IN WHICH THEINTERACTION BETWEEN THE COMPONENTS OF THE VECTOR ARE ELIMINATEDTHE VARIABLE ZBF CAN BE INTERPRETED GEOMETRICALLY IN TWODIMENSIONS BY OBSERVING THAT WHEN ZBF  LEFTBEGINSMALLMATRIX 1     0ENDSMALLMATRIXRIGHT THEN YBF  QBF1 THE FIRSTEIGENVECTOR OF A AND WHEN ZBF  LEFTBEGINSMALLMATRIX  0     1ENDSMALLMATRIXRIGHT THEN YBF  QBF2  THE EM  ORTHOGONAL EIGENVECTORS OF A THUS PROVIDE THE ORTHOGONAL BASES OF  A NEW COORDINATE SYSTEM  FIGURE REFFIGEIGGEOM1 ILLUSTRATES THECONCEPT  IN FIGURE REFFIGEIGGEOM1A LEVEL CURVES OF THEQUADRATIC FORM XBFXBF0T AXBFXBF0 ARE SHOWN WHEREBEGINEQUATION A  BEGINBMATRIX388  384 384  612 ENDBMATRIXLABELEQEIGGEOMAENDEQUATIONHAS THE EIGENDECOMPOSITION BEGINALIGNEDLAMBDA1  9  QQUADQQUADXBF1  FRAC1534T EXMATSPLAMBDA2  1  QQUADQQUAD XBF2  FRAC1543TENDALIGNEDAND XBF0  21T  ALSO SHOWN IN FIGURE REFFIGEIGGEOM1AARE THE NEW COORDINATES Y1 AND Y2 OBTAINED BY THE TRANSLATIONYBF  XBF  XBF0  THESE COORDINATES HAVE THEIR ORIGIN AT THEBOTTOM OF THE QUADRATIC BOWL  IN FIGURE REFFIGEIGGEOM1B WEUSE THE NEW COORDINATES Z1 AND Z2 IN THE EIGENVECTOR DIRECTIONSOF A  THESE COORDINATES POINT IN THE EIGENVECTOR DIRECTIONS OFA  THE LEVEL CURVES IN THE Z COORDINATES CORRESPOND TO THEEQUATION Z12 LAMBDA1  Z22 LAMBDA2  COR 9Z12 Z22  COR FRACZ121  FRACZ229  CWHICH IS THE EQUATION FOR AN ELLIPSE  THE STEEPEST DIRECTION OUT OFTHE BOWL ALONG THE Z1 AXIS CORRESPONDS TO THE LARGESTEIGENVALUE  BEGINFIGUREHTBPCENTERINGMBOXSUBFIGUREEPSFIGFILEPICTUREDIREIGDIR1EPS WIDTH045TEXTWIDTHQUAD      SUBFIGUREEPSFIGFILEPICTUREDIREIGDIR2EPS WIDTH045TEXTWIDTH  CAPTIONTHE GEOMETRY OF QUADRATIC FORMSTHE GEOMETRY OF QUADRATIC FORMS A THE ORIGINAL AND TRANSLATED COORDINATES B THE ROTATED COORDINATES  LABELFIGEIGGEOM1 EIGDIRMENDFIGUREIN THE GENERAL TWODIMENSIONAL CASE THE LEVEL CURVES OF THE QUADRATICFORM QAXBFC ARE OF THE FORMBEGINEQUATION Z12 LAMBDA1  Z22 LAMBDA2  CLABELEQEIGGEOM2AENDEQUATIONIF LAMBDA1LAMBDA2 0 THIS EQUATION DESCRIBES AN ELLIPSE WITHMAJOR AND MINOR AXES IN THE DIRECTIONS OF THE EIGENVECTORS OF AFOR LAMBDA1 LAMBDA2  0 REFEQEIGGEOM2A DEFINES A HYPERBOLAIF THE EIGENVALUES DIFFER GREATLY IN MAGNITUDE SUCH AS LAMBDA1 GGLAMBDA2 THEN THE MATRIX A IS SAID TO HAVE A BIG EM EIGENVALUE  DISPARITY  THIS CORRESPONDS TO THE MATRIX BEING POORLYCONDITIONED INDEXEIGENVALUE DISPARITY SPREADINDEXILLCONDITIONEDBEGINEXAMPLE  RETURNING TO EXAMPLE REFEXMGAUSCONTOUR WE WANT  TO MAKE PLOTS OF THE CONTOURS OF CONSTANT PROBABILITY WHERE YBFT R1 YBF  CLET US WRITE THE COVARIANCE MATRIX R AS R  U LAMBDA UHTHEN R1 HAS THE DECOMPOSITION R1  U LAMBDA1 UHAND REFEQGAUSCONTOUR CAN BE WRITTEN AS ZBFT LAMBDA1 ZBF  CSINCE THE EIGENVALUES OF R1 ARE THE RECIPROCALS OF THEEIGENVALUES OF R SEE EXERCISE REFEXEIGINV  IN TWO DIMENSIONSTHIS IS FRACZ12LAMBDA1  FRACZ22LAMBDA2  CWHEN C1 THIS DEFINES AN ELLIPSE WITH MAJOR AND MINOR AXESSQRTLAMBDA1 AND SQRTLAMBDA2  FIGURE REFFIGEIGGEOM2ILLUSTRATES THE CASE FOR R  BEGINBMATRIX388  384 384  612 ENDBMATRIXTHE SAME AS IN REFEQEIGGEOMA  THE LEVEL CURVES ARE OF THE FORM FRACZ129  FRACZ221  CIN THIS CASE Z1 POINTS IN THE DIRECTION OF EM SLOWEST INCREASEAS IT IS SCALED BY THE EM INVERSE OF THE EIGENVALUE  LARGEEIGENVALUES CORRESPOND TO LARGE VARIANCES AND HENCE THE BROAD SPREADIN THE DISTRIBUTIONBEGINFIGUREHTBPCENTERINGMBOXEPSFIGFILEPICTUREDIREIGDIR3EPSWIDTH045TEXTWIDTH  CAPTIONLEVEL CURVES FOR A GAUSSIAN DISTRIBUTION  LABELFIGEIGGEOM2 EIGDIR2MENDFIGUREENDEXAMPLEIN HIGHER DIMENSIONS THE SAME GEOMETRIC PRINCIPLE APPLIES  BEGINFACTBOXQUADRATIC FORMS OF A MATRIX A GIVE RISE TO  CLASSICAL CONIC SECTIONS IN TWO DIMENSIONS  ELLIPSES HYPERBOLAS  AND INTERSECTING LINES  AND MULTIDIMENSIONAL GENERALIZATIONS OF  THE CONIC SECTIONS FOR HIGHER DIMENSIONS WITH ORTHOGONAL AXIS  DIRECTIONS DETERMINED BY THE EIGENVECTORS OF AENDFACTBOXTHE QUADRATIC FORMS OF AN MATSIZEMM MATRIX WITH ALL POSITIVEEIGENVALUES FORM AN ELLIPSOID IN M DIMENSIONS IN THREE DIMENSIONSIT HELPS TO ENVISION AN AMERICAN FOOTBALL AN ELLIPSOID  FIGUREREFFIGFOOTBALLA SHOWS THE LOCUS OF POINTS PRODUCED BYQAXBF FOR XBF  1 THE UNIT BALL WHERE A IS A POSITIVEDEFINITE MATRIX ALL EIGENVALUES 0  THE L2 NORM OF THE MATRIXCORRESPONDS TO THE AMOUNT OF STRETCH IN THE DIRECTION THAT THE UNITBALL IS STRETCHED THE FARTHEST  THE DIRECTION OF THE EIGENVECTORASSOCIATED WITH THE LARGEST EIGENVALUE CALL IT XBF1  IF WE SLICETHE ELLIPSOID THROUGH THE LARGEST CROSS SECTION PERPENDICULAR TOXBF1 AS SHOWN IN FIGURE REFFIGFOOTBALLB THE LOCUS IS ANELLIPSE  THE LARGEST DIRECTION OF THE ELLIPSE ON THIS PLANECORRESPONDS TO THE NEXT LARGEST EIGENVALUE AND SO FORTHBEGINFIGUREHTBPCENTERINGMBOXSUBFIGUREEPSFIGFILEPICTUREDIRELLIPSOID1EPS WIDTH045TEXTWIDTHQUAD      SUBFIGUREEPSFIGFILEPICTUREDIRELLIPSOID2EPS WIDTH045TEXTWIDTH  CAPTIONTHE MAXIMUM PRINCIPALTHE MAXIMUM PRINCIPAL  A AN ELLIPSOID IN    THREEDIMENSIONS B THE PLANE ORTHOGONAL TO THE PRINCIPAL    EIGENVECTOR  LABELFIGFOOTBALL SURF1MENDFIGURETHE EIGENVALUES OF A SELFADJOINT MATRIX CAN BE ORDERED SO THAT LAMBDA1 GEQ LAMBDA2 GEQ LAMBDA3 GEQ CDOTS GEQ LAMBDAMWITH THIS ORDERING LET THE ASSOCIATED EIGENVECTORS BEXBF1XBF2LDOTSXBFM  IT IS ALSO CONVENIENT TO ASSUME THATTHE EIGENVECTORS HAVE BEEN NORMALIZED SO THAT XBFI21I12LDOTSM  WITH THIS ORDERING THE GEOMETRICAL REASONING ABOUTTHE ELLIPSOID CAN BE SUMMARIZED AND GENERALIZED TO M DIMENSIONS BYTHE FOLLOWING THEOREMBEGINTHEOREM LABELTHMMAXEIG  MAXIMUM PRINCIPLE FOR A POSITIVE SEMIDEFINITE SELFADJOINT MATRIX  A WITH QAXBF  LA AXBFXBF RA  XBFH A XBF THE  MAXIMUM MAX XBF2  1 QAXBFIS LAMBDA1 THE LARGEST EIGENVALUE OF A AND THE MAXIMIZINGXBF IS XBF  XBF1 THE EIGENVECTOR CORRESPONDING TO LAMBDA1FURTHERMORE IF WE MAXIMIZE QAXBF SUBJECT TO THE CONSTRAINTS THATBEGINENUMERATEITEM LA XBFXBFJRA  0 J12LDOTSK1 ANDITEM XBF2  1ENDENUMERATETHEN LAMBDAK IS THEMAXIMIZED VALUE SUBJECT TO THE CONSTRAINTS AND XBFK IS THECORRESPONDING VALUE OF XBFENDTHEOREMTHE CONSTRAINT LA XBFXBFJ RA  0 SERVES TO PROJECT THE SEARCHTO THE SPACE ORTHOGONAL TO THE PREVIOUSLYDETERMINED EIGENDIRECTIONSEG THE SLICE THROUGH THE ELLIPSOIDBEGINPROOF  THE PROOF IS BY CONSTRAINED OPTIMIZATION USING LAGRANGE MULTIPLIERS  SEE SECTION REFSECBASICOPT  INDEXCONSTRAINED OPTIMIZATION  INDEXLAGRANGE MULTIPLIER WE HAVE ALREADY SEEN THE FIRST PART OF  THE PROOF IN THE CONTEXT OF THE SPECTRAL NORM  FORM THE FUNCTION JXBF  XBFH A XBF  LAMBDAXBFH XBFWHERE LAMBDA IS A LAGRANGE MULTIPLIER  TAKING THE GRADIENT WITHRESPECT TO XBF SEE SECTION REFSECIMPGRAD AND EQUATING TO ZEROWE OBTAIN PARTIALDXBF JXBF  AXBF  LAMBDA XBF  0WE SEE THAT THE MAXIMIZING SOLUTIONFOOTNOTETHE  HERE INDICATES  AN EXTREMIZING VALUE NOT AN ADJOINT  NOTATIONALLY THERE SHOULD BE  LITTLE AMBIGUITY SINCE XBF IS A VECTOR NOT AN OPERATORXBF MUST SATISFY AXBF  LAMBDAXBFTHUS XBF MUST BE AN EIGENVECTOR OF A AND LAMBDA MUST BEEIGENVALUE  FOR THIS XBF WE HAVE QXBF  XBFHAXBF  LAMBDA XBFH XBF  MAXIMIZATION OF THIS SUBJECT TOTHE CONSTRAINT XBF1 REQUIRES THAT WE CHOOSE LAMBDA TO BETHE LARGEST EIGENVALUE AND XBF  XBF1 THE EIGENVECTORASSOCIATED WITH THE LARGEST EIGENVALUETO PROVE THE SECOND PART OBSERVE THAT SINCE THE EIGENVECTORS OF ASELFADJOINT MATRIX ARE ORTHOGONAL THE MAXIMIZING SOLUTION XBFSUBJECT TO THE CONSTRAINTS LA XBFXBFJRA 0QQUAD J12LDOTSK1MUST LIE IN SKM  LSPANXBFKXBFK1LDOTSXBFM  LET BEGINALIGNEDXBF  FRACXBFK  ALPHAK1 XBFK1  ALPHAK2  XBFK2  CDOTS  ALPHAM XBFM  XBFK  ALPHAK1 XBFK1  ALPHAK2 XBFK2   CDOTS  ALPHAMXBFM  EXMATSP FRACXBFK  ALPHAK1 XBFK1  ALPHAK1 XBFK2   CDOTS  ALPHAM XBFMSQRT1  ALPHAK12  ALPHAK22     CDOTS  ALPHAM2ENDALIGNEDBE A NORMALIZED CANDIDATE SOLUTION  THENBEGINEQUATION  QAXBF  LAMBDAK FRAC1  ALPHAK12    FRACLAMBDAK1LAMBDAK  CDOTS  ALPHAM2    FRACLAMBDAMLAMBDAK 1ALPHAK12  CDOTS     ALPHAM2LABELEQQMAX1ENDEQUATIONSINCE LAMBDAK GEQ LAMBDAK1 GEQ CDOTS GEQ LAMBDAM GEQ 0QAXBF IS MAXIMIZED WHEN ALPHAK1  ALPHAK2  CDOTS ALPHAM  0 SEE EXERCISE REFEXEIGMAX  THUS QXBF HASTHE MAXIMUM VALUE LAMBDAK AND XBF0  XBFKENDPROOFTHE QUOTIENT RXBF  FRACXBFT AXBFXBFTXBFIS KNOWN AS A EM RAYLEIGH QUOTIENT  INDEXRAYLEIGH QUOTIENT FROMTHEOREM REFTHMMAXEIG WE CAN CONCLUDE THAT MAXXBFNEQ 0 RXBF  LAMBDA1AND THAT THE MAXIMIZING VALUE IS XBF  XBF1 AND THAT MINXBFNEQ 0 RXBF  LAMBDAMWHERE THE MINIMIZING VALUE IS XBF  XBFM  SOME LEASTSQUARESPROBLEMS CAN BE COUCHED IN TERMS OF RAYLEIGH QUOTIENTS AS WILL BESHOWN IN SECTION REFSECEIGFILTAPPLICATION OF THEOREM REFTHMMAXEIG REQUIRES KNOWING THE FIRST K1EIGENVECTORS IN ORDER TO FIND THE KTH EIGENVALUE AND EIGENVECTORTHE FOLLOWING THEOREM PROVIDES A MEANS OF CHARACTERIZING THEEIGENVALUES WITHOUT KNOWING THE FIRST K1 EIGENVECTORS  IT IS OFTENUSEFUL IN DETERMINING APPROXIMATE VALUES FOR THE EIGENVALUESBEGINTHEOREM LABELTHMCOURANT INDEXMINIMAX PRINCIPLE FOR EIGENVALUES  COURANT MINIMAX PRINCIPLE FOR ANY SELFADJOINT MATSIZEMM  MATRIX A LAMBDAK  MINC MAXBEGINSUBARRAYC  XBF2  1  CXBF   0ENDSUBARRAY LA AXBFXBF RAWHERE C IS EM ANY MATSIZEK1M MATRIXENDTHEOREMGEOMETRICALLY THE REQUIREMENT THAT CXBF  0 MEANS THAT XBF LIESON SOME MK1DIMENSIONAL HYPERPLANE IN RBBM  WE FINDLAMBDAK BY MAXIMIZING QAXBF FOR XBF LYING ON THEHYPERPLANE SUBJECT TO THE CONSTRAINT XBF21 THEN MOVE THEHYPERPLANE AROUND UNTIL THE MAXIMUM VALUE QAXBF IS AS SMALL ASPOSSIBLE  FOR EXAMPLE TO FIND LAMBDA2 THINK OF MOVING THEPLANE AROUND IN FIGURE REFFIGFOOTBALLBBEGINPROOF  FOR AULAMBDA UH WE HAVE LA AXBFXBFRA  LA LAMBDAYBFYBF RA  SUMI1MLAMBDAI YI2WHERE YBF  QH XBF  NOTE THAT CXBF  0 IF AND ONLY IF CQYBF 0  LET B  CQLET MU  MINC MAXBEGINSUBARRAYC  XBF2  1  CXBF   0ENDSUBARRAY LA AXBFXBF RA  MINB MAXBEGINSUBARRAYC  YBF21    BYBF  0ENDSUBARRAY SUMI1M LAMBDAI YI2THE PROOF IS GIVEN BY SHOWING THAT MU LEQ LAMBDAK AND MU GEQLAMBDAK SO THE ONLY ALTERNATIVE IS MU  LAMBDAKIT IS POSSIBLE TO CHOOSE A FULLRANK B SO THAT BYBF 0 IMPLIESTHAT Y1  Y2  CDOTS  YK10  FOR SUCH A B MU LEQ MAXYBF2  1 SUMIKMLAMBDAI YI2 LAMBDAKWHERE THE INEQUALITY COMES BECAUSE THE MINIMUM OVER B IS NOTTAKENTO GET THE OTHER INEQUALITY ASSUME THAT YK1  YK2  CDOTS YM  0  WITH THESE MK CONSTRAINTS THE EQUATION BYBF  0 ISA SYSTEM OF K1 EQUATIONS IN THE K UNKNOWNS Y1Y2DOTSYKTHIS ALWAYS HAS NONTRIVIAL SOLUTIONS  THEN MU GEQ MINB MAXBEGINSUBARRAYC YBF2  1  YK1   CDOTS    YM  0  BYBF  0ENDSUBARRAY SUMI1MLAMBDAI YI2 GEQ MINB MAXBEGINSUBARRAYC YBF2  1  YK1   CDOTS    YM  0  BYBF  0ENDSUBARRAY  LAMBDAKSUMI1K YI2  LAMBDAKWHERE THE FIRST INEQUALITY COMES BY VIRTUE OF THE EXTRA CONSTRAINTS ONTHE MAX AND THE SECOND INEQUALITY FOLLOWS SINCE LAMBDAK IS THESMALLEST OF THE EIGENVALUES IN THE SUMENDPROOFBEGINEXERCISESITEM LET  R  BEGINBMATRIX515385  323077 323077  784615 ENDBMATRIXBEGINENUMERATEITEM DETERMINE THE EIGENVALUES AND EIGENVECTORS OF RITEM DRAW LEVEL CURVES OF THE QUADRATIC FORM QRXBF  IDENTIFY  THE EIGENVECTOR DIRECTIONS ON THE PLOT AND ASSOCIATE THESE WITH TH  EIGENVALUES ITEM DRAW THE LEVEL CURVES OF THE QUADRATIC FORM QR1XBF  IDENTIFYING EIGENVECTOR DIRECTIONS AND THE EIGENVALUESENDENUMERATEITEM LABELEXEIGMAX IN THE PROOF OF THEOREM REFTHMMAXEIG  BEGINENUMERATE  ITEM SHOW THAT REFEQQMAX1 IS TRUE  ITEM SHOW THAT QXBF OF REFEQQMAX1 IS MAXIMIZED WHEN    ALPHAK1  ALPHAK2  CDOTS  ALPHAM  0  HINT    LET ABF  ALPHA0ALPHAK1ALPHAK2LDOTSALPHAM    QQUADBBF  B0BKBK1LDOTSBMWHERE ALPHA0  B0 1 AND BI  LAMBDAILAMBDAK THEN USETHE CAUCHYSWARTZ INEQUALITYENDENUMERATEITEM WRITE AND TEST A SC MATLAB FUNCTION TT PLOTELLIPSEAX0C  THAT COMPUTES POINTS ON THE ELLIPSE DESCRIBED BY XBFXBF0T A  XBFXBF0  C SUITABLE FOR PLOTTINGENDEXERCISESSECTIONEXTREMAL QUADRATIC FORMS SUBJECT TO LINEAR CONSTRAINTSLABELSECCONEIGTHE OPTIMIZATION PROBLEMS OF THE PREVIOUS SECTION FOUND EXTREMA OFQUADRATIC FORMS SUBJECT TO THE CONSTRAINT THAT THE SOLUTION ISORTHOGONAL TO PREVIOUS SOLUTIONS  IN THIS SECTION WE MODIFY THECONSTRAINT SOMEWHAT AND CONSIDER GENERAL LINEAR CONSTRAINTS  IMAGINEAN ELLIPSOID IN THREE DIMENSIONS AS IN FIGURE REFFIGFOOTBALLATHE AXES OF THE ELLIPSE CORRESPOND TO THE EIGENVECTORS OF A MATRIXWITH THE LENGTH DETERMINED BY THE EIGENVALUES  NOW IMAGINE THEELLIPSOID IS SLICED BY A PLANE THROUGH THE ORIGIN AS INREFFIGFOOTBALLB BUT WITH THE PLANE FREE TO CROSS AT ANY ANGLETHE INTERSECTION OF THE ELLIPSOID AND THE PLANE FORMS AN ELLIPSEWHAT ARE THE MAJOR AND MINOR AXES OF THIS INTERSECTING ELLIPSEPOINTS ON THE PLANE CAN BE DESCRIBED AS XBFT CBF  0 WHERECBF IS THE VECTOR ORTHOGONAL TO THE PLANE  THE PROBLEM IS TODETERMINE THE STATIONARY POINTS EIGENVECTORS AND EIGENVALUES OFXBFH A XBF THE ELLIPSOID SUBJECT TO THE CONSTRAINTS XBFHXBF  1 AND XBFH CBF  0  THE PROBLEM AS STATED IN THREEDIMENSIONS CAN OBVIOUSLY BE GENERALIZED TO HIGHER DIMENSIONS  WITHOUTLOSS OF GENERALITY ASSUME THAT CBF IS SCALED SO THAT CBF2 1  A SOLUTION MAY BE FOUND USING LAGRANGE MULTIPLIERS  LET JXBF  XBFH A XBF  LAMBDAXBFH XBF  MU XBFH CBFWHERE LAMBDA AND MU ARE LAGRANGE MULTIPLIERS  TAKING THEGRADIENT AND EQUATING TO ZERO LEADS TO BEGINEQUATIONAXBF   LAMBDA XBF  MU CBF  0LABELEQCONEIG1ENDEQUATIONMULTIPLYING BY CBFH AND USING CBF2  1 LEADS TO MU CBFH AXBF  SUBSTITUTING THIS INTO REFEQCONEIG1 LEADS TOBEGINEQUATIONI  CBF CBFHAXBF  LAMBDA XBFLABELEQCONEIG2ENDEQUATIONLET P  ICBFCBFH  IT IS APPARENT THAT P IS A PROJECTIONMATRIX SO P2P P  THEN PA XBF  LAMBDA XBF IS AN EIGENVALUEPROBLEM BUT PA MAY NOT BE HERMITIAN SYMMETRIC EVEN THOUGH BOTH PAND A ARE HERMITIAN SYMMETRIC  SINCE IT IS EASIER TO COMPUTEEIGENVALUES FOR SYMMETRIC MATRICES IT IS WORTHWHILE FINDING A WAY TOSYMMETRICIZE THE PROBLEM  USING THE FACT THAT THE EIGENVALUES OF PAARE THE SAME AS THE EIGENVALUES OF AP WE WRITE LAMBDAPA  LAMBDAP2 A  LAMBDAPAPLET KPAP  THEN FOR AN EIGENVECTOR ZBF IN KZBF  LAMBDA ZBFTHE VECTOR XBF  PZBF IS AN EIGENVECTOR OF PAMORE GENERALLY THE EIGENPROBLEM MAY HAVE SEVERAL CONSTRAINTSBEGINEQUATION  LABELEQCONEIGCON  BEGINSPLITCH XBF  0 XBFH XBF  1ENDSPLITENDEQUATIONTHEN IF P  I  CCHC1CH THE STATIONARY VALUES OF XBFHAXBF SUBJECT REFEQCONEIGCON ARE FOUND FROM THE EIGENVALUES OFKPAP SEE EXERCISE REFEXLINCONEIGBEGINEXERCISESITEM DETERMINE STATIONARY VALUES EIGENVALUES AND EIGENVECTORS OF  XBFT R XBF SUBJECT TO XBFT CBF  0 WHERE R  BEGINBMATRIX515385  323077 323077  784615 ENDBMATRIX QQUAD CBF  12TITEM LABELEXLINCONEIG SHOW THAT THE STATIONARY VALUES OF XBFH  R XBF SUBJECT TO REFEQCONEIGCON ARE FOUND FROM THE  EIGENVALUES OF PAP WHERE P  ICCHC1CHENDEXERCISESINPUTLINALGDIRGERSHCHAPTERPARTAPPLICATION OF EIGENDECOMPOSITION METHODSSECTIONLOWRANK APPROXIMATIONSKARHUNENLOEVE LOWRANK  APPROXIMATIONS AND PRINCIPAL COMPONENT METHODSLABELSECKARHUNEN1LET XBF BE A ZEROMEAN MATSIZEM1 RANDOM VECTOR AND LET R EXBF XBFH  LET R HAVE THE FACTORIZATION R  ULAMBDA UHWHERE THE COLUMNS OF U ARE THE NORMALIZED EIGENVECTORS OF R  LETYBF  UH XBF  THEN YBF IS A ZEROMEAN RANDOM VECTOR WITHUNCORRELATED COMPONENTS EYBF YBFH   LAMBDAWE CAN THUS VIEW THE MATRIX UH AS A WHITENING FILTER  TURNINGTHE EXPRESSION AROUND WE CAN WRITE BEGINEQUATION XBF  UYBF  SUMI1M UBFI YILABELEQKHL1ENDEQUATIONTHIS SYNTHESIS EXPRESSION SAYS THAT WE CAN CONSTRUCT THE RANDOMVARIABLE XBF AS A LINEAR COMBINATION OF ORTHOGONAL VECTORS WHERETHE COEFFICIENTS ARE UNCORRELATED RANDOM VARIABLES  THEREPRESENTATION IN REFEQKHL1 IS CALLED THE EM  KARHUNENLOEVE EXPANSION OF XBF INDEXKARHUNENLOEVEIN THIS EXPANSION THE EIGENVECTORS OF THE CORRELATION MATRIX R AREUSED AS THE BASIS VECTORS OF THE EXPANSION  THE KARHUNENLOEVE EXPANSION COULD BE USED TO TRANSMIT THE VECTORXBF  IF BY SOME MEANS THE AUTOCORRELATION MATRIX AND ITSEIGENDECOMPOSITION WERE KNOWN AT BOTH THE TRANSMITTER AND RECEIVERTHEN SENDING THE COMPONENTS YI WOULD PROVIDE BY REFEQKHL1 AREPRESENTATION OF XBF  IN THIS REPRESENTATION M DIFFERENTNUMBERS ARE NEEDEDSUPPOSE NOW THAT WE WANTED TO PROVIDE AN APPROXIMATE REPRESENTATION OFXBF USING FEWER COMPONENTS  WHAT IS THE BEST REPRESENTATIONPOSSIBLE GIVEN THE CONSTRAINT THAT FEWER THAN M COMPONENTS CAN BEUSED  LET XBFHAT IN CBBM  BE THE APPROXIMATION OFXBF OBTAINED BY XBFHAT  KXBFWHERE K IS A MATSIZEMM MATRIX OF RANK R M  SUCH AREPRESENTATION IS SOMETIMES CALLED A RANKR REPRESENTATION ONLY RPIECES OF INFORMATION ARE USED TO APPROXIMATE XBF  WE DESIRE TODETERMINE K SO THAT XBFHAT IS THE BEST APPROXIMATION OF XBFIN A MINIMUM MEANSQUARED ERROR SENSE  SUCH AN APPROXIMATION ISSOMETIMES REFERRED TO AS A EM LOWRANK  APPROXIMATION INDEXLOWRANK APPROXIMATIONLET R  EXBF XBFH HAVEEIGENVALUES LAMBDA1LAMBDA2LDOTSLAMBDAM WITH CORRESPONDINGEIGENVECTORS XBF1XBF2LDOTSXBFM THE MEANSQUARED ERROR ASA FUNCTION OF K ISBEGINALIGN E2K  EXBFXBFHATH XBFXBFHAT NONUMBER   TRACE EXBFXBFHATXBFXBFHATH NONUMBER   TRACE IKRIKH LABELEQE2KENDALIGNSINCE E2K  E2KH WE MAY ASSUME THAT K IS HERMITIAN  WECAN WRITE K WITH AN ORTHOGONAL DECOMPOSITIONBEGINEQUATION K  SUMI1R MUI UBFI UBFIH  UMR UHLABELEQKLOWRANKENDEQUATIONWHERE MR  BEGINBMATRIX MU1   MU2   DDOTS   MUR  0  DDOTS  0 ENDBMATRIXAND U IS A UNITARY MATRIX  SUBSTITUTING REFEQKLOWRANK INTO REFEQE2K WE FIND SEE EXERCISE REFEXLOWRANK1BEGINEQUATION E2K  SUMI1R UBFIH R UBFI1MUI2  SUMIR1MUBFIH R UBFILABELEQLR2ENDEQUATIONTO MINIMIZE THIS CLEARLY WE CAN SET MUI1I12LDOTSR  THENWE MUST MINIMIZE SUMIR1M UBFIH R UBFISUBJECT TO THE CONSTRAINTS THAT UBFIH UBFJ  DELTAIJ  BUTFROM THE DISCUSSION OF SECTION REFSECGEOSYM UBFIIR1R2LDOTSM MUST BE THE EIGENVECTORS OF R CORRESPONDING TOTHE MR EM SMALLEST EIGENVALUES OF R  THE EIGENVECTORSUBFII12LDOTSR WHICH ARE ORTHOGONAL TO THESE FORM THE COLUMNSOF U SO BEGINEQUATION K  SUMI1R UBFI UBFIH  U IR UHLABELEQKREDUCE1ENDEQUATIONWHERE IR HAS R ONES ON THE DIAGONAL IS THE REST ZEROSTHE MATRIX K IS A RANKR PROJECTION MATRIX  THE INTERPRETATION OF THIS RESULT IS THIS  TO OBTAIN THE BESTAPPROXIMATION TO XBF USING ONLY R PIECES OF INFORMATION SEND THEVALUES OF YI CORRESPONDING TO THE R LARGEST EIGENVALUES OF RLOWRANK APPROXIMATIONS AND KARHUNENLOEVE EXPANSIONS HAVETHEORETICAL APPLICATION IN TRANSFORMCODING FOR DATA COMPRESSION  AVECTOR XBF IS REPRESENTED BY ITS COEFFICIENTS IN THEKARHUNENLOEVE TRANSFORM WITH THE COEFFICIENTS LISTED IN ORDER OFDECREASING EIGENVALUE STRENGTH  THE FIRST R OF THESE COEFFICIENTSARE QUANTIZED AND THE REMAINING COEFFICIENTS ARE SET TO ZERO  THE RCOEFFICIENTS PROVIDE THE REPRESENTATION FOR THE ORIGINAL SIGNAL  THECORRESPONDING SIGNIFICANT EIGENVECTORS OF THE CORRELATION MATRIX AREASSUMED SOMEHOW TO BE KNOWN  SINCE THE KARHUNENLOEVE TRANSFORMPROVIDES THE OPTIMUM LOWRANK APPROXIMATION THE RECONSTRUCTED DATASHOULD BE A GOOD REPRESENTATION OF THE ORIGINAL DATA  HOWEVER THEKARHUNENLOEVE TRANSFORM IS RARELY USED IN PRACTICE  FIRST THEREIS THE PROBLEM OF DETERMINING R AND ITS EIGENVECTORS FOR A GIVENSIGNAL AND SECOND FOR EACH SIGNAL SET THE EIGENVECTORS SELECTED MUSTSOMEHOW BE COMMUNICATED TO THE DECODING SIDESUBSECTIONPRINCIPAL COMPONENT METHODSINDEXPRINCIPAL COMPONENTRELATED TO LOWRANK APPROXIMATIONS ARE PRINCIPAL COMPONENT METHODSLET XBF BE AN MDIMENSIONAL ZEROMEAN RANDOM VECTOR ASSUMED TO BEREAL FOR CONVENIENCE AND LET XBF1ALLOWBREAK XBF2ALLOWBREAKLDOTSALLOWBREAK XBFN BE NOBSERVATIONS OF XBF  WE FORM THE EM SAMPLE COVARIANCE MATRIXS BY S  FRAC1N1 SUMI1N XBFI XBFITTHE PRINCIPAL COMPONENTS OF THE DATA ARE SELECTED SO THAT THE ITHPRINCIPAL COMPONENT IS THE LINEAR COMBINATION OF THE OBSERVED DATAWHICH ACCOUNTS FOR THE ITH LARGEST PORTION OF THE VARIANCE IN THEOBSERVATIONS CITEPAGE 268MORRISON1976  LET Y1 Y2 LDOTSYR BE THE PRINCIPAL COMPONENTS OF THE DATA  THE FIRST PRINCIPALCOMPONENT IS FORMED AS A LINEAR COMBINATION Y1  ABF1T XBFWHERE ABF1 IS CHOSEN SO THAT THE SAMPLE VARIANCE OF Y1 ISMAXIMIZED SUBJECT TO THE CONSTRAINT THAT  ABF11  THEPRINCIPAL COMPONENT VALUES OBTAINED FROM THE OBSERVATIONS ARE Y1I ABFIT XBFI AND THE SAMPLE VARIANCE IS SIGMAY12  FRAC1N1 SUMI1N Y1I2  FRAC1N1SUMI1N ABF1T XBFI2 ABF1T S ABF1MAXIMIZING ABF1T S ABF1 SUBJECT TO ABF11 IS A PROBLEMWE HAVE MET BEFORE ABF1 IS THE NORMALIZED EIGENVECTORCORRESPONDING TO THE LARGEST EIGENVALUE OF S LAMBDA1  IN THISCASE SIGMAY12  ABF1T SABF1  LAMBDA1 ABF1T ABF1 LAMBDA1THE SECOND PRINCIPAL COMPONENT IS CHOSEN SO THAT Y2 IS UNCORRELATEDWITH Y1 WHICH LEADS TO THE CONSTRAINT ABF2T ABF1  0GIVEN THE DISCUSSION IN SECTION REFSECGEOSYM ABF2 IS THEEIGENVECTOR CORRESPONDING TO THE SECONDLARGEST EIGENVALUE OF S ANDSO FORTH  THE EIGENVECTORS USED TO COMPUTE THE PRINCIPAL COMPONENTSARE CALLED THE PRINCIPAL COMPONENT DIRECTIONS  IF MOST OF THEVARIANCE OF THE SIGNAL IS CONTAINED IN THE PRINCIPAL COMPONENTS THESEPRINCIPAL COMPONENTS CAN BE USED INSTEAD OF THE DATA FOR MANYSTATISTICAL PURPOSESBEGINEXAMPLE  FIGURE REFFIGSCATTER1 SHOWS 200 SAMPLE POINTS  FROM SOME MEASURED 2DIMENSIONAL ZEROMEAN DATA  XBF1XBF2LDOTSXBF200  FOR THIS DATA THE COVARIANCE  MATRIX IS ESTIMATED AS S  FRAC12001SUMI1200 XBFI XBFIT BEGINBMATRIXHFILL241893  HFILL106075 HFILL106075  HFILL638059 ENDBMATRIXLET SBF1 AND SBF2 DENOTE THE NORMALIZED EIGENVALUES OF S  THENTHE EIGENDECOMPOSITION OF THIS DATA IS SBF1  BEGINBMATRIX 9064  4225 ENDBMATRIXT QQUAD SBF2 BEGINBMATRIX 4225  9064 ENDBMATRIXT LAMBDA1  291343 QQUAD LAMBDA2  14355FIGURE REFFIGSCATTER1 ALSO SHOWS A PLOT OF THESE EIGENVECTORS THEPRINCIPAL COMPONENT DIRECTIONS OF THE DATA SCALED BY THE SQUARE ROOTOF THE CORRESPONDING EIGENVALUETHE SCALAR VARIABLE Y1  9063 X1  4225 X2ACCOUNTS FOR 1002913432913431435595 OF THE TOTALVARIANCE OF THE RANDOM VECTOR XBF  X1X2 AND HENCE IS A GOODAPPROXIMATION TO XBF FOR MANY STATISTICAL PURPOSESENDEXAMPLE DATA FROM SCATTERMBEGINFIGUREHTBP  CENTERINGMBOXEPSFIGFILEPICTUREDIRSCATTER1EPS  CAPTIONSCATTER DATA FOR PRINCIPAL COMPONENT ANALYSIS  LABELFIGSCATTER1ENDFIGUREBEGINEXERCISESITEM LABELEXLOWRANK1 SHOW USING REFEQKLOWRANK  THAT  E2K CAN BE WRITTEN AS IN REFEQLR2ITEM LABELEXPC1 LET XBF BE A PELEMENT ZEROMEAN RANDOM  VECTOR WITH   COVARIANCE R AND LET YBF BE A QELEMENT RANDOM VECTOR YBF  BT XBFWHERE B IN MPQ AND Q  P  LET RY  BT R B BE THECOVARIANCE MATRIX OF YBF  SHOW THAT TRACERY IS MAXIMIZED BYTAKING B  XBF1 XBF2 LDOTS XBFQ  XQWHERE LABELEXPC2 XBFI IS THE ITH NORMALIZED EIGENVECTOR OF RITEM LET YBF  BT XBF AS IN THE PREVIOUS EXERCISE  SHOW THAT  DETRY IS MAXIMIZED WHEN B  XQ AS BEFOREITEM FOR A DATA COMPRESSION APPLICATION IT IS DESIRED TO ROTATE A  SET OF NDIMENSIONAL ZEROMEAN DATA Y  YBF1YBF2LDOTS  YBFN SO THAT IT MATCHES WITH ANOTHER SET OF NDIMENSIONAL  DATA Z   ZBF1ZBF2LDOTSZBFM  DESCRIBE HOW TO  PERFORM THE ROTATION IF THE MATCH IS DESIRED IN THE DOMINANT Q  COMPONENTS OF THE DATAITEM BF COMPUTER EXERCISE THIS EXERCISE WILL DEMONSTRATE SOME  CONCEPTS OF PRINCIPAL COMPONENTS  BEGINENUMERATE  ITEM CONSTRUCT A SYMMETRIC MATRIX R IN M2 THAT HAS UNNORMALIZED    EIGENVECTORS  XBF1  BEGINBMATRIX 1  5 ENDBMATRIXQQUAD XBF2 BEGINBMATRIX5  1 ENDBMATRIXWITH CORRESPONDING EIGENVALUES LAMBDA1  10 LAMBDA2  2ITEM GENERATE AND PLOT 200 POINTS OF ZEROMEAN GAUSSIAN DATA THAT HAS  THE COVARIANCE RITEM FORM AN ESTIMATE OF THE COVARIANCE OF THE GENERATED DATA AND  COMPUTE THE PRINCIPAL COMPONENTS OF THE DATAITEM PLOT THE PRINCIPAL COMPONENT INFORMATION OVER THE DATA AND  VERIFY THAT THE PRINCIPAL COMPONENT VECTORS LIE AS ANTICIPATED  ENDENUMERATEITEM CITESCARFTUFTS1987 INDEXLOWRANK APPROXIMATION LOWRANK  APPROXIMATION CAN SOMETIMES BE USED TO OBTAIN A BETTER  REPRESENTATION OF A NOISY SIGNAL  SUPPOSE THAT AN MDIMENSIONAL  ZEROMEAN SIGNAL XBF WITH RX  EXBF XBFH IS TRANSMITTED  THROUGH A NOISY CHANNEL SO THAT THE RECEIVED SIGNAL IS RBF  XBF  NUBFAS SHOWN IN FIGURE REFFIGLOWRANK1A  LET ENUBFNUBFH  RNU   SIGMANU2I  THE MS ERROR IN THIS SIGNAL IS  E2TEXTDIRECT  ERBF  XBFHRBFXBF  M SIGMANU2ALTERNATIVELY WE CAN SEND THE SIGNAL XBF1  UH XBF WHERE U ISTHE MATRIX OF EIGENVECTORS OF RX AS IN REFEQKREDUCE1  THERECEIVED SIGNAL IN THIS CASE IS RBFR  XBF1  NUBFFROM WHICH AN APPROXIMATION TO XBFR IS OBTAINED BY XBFHATR  UIRRBFRSHOW THAT BEGINALIGNEDE2TEXTINDIRECT  EXBFXBFHATRHXBFXBFHATR  SUMIR1M LAMBDAI  R SIGMANU2ENDALIGNEDHENCE CONCLUDE THAT FOR SOME VALUES OF R THE REDUCED RANK METHODMAY HAVE LOWER MS ERROR THAN THE DIRECT ERRORBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    SUBFIGUREDIRECTINPUTPICTUREDIRLR1    SUBFIGUREDIRECTINPUTPICTUREDIRLR2    CAPTIONDIRECT AND INDIRECT TRANSMISSION OF A VECTOR THROUGH A    NOISY CHANNEL    LABELFIGLOWRANK1  ENDCENTERENDFIGUREENDEXERCISESSECTIONEIGENFILTERSLABELSECEIGFILTINDEXEIGENFILTERINDEXFILTER DESIGNEIGENFILTERSEIGENFILTERS ARE FIR FILTERS WHOSE COEFFICIENTS ARE DETERMINED BYMINIMIZING OR MAXIMIZING A QUADRATIC FORM SUBJECT TO SOME CONSTRAINTIN THIS SECTION TWO DIFFERENT TYPES OF EIGENFILTER DESIGNS AREPRESENTED  THE FIRST IS FOR A RANDOM SIGNAL IN RANDOM NOISE AND THEFILTER IS DESIGNED IN SUCH A WAY AS TO MAXIMIZE THE SIGNALTONOISERATIO AT THE OUTPUT OF THE FILTER  THE SECOND IS FOR DESIGN OF FIRFILTERS WITH A SPECIFIED FREQUENCY RESPONSE  AS SUCH THEY PROVIDE ANALTERNATIVE TO THE STANDARD PARKSMCCLELLAN FILTER DESIGN APPROACHSUBSECTIONEIGENFILTERS FOR RANDOM SIGNALSINDEXEIGENFILTERRANDOM SIGNALSIN THE SYSTEM SHOWN IN FIGURE REFFIGEIGFILTR LET FT DENOTETHE INPUT SIGNAL WHICH IS ASSUMED TO BE A STATIONARY ZEROMEANRANDOM PROCESS  THE INPUT IS CORRUPTED BY ADDITIVE WHITE NOISE NUTWITH VARIANCE SIGMA2  THE SIGNAL THEN PASSES THROUGH AN FIRFILTER OF LENGTH M REPRESENTED BY THE VECTOR HBF TO PRODUCE THEOUTPUT YT  IT IS DESIRED TO DESIGN THE FILTER HBF TO MAXIMIZETHE SIGNALTONOISE RATIO BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIREIGFILR1    CAPTIONNOISY SIGNAL TO BE FILTERED USING AN EIGENFILTER HBF    LABELFIGEIGFILTR  ENDCENTERENDFIGURELET FBFT  BEGINBMATRIX FT  TT1  VDOTS  FTM1ENDBMATRIXTHEN THE FILTER OUTPUT CAN BE WRITTEN AS YT  HBFH FBFTTHE POWER OUTPUT DUE TO THE INPUT SIGNAL IS P0  E YK2  E HBFH FBFTFBFHT HBF  HBFH R HBFWHERE R  E FBFT FBFHT IS THE AUTOCORRELATION MATRIX OF FLET NUBFT  BEGINBMATRIX NUT  NUT1  VDOTS  NUTM1ENDBMATRIXTHEN THE OUTPUT OF THE FILTER DUE ONLY TO THE NOISE IS HBFH NUBFT AND THE AVERAGE NOISE POWER OUTPUT IS N0  EHBFH NUBFT NUBFTH HBF  SIGMA2 HBFH HBFTHE SIGNALTONOISE RATIO SNR IS SNR  FRACP0N0  FRACHBFH R HBFSIGMA2 HBFH HBFTHE PROBLEM NOW IS TO CHOOSE THE COEFFICIENTS OF THE FILTER HBF INSUCH A WAY AS TO MAXIMIZE THE SNR  HOWEVER THIS IS SIMPLY A RAYLEIGHQUOTIENT INDEXRAYLEIGH QUOTIENT WHICH IS MAXIMIZED BY TAKING HBF  XBF1WHERE XBF1 IS THE EIGENVECTOR OF R CORRESPONDING TO THE LARGESTEIGENVALUE LAMBDA1  THE MAXIMUM SNR IS SNRMAX  FRACLAMBDA1SIGMA2IT IS INTERESTING TO CONTRAST THIS EIGENFILTER WHICH MAXIMIZES THESNR FOR A RANDOM INPUT SIGNAL WITH THE MATCHED FILTER DISCUSSED INSECTION REFSECDIGCOM  THE OPERATION OF THE MATCHED FILTER AND THEEIGENFILTER ARE IDENTICAL THEY BOTH PERFORM AN INNER PRODUCTCOMPUTATION  HOWEVER IN THE CASE OF THE MATCHED FILTER THE FILTERCOEFFICIENTS ARE EXACTLY THE CONJUGATE OF THE KNOWN SIGNAL  IN THERANDOM SIGNAL CASE THE SIGNAL CAN ONLY BE KNOWN BY ITS STATISTICSTHE OPTIMAL FILTER IN THIS CASE SELECTS THAT COMPONENT OF THEAUTOCORRELATION THAT IS MOST SIGNIFICANTFOR THIS EIGENFILTER THE IMPORTANT INFORMATION NEEDED IS THEEIGENVECTOR CORRESPONDING TO THE LARGEST EIGENVALUE OF A HERMITIANMATRIX  INFORMATION ABOUT THE PERFORMANCE OF THE FILTER SUCH AS THESNR MAY BE OBTAINED FROM THE LARGEST EIGENVALUE  WHILE COMPUTING ACOMPLETE EIGENDECOMPOSITION OF A GENERAL MATRIX MAY BE DIFFICULT ITIS NOT TOO DIFFICULT TO COMPUTE THE LARGEST EIGENVALUE AND ITSASSOCIATED EIGENVECTOR  A MEANS OF DOING THIS IS PRESENTED IN SECTIONREFSECPOWERMETHODBEGINEXERCISES ITEM LABELEXLCMV1 SHOW THAT REFEQLCMV1 IS CORRECTITEM FOR AN INPUT SIGNAL WITH CORRELATION MATRIX R  BEGINBMATRIX 2  3  2 3  4  1 2  1  6ENDBMATRIXBEGINENUMERATEITEM DESIGN AN EIGENFILTER WITH 3 TAPS THAT MAXIMIZES THE SNR AT THE  OUTPUT OF THE FILTERITEM PLOT THE FREQUENCY RESPONSE OF THIS FILTERITEM DESIGN AN EIGENFILTER THAT EM MINIMIZES THE OUTPUT ENERGY  SUBJECT TO THE CONSTRAINT THAT ENDENUMERATEENDEXERCISESSUBSECTIONEIGENFILTER FOR DESIGNED SPECTRAL RESPONSELABELSECEIGFSRINDEXEIGENFILTERDESIRED SPECTRAL RESPONSE A VARIETY OF FILTERDESIGN TECHNIQUES EXIST THE MOST POPULAR OF WHICH IS PROBABLY THEPARKSMCCLELLAN ALGORITHM IN WHICH THE MAXIMUM ERROR BETWEEN ADESIRED SIGNAL SPECTRUM HDEJOMEGA AND THE FILTER SPECTRUMHEJOMEGA IS MINIMIZED  IN THIS SECTION WE PRESENT ANALTERNATIVE FILTER DESIGN TECHNIQUE WHICH MINIMIZES A QUADRATICFUNCTION RELATED TO THE ERROR HDEJOMEGA  HEJOMEGA2WHILE IT DOES NOT GUARANTEE TO MINIMIZE THE MAXIMUM ERROR THE METHODDOES PRODUCE GOOD DESIGNS AND IS OF REASONABLE COMPUTATIONALCOMPLEXITY  FURTHERMORE IT IS STRAIGHTFORWARD TO IMPOSE SOMECONSTRAINTS ON THE FILTER DESIGN  THE DESIGN IS EXEMPLIFIED FORLINEAR PHASE LOWPASS FILTERS ALTHOUGH IT CAN BE EXTENDED BEYOND THESERESTRICTIONSIT IS DESIRED TO DESIGN A LOWPASS LINEAR PHASE FIR FILTER WITHMN1 COEFFICIENTS THAT APPROXIMATES A DESIRED SPECTRAL RESPONSEHDEJOMEGA WHERE N IS EVEN  THE DESIRED SPECTRAL RESPONSEHAS A LOWPASS CHARACTERISTIC  HDEJOMEGA  BEGINCASES  1  0 LEQ OMEGA LEQ OMEGAP   0  OMEGAS LEQ OMEGA LEQ PIENDCASESWITH OMEGAP  OMEGAS  SEE FIGURE REFFIGEIGFILTSPEC  THE FILTER ISSCALED SO THAT THE MAGNITUDE RESPONSE AT OMEGA0 IS 1BEGINFIGUREHTBPBEGINCENTERINPUTPICTUREDIRHD1  ENDCENTERCAPTIONMAGNITUDE RESPONSE SPECIFICATIONS FOR A LOWPASS FILTER  LABELFIGEIGFILTSPECENDFIGURETHE TRANSFER FUNCTION OF THE ACTUAL IN CONTRAST TO THE DESIREDFILTER IS HZ  SUMN0N HN ZNWHERE THE CONSTRAINT HN  HNN IS IMPOSED TO ACHIEVE LINEARPHASE  LET MN2  THE FREQUENCYRESPONSE CAN BE WRITTEN ASBEGINEQUATIONHEJOMEGA  EJNOMEGA2 HROMEGALABELEQEIGF1ENDEQUATIONWHEREBEGINEQUATIONHROMEGA  SUMN0M BN COSOMEGA N  BBFT CBFOMEGALABELEQEIGF2ENDEQUATIONANDBEGINEQUATION BBF  BEGINBMATRIX B0  B1  VDOTS  BM ENDBMATRIXBEGINBMATRIX HM  2HM1  VDOTS  2H0 ENDBMATRIX QQUADCBFOMEGA  BEGINBMATRIX 1  COS OMEGA  VDOTS  COS MOMEGAENDBMATRIX LABELEQEIGFDENDEQUATION SEE EXERCISE REFEXEIGFILT  THE SQUARED MAGNITUDE RESPONSE OFTHE FILTER IS HEJOMEGA2  HR2OMEGA  BBFT CBFOMEGACBFTOMEGA BBFSUBSUBSECTIONSTOPBAND ENERGYTHE ENERGY THAT PASSES IN THE STOPBAND WHICH WE WANT TO MINIMIZE IS ES  FRAC1PI INTOMEGASPIHEJOMEGA  HDEJOMEGA2DOMEGA  FRAC1PI BBFTINTOMEGASPI CBFOMEGA CBFTOMEGADOMEGA BBFLET P  FRAC1PIINTOMEGASPI CBFOMEGACBFTOMEGADOMEGAWHERE THE JKTH ELEMENT OF P IS PJK  FRAC1PIINTOMEGASPI COSJ OMEGACOSKOMEGA  DOMEGATHIS CAN BE READILY COMPUTED IN CLOSED FORMSUBSUBSECTIONPASSBAND DEVIATION THE DESIRED DC RESPONSE HDEJ0  1 CORRESPONDS TO THE CONDITION BBFT ONEBF  1WHERE ONEBF IS THE VECTOR OF ALL 1S  THROUGHOUT THE PASSBAND WEDESIRE THE MAGNITUDE RESPONSE TO BE 1 THE DEVIATION FROM THE DESIREDRESPONSE IS 1  BBFT CBFOMEGA  BBFT ONEBF  BBFTONEBF CBFOMEGATHE SQUARE OF THIS DEVIATION CAN BE INTEGRATED OVER THE FREQUENCIES INTHE PASSBAND AS A MEASURE OF THE QUALITY OF THE PASSBAND ERROR OF THEFILTER  LET EP  FRAC1PIINT0OMEGAP BBFTONEBF CBFOMEGAONEBFCBFOMEGAT BBF DOMEGA  BBFT Q BBFWITH Q  FRAC1PIINT0OMEGAP ONEBF  CBFOMEGAONEBF CBFOMEGAT DOMEGAPTHIS MATRIX CAN ALSO READILY COMPUTED IN CLOSED FORMSUBSUBSECTIONOVERALL RESPONSE FUNCTIONAL  LET JALPHA  ALPHA ES  1ALPHA EPBE AN OBJECTIVE FUNCTION WHICH TRADES OFF THE IMPORTANCE OF THESTOPBAND ERROR WITH THE PASSBAND ERROR USING THE PARAMETER ALPHA0  ALPHA  1  COMBINING THE ERRORS TOGETHER WE OBTAIN JALPHA  BBFT R BBFWHERE R  ALPHA P  1ALPHA Q OBVIOUSLY JALPHA CAN BEMINIMIZED BY SETTING BBF  ZEROBF  THIS CORRESPONDS TO ZERO INTHE PASSBAND AS WELL WHICH MATCHES THE DEVIATION REQUIREMENT BUTFAILS TO BE PHYSICALLY USEFUL  TO ELIMINATE THE TRIVIAL FILTER WEIMPOSE THE CONSTRAINT THAT BBF HAS UNIT NORM  THE FINAL FILTERCOEFFICIENTS CAN BE SCALED FROM BBF IF DESIRED  THE DESIGN PROBLEMTHUS REDUCES TO  BEGINALIGNEDTEXTMINIMIZE     BBFT R BBF TEXTSUBJECT TO   BBFT BBF  1ENDALIGNEDTHIS IS EQUIVALENT TO  MINBBF NEQ 0FRACBBFT R BBFBBFT BBFA RAYLEIGH QUOTIENT INDEXRAYLEIGH QUOTIENT WHICH IS SOLVED BYTAKING BBF TO BE THE EIGENVECTOR CORRESPONDING TO THE EM  SMALLEST EIGENVALUE OF THE SYMMETRIC MATRIX RFIGURE REFFIGEIGFILT1 ILLUSTRATES THE MAGNITUDE RESPONSE OF AFILTER DESIGNS WITH 45 COEFFICIENTS WHERE OMEGAP  02PIOMEGAS  025PI  THE SOLID LINE SHOWS THE EIGENFILTER RESPONSEWHEN ALPHA  02 PLACING MORE EMPHASIS IN THE PASSBAND  THEDOTTED LINE SHOWS THE EIGENFILTER RESPONSE WHEN ALPHA  08PLACING MORE EMPHASIS IN THE STOPBAND  FOR COMPARATIVE PURPOSES THERESPONSE OF A 45COEFFICIENT FILTER DESIGNED USING THE PARKSMCCLELLANALGORITHM IS ALSO SHOWN WITH A DASHDOT LINE  THE EIGENFILTER WITHALPHA08 HAS BETTER ATTENUATION PROPERTIES IN THE STOPBAND BUTDOES NOT HAVE THE EQUIRIPPLE PROPERTYBEGINFIGUREHTBP TESTEIGFILM    CENTERINGMBOXEPSFIGFILEPICTUREDIRTESTEIGF1EPSWIDTH09TEXTWIDTH    CAPTIONEIGENFILTER RESPONSE   LABELFIGEIGFILT1ENDFIGURESC MATLAB CODE THAT DESIGNS THE FREQUENCY RESPONSE IS SHOWN INALGORITHM REFALGEIGFILT  BEGINNEWPROGENVEIGENFILTER    DESIGNEIGFILMEIGFILTEIGENFILTER DESIGNEIGMAKEPQMENDNEWPROGENVSUBSECTIONCONSTRAINED EIGENFILTERSONE POTENTIAL ADVANTAGE OF THE EIGENFILTER METHOD OVER THEPARKSMCCLELLAN ALGORITHM IS THAT IT IS FAIRLY STRAIGHTFORWARD TOINCORPORATE A VARIETY OF CONSTRAINTS INTO THE DESIGN  REFERENCES ONSOME APPROACHES ARE GIVEN AT THE END OF THIS CHAPTER  WE CONSIDERHERE THE PROBLEM OF ADDING CONSTRAINTS TO FIX THE RESPONSE AT CERTAINFREQUENCIES  SUPPOSE THAT WE DESIRE TO SPECIFY THE MAGNITUDE RESPONSEAT R DIFFERENT FREQUENCIES SO THAT HROMEGAI  DIFOR I12LDOTSR  THIS CAN BE WRITTEN AS BBFT CBFOMEGAI  DIFOR I12LDOTSR  STACKING THE CONSTRAINTS WE HAVE CT BBF  DBFWHEREBEGINEQUATION C  BEGINBMATRIX CBFOMEGA1  CBFOMEGA2  CDOTS   CBFOMEGAR  ENDBMATRIX QQUAD QQUADDBF  BEGINBMATRIXD1  D2  VDOTS  DR ENDBMATRIXLABELEQCCONEIGENDEQUATIONTHE PROBLEM CAN NOW BE STATED AS BEGINALIGNEDTEXTMINIMIZE   BBFT R BBF TEXTSUBJECT TO   CT BBF  DBF ENDALIGNEDA COST FUNCTIONAL INCLUDING THE R CONSTRAINTS CAN BE WRITTEN ASBEGINEQUATION JBBF  BBFT R BBF  LAMBDABFT CTBBFLABELEQJEIGF2ENDEQUATIONWHERE LAMBDABF  LAMBDA1LAMBDA2LDOTSLAMBDART  THISLEADS TO THE SOLUTION SEE EXERCISE REFEXEIGF2BEGINEQUATION  LABELEQBEIGF2  BBF  R1 CLEFTCT R1CRIGHT1 DBFENDEQUATIONALGORITHM REFALGCONEIG SHOWS THE CODE THAT COMPUTES THECOEFFICIENTS  FIGURE REFFIGEIGFILT2A SHOWS THE MAGNITUDE RESPONSEOF A 45COEFFICIENT EIGENFILTER WITH OMEGAP  02PI OMEGAS 025 PI WITH CONSTRAINTS SO THAT HREJ0  1 QQUAD HREJ4PI  1 QQUADHREJ5PI  0 QQUAD HREJ8PI  0BECAUSE OF THE ZERO OUTPUTS THE RESPONSE IS NOT SHOWN ON A DB SCALEFIGURE REFFIGEIGFILT2B SHOWS THE DB SCALE  FOR COMPARISON THERESPONSE OF AN EIGENFILTER WITH THE SAME OMEGAS AND OMEGAP BUTWITHOUT THE EXTRA CONSTRAINTS IS SHOWN WITH A DOTTED LINEBEGINFIGUREHTBPCENTERING TESTEIGFIL2MMBOXSUBFIGURELINEAR SCALEEPSFIGFILEPICTUREDIRTESTEIGF2AEPS WIDTH045TEXTWIDTHQUAD      SUBFIGUREDB SCALEEPSFIGFILEPICTUREDIRTESTEIGF2BEPS WIDTH045TEXTWIDTH  CAPTIONRESPONSE OF A CONSTRAINED EIGENFILTER  LABELFIGEIGFILT2ENDFIGUREBEGINNEWPROGENVCONSTRAINED EIGENFILTER    DESIGNEIGFILCONMCONEIGCONSTRAINED EIGENFILTER DESIGNENDNEWPROGENVBEGINEXERCISESITEM SHOW THAT REFEQEIGF2 IS CORRECT USING THE DEFINITIONS OF  REFEQEIGFDITEM LABELEXEIGF2 SHOW THAT MINIMIZING REFEQJEIGF2 SUBJECT  TO CT BBF  DBF LEADS TO REFEQBEIGF2ITEM SHOW THAT REFEQEIGF1 AND REFEQEIGF2 ARE CORRECTITEM DEVISE A MEANS OF MATCHING A DESIRED RESPONSE BY MINIMIZING  BBFT R BBF SUBJECT TO THE FOLLOWING CONSTRAINTS BEGINALIGNEDBBFT BBF  1 CT BBF  ZEROBFENDALIGNEDWHERE C IS AS IN REFEQCCONEIG  THAT IS THE FILTERCOEFFICIENTS ARE CONSTRAINED IN ENERGY BUT THERE ARE FREQUENCIES ATWHICH THE RESPONSE SHOULD BE EXACTLY 0  HINT SEE SECTIONREFSECCONEIGITEM CONSIDER THE INTERPOLATION SCHEME SHOWN IN FIGURE  REFFIGMULTIRATEL  THE OUTPUT CAN BE WRITTEN AS YZ   XZLHZ    BEGINENUMERATE  ITEM SHOW THAT IF  HLT  BEGINCASESC  T  0  0  TEXTOTHERWISEENDCASESTHEN YLT  CXT  THIS MEANS THAT THE INPUT SAMPLES ARE CONVEYEDEM WITHOUT DISTORTION BUT POSSIBLY WITH A SCALE FACTOR TO THEOUTPUT  SUCH FILTERS ARE CALLED EM NYQUIST OR EM LTH BANDFILTERS CITEVAIDYANATHAN INDEXNYQUIST FILTER ITEM EXPLAIN HOW TO USE THE EIGENFILTER DESIGN TECHNIQUE TO DESIGN AN  OPTIMAL MEANSQUARE LTH BAND FILTERBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRMULTIRATE1    CAPTIONEXPANSION AND INTERPOLATION USING MULTIRATE PROCESSING    LABELFIGMULTIRATEL  ENDCENTERENDFIGUREENDENUMERATEITEM WRITE AND TEST A SC MATLAB PROGRAM WHICH ACCEPTS A PASSBAND  UPPER FREQUENCY OMEGAP AND A STOPBAND LOWER FREQUENCY OMEGAS  AND COMPUTES N FILTER COEFFICIENTS USING THE EIGENFILTER  APPROACH  ITEM IT IS DESIRED TO DEVELOP A LOWPASS FILTER HBF SUCH THAT THE  MAGNITUDE RESPONSE OF THE FILTER AT A PARTICULAR FREQUENCY OMEGA0  IS PRECISELY 0  USING THE NOTATION OF REFEQEIGF2 HOMEGA0  BBFT CBFOMEGA0  0DEVELOP AN EIGENBASED SOLUTION TO THE PROBLEMITEM A FILTER IS TO BE DESIGNED SO THAT BBFT P BBFIS MINIMIZED WHERE P CONTAINS PASSBAND AND STOPBAND INFORMATION ASIN REFEQEIGF2 SUBJECT TO A PASSFREQUENCY CONSTRAINT BBFT CBFOMEGA0  BETAFOR SOME BETA  SHOW THAT THE OPTIMAL FILTER IS BBF  BETA FRACP1 CBFOMEGA0CBFOMEGA0T A  CBFOMEGA0ENDEXERCISESSECTIONSIGNAL SUBSPACE TECHNIQUESLABELSECMUSICIN SECTION REFSECMODAL1 WE EXAMINED METHODS OF DETERMININGWHICH SINUSOIDAL SIGNALS ARE PRESENT IN A SIGNAL BASED UPON FINDING ACHARACTERISTIC EQUATION THEN FINDING ITS ROOTS  AS POINTED OUT INTHAT SECTION THESE METHODS CAN PROVIDE GOOD SPECTRAL RESOLUTION BUTBREAK DOWN QUICKLY IN THE PRESENCE OF NOISE  IN THIS SECTION WECONTINUE IN THAT SPIRIT BUT ACCOUNT EXPLICITLY FOR THE POSSIBILITY OFNOISE BREAKING THE SIGNAL OUT IN TERMS OF A EM SIGNAL SUBSPACECOMPONENT AND A EM NOISE SUBSPACE COMPONENT INDEXSIGNAL  SUBSPACEINDEXNOISE SUBSPACESUBSECTIONTHE SIGNAL MODELSUPPOSE THAT A SIGNAL XT CONSISTS OF THE SUM OF P COMPLEXEXPONENTIALS IN NOISE XT  SUMI1P AI EJ2PI FI T  PHIIWHERE FI IN 05 IS THE FREQUENCY WE ASSUME HERE THAT ALLFREQUENCIES ARE DISTINCT AI IS THE AMPLITUDE AND PHII IS THEPHASE OF THE ITH SIGNAL  THE PHASES ARE ASSUMED TO BE STATIONARYSTATISTICALLY INDEPENDENT AND UNIFORMLY DISTRIBUTED OVER 02PITHE AUTOCORRELATION FUNCTION FOR XT IS SEE EXERCISEREFEXSINACBEGINEQUATIONRXXK  EXTXBARTK  SUMI1P PI EJ2PI FI KLABELEQSINACENDEQUATIONWHERE PI  AI2  LET XBFT  BEGINBMATRIX XT  XT1  VDOTS  XTM1ENDBMATRIXAND LET RXX BE THE MATSIZEMM AUTOCORRELATION MATRIX FORXT RXX  EXBFTXBFHT  BEGINBMATRIX RXX0   RRXX1    CDOTS  RXXM1 RXX1  RXX0  CDOTS  RXXM2 VDOTS RXXM1  RXXM2  CDOTS  RXX0 ENDBMATRIXTHE AUTOCORRELATION MATRIX CAN BE WRITTEN AS BEGINEQUATIONRXX  SUMK1P PK SBFK SBFKHLABELEQMUSIC0ENDEQUATIONWHERE SBFI  BEGINBMATRIX 1  EJ2PI FI  EJ2PI 2FI  VDOTS  E2JPI M1FI ENDBMATRIXEQUATION REFEQMUSIC0 CAN ALSO BE WRITTEN AS RXX  SP SHWHERE BEGINEQUATIONS  BEGINBMATRIXSBF1 SBF2  LDOTS  SBFPENDBMATRIXQQUADTEXTANDQQUAD P  DIAGP1P2LDOTSPPLABELEQMUSIC3ENDEQUATIONTHE MATRIX S IS A VANDERMONDE MATRIX   INDEXVANDERMONDE MATRIXTHE VECTOR SPACEBEGINEQUATION SIGSPACE  LSPANSBF1SBF2LDOTSSBFPLABELEQDEFSIGSPACEENDEQUATIONIS SAID TO BE THE EM SIGNAL SUBSPACE OF THE SIGNAL XT  THISNAME IS APPROPRIATE SINCE EVERY XBFT CAN BE EXPRESSED AS A LINEARCOMBINATION OF THE COLUMNS OF S HENCE XBFT IN SIGSPACEIT CAN BE SHOWN FOR MP THAT RXX HAS RANK P  LETLAMBDARXX DENOTE THE EIGENVALUES OF RXX ORDERED SO THATLAMBDA1 GEQ LAMBDA2 GEQ CDOTS GEQ LAMBDAM AND LETUBF1UBF2LDOTSUBFM BE THE CORRESPONDING EIGENVECTORS WHICHARE NORMALIZED SO THAT UBFITUBFJ  DELTAIJ  THENBEGINEQUATIONRXX UBFI  LAMBDAI UBFILABELEQMUSIC1ENDEQUATIONRECALL FROM  LEMMA LEMZEROEIG THAT IF RXX HAS RANK PTHEN LAMBDAP1  LAMBDAP2  CDOTS  LAMBDAM  0SO WE CAN WRITE RXX  SUMI1P LAMBDAI UBFI UBFIHTHE EIGENVECTORS UBF1UBF2LDOTSUBFP ARE CALLED THE EM  PRINCIPAL EIGENVECTORS OF RXXBEGINLEMMA  THE PRINCIPAL EIGENVECTORS OF RXX SPAN THE SIGNAL SUBSPACE SIGSPACE LSPANUBF1UBF2LDOTSUBFP LSPANSBF1SBF2LDOTSSBFP ENDLEMMABEGINPROOF  SUBSTITUTE REFEQMUSIC0 INTO REFEQMUSIC1  LEFTSUMI1P PI SBFI SBFIHRIGHT UBFJ  LAMBDAJ UBFJTHUS UBFJ  FRAC1LAMBDAJ SUMI1P PI SBFI SBFIHUBFJ  SUMI1P BETAIJ SBFIWHERE BETAIJ  FRAC1LAMBDAI PK SBFIH UBFJSINCE EVERY UBFJ CAN BE EXPRESSED AS A LINEAR COMBINATION OFSBFII12LDOTSP AND SINCE THEY ARE BOTH P DIMENSIONALVECTOR SPACES THE SPAN OF BOTH SETS ARE THE SAMEENDPROOFGIVEN A SEQUENCE OF OBSERVATIONS OF XT WE CAN DETERMINE ESTIMATERXX AND FIND ITS EIGENVECTORS UBFI  KNOWING THE FIRST PEIGENVECTORS WE CAN DETERMINE THE SPACE IN WHICH THE SIGNALS RESIDEEVEN THOUGH AT THIS POINT WE DONT KNOW WHAT THE SIGNAL FREQUENCIESARE SUBSECTIONTHE NOISE MODELASSUME THAT XT IS OBSERVED IN NOISE YT  XT  WTWHERE WT IS A STATIONARY ZERO MEAN WHITENOISE SIGNALINDEPENDENT OF XT WITH EWTWBART  SIGMAW2  THENBEGINALIGNRYYK  RXXK  SIGMA2W DELTAK NONUMBER  SUMI1P PI EJ2PI FI K  SIGMAW2 DELTAKLABELEQMUSIC6ENDALIGNLETYBFT  BEGINBMATRIX YT  YT1  VDOTS  YTM1ENDBMATRIX QQUADTEXTAND QQUADWBFT  BEGINBMATRIXWT  WT1  VDOTS  WTM1ENDBMATRIXTHEN RYY  EYBFTYBFHT CAN BE WRITTEN ASBEGINEQUATION RYY  RXX  SIGMA2W ILABELEQCORRENDEQUATIONTHE AUTOCORRELATION MATRIX RYY IS FULL RANK BECAUSESIGMA2W I IS FULL RANK  LET RYY UBFI  MUI UBFI BETHE EIGENEQUATION FOR RYY WITH THE EIGENVALUES SORTED AS MU1GEQ MU2 GEQ CDOTS GEQ MUM  THE FIRST P EIGENVALUES OF RYY ARE RELATED TO THE FIRST P EIGENVALUES OF RXX BY MUI  LAMBDAI  SIGMAW2AND THE CORRESPONDING EIGENVECTORS ARE EQUAL  SEE EXERCISEREFEXEIGSHIFTMAT  FURTHERMORE EIGENVALUESMUP1MUP2LDOTSMUM ARE ALL EQUAL TO SIGMAW2THUS WE CAN WRITE RYY  SUMI1P LAMBDAI  SIGMAW2 UBFI UBFIH SUMIP1M SIGMAW2 UBFI UBFIHTHE SPACE NOISESPACE  LSPANUBFP1UBFP2 LDOTS UBFMIS CALLED THE EM NOISE SUBSPACE  ANY VECTOR FROM THE SIGNALSUBSPACE IS ORTHOGONAL TO NOISESPACESUBSECTIONPISARENKO HARMONIC DECOMPOSITIONINDEXPISARENKO HARMONIC DECOMPOSITIONBASED ON THE OBSERVATION THAT THE SIGNAL SPACE IS ORTHOGONAL TO THENOISE SPACE THERE ARE VARIOUS MEANS THAT CAN BE EMPLOYED TO ESTIMATETHE THE SIGNAL COMPONENTS IN THE PRESENCE OF NOISE  IN THE PISARENKOHARMONIC DECOMPOSITION PHD THE ORTHOGONALITY IS EXPLOITED DIRECTLYSUPPOSE THAT THE NUMBER OF MODES P IS KNOWN  THEN SETTING MP1THE NOISE SUBSPACE IS SPANNED BY THE SINGLE VECTOR UBFM WHICHMUST BE ORTHOGONAL TO ALL OF THE SIGNAL SPACE VECTORSBEGINEQUATION SBFIH UBFM  0QQUAD I12LDOTSPLABELEQPIS1ENDEQUATIONLETTING UBFM  UM0UM2LDOTSUMM1TREFEQPIS1 CAN BE WRITTEN AS SUMK0M1 UMKEJ2PI FI K  0THIS IS A POLYNOMIAL IN EJ2PI FI  THE M1P ROOTS OF THISPOLYNOMIAL WHICH LIE ON THE UNIT CIRCLE CORRESPOND TO THEFREQUENCIES OF THE SINUSOIDAL SIGNAL  ONCE THE FREQUENCIES AREOBTAINED FROM THE ROOTS OF THE POLYNOMIAL THE SQUARED AMPLITUDES CANBE OBTAINED BY SETTING UP A SYSTEM OF EQUATIONS FROM REFEQMUSIC6FOR M12LDOTSPBEGINEQUATION BEGINBMATRIX EJ2PI F1  EJ2PI F2  CDOTS  EJ2PI FP EJ2PI 2F1  EJ2PI 2F2  CDOTS  EJ2PI 2FP VDOTS EJ2PI P F1  EJ2PI PF2  CDOTS  EJ2PI PFP ENDBMATRIXBEGINBMATRIXP1  P2  VDOTS  PP ENDBMATRIX  BEGINBMATRIXRYY1  RYY2  VDOTS    RYYP ENDBMATRIXLABELEQPHDAMPENDEQUATIONTHE NOISE STRENGTH IS OBTAINED FROM THE MTH EIGENVALUE OF RYYOF COURSE IN PRACTICE THE CORRELATION MATRIX RYY MUST BEESTIMATED BASED ON RECEIVED SIGNALSBEGINEXAMPLE LABELEXMPISA SOURCE XT IS KNOWN TO PRODUCE P3 SINUSOIDS  THE CORRELATIONMATRIX RYY IS ESTIMATED TO BE RYY  BEGINBMATRIXHFILL 64000  HFILL   27361  46165J  HFILL15000  34410J HFILL17361  10898J HFILL27361  46165J   HFILL 64000 HFILL  27361  46165J HFILL 15000  34410J HFILL 15000  34410J  HFILL 27361  46165J   HFILL64000  HFILL  27361  46165J HFILL 17361  10898J HFILL 15000  34410J  HFILL27361 46165J  HFILL 64000 00000J ENDBMATRIXALGORITHM REFALGPISARENKO CAN BE USED TO DETERMINE THE FREQUENCIESOF THE SOURCETHE RESULT OF THIS COMPUTATIONIS SIGMA2  04 AND F  02 03 04T  THE AMPLITUDES AREPBF  123TENDEXAMPLEBEGINNEWPROGENVPISARENKO HARMONIC DECOMPOSITION    PISARENKOMPISARENKOPISARENKO HARMONIC    DECOMPOSITIONENDNEWPROGENVSUBSECTIONMUSICINDEXMUSICMUSIC STANDS FOR MULTIPLE SIGNAL CLASSIFICATION  LIKE THE PHD ITRELIES ON THE FACT THAT THE SIGNAL SUBSPACE IS ORTHOGONAL TO THE NOISESUBSPACE  LETBEGINEQUATION SBFTF  1  EJ2PI F  EJ2PI2F  CDOTS  EJ2PI  M1FLABELEQMUSICSENDEQUATIONWHEN F  FI ONE OF THE INPUT FREQUENCIES THEN FOR ANY VECTORXBF IN THE NOISE SUBSPACE SBFHF XBF  0SINCE THEY ARE ORTHOGONAL  LET MF  SUMKP1M SBFHF UBFK2THEN THEORETICALLY WHEN FFI THEN MF  0 AND 1MF ISINFINITE  THUS A PLOT OF 1MF SHOULD HAVE A TALL PEAK AT FFIFOR EACH OF THE INPUT FREQUENCIES  THE FUNCTIONBEGINEQUATIONPF   FRAC1SUMKP1M SBFHF UBFK2LABELEQMUSIC7ENDEQUATIONIS SOMETIMES REFERRED TO AS THE MUSIC SPECTRUM OF F  BY LOCATINGTHE PEAKS THE FREQUENCIES CAN BE IDENTIFIED  KNOWING THE FREQUENCIESTHE SIGNAL STRENGTHS CAN BE COMPUTED USED REFEQPHDAMP AS FORTHE PISARENKO METHOD  THE MUSIC SPECTRUM CAN BE COMPUTED USINGALGORITHM REFALGMUSICBEGINEXAMPLE LABELEXMMUSIC1  USING THE DATA FROM BEFORE COMPUTE THE SPECTRUM USING THE MUSIC  METHOD    FIRST WE USE THE FOLLOWING CODE TO COMPUTE THE VALUE AT A POINT OF  THE MUSIC SPECTRUM GIVEN THE EIGENVECTORS OF THE AUTOCORRELATION MATRIXBEGINNEWPROGENVCOMPUTE THE MUSIC SPECTRUM    MUSICFUNMMUSICCOMPUTE THE MUSIC SPECTRUMENDNEWPROGENVTHEN THE MUSIC SPECTRUM CAN BE PLOTTED WITH THE FOLLOWING SC MATLAB CODEASSUMING THAT RYY IS ALREADY ENTERED INTO SC MATLAB 5EM SEE TESTMUSICMHRULEBEGINVERBATIMVU  EIGRYYF00015PF  MUSICFUNF3VPLOTFPFENDVERBATIMHRULE VSKIP 5EMNOINDENT THE PLOT OF THE MUSIC SPECTRUM IS SHOWN IN FIGUREREFFIGMUSIC1  THE PEAKS ARE CLEARLY AT 02 03 AND 04COMPUTATION OF THE SIGNAL STRENGTHS IS AS IN EXAMPLE REFEXMPISBEGINFIGURETBPCENTERINGMBOXPSFIGFILEPICTUREDIRMUSIC1EPS  CAPTIONTHE MUSIC SPECTRUM FOR EXAMPLE REFEXMMUSIC1  LABELFIGMUSIC1ENDFIGUREENDEXAMPLE TESTMUSICMSECTIONGENERALIZED EIGENVALUESLABELSECGENEIGINDEXGENERALIZED EIGENVALUESIN ADDITION TO THE MANY APPLICATIONS OF EIGENVALUES TO SIGNALPROCESSING THERE HAS ARISEN RECENTLY AN INTEREST IN GENERALIZEDEIGENVALUE PROBLEMS OF THE FORM A UBF  LAMBDA BUBFWHERE A AND B ARE MATSIZEMM MATRICES  THE SET OF MATRICESA  LAMBDA B IS SAID TO FORM A EM MATRIX PENCIL  INDEXMATRIX  PENCIL THE EIGENVALUES OF THE MATRIX PENCIL DENOTEDLAMBDAAB ARE THOSE VALUES OF LAMBDA FOR WHICH DETALAMBDA B  0FOR AN EIGENVALUE OF THE PENCIL LAMBDA IN LAMBDAAB A VECTORUBF NEQ 0 SUCH THAT AUBF  LAMBDA BUBFIS SAID TO BE AN EIGENVECTOR OF ALAMBDA BNOTE THAT IF B IS NONSINGULAR THEN THERE ARE N EIGENVALUES ANDLAMBDAAB  LAMBDAB1A  THIS PROVIDES ONE MEANS OF FINDINGTHE EIGENVALUES OF THE MATRIX PENCIL  HOWEVER IT IT NOT PARTICULARLYWELLCONDITIONED  THE BOOK CITEGVL CONTAINS AN EXTENSIVEDISCUSSION OF NUMERICALLY STABLE MEANS OF COMPUTING GENERALIZEDEIGENVALUES   SC MATLAB CAN COMPUTE THE GENERALIZED EIGENVALUESUSING THE TT EIG COMMAND WITH TWO ARGUMENTS AS TT EIGABSUBSECTIONAN APPLICATION ESPRITLABELSECESPRITONE APPLICATION OF GENERALIZED EIGENVALUE DECOMPOSITIONS IS TOSINUSOIDAL ESTIMATION USING ESPRIT INDEXESPRIT ESTIMATION OFSIGNAL PARAMETERS VIA ROTATIONAL INVARIANCE TECHNIQUES  LIKE MUSICTHE METHOD ASSUMES THAT THERE ARE P SINUSOIDS IN WHITE NOISE ANDDEALS WITH AN EIGENDECOMPOSITION  THE SAME NOTATION AS IN SECTIONREFSECMUSIC IS EMPLOYED XBFT IS A VECTOR OF SIGNAL SAMPLESWBFT IS A VECTOR OF NOISE SAMPLES AND YBFT  XBFT WBFT  WE ALSO INTRODUCE A NEW VECTOR OF DELAYED SAMPLES ZBFT YBFT1 OR ZBFT  BEGINBMATRIX YT1  YT2  VDOTS  YTMENDBMATRIXAS BEFORE WE CAN WRITE RYY  EYBFTYBFHT  SPSH  SIGMA2W IALSO SEE EXERCISE REFEXESPRITBEGINEQUATION RYZ  EYBFTZBFHT  SPPHIH SH  RWLABELEQRYZENDEQUATIONWHERE RW  EWBFT WBFHT1  SIGMAW2BEGINBMATRIX0  0   0  CDOTS  0  01  0  0  CDOTS  0  0 0  1  0  CDOTS  0  0  VDOTS 0  0  0  CDOTS  1  0ENDBMATRIXAND PHI REPRESENTS THE PHASE SHIFT BETWEEN SUCCESSIVE SAMPLES PHI  DIAGEJ2PI F1 EJ2PI F2LDOTS EJ2PI FPIF M  P THE MATRIX RXX  RYY  SIGMA2W I  SPSHHAS RANK P  LET CYZ  RYZ  RW  SPPHIH SHNOW CONSIDER THE GENERALIZED EIGENVALUE PROBLEMBEGINEQUATION RXX UBF  LAMBDA CYZ UBF LABELEQESPRIT1ENDEQUATIONTHAT IS RXX  LAMBDA CYZ UBF  0THIS CAN BE WRITTEN AS SPILAMBDA PHIH SH UBF  0SINCE PHI IS DIAGONAL IT IS CLEAR THAT LAMBDA  EJ2PI FIIS AN EIGENVALUE OF REFEQESPRIT1  FROM THE P GENERALIZEDEIGENVALUES THAT LIE ON THE CIRCLE CAN BE OBTAINED THE FREQUENCIESFII12LDOTSP  BEGINEXAMPLE LABELEXMESPRIT  FOR THE DATA CORRELATION MATRIX OF  EXAMPLE REFEXMPIS THE CROSS CORRELATION MATRIX RYZ IS  DETERMINED TO BE RYZ  BEGINBMATRIXHFILL 27361  46165I   HFILL60000    HFILL 27361  46165IHFILL 15000  34410I  HFILL 11000  34410I  HFILL 27361  46165I   HFILL 60000HFILL 27361 46165I HFILL 17361  10898I  HFILL 11000  34410I HFILL 27361 46165I   HFILL 60000 HFILL 15000  08123I HFILL  17361  10898I  HFILL 11000 34410I HFILL 27361  46165I ENDBMATRIXTHE FOLLOWING ALGORITHM IMPLEMENTS THE ESPRIT ALGORITHMBEGINNEWPROGENVCOMPUTE THE FREQUENCY SPECTRUM OF A SIGNAL USING ESPRIT    ESPRITMESPRITCOMPUTE THE FREQUENCY SPECTRUM OF    A SIGNAL USING ESPRITENDNEWPROGENVTHE RESULTS OF THE COMPUTATION ARE F  020304 AS BEFOREENDEXAMPLEBEGINEXERCISESITEM LABELEXSINAC  SHOW THAT REFEQSINAC IS TRUE SHOW THAT  REFEQMUSIC0 IS TRUEITEM LABELEXESPRIT SHOW  THAT REFEQRYZ IS TRUEITEM SHOW THAT EVERY XBFT IN SIGSPACE WHERE SIGSPACE IS  DEFINED IN REFEQDEFSIGSPACEITEM SHOW FOR MP THAT RXX DEFINED IN REFEQMUSIC0 HAS  RANK PITEM SHOW THAT THE SIGNAL STRENGTH OF THE ITH SINUSOIDAL COMPONENT  CAN BE DETERMINED FROM  AI  FRACUBFIHRYY  SIGMAW2 IUBFIUBFIH SBFI2WHERE UBFI IS THE GENERALIZED EIGENVECTOR CORRESPONDING TOLAMBDAI AND SBFI IS DETERMINED FROM REFEQMUSICSITEM SHOW THAT THE CROSS COVARIANCE MATRIX RYZ DEFINED IN  REFEQRYZ CAN BE WRITTEN AS RYZ  BEGINBMATRIX RYY1  RYY2   CDOTS  RYYM RYY0  RYY1  CDOTS  RYYM1 VDOTS RBARYYM2  RBARYYM3  CDOTS  RYY1 ENDBMATRIXITEM NUMERICALLY COMPUTE USING EG SC MATLAB THE GENERALIZED  EIGENVALUES AND EIGENVECTORS FOR A UBF  LAMBDA B UBFWHERE A  BEGINBMATRIX 3  2 4  1  7  9  6  2  1ENDBMATRIXQQUADB  BEGINBMATRIX 5  2  1  5  3  0  2  1  7ENDBMATRIXITEM CITEKAILATH80 LET ABBF CBFT D BE A STATESPACE  DESCRIPTION OF A SYSTEM SEE CHAPTER REFCHAPINTRO  SHOW THAT  THE ZEROS OF THE TRANSFER FUNCTION HS  CBFTSIA1 BBF   D CAN BE COMPUTED AS THE EIGENVALUES OF ABBF D1 CBFTITEM CONTINUING THE PREVIOUS PROBLEM SHOW THAT THE ZEROS CAN BE  COMPUTED BY SOLVING THE GENERALIZED EIGENVALUE PROBLEM LAMBDA E  F PBF  0WHERE E  BEGINBMATRIXI  0  0  0 ENDBMATRIX QQUADF  BEGINBMATRIX A BBF  CBFT  D ENDBMATRIXITEM LET ABBFCBFD REPRESENT A SYSTEM IN STATEVARIABLE  FORM HAVING TRANSFER FUNCTION HS  CBFTSIA1BBF  D  SHOW THAT THE ZEROS CAN BE COMPUTED BY SOLVING THE GENERALIZED  EIGENVALUE PROBLEM LAMBDA E  F PBF  0WHERE E  BEGINBMATRIX I  0  0  0  ENDBMATRIXQQUAD  F  BEGINBMATRIXA  BBF  CBFT  D ENDBMATRIXENDEXERCISESINPUTLINALGDIRCHARPOLYINPUTLINALGDIRMOVEEIGINPUTLINALGDIRCONCHANNELSECTIONCOMPUTATION OF EIGENVALUES AND EIGENVECTORSLABELSECCOMPEIGTHE AREA OF NUMERICAL ANALYSIS DEALING WITH THE COMPUTATION OFEIGENVALUES AND EIGENVECTORS IS BOTH BROAD AND DEEP AND WE CANPROVIDE ONLY AN INTRODUCTION HERE  BECAUSE OF ITS IMPORTANCE INEIGENFILTER AND PRINCIPAL COMPONENT ANALYSIS WE DISCUSS MEANS OFCOMPUTING THE LARGEST AND SMALLEST EIGENVALUES USING THE POWER METHODATTENTION IS THEN DIRECTED TO THE CASE OF SYMMETRIC MATRICESBECAUSE OF THEIR IMPORTANCE IN SIGNAL PROCESSING   SUBSECTIONCOMPUTING THE LARGEST AND SMALLEST EIGENVALUESLABELSECPOWERMETHODA SIMPLE ITERATIVE ALGORITHM KNOWN AS THE POWER METHOD INDEXPOWER  METHOD CAN BE USED TO FIND THE LARGEST EIGENVALUE AND ITSASSOCIATED EIGENVECTOR  ALGORITHMS REQUIRING ONLY THE LARGESTEIGENVALUE CAN BENEFIT BY AVOIDING THE OVERHEAD OF COMPUTINGEIGENVALUES WHICH ARE NOT NEEDED  THE POWER METHOD WORKS FOR BOTHSELFADJOINT AND NONSELFADJOINT MATRICESLET A BE A MATSIZEMM DIAGONALIZABLE MATRIX WITH POSSIBLYCOMPLEX EIGENVALUES ORDERED AS LAMBDA1 GEQ LAMBDA2 GEQCDOTS GEQ LAMBDAM WITH CORRESPONDING EIGENVECTORSXBF1XBF2LDOTSXBFM  LET XBF0 BE A NORMALIZEDVECTOR WHICH IS ASSUMED NOT TO BE ORTHOGONAL TO XBF1  THE VECTORXBF0 CAN BE WRITTEN IN TERMS OF THE EIGENVECTORS AS XBF0  A1 XBF1  A2 XBF2  CDOTS  AM XBFMFOR SOME SET OF COEFFICIENTS  AI WHERE A1 NEQ 0 WE DEFINETHE POWER METHOD RECURSION BYBEGINEQUATION  LABELEQPOWERMETHODXBFK1  AXBFK  ENDEQUATIONTHENBEGINALIGNEDXBF1  AXBF0  A1 LAMBDA1LEFTXBF1  FRACA2A1  FRACLAMBDA2LAMBDA1  CDOTS   FRACAMA1FRACLAMBDAMLAMBDA1 RIGHT XBF2  AXBF1  A1 LAMBDA12LEFTXBF1   FRACA2A1  LEFTFRACLAMBDA2LAMBDA1RIGHT2  CDOTS   FRACAMA1LEFTFRACLAMBDAMLAMBDA1RIGHT2 RIGHT VDOTSENDALIGNEDBECAUSE OF THE ORDERING OF THE EIGENVALUES AS K RIGHTARROW INFTY  XBFK RIGHTARROW A1 XBF1WHICH IS THE EIGENVECTOR OF A CORRESPONDING TO THE LARGESTEIGENVALUE  THE EIGENVALUE ITSELF IS FOUND BY A RAYLEIGH QUOTIENT XBFKH A XBFKXBFK RIGHTARROW LAMBDA1THE ALGORITHM IS ILLUSTRATED IN ALGORITHM REFALGMAXEIGBEGINNEWPROGENVCOMPUTE THE LARGEST EIGENVALUE USING THE POWER METHOD    MAXEIGMMAXEIGCOMPUTE THE LARGEST EIGENVALUE    USING THE POWER METHODENDNEWPROGENVAN APPROACH SUGGESTED IN CITEMORRISON1976 FOR FINDING THE SECONDLARGEST EIGENVALUE AND ITS EIGENVECTOR AFTER KNOWING THE LARGESTEIGENVALUE IS TO FORM THE MATRIX A1  A  LAMBDA1 XBF1 XBF1HWHERE XBF1 IS THE NORMALIZED EIGENVECTOR  ALGEBRAICLY THELARGEST ROOT OF DETA1  LAMBDA I  0IS THE SECOND LARGEST EIGENVALUE OF A  COMPUTATION OF THEEIGENVALUE CAN BE OBTAINED BY THE POWER METHOD APPLIED TO A1  THERESULT DEPENDS ON CORRECT COMPUTATION OF LAMBDA1 SO THERE ISPOTENTIAL FOR NUMERICAL DIFFICULTY  EXTENDING THIS TECHNIQUE THEITH PRINCIPAL COMPONENT CAN BE FOUND AFTER THE FIRST I1 AREDETERMINED BY FORMING AI1  A  SUMJ1I1 LAMBDAI XBFI XBFIHAND USING THE POWER METHOD ON AI1  IF MANY EIGENVALUES ANDEIGENVECTORS ARE NEEDED IT MAY BE MORE EFFICIENT COMPUTATIONALLY ANDNUMERICALLY TO COMPUTE A COMPLETE EIGENDECOMPOSITIONFINDING THE SMALLEST EIGENVALUE CAN BE ACCOMPLISHED IN AT LEAST TWOWAYS  IF LAMBDA1 IS AN EIGENVALUE OF A THEN 1LAMBDA1 IS ANEIGENVALUE OF A1 AND THE EIGENVECTORS IN EACH CASE ARE THESAME  THE LARGEST EIGENVALUE LAMBDA OF A1 IS THUS THERECIPROCAL OF THE SMALLEST EIGENVALUE OF A  THIS METHOD WOULDREQUIRE INVERTING AALTERNATIVELY WE CAN FORM B  LAMBDA1I  A WHICH HAS LARGESTEIGENVALUE LAMBDA1  LAMBDAM  THE POWER METHOD CAN BE APPLIEDTO B TO FIND LAMBDA1  LAMBDAM FROM WHICH LAMBDAM CAN BEOBTAINED  SEE ALGORITHM REFALGMINEIGBEGINNEWPROGENVCOMPUTE THE SMALLEST EIGENVALUE USING THE POWER METHOD    MINEIGMMINEIGCOMPUTE THE SMALLEST EIGENVALUE    USING THE POWER METHODENDNEWPROGENVSUBSECTIONCOMPUTING THE EIGENVALUES OF A SYMMETRIC MATRIXLABELSECEIGCOMP2FINDING THE FULL SET OF EIGENVALUES AND EIGENVECTORS OF A MATRIX HASBEEN A MATTER OF CONSIDERABLE STUDY  THOROUGH DISCUSSIONS AREPROVIDED IN CITEWILKINSONAEP AND CITEGVL WHILE SOME NUMERICALIMPLEMENTATIONS ARE DISCUSSED IN CITEPRESSETALA EM REAL SYMMETRIC MATRIX A IS ORTHOGONALLY SIMILAR TO ADIAGONAL MATRIX LAMBDA A  Q LAMBDA QTTHE EIGENVALUES OF A ARE THEN FOUND EXPLICITLY ON THE DIAGONAL OFLAMBDA AND THE EIGENVECTORS ARE FOUND IN Q  ONE STRATEGY TOFINDING THE EIGENVALUES AND EIGENVECTORS IS TO MOVE A TOWARD BEING ADIAGONAL MATRIX BY A SERIES OF ORTHOGONAL TRANSFORMATIONS SUCH ASHOUSEHOLDER TRANSFORMATIONS OR GIVENS ROTATIONS WHICH WERE DISCUSSEDIN CONJUNCTION WITH THE QR FACTORIZATION IN SECTION REFSECQRONE APPROACH TO THIS STRATEGY IS TO FIRST REDUCE A TO BEING A EM  TRIDIAGONAL  MATRIX BY A SERIES OF HOUSEHOLDER TRANSFORMATIONSTHEN APPLY A SERIES OF GIVENS ROTATIONS THAT EFFICIENTLY DIAGONALIZETHE TRIDIAGONAL MATRIX  THIS TECHNIQUE HAS BEEN SHOWN TO PROVIDE AGOOD MIX OF COMPUTATIONAL SPEED BY MEANS OF THE TRIDIAGONALIZATIONWITH NUMERICAL ACCURACY USING THE ROTATIONS  THROUGHOUT THEDISCUSSION SC MATLAB CODE IS PROVIDED TO MAKE THE PRESENTATIONCONCRETE  SC MATLAB OF COURSE PROVIDES EIGENVALUES ANDEIGENVECTORS VIA THE FUNCTION TT EIGSUBSUBSECTIONTRIDIAGONALIZATION OF AINDEXTRIDIAGONALIZATIONLET A BE AN MATSIZEMM SYMMETRIC MATRIX  LET  Q1  BEGINBMATRIX 1   H1 ENDBMATRIXBE AN ORTHOGONAL MATRIX WHERE H1 IS A HOUSEHOLDER TRANSFORMATIONTHE TRANSFORMATION IS CHOSEN SO THAT Q1 A HAS ZEROS DOWN THE FIRSTCOLUMN IN POSITIONS 3MC M  SINCE A IS SYMMETRICBEGINEQUATION Q1 A Q1T  BEGINBMATRIX A11  TIMES  0  CDOTS  0    TIMES       0            VDOTS     MULTICOLUMN4CRAISEBOX15EX0CM0CMB1   0 ENDBMATRIXLABELEQTRIDIAG1ENDEQUATIONWHERE TIMES INDICATES AN ELEMENT WHICH IS NOT ZERO AND B1 IS ANMATSIZEM1M1 MATRIX WE CONTINUE ITERATIVELY APPLYINGHOUSEHOLDER TRANSFORMATIONS TO SET THE SUBDIAGONALS AND SUPERDIAGONALSTO ZERO  THEN T   QM2CDOTS Q2Q1 A Q1T Q2T CDOTS QM2T  QTA QIS TRIDIAGONAL  ALGORITHM REFALGTRIDIAG ILLUSTRATEDTRIDIAGONALIZATION USING HOUSEHOLDER TRANSFORMATIONS  THECOMPUTATION COST CAN BE REDUCED BY EXPLOITING THE SYMMETRY OF A SEEEXERCISE REFEXTRIDIAG1 TO 4M33 FLOATING OPERATIONS IF THEMATRIX Q IS NOT RETURNED KEEPING TRACK OF Q REQUIRES ANOTHER4M33 FLOATING OPERATIONSBEGINNEWPROGENVTRIDIAGONALIZATION OF A REAL SYMMETRIC MATRIX    TRIDIAGMTRIDIAGTRIDIAGONALIZATION OF A REAL    SYMMETRIC MATRIXENDNEWPROGENVBEGINEXAMPLE  FOR THE MATRIX  A  BEGINBMATRIX315447  0516339  0348363  00191762 0516339  237035  0318412  0794899 0348363  0318412  229206  104116 00191762  0794899  104116  218312 ENDBMATRIX THE TRIDIAGONAL FORM T AND Q ARE T  BEGINBMATRIX315447  0623162  0  0 0623162  256432  125495  0 0  125495  230604  0404962 0  0  0404962  197516 ENDBMATRIXQ  BEGINBMATRIX1  0  0  0 0  0828579  000572494  0559842 0  0559025  00634203  0826722 0  00307724  099797  00557491 ENDBMATRIXENDEXAMPLESUBSECTIONTHE QR ITERATIONHAVING FOUND THE TRIDIAGONAL FORM WE REDUCE THE MATRIX FURTHER TOWARDA DIAGONAL FORM USING QR ITERATION  WE FORM THE QR FACTORIZATION OFT AS T  Q0 R0THEN WE OBSERVE THAT  Q0T T Q0  Q0TQ0 R0 Q0  R0 Q0LET T1  R0 Q0  WE THEN PROCEED ITERATIVELY ALTERNATING A QRFACTORIZATION STEP WITH A REVERSAL OF THE PRODUCT BEGINALIGNEDT0  T  Q0 R0 T1  R0 Q0      Q1 R1 VDOTS TK  QK RK TK1  RK QKENDALIGNEDTHE KEY  RESULT IS PROVIDED BY THE FOLLOWING THEOREMBEGINTHEOREM LABELTHMQRALG  IF THE EIGENVALUES OF A AND HENCE OF T ARE OF DIFFERENT  ABSOLUTE VALUE LAMBDAI THEN TK APPROACHES A DIAGONAL  MATRIX AS KRIGHTARROW INFTY  IN THIS MATRIX THE EIGENVALUES  ARE ORDERED DOWN THE DIAGONAL SO THAT  TK11  TK22  CDOTS  TKMMENDTHEOREMTHE PROOF OF THE THEOREM IS TOO LENGTH TO FIT WITHIN THE SCOPE OF THISBOOK SEE EG CITECHAPTER 6STOER1993 OR CITECHAPTER7GVL  SINCE TK APPROACHES A DIAGONAL MATRIX WECAN READ THE EIGENVALUES OF A DIRECTLY OFF THE DIAGONAL OFTK FOR K SUFFICIENTLY LARGE  SINCE T0 T ISTRIDIAGONAL T0 CAN BE CONVERTED TO UPPER TRIANGULAR USING ONLYM1 GIVENS ROTATIONS  THIS IS AN IMPORTANT REASON FORTRIDIAGONALIZING FIRST SINCE GIVEN PROPER ATTENTION THE NUMBER OFCOMPUTATIONS CAN BE GREATLY REDUCEDIN THE PROOF OF THEOREM REFTHMQRALG IT IS SHOWN THAT ASUPERDIAGONAL ELEMENT OF TK CONVERGES TO ZERO AS TKIJ APPROX LEFTFRACLAMBDAILAMBDAJRIGHTKSINCE LAMBDAI  LAMBDAJ THIS DOES CONVERGE  HOWEVER IFLAMBDAI IS NEAR TO LAMBDAJ CONVERGENCE IS SLOW  THECONVERGENCE CAN BE ACCELERATED BY MEANS OF SHIFTING WHICH RELIES ONTHE OBSERVATION THAT IF LAMBDA IS AN EIGENVALUE OF T THENLAMBDA TAU IS AN EIGENVALUE OF TTAU I  BASED ON THIS WEFACTORIZE TK  TAUK I  QK RKTHEN WRITE TK1  RK QK  TAUK ITHIS IS KNOWN AS AN EM EXPLICIT SHIFT QR ITERATION  WITH THE SHIFTTHE CONVERGENCE CAN BE SHOWN TO BE DETERMINED BY THE RATIO FRACLAMBDAI  TAUKLAMBDAJ  TAUKTHEN THE SHIFT TAUK IS SELECTED AT EACH K TO MAXIMIZE THE RATEOF CONVERGENCE  A GOOD CHOICE COULD BE TO SELECT TAUK CLOSE TOTHE SMALLEST EIGENVALUE LAMBDAM HOWEVER THIS IS NOT GENERALLYKNOWN IN ADVANCE  AN EFFECTIVE ALTERNATIVE STRATEGY IS THE COMPUTETHE EIGENVALUES OF THE MATSIZE22 SUBMATRIX IN THE LOWER RIGHTOF T AND USING THAT EIGENVALUE WHICH IS CLOSEST TO TKMMTHIS IS KNOWN AS THE WILKINSON SHIFT INDEXWILKINSON SHIFTWHILE THE EXPLICIT SHIFT USUALLY WORKS WELL SUBTRACTING A LARGETAUK FROM THE DIAGONAL ELEMENTS CAN LEAD TO A LOSS OF ACCURACY FORTHE SMALL EIGENVALUES  WHAT IS PREFERRED IS THE EM IMPLICIT QR  SHIFT INDEXIMPLICIT QR SHIFT ALGORITHM  BRIEFLY HOW THIS WORKS ISTHAT A GIVENS ROTATION MATRIX IS FOUND SO THAT BEGINBMATRIXC  S  S  CENDBMATRIXBEGINBMATRIXTK11 TAUK  B  ENDBMATRIX  BEGINBMATRIX TIMES  0 ENDBMATRIXTHAT IS THE ROTATION ZEROS OUT AN ELEMENT BELOW THE DIAGONAL OF THEEM SHIFTED MATRIX  HOWEVER THE SHIFT IS NEVER EXPLICITLYCOMPUTED ONLY THE APPROPRIATE GIVENS MATRIX  APPLICATION OF THEROTATION FOR THE SHIFT INTRODUCES NEW ELEMENTS IN THE OFF DIAGONALSFOR EXAMPLE THE  MATSIZE55 MATRIX T  BEGINBMATRIXTIMES  TIMES 00 0 TIMES  TIMES  TIMES  0 0 0 TIMES TIMES TIMES 000TIMES  TIMES  TIMES 000TIMES  TIMES  ENDBMATRIXWHERE TIMES INDICATES NONZERO ELEMENTSWHEN OPERATED ON BY THE GIVENS ROTATION G1 DESIGNED TO ZERO OUT THE12 ELEMENT OF THE EM SHIFTED MATRIX BECOMESBECOMES G1 T G1T  BEGINBMATRIX TIMES  TIMES    0 0TIMES TIMESTIMES  00  TIMES  TIMES  TIMES0 0  0TIMESTIMESTIMES00  0TIMESTIMES ENDBMATRIXWHERE  INDICATES NONZERO ELEMENTS WHICH ARE INTRODUCED  A SERIESOF GIVENS ROTATIONS THAT DO NOT OPERATE ON THE SHIFTED MATRIX IS THENAPPLIED TO CHASE THESE NONZERO ELEMENTS DOWN THE DIAGONAL BEGINALIGNEDBEGINBMATRIX TIMES  TIMES    0 0TIMES TIMESTIMES  00  TIMES  TIMES  TIMES0 0  0TIMESTIMESTIMES00  0TIMESTIMES ENDBMATRIX STACKRELG2LONGRIGHTARROWBEGINBMATRIX TIMES  TIMES  0  0 0TIMES TIMESTIMES  0 0 TIMES  TIMES  TIMES0 0  TIMESTIMESTIMES00  0TIMESTIMES ENDBMATRIX STACKRELG3LONGRIGHTARROW EXMATSPBEGINBMATRIX TIMES  TIMES  0  0 0TIMES TIMESTIMES  00 0 TIMES  TIMES  TIMES 0  0TIMESTIMESTIMES00  TIMESTIMES ENDBMATRIX STACKRELG4LONGRIGHTARROWBEGINBMATRIX TIMES  TIMES  0  0 0TIMES TIMESTIMES  00 0 TIMES  TIMES  TIMES0 0  0TIMESTIMESTIMES00  0 TIMESTIMES ENDBMATRIX ENDALIGNEDTHE STEPS OF INTRODUCING THE SHIFTED GIVENS ROTATION FOLLOWED BY THEGIVENS ROTATIONS WHICH RESTORE THE TRIDIAGONAL FORM ARE COLLECTIVELYCALLED AN EM IMPLICIT QR SHIFT  CODE WHICH IMPLEMENTS THISIMPLICIT QR SHIFT IS SHOWN IN ALGORITHM REFALGIMPLICITQRBEGINNEWPROGENVIMPLICIT QR SHIFT    EIGQRSHIFTSTEPMIMPLICITQRIMPLICIT QR SHIFTENDNEWPROGENVCOMBINING THE TRIDIAGONALIZATION AND THE IMPLICIT QR SHIFT IS SHOWN INALGORITHM REFALGNEWEIG  FOLLOWING THE INITIAL TRIDIAGONALIZATIONTHE MATRIX T IS DRIVEN TOWARD A DIAGONAL FORM WITH THE LOWER RIGHTCORNER PROBABLY CONVERGING FIRST  THE MATRIX T IS SPLIT INTOTHREE PIECES T  BEGINBMATRIX T1   T2    T3 ENDBMATRIXWHERE T3 IS DIAGONAL AS DETERMINED BY A COMPARISON WITH ATHRESHOLD EPSILON AND T1 IS ALSO  THE IMPLICIT QR SHIFT ISAPPLIED ONLY TO T2  THE ALGORITHM ITERATES UNTIL T IS FULLYDIAGONALIZEDBEGINNEWPROGENVCOMPLETE EIGENVALUEEIGENVECTOR FUNCTION    NEWEIGMNEWEIGCOMPLETE EIGENVALUEEIGENVECTOR FUNCTIONENDNEWPROGENVBEGINEXAMPLE  FOR THE MATRIX A  BEGINBMATRIX315447  0516339  0348363  00191762 0516339  237035  0318412  0794899 0348363  0318412  229206  104116 00191762  0794899  104116  218312 ENDBMATRIXTHE STATEMENT TT TX  NEWEIGA RETURNS THE EIGENVALUES IN TAND THE EIGENVECTORS IN X AS T  BEGINBMATRIX4  0  0  0 0  3  0  0 0  0  1  0 0  0  0  2 ENDBMATRIX QQUADQQUADX  BEGINBMATRIX05  0829341  0182574  0169882 05  00606835  0365148  0782933 05  0303418  0547723  0598279 05  046524  0730297  00147723 ENDBMATRIXENDEXAMPLEBEGINEXERCISESITEM THE COMPUTATIONAL ROUTINES DESCRIBED IN THIS SECTION APPLY TO  EM REAL  MATRICES  IN THIS PROBLEM WE EXAMINE HOW TO EXTEND REAL  COMPUTATIONAL ROUTINES TO HERMITIAN COMPLEX MATRICES  LET A BE  A HERMITIAN MATRIX AND LET A  AR  J AI WHERE AR IS THE  REAL PART AND AI IS THE IMAGINARY PART  LET XBF  XBFR  J  XBFI BE AN EIGENVECTOR OF A WITH ITS CORRESPONDING REAL AND  IMAGINARY PARTS  BEGINENUMERATE  ITEM SHOW THAT A XBF  LAMBDA XBF CAN BE REWRITTEN AS BEGINBMATRIXAR  AI  AI  AR ENDBMATRIXBEGINBMATRIXXBFR  XBFI ENDBMATRIX  LAMBDABEGINBMATRIX XBFR   XBFIENDBMATRIXITEM LET ABB  LEFTBEGINSMALLMATRIX AR  AI  AI       ARENDSMALLMATRIXRIGHT  SHOW THAT ABB IS SYMMETRICITEM SHOW THAT IF XBFRT XBFITT IS AN EIGENVECTOR OF  ABB CORRESPONDING TO LAMBDA THEN SO IS XBFIT  XBFITT  THE LATTER EIGENVECTOR CORRESPONDS TO XBF   JXBFR  JXBFIITEM CONCLUDE THAT EACH EIGENVALUE OF ABB HAS MULTIPLICITY 2 AND  THAT THE EIGENVALUES OF A CAN BE OBTAINED BY SELECTING ONE  EIGENVECTOR CORRESPONDING TO EACH PAIR OF REPEATED EIGENVALUES  ENDENUMERATEITEM CITEGVL LABELEXTRIDIAG1 IN THE HOUSEHOLDER  TRIDIAGONALIZATION ILLUSTRATED IN REFEQTRIDIAG1 THE MATRIX  B1 IS OPERATED ON BY THE HOUSEHOLDER MATRIX HV TO PRODUCE HV  B1 HV FOR A HOUSEHOLDER VECTOR VBF  SHOW THAT HV B1 HV  CAN BE COMPUTED BY HV B1 HV  B VBFWBFT  WBF VBFTWHERE  PBF  FRAC2VBFT VBF B1 VBF QQUADQQUAD WBF  PBF  FRACPBFT VBFVBFT VBF VBFITEM WILKINSON SHIFT IF T  LEFTBEGINSMALLMATRIX AN1  BN1       BN1AN ENDSMALLMATRIXRIGHT SHOW THAT THE EIGENVALUES      ARE OBTAINED BY MU  AN  D PM SQRTD2  BN12WHERE D  AN1  AN2ITEM SHOW THAT THE EIGENVALUES  OF A  PERMUTATION MATRIX ARE SUCH  THAT LAMBDAI1ENDEXERCISESSETEXSECTREFSECEIG1BEGINEXERCISESITEM EIGENFUNCTIONS AND EIGENVECTORS  BEGINENUMERATE  ITEM LABELEXEIGFUNC LET LC BE THE OPERATOR WHICH COMPUTES THE NEGATIVE OF THE    SECOND DERIVATIVE LC U  FRACD2DT2 U DEFINED FOR    FUNCTIONS ON 01  SHOW THAT UNT  SINNPI TIS AN EIGENFUNCTION OF LC WITH EIGENVALUE LAMBDAN  NPI2ITEM IN MANY NUMERICAL PROBLEMS A DIFFERENTIATION OPERATOR IS  DISCRETIZED  SHOW THAT WE CAN APPROXIMATE THE SECOND DERIVATIVE  OPERATOR BY FRACD2DT2 APPROX FRACUTH  2UT  UTHH2WHERE H IS SOME SMALL NUMBERITEM DISCRETIZE THE INTERVAL 01 INTO 0T1T2LDOTSTN  WHERE TI  IN  LET UBF   UT1LDOTSUTN1T AND SHOW THAT THE OPERATOR  LC U CAN BE APPROXIMATED BY THE OPERATOR FRAC1N2 LUBF WHERE  L  BEGINBMATRIX2  1 121 0121  DDOTS DDOTS  DDOTS 121 21 ENDBMATRIXITEM SHOW THAT THE EIGENVECTORS OF L ARE XBFN  BEGINBMATRIX SINNPIN SIN2NPIN  CDOTS   SINN1NPINENDBMATRIXT QQUAD N12LDOTSNWHERE LAMBDAN  4 SIN2NPI2N  NOTE THAT XBFN IS SIMPLYA SAMPLED VERSION OF XNTENDENUMERATEITEM FIND THE EIGENVALUES OF THE  FOLLOWING MATRICES  BEGINENUMERATE  ITEM A DIAGONAL MATRIX A  BEGINBMATRIXA11   A22   DDOTS   ANNENDBMATRIXITEM A TRIANGULAR MATRIX EITHER UPPER OR LOWER UPPER IN THIS  EXERCISE A  BEGINBMATRIXA11  A12  A13  CDOTS  A1N  0  A22  A23  CDOTS A2N  VDOTS  DDOTS 0 0  0  CDOTS  ANN ENDBMATRIXITEM FROM THESE EXERCISES CONCLUDEBEGINFACTBOXTHE DIAGONAL ELEMENTS FORM THE EIGENVALUES OF A IF    A IS TRIANGULAR  ENDFACTBOXINDEXEIGENVALUESTRIANGULAR MATRIXINDEXTRIANGULAR MATRIXEIGENVALUESENDENUMERATEITEM FOR MATRIX T IN BLOCK TRIANGULAR FORM T  BEGINBMATRIXT11  T12  0  T22 ENDBMATRIXSHOW THAT LAMBDAT  LAMBDAT11 CUP LAMBDAT22ITEM SHOW THAT THE DETERMINANT OF AN MATSIZENN MATRIX IS THE  PRODUCT OF THE EIGENVALUES THAT IS BOXED DETA  PRODI1N LAMBDAIINDEXDETERMINANTPRODUCT OF EIGENVALUESITEM SHOW THAT THE TRACE OF A MATRIX IS THE SUM OF THE EIGENVALUES BOXED TRACEA  SUMI1N LAMBDAIINDEXTRACESUM OF EIGENVALUESITEM WE WILL USE THE PREVIOUS TWO RESULTS TO PROVE A USEFUL INEQUALITY  BF HADAMARDS INEQUALITY INDEXHADAMARDS INEQUALITY  INDEXINEQUALITIESHARAMARDS   LET A BE A SYMMETRIC POSITIVE DEFINITE  MATSIZEMM  MATRIX  THEN BOXEDDETA LEQ PRODI1M AIIWITH EQUALITY IF AND ONLY IF A IS DIAGONALBEGINENUMERATEITEM SHOW THAT WE CAN WRITE ADBD WHERE D IS DIAGONAL AND B  HAS ONLY 1S ON THE DIAGONAL  DETERMINE DITEM EXPLAIN THE FOLLOWING EQUALITIES AND INEQUALITIES HINT USE THEINDEXARITHMETICGEOMETRIC MEAN INEQUALITYINDEXINEQUALITIESARITHMETICGEOMETRIC MEAN  ARITHMETICGEOMETRIC INEQUALITY    SEE EQUATION  REFEQGEOARINEQ  WHAT IS LAMBDAI HERE BEGINALIGNEDDETA  DETDBD  LEFTPRODI1M AIIRIGHT  DETB   LEFTPRODI1M AII RIGHT PRODI1M LAMBDAI LEQLEFTPRODI1M AII RIGHT LEFTFRAC1MSUMI1M  LAMBDAIRIGHTN  LEFTPRODI1M AII RIGHTLEFTFRAC1N  TRACEBRIGHTM  LEFTPRODI1M AII RIGHTENDALIGNEDENDENUMERATEITEM SUPPOSE A IS A RANK1 MATRIX FORMED BY A  ABF BBFT  FIND THE EIGENVALUES AND EIGENVECTORS OF A  ALSO SHOW THAT IF A  IS RANK 1 THEN DETIA  1TRACEAITEM LABELEXEIGSHIFTMAT SHOW THAT IF LAMBDA IS AN EIGENVALUE  OF A THEN LAMBDAR IS AN EIGENVALUE OF A RI AND THAT A  AND A  RI HAVE THE SAME EIGENVECTORSITEM SHOW THAT BEGINFACTBOXIF LAMBDA IS AN EIGENVALUE OF A THEN LAMBDAN  IS AN EIGENVALUE OF AN AND AN HAS THE SAME EIGENVECTORS AS  AENDFACTBOXITEM SHOW THAT LABELEXEIGINVBEGINSFACTBOXIF LAMBDA IS A NONZERO EIGENVALUE OF A THEN 1LAMBDA    IS AN EIGENVALUE OF A1  ENDSFACTBOXBEGINFACTBOXTHE EIGENVECTORS OF A CORRESPONDING TO NONZERO    EIGENVALUES ARE EIGENVECTORS OF A1  ENDFACTBOXITEM LABELEXPOLYEIG GENERALIZING THE PREVIOUS TWO PROBLEMS SHOW THAT  IF LAMBDA1 LAMBDA2 LDOTS LAMBDAM ARE THE EIGENVALUES OF  A AND IF GX IS A SCALAR POLYNOMIAL THEN THE EIGENVALUES OF  GA ARE  GLAMBDA1 GLAMBDA2 LDOTS GLAMBDAM  GANTMACHER V1P84ITEM SHOW THAT THE EIGENVALUES OF A PROJECTION MATRIX P  ARE EITHER 1 OR 0ITEM IN THIS PROBLEM YOU WILL ESTABLISH SOME RESULTS ON EIGENVALUES  OF PRODUCTS OF MATRICES  BEGINENUMERATE  ITEM IF A AND B ARE BOTH SQUARE SHOW THAT THE EIGENVALUES OF    AB ARE THE SAME AS THE EIGENVALUES OF BA  ITEM SHOW THAT IF THE MATSIZENN MATRICES A AND B HAVE A    COMMON SET OF N LINEARLY INDEPENDENT EIGENVECTORS THEN ABBA  ENDENUMERATEA THOROUGH STUDY OF WHEN AB  BA AS INTRODUCED IN THIS PROBLEM ISTREATED IN CITEORTEGA1988 ORGEGA P 249ITEM SHOW THAT A STOCHASTIC MATRIX HAS LAMBDA1 AS AN EIGENVALUE AND THAT XBF  11LDOTS1T IS THE CORRESPONDING EIGENVECTOR IT CAN BE SHOWN CITEORTEGA1988 THAT LAMBDA1 IS THE LARGEST EIGENVALUEITEM LINEAR FIXEDPOINT PROBLEMS  SOME PROBLEMS ARE OF THE FORMINDEXFIXEDPOINT PROBLEMS A XBF  XBFIF A HAS AN EIGENVALUE EQUAL TO 1 THEN THIS PROBLEM HAS A SOLUTIONCONDITIONS GUARANTEEING THAT A HAS AN EIGENVALUE OF 1 ARE DESCRIBEDIN CITEMINC1988  EXAMPLE PROBLEMS OF THIS SORT ARE THESTEADYSTATE PROBABILITIES FOR A MARKOV CHAIN AND DETERMININGVALUES FOR A COMPACTLYSUPPORTED WAVELET AT INTEGER VALUES OF THE ARGUMENTBEGINENUMERATEITEM LET  A BEGINBMATRIX 5  3  2  2  0  7  3  7  1ENDBMATRIXBE THE STATETRANSITION PROBABILITY MATRIX FOR A MARKOV MODELDETERMINE THE STEADYSTATE PROBABILITY PBF SUCH THAT A PBF PBFITEM THE TWOSCALE EQUATION FOR A SCALING FUNCTION INDEXSCALING    FUNCTION REFEQTWOSCALE3 IS PHIT  SUMK0N1 CK  PHI2TK  GIVEN THAT WE KNOW THAT THE PHIT IS ZERO FOR T  LEQ 0 AND FOR T GEQ N1 WRITE AN EQUATION OF THE FORM BEGINBMATRIX PHI1  PHI2  VDOTS  PHIN2  ENDBMATRIX A BEGINBMATRIX PHI1  PHI2  VDOTS  PHIN2  ENDBMATRIXWHERE A IS A MATRIX OF WAVELET COEFFICIENTS CK  GIVEN THE SET OFCOEFFICIENTS CK SPECIFY A AND DESCRIBE HOW TO SOLVE THISEQUATION  DESCRIBE HOW TO FIND PHIT AT ALL DYADIC RATIONALNUMBERS NUMBERS OF THE FORM K2I FOR INTEGERS K AND IINDEXDYADIC NUMBERENDENUMERATE ITEM CITEKAILATH80 LET ABBFCBFD REPRESEN A SYSTEM IN   STATESPACE FORM HAVING TRANSFER FUNCTION HS  CBFTSIA1   BBF  D  SHOW THAT THE ZEROS OF HS CAN BE COMPUTED AS THE   EIGENVALUES OF THE MATRIX A  BBF D1 CBFTEXSKIPSETEXSECTREFSECDIAGONAL ITEM PROVE IF A AND B ARE DIAGONALIZABLE THEY SHARE THE SAME   EIGENVECTOR MATRIX S IF AND ONLY IF ABBAITEM SHOW THAT THE INERTIA OF A HERMITIAN MATRIX A IS UNIQUELY  DETERMINED IF THE SIGNATURE AND RANK OF A ARE KNOWNITEM SYLVESTERS LAW OF INERTIA SHOW THAT IF A AND B HAVE THE  SAME INERTIA THEN THERE IS A MATRIX S SUCH THAT A  SBSH  HINT DIAGONALIZE A  UA LAMBDAA UAH   UA DA SIGMAA DA  UAH WHERE SIGMAA IS DIAGONAL WITH PM 10 ELEMENTS  SIMILARLY FOR BITEM SHOW THAT IF A AND B ARE SIMILAR SO THAT B  T1AT  BEGINENUMERATE  ITEM  A AND B HAVE THE HAVE THE SAME EIGENVALUES AND THE SAME    CHARACTERISTIC EQUATION  ITEM IF XBF IS AN EIGENVECTOR OF A THEN ZBF  T1 XBF    IS AN EIGENVECTOR OF B  ITEM IF IN ADDITION C AND D ARE SIMILAR WITH D  T1CT    THEN AC IS SIMILAR TO BD  ENDENUMERATE  ITEM DETERMINE THE JORDAN FORM OF A1  BEGINBMATRIX 2  1  2  0  2  3  002 ENDBMATRIXAND A2  BEGINBMATRIX 202  023  0 02 ENDBMATRIXITEM SHOW THAT REFEQJORDANPOW IS TRUE FOR THE MATSIZE33  MATRIX SHOWN  THEN GENERALIZE BY INDUCTION TO AN MATSIZEMM  JORDAN BLOCK ITEM SHOW THAT IF J IS A MATSIZE33 JORDAN BLOCK THAT  EJT  BEGINBMATRIXELAMBDA T  TELAMBDA T    FRAC12 T2 ELAMBDA T  0ELAMBDA T  TELAMBDA T  00 ELAMBDA T  ENDBMATRIX  THEN GENERALIZE BY INDUCTION TO A MATSIZEMM JORDAN BLOCKITEM SHOW THAT BEGINFACTBOX  A SELFADJOINT MATRIX IS POSITIVE SEMIDEFINITE IF AND  INDEXPOSITIVE SEMIDEFINITE ONLY IF ALL OF ITS EIGENVALUES ARE  GEQ 0ENDFACTBOXALSO SHOW THAT IF ALL THE EIGENVALUES ARE POSITIVE THEN THE MATRIX ISPOSITIVE DEFINITE INDEXPOSITIVE DEFINITE THE CONVERSE IS NOT TRUEA MATRIX WITH POSITIVE EIGENVALUES IS NOT NECESSARILY POSITIVEDEFINITEITEM SHOW THAT IF A HERMITIAN MATRIX A IS POSITIVE DEFINITE THEN  SO IS AK FOR K IN ZBB POSITIVE AS WELL AS NEGATIVE POWERSITEM SHOW THAT IF A IS NONSINGULAR THEN A AH IS POSITIVE DEFINITEITEM LABELEXSPECTRH PROVE THEOREM REFTHMDIAGSYM BY ESTABLISHING  THE FOLLOWING   TWO STEPS  BEGINENUMERATE  ITEM SHOW THAT IF A IS SELFADJOINT AND U IS UNITARY THEN     T  UH A U IS ALSO SELFADJOINT  ITEM SHOW THAT IF A SELFADJOINT MATRIX IS TRIANGULAR THEN IT MUST    BE DIAGONAL  ENDENUMERATEITEM LABELEXPROVEZEROEIG PROVE LEMMA REFLEMZEROEIGITEM A MATRIX N IS BF NORMAL IF IT COMMUTES WITH    NH NHN  NNH INDEXNORMAL MATRIX  BEGINENUMERATE  ITEM SHOW THAT UNITARY SYMMETRIC HERMITIAN AND SKEW SYMMETRIC    AND SKEW HERMITIAN MATRICES ARE NORMAL INDEXSKEW SYMMETRIC    INDEXSKEW HERMITIAN A MATRIX A IS SKEW SYMMETRIC IF AT     A  IT IS SKEW HERMITIAN IF AH  A  ITEM SHOW THAT FOR A NORMAL MATRIX THE TRIANGULAR MATRIX    DETERMINED BY THE SCHUR LEMMA IS DIAGONAL  ENDENUMERATEITEM SHOW THAT FOR A HERMITIAN MATRIX A IF A2  A THEN  RANKATRACEA  FROM CAMPBELL AND MEYER P 2ITEM LABELEXCYCLICMAT LETF  BEGINBMATRIX1  1             CDOTS  1 1  EJ2PIN  CDOTS  EJ2PIN1N 1  EJ4PIN  CDOTS  EJ4PIN1N VDOTS                   VDOTS 1  EJPIN1N   CDOTS  EJPIN12NENDBMATRIXTHE IJTH ELEMENT OF THIS IS FIJ  EJ2PI IJN  FOR AVECTOR XBF  X0 LDOTS XN1T THE PRODUCT XBF  F XBFIS THE DFT OF XBFBEGINENUMERATEITEM LABELEXFUNIT PROVE THAT THE MATRIX FSQRTN IS  UNITARY  HINT SHOW THAT BOXEDSUMN0N1 EJ2PI NKN  BEGINCASES N  K EQUIV0 BMOD N 0  K NOT EQUIV 0 BMOD N ENDCASESITEM LABELEXCIRCDIAG A MATRIX C  BEGINBMATRIXC0  C1  C2  LDOTS  CN1 CN1  C0  C1  LDOTS  CN2 VDOTS C1C2   C3   LDOTS    C0 ENDBMATRIXIS SAID TO BE A EM CIRCULANT MATRIX  SHOW THAT C IS DIAGONALIZEDBY F CF  FLAMBDA WHERE LAMBDA IS DIAGONAL  COMMENT ON THEEIGENVALUES AND EIGENVECTORS OF A CIRCULANT MATRIX  THE FFTBASED APPROACH TO CYCLIC CONVOLUTION WORKS ESSENTIALLY BYTRANSFORMING THE CYCLIC MATRIX TO A DIAGONAL MATRIX WHEREMULTIPLICATION POINTBYPOINT CAN OCCUR FOLLOWED BY TRANSFORMATIONBACK TO THE ORIGINAL SPACEENDENUMERATEEXSKIPSETEXSECTREFSECGEOINVSUBITEM PROVE THEOREM REFTHMADECOMP  HINT START WITH A  U  LAMBDA UH  ITEM CONSTRUCT MATSIZE33 MATRICES ACCORDING THE FOLLOWING SETS  OF SPECIFICATIONS SEE SPECEIGM  BEGINENUMERATE  ITEM LAMBDA1 LAMBDA21 LAMBDA3  2 WITH INVARIANT    SUBSPACES R1  LSPAN121T210TQQUADQQUAD R2     LSPAN125TIN THIS CASE DETERMINE THE EIGENVALUES AND EIGENVECTORS OF THE MATRIXYOU CONSTRUCT AND COMMENT ON THE RESULTSITEM LAMBDA1  1 LAMBDA2  4 LAMBDA3  9 WITH  CORRESPONDING EIGENVECTORS XBF1  FRAC1SQRT14BEGINBMATRIX1  23  ENDBMATRIX  QQUAD XBF2  FRAC1SQRT5BEGINBMATRIX 2  1  0  ENDBMATRIXQQUAD XBF3  FRAC1SQRT70BEGINBMATRIX365ENDBMATRIX  ENDENUMERATEITEM LET A BE MATSIZEMM HERMITIAN WITH K DISTINT  EIGENVALUES AND SPECTRAL REPRESENTATION A  SUMI1K LAMBDAI  PI WHERE PI IS THE PROJECTOR ONTO THE ITH INVARIANCE  SUBSPACE  SHOW THAT SI  A1  SUMI1K FRACPISLAMBDAIITEM CITEPAGE 663KAILATH80 THE DIAGONALIZATION OF SELFADJOINT  MATRICES CAN BE EXTENDED TO MORE GENERAL MATRICES  LET A BE AN  MATSIZEMM MATRIX WITH M LINEARLY INDEPENDENT EIGENVECTORS  XBF1XBF2LDOTS XBFM AND LET S  XBF1 XBF2  LDOTS XBFM  LET T  S1  THEN WE HAVE A  S LAMBDA  T WHERE LAMBDA IS THE DIAGONAL MATRIX OF EIGENVALUES  BEGINENUMERATE  ITEM LET TBFIT BE A ROW OF T  SHOW THAT TBFIT XBFJ     DELTAIJ  ITEM SHOW THAT A  SUMI1M LAMBDAI XBFI TBFIT  ITEM LET PI  XBFI TBFIT  SHOW THAT PI PJ  PI    DELTAIJ  ITEM SHOW THAT I  SUMI1M PI RESOLUTION OF IDENTITY  ITEM SHOW THAT  SIA1  SUMI1M FRACPISLAMBDAIENDENUMERATEEXSKIPSETEXSECTREFSECGEOSYMITEM LET  R  BEGINBMATRIX515385  323077 323077  784615 ENDBMATRIXBEGINENUMERATEITEM DETERMINE THE EIGENVALUES AND EIGENVECTORS OF RITEM DRAW LEVEL CURVES OF THE QUADRATIC FORM QRXBF  IDENTIFY  THE EIGENVECTOR DIRECTIONS ON THE PLOT AND ASSOCIATE THESE WITH TH  EIGENVALUES ITEM DRAW THE LEVEL CURVES OF THE QUADRATIC FORM QR1XBF  IDENTIFYING EIGENVECTOR DIRECTIONS AND THE EIGENVALUESENDENUMERATEITEM LABELEXSYLV2  SHOW THAT  IF A IS A HERMITIAN  HJ P 192  MATSIZEMM MATRIX AND IF XBFH A XBF GEQ 0 FOR ALL  VECTORS XBF IN A KDIMENSIONAL SPACE WITH K LEQ M THEN  A HAS AT LEAST K NONNEGATIVE EIGENVALUES  IF XBFH A XBF0  FOR ALL NONZERO XBF IN A KDIMENSIONAL SPACE THEN A HAS AT  LEAST K POSITIVE EIGENVALUES  HINT LET SK BE THE  KDIMENSIONAL SPACE AND LET UBF1 LDOTSUBFNK SPANE  SKPERP  LET C  UBF1LDOTS UBFNK AND USE THE COURANT MINIMAX PRINCIPLE BY CONSIDERING MINBEGINSUBARRAYC XBF NEQ 0  C XBF  0ENDSUBARRAY FRACXBFH  AXBFXBFH XBFITEM LABELEXEIGMAX IN THE PROOF OF THEOREM REFTHMMAXEIG  BEGINENUMERATE  ITEM SHOW THAT REFEQQMAX1 IS TRUE  ITEM SHOW THAT QXBF OF REFEQQMAX1 IS MAXIMIZED WHEN    ALPHAK1  ALPHAK2  CDOTS  ALPHAM  0 ENDENUMERATEITEM WRITE AND TEST A SC MATLAB FUNCTION TT PLOTELLIPSEAX0C  THAT COMPUTES POINTS ON THE ELLIPSE DESCRIBED BY XBFXBF0T A  XBFXBF0  C SUITABLE FOR PLOTTINGEXSKIPSETEXSECTREFSECCONEIGITEM DETERMINE STATIONARY VALUES EIGENVALUES AND EIGENVECTORS OF  XBFT R XBF SUBJECT TO XBFT CBF  0 WHERE R  BEGINBMATRIX515385  323077 323077  784615 ENDBMATRIX QQUAD CBF  12TITEM LABELEXLINCONEIG SHOW THAT THE STATIONARY VALUES OF XBFH  R XBF SUBJECT TO REFEQCONEIGCON ARE FOUND FROM THE  EIGENVALUES OF PRP WHERE P  ICCHC1CHEXSKIPSETEXSECTREFSECGERSHITEM DETERMINE REGIONS IN THE COMPLEX PLANE WHERE THE EIGENVALUES OF  A ARE FOR A  BEGINBMATRIX 311  142  127ENDBMATRIX AND  A2  BEGINBMATRIX3  1  1  1  4  2  1  2  7 ENDBMATRIX ITEM SHOW THAT ALL THE EIGENVALUES OF A LIE IN GA CAP GATITEM FOR A REAL MATRIX MATSIZEMM MATRIX A WITH DISJOINT  GERSHGORIN CIRCLES SHOW THAT ALL THE EIGENVALUES OF A ARE REALITEM CITEHORNJOHNSON IN THIS EXERCISE YOU WILL PROVE A SIMPLE  VERSION OF THE EM HOFFMANWIELANDT INDEXHOFFMANWIELANDT    THEOREM THEOREM A THEOREM FREQUENTLY USED FOR PERTURBATION  STUDIES  LET A AND E BE MATSIZEMM HERMITIAN MATRICES  AND LET A AND AE HAVE EIGENVALUES LAMBDAI AND  LAMBDAHATI RESPECTIVELY I12LDOTSM RESPECTIVELY  ARRANGED IN INCREASING ORDER LAMBDA1 LEQ LAMBDA2 LEQ CDOTS LEQ LAMBDAM QQUADQQUADLAMBDAHAT1 LEQ LAMBDAHAT2 LEQ CDOTS LEQ LAMBDAHATM  LET A  QLAMBDA QH AND LET  AE  WLAMBDAHAT WH WHERE Q AND W ARE UNITARY MATRICES    BEGINENUMERATE  ITEM STARTING  FROM  EF2  AEAF2  WLAMBDAHAT WH  QLAMBDA QHF2 SHOW THAT EF2  SUMI1M LAMBDAI2  MUI2  2 REAL    TRACE ZLAMBDA ZHLAMBDAHATWHERE Z  QH W ITEM THUS SHOW THAT EF2  GEQ SUMI1M LAMBDAI2  MUI2  2MAXU  TEXT UNITARY REAL  TRACE ULAMBDA UH LAMBDAHATIT CAN BE SHOWN THAT THE MAXIMUM OF MAXU TEXT UNITARY  REAL TRACE ULAMBDA UH LAMBDAHAT OCCURS WHEN U IS A PERMUTATION  MATRIX   ENDENUMERATEEXSKIPSETEXSECTREFSECKARHUNEN1ITEM LABELEXLOWRANK1 SHOW USING REFEQKLOWRANK  THAT  E2K CAN BE WRITTEN AS IN REFEQLR2  HINT REMEMBER HOW TO  COMMUTE INSIDE A TRACE ITEM LABELEXPC1 LET XBF BE A PELEMENT ZEROMEAN RANDOM   VECTOR WITH    COVARIANCE R AND LET YBF BE A QELEMENT RANDOM VECTOR  YBF  BT XBF  WHERE B IN MPQ AND Q  P  LET RY  BT R B BE THE COVARIANCE MATRIX OF YBF  SHOW THAT TRACERY IS MAXIMIZED BY TAKING  B  XBF1 XBF2 LDOTS XBFQ  XQ  WHERE LABELEXPC2 XBFI IS THE ITH NORMALIZED EIGENVECTOR OF R ITEM LET YBF  BT XBF AS IN THE PREVIOUS EXERCISE  SHOW THAT   DETRY IS MAXIMIZED WHEN B  XQ AS BEFOREITEM FOR A DATA COMPRESSION APPLICATION IT IS DESIRED TO ROTATE A  SET OF NDIMENSIONAL ZEROMEAN DATA Y  YBF1YBF2LDOTS  YBFN SO THAT IT MATCHES WITH ANOTHER SET OF NDIMENSIONAL  DATA Z   ZBF1ZBF2LDOTSZBFM  DESCRIBE HOW TO  PERFORM THE ROTATION IF THE MATCH IS DESIRED IN THE DOMINANT Q  COMPONENTS OF THE DATAITEM BF COMPUTER EXERCISE THIS EXERCISE WILL DEMONSTRATE SOME  CONCEPTS OF PRINCIPAL COMPONENTS  BEGINENUMERATE  ITEM CONSTRUCT A SYMMETRIC MATRIX R IN M2 THAT HAS UNNORMALIZED    EIGENVECTORS  XBF1  BEGINBMATRIX 1  5 ENDBMATRIXQQUAD XBF2 BEGINBMATRIX5  1 ENDBMATRIXWITH CORRESPONDING EIGENVALUES LAMBDA1  10 LAMBDA2  2ITEM GENERATE AND PLOT 200 POINTS OF ZEROMEAN GAUSSIAN DATA THAT HAS  THE COVARIANCE RITEM FORM AN ESTIMATE OF THE COVARIANCE OF THE GENERATED DATA AND  COMPUTE THE PRINCIPAL COMPONENTS OF THE DATAITEM PLOT THE PRINCIPAL COMPONENT INFORMATION OVER THE DATA AND  VERIFY THAT THE PRINCIPAL COMPONENT VECTORS LIE AS ANTICIPATED  ENDENUMERATEITEM CITESCHARFTUFTS1987 INDEXLOWRANK APPROXIMATION LOWRANK  APPROXIMATION CAN SOMETIMES BE USED TO OBTAIN A BETTER  REPRESENTATION OF A NOISY SIGNAL  SUPPOSE THAT AN MDIMENSIONAL  ZEROMEAN SIGNAL XBF WITH RX  EXBF XBFH IS TRANSMITTED  THROUGH A NOISY CHANNEL SO THAT THE RECEIVED SIGNAL IS RBF  XBF  NUBFAS SHOWN IN FIGURE REFFIGLOWRANK1A  LET ENUBFNUBFH  RNU   SIGMANU2I  THE MS ERROR IN THIS SIGNAL IS  E2TEXTDIRECT  ERBF  XBFHRBFXBF  M SIGMANU2ALTERNATIVELY WE CAN SEND THE SIGNAL XBF1  UH XBF WHERE U ISTHE MATRIX OF EIGENVECTORS OF RX AS IN REFEQKREDUCE1  THERECEIVED SIGNAL IN THIS CASE IS RBFR  XBF1  NUBFFROM WHICH AN APPROXIMATION TO XBFR IS OBTAINED BY XBFHATR  UIRRBFRWHERE IR  DIAG11LDOTS100LDOTS0 WITH R ONES  SHOW THAT BEGINALIGNEDE2TEXTINDIRECT  EXBFXBFHATRHXBFXBFHATR  SUMIR1M LAMBDAI  R SIGMANU2ENDALIGNEDHENCE CONCLUDE THAT FOR SOME VALUES OF R THE REDUCED RANK METHODMAY HAVE LOWER MS ERROR THAN THE DIRECT ERRORBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    SUBFIGUREDIRECTINPUTPICTUREDIRLR1 QQUAD    SUBFIGUREINDIRECTINPUTPICTUREDIRLR2    CAPTIONDIRECT AND INDIRECT TRANSMISSION THROUGH A      NOISY CHANNEL    LABELFIGLOWRANK1  ENDCENTERENDFIGUREEXSKIPSETEXSECTREFSECEIGFILT ITEM LABELEXLCMV1 SHOW THAT REFEQLCMV1 IS CORRECTITEM FOR AN INPUT SIGNAL WITH CORRELATION MATRIX R  BEGINBMATRIX 2  3  2 3  4  1 2  1  6ENDBMATRIXBEGINENUMERATEITEM DESIGN AN EIGENFILTER WITH 3 TAPS THAT MAXIMIZES THE SNR AT THE  OUTPUT OF THE FILTERITEM PLOT THE FREQUENCY RESPONSE OF THIS FILTERITEM DESIGN AN EIGENFILTER THAT EM MINIMIZES THE OUTPUT ENERGY  SUBJECT TO THE CONSTRAINT THAT ENDENUMERATEEXSKIP ITEM LABELEXEIGFILT SHOW THAT REFEQEIGF2 IS CORRECT USING THE DEFINITIONS OF   REFEQEIGFDITEM LABELEXEIGF2 SHOW THAT MINIMIZING REFEQJEIGF2 SUBJECT  TO CT BBF  DBF LEADS TO REFEQBEIGF2ITEM SHOW THAT REFEQEIGF1 AND REFEQEIGF2 ARE CORRECTITEM DEVISE A MEANS OF MATCHING A DESIRED RESPONSE BY MINIMIZING  BBFT R BBF SUBJECT TO THE FOLLOWING CONSTRAINTS BEGINALIGNEDBBFT BBF  1 CT BBF  ZEROBFENDALIGNEDWHERE C IS AS IN REFEQCCONEIG  THAT IS THE FILTERCOEFFICIENTS ARE CONSTRAINED IN ENERGY BUT THERE ARE FREQUENCIES ATWHICH THE RESPONSE SHOULD BE EXACTLY 0  HINT SEE SECTIONREFSECCONEIGITEM CONSIDER THE INTERPOLATION SCHEME SHOWN IN FIGURE  REFFIGMULTIRATEL  THE OUTPUT CAN BE WRITTEN AS YZ   XZLHZ    BEGINENUMERATE  ITEM SHOW THAT IF  HLT  BEGINCASESC  T  0  0  TEXTOTHERWISEENDCASESTHEN YLT  CXT  THIS MEANS THAT THE INPUT SAMPLES ARE CONVEYEDEM WITHOUT DISTORTION BUT POSSIBLY WITH A SCALE FACTOR TO THEOUTPUT  SUCH FILTERS ARE CALLED EM NYQUIST OR EM LTH BANDFILTERS CITEVAIDYANATHAN1993 INDEXNYQUIST FILTER ITEM EXPLAIN HOW TO USE THE EIGENFILTER DESIGN TECHNIQUE TO DESIGN AN  OPTIMAL MEANSQUARE LTH BAND FILTERBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRMULTIRATE1    CAPTIONEXPANSION AND INTERPOLATION USING MULTIRATE PROCESSING    LABELFIGMULTIRATEL  ENDCENTERENDFIGUREENDENUMERATEITEM WRITE AND TEST A SC MATLAB PROGRAM WHICH ACCEPTS A PASSBAND  UPPER FREQUENCY OMEGAP AND A STOPBAND LOWER FREQUENCY OMEGAS  AND COMPUTES N FILTER COEFFICIENTS USING THE EIGENFILTER  APPROACH  ITEM IT IS DESIRED TO DEVELOP A LOWPASS FILTER HBF SUCH THAT THE  MAGNITUDE RESPONSE OF THE FILTER AT A PARTICULAR FREQUENCY OMEGA0  IS PRECISELY 0  USING THE NOTATION OF REFEQEIGF2 HOMEGA0  BBFT CBFOMEGA0  0DEVELOP AN EIGENBASED SOLUTION TO THE PROBLEMITEM A FILTER IS TO BE DESIGNED SO THAT BBFT P BBFIS MINIMIZED WHERE P CONTAINS PASSBAND AND STOPBAND INFORMATION ASIN REFEQEIGF2 SUBJECT TO A PASSFREQUENCY CONSTRAINT BBFT CBFOMEGA0  BETAFOR SOME BETA  SHOW THAT THE OPTIMAL FILTER IS BBF  BETA FRACP1 CBFOMEGA0CBFOMEGA0T A  CBFOMEGA0EXSKIPSETEXSECTREFSECMUSICITEM LABELEXSINAC  SHOW THAT REFEQSINAC IS TRUE SHOW THAT  REFEQMUSIC0 IS TRUEITEM SHOW FOR MP THAT RXX DEFINED IN REFEQMUSIC0 HAS  RANK P  HINT SEE PROPERTIES OF RANK ON PAGE PAGEREFPAGERANKPAGE ITEM SHOW THAT EVERY XBFT IN SIGSPACE WHERE SIGSPACE IS  DEFINED IN REFEQDEFSIGSPACEEXSKIPSETEXSECTREFSECGENEIGITEM NUMERICALLY COMPUTE USING EG SC MATLAB THE GENERALIZED  EIGENVALUES AND EIGENVECTORS FOR A UBF  LAMBDA B UBFWHERE A  BEGINBMATRIX 3  2 4  1  7  9  6  2  1ENDBMATRIXQQUADB  BEGINBMATRIX 5  2  1  5  3  0  2  1  7ENDBMATRIXITEM LABELEXESPRIT SHOW  THAT REFEQRYZ IS TRUE ITEM IN THE ESPRIT APPROACH SHOW THAT THE SIGNAL STRENGTH OF THE   ITH SINUSOIDAL COMPONENT CAN BE DETERMINED FROM  AI  FRACUBFIHRYY  SIGMAW2 IUBFIUBFIH SBFI2  WHERE UBFI IS THE GENERALIZED EIGENVECTOR CORRESPONDING TO LAMBDAI AND SBFI IS DETERMINED FROM REFEQMUSICS ITEM SHOW THAT THE CROSS COVARIANCE MATRIX RYZ DEFINED IN   REFEQRYZ CAN BE WRITTEN AS  RYZ  BEGINBMATRIX RYY1  RYY2    CDOTS  RYYM  RYY0  RYY1  CDOTS  RYYM1  VDOTS  RBARYYM2  RBARYYM3  CDOTS  RYY1  ENDBMATRIX  ITEM CITEKAILATH80 LET ABBF CBFT D BE A STATESPACE   DESCRIPTION OF A SYSTEM SEE CHAPTER REFCHAPINTRO  SHOW THAT   THE ZEROS OF THE TRANSFER FUNCTION HS  CBFTSIA1 BBF    D CAN BE COMPUTED AS THE EIGENVALUES OF ABBF D1 CBFT ITEM CONTINUING THE PREVIOUS PROBLEM SHOW THAT THE ZEROS CAN BE   COMPUTED BY SOLVING THE GENERALIZED EIGENVALUE PROBLEM  LAMBDA E  F PBF  0  WHERE  E  BEGINBMATRIXI  0  0  0 ENDBMATRIX QQUAD F  BEGINBMATRIX A BBF  CBFT  D ENDBMATRIX  ITEM LET ABBFCBFD REPRESENT A SYSTEM IN STATEVARIABLE   FORM HAVING TRANSFER FUNCTION HS  CBFTSIA1BBF  D   SHOW THAT THE ZEROS CAN BE COMPUTED BY SOLVING THE GENERALIZED   EIGENVALUE PROBLEM  LAMBDA E  F PBF  0  WHERE  E  BEGINBMATRIX I  0  0  0  ENDBMATRIXQQUAD   F  BEGINBMATRIXA  BBF  CBFT  D ENDBMATRIX EXSKIPSETEXSECTREFSECMINPOLY  ITEM SHOW THAT THE MINIMAL POLYNOMIAL IS UNIQUE  HINT SUBTRACT    FX  GX  ITEM SHOW THAT THE MINIMAL POLYNOMIAL DIVIDES EVERY ANNIHILATING    POLYNOMIAL WITHOUT REMAINDER    ITEM CITEGANTMACHERI DETERMINE THE MINIMAL POLYNOMIAL OF A  BEGINBMATRIX 332  152  130 ENDBMATRIXITEM CITEPAGE 657KAILATH80 RESOLVENT IDENTITIES INDEXRESOLVENT    IDENTITIES LET A BE A MATSIZEMM MATRIX WITH  CHARACTERISTIC POLYNOMIAL CHIAS  DETSIA  SM  AM1 SM1  CDOTS  A1 S   A0THE MATRIX SIA1 IS KNOWN AS THE RESOLVENT OFA INDEXRESOLVENT OF A MATRIXBEGINENUMERATEITEM SHOW THATBEGINEQUATIONLABELEQRESOL1BEGINSPLIT ADJSIA  ISM1  AAM1 I SM2  CDOTS   QQUAD AM1 AM1 AM2  CDOTS  A1 IENDSPLITENDEQUATIONHINT MULTIPLY BOTH SIDES BY SIA AND USE THE CAYLEYHAMILTON THEOREMITEM SHOW THATBEGINEQUATIONLABELEQRESOL2BEGINSPLIT ADJSIA  AM1  SAM1 AM2  CDOTS   QQUAD SM1 AM1 SM2  CDOTS  A1 IENDSPLITENDEQUATIONITEM LET SI BE THE COEFFICIENT OF SMI IN REFEQRESOL1  SHOW THAT THE SI  CAN BE RECURSIVELY COMPUTED ASBEGINALIGNED S1  I  QQUAD S2  S1 A  AM1 I QQUADS3  S2 A  AM2 I QQUAD LDOTS SM  SM1A  A1I QQUAD 0  SM  A0 IENDALIGNEDITEM SHOW THAT BOTH THE COEFFICIENTS OF THE CHARACTERISTIC POLYNOMIAL  AI AND THE COEFFICIENTS SI OF THE ADJOINT OF THE RESOLVENT CAN  BE RECURSIVELY COMPUTED AS FOLLOWSBEGINALIGNAT3S1I                QQUAD  A1  S1 A   QQUAD  AM1  TRACEA1 S2 A1  AM1 I  QQUAD A2  S2 A   QQUAD AM2 FRAC12 TRACEA2 S3 A2  AM2 I  QQUAD A3  S3 A    QQUAD AM3  FRAC13 TRACEA3VDOTS                QQUAD VDOTS           QQUAD  VDOTS SM1 AM1  A2I  QQUAD AM1  SM1A  QQUAD A1  FRAC1M1TRACEAM1 SM SM1 A  A1I  QQUAD AM  SM A  QQUAD A0  FRAC1M TRACEAMENDALIGNATTHESE RECURSIVE FORMULAS ARE KNOWN AS THELEVERRIERSOURIAUFADDEEVAFRAME FORMULAS CITEP 88GANTMACHERIHINT USE THE NEWTON IDENTITIES CITEP 436CHRYSTAL1926INDEXNEWTON IDENTITIES FOR THE POLYNOMIALS PX  XM AM1XM1  CDOTS  A0 LET SI DENOTE THE SUM OF THE ITHPOWER OF THE ROOTS OF PX  THUS S1 IS THE SUM OF THE ROOTSS2 IS THE SUM OF THE SQUARES OF THE ROOTS ETC  THEN BEGINALIGNED S1  AM1  0    S2  AM1 S1  2AM2  0    VDOTS    SM1  AM1SM2  CDOTS  M1 A1  0ENDALIGNEDALSO USE THE FACT THAT POWERS LAMBDAAK  LAMBDAKA SO THAT TRACEAK   SUMI1M LAMBDAIK  SKSHOW THAT AK  AK  AM1 AK1  CDOTS   AMK ATHEN TAKE THE TRACE OF EACH SIDEINDEXLEVERRIER FORMULASENDENUMERATEEXSKIPSETEXSECTREFSECMOVEEIGITEM LABELEXCOMPANDET EXPANDING BY COFACTORS SHOW THAT THE  CHARACTERISTIC EQUATION OF THE MATRIX IN FIRST COMPANION FORM  REFEQCONTROL4 IS SM  AM1 SM1  A1 S  A0ITEM LET A BE AN MATSIZEMM MATRIX  REFER TO FIGURE  REFFIGPLANT2   BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODE      INPUTPICTUREDIRPLANT2      CAPTIONTRANSFORMATION FROM A GENERAL MATRIX TO 1ST COMPANION FORM      LABELFIGPLANT2    ENDCENTER  ENDFIGURE    BEGINENUMERATE    ITEM IF AHAT  UAU1 WHEREU1  Q  BEGINBMATRIX BBF  ABBF  CDOTS  AM1BBFENDBMATRIXTHEN SHOW THAT AHAT HAS THE SECOND COMPANION FORMBEGINEQUATION AHAT  BEGINBMATRIX 000CDOTS  A0 1 00 CDOTS  A1 010CDOTS A2 VDOTS DDOTS 000CDOTS AM1ENDBMATRIXLABELEQSECONDCOMPANENDEQUATIONHINT SHOW THAT U1AHAT  AU1  USE THE CAYLEYHAMILTONTHEOREM INDEXCAYLEYHAMILTON THEOREMITEM SHOW THAT BBFHAT  UBBF  10LDOTS0TITEM SHOW THAT IF ATILDE  VAHAT V1 WHERE AHAT IS IN  SECOND COMPANION FORM REFEQSECONDCOMPAN AND V1 IS THE  HANKEL MATRIX INDEXHANKEL MATRIX V1  BEGINBMATRIX A1  CDOTS  AM3 AM2  AM1  1 A2   CDOTS  AM2 AM1   1  0 A3   CDOTS  AM1   1  0 0VDOTS   1 ENDBMATRIX  WTHEN ATILDE HAS THE FIRSTCOMPANION FORM SHOWN IN REFEQCONTROL4    ENDENUMERATEITEM FOR THE SYSTEM WITH A  BEGINBMATRIX012  321  24 5 ENDBMATRIXQQUADQQUAD BBF  BEGINBMATRIX 2  0  1 ENDBMATRIXDETERMINE THE GAIN MATRIX GBF SO THAT THE EIGENVALUES OF AC  A BBFGBFT ARE AT 3 2 PM J3SQRTFRAC32ITEM LINEAR OBSERVERS INDEXLINEAR OBSERVER IN ORDER TO DO POLE  PLACEMENT AS DESCRIBED IN THIS SECTION THE STATE XBF MUST BE  KNOWN  MORE COMMONLY ONLY AN OUTPUT Y IS AVAILABLE WHERE Y   CBFT XBF  IN THIS CASE AN EM OBSERVER MUST BE CONSTRUCTED  TO ESTIMATE THE STATE      ASSUME THAT THE SYSTEM SATISFIES XBFDOT A XBF  BBF U  THEN THE  OBSERVER IS OF THE FORM IN CONTINUOUS TIME XBFHATDOT  AHAT XBFHAT  BBFHAT U  KBF YLET EBF  XBFXBFHAT DENOTE THE DIFFERENCE BETWEEN THE TRUE STATEXBF AND THE ESTIMATED STATE XBFHATBEGINENUMERATEITEM WRITE THE DIFFERENTIAL EQUATION FOR EBFDOT AND SHOW THAT IN  ORDER FOR THE ERROR EBFDOT TO BE INDEPENDENT OF THE STATE XBF  AND THE INPUT U THAT AHAT  AKBFCBFT QQUADTEXTANDQQUAD BHAT  BITEM BASED ON THIS DETERMINE A MEANS TO PLACE THE EIGENVALUES OF THE  OBSERVER MATRIX AHAT AT ANY DESIRED LOCATION  HINT CONSIDER THE  DUALITY BETWEEN AC  ABBF GBFT AND AHAT  A  KBF  CBFT   YOU SHOULD FIND THAT THE SOLUTION INVOLVES A MATRIX OF  THE FORM N  BEGINBMATRIX CBFT   CBFTA  VDOTS  CBFT AM1  ENDBMATRIXCALLED THE EM OBSERVABILITY TEST MATRIX  INDEXOBSERVABILITY TEST  MATRIXENDENUMERATEITEM LET Q  BEGINBMATRIXBBF  A BBF  A2 BBF  LDOTS AM1 BBFENDBMATRIXQQUAD TEXTAND QQUADN  BEGINBMATRIX CBFT   CBFTA  VDOTS  CBFT AM1  ENDBMATRIXBE THE CONTROLLABILITY AND OBSERVABILITY TEST MATRICES RESPECTIVELYOF A SYSTEM A BBF CBF  DETERMINE THE PRODUCT H  NQNOTE THAT THE ELEMENTS OF H ARE THE MARKOV PARAMETERS INDEXMARKOV  PARAMETERS INTRODUCED IN SECTION REFSECLTI  IF RANKH  M  WHAT CAN YOU CONCLUDE ABOUT RANKN AND RANKQITEM CITEPAGE 660KAILATH80 SOME PROPERTIES OF COMPANION  MATRICES INDEXCOMPANION MATRIXPROPERTIES LET A  BEGINBMATRIX AM1  AM2  CDOTS  A1  A0  1  0  CDOTS  00 0  1  CDOTS  00 VDOTS 0  0  CDOTS  10 ENDBMATRIXBE AN MATSIZEMM COMPANION MATRIX SOMETIMES CALLED ATOPCOMPANION MATRIXBEGINENUMERATEITEM SHOW THAT RANKLAMBDAI I  A LEQ M1 WHERE LAMBDAI IS  AN EIGENVALUE OF AITEM SHIFTING SHOW THAT EBFIT A  EBFI1T FOR 2 LEQ I LEQ  M WHERE EBFI IS THE UNIT VECTOR WITH 1 IN THE ITH  POSITION  ALSO EBF1T A  BEGINBMATRIX AM1  AM2   CDOTS  A0 ENDBMATRIXITEM SHOW THAT IF A IS NONSINGULAR THEN A1 IS A BOTTOM  COMPANION MATRIX WITH LAST ROW 1A0 AM1A0 CDOTS A1A0ITEM INVERSE IS COMPANION SHOW THAT A IS NONSINGULAR IF AND ONLY  IF A0 NEQ 0ENDENUMERATEEXSKIPSETEXSECTREFSECCHANNELCAPITEM FOR A CHANNEL CONSTRAINED TO HAVE AT LEAST ONE 0 BETWEEN  EVERY 1 AND RUNS OF ZERO NO LONGER THAN 3NO RUNS OF 1S OR 0S LONGER  THAN 3 INDEXRUNLENGTHCONSTRAINED CODING  BEGINENUMERATE  ITEM DRAW THE STATETRANSITION DIAGRAM  ITEM DETERMINE THE STATETRANSITION MATRIX  ITEM DETERMINE THE CAPACITY  ENDENUMERATEEXSKIPSETEXSECTREFSECCOMPEIGITEM LABELEXEIGCPLX THE COMPUTATIONAL ROUTINES DESCRIBED IN THIS  SECTION APPLY TO EM REAL  MATRICES  IN THIS PROBLEM WE EXAMINE  HOW TO EXTEND REAL COMPUTATIONAL ROUTINES TO HERMITIAN COMPLEX  MATRICES  LET A BE A HERMITIAN MATRIX AND LET A  AR  JJ  AI WHERE AR IS THE REAL PART AND AI IS THE IMAGINARY PART  LET XBF  XBFR  J XBFI BE AN EIGENVECTOR OF A WITH ITS  CORRESPONDING REAL AND IMAGINARY PARTS  BEGINENUMERATE  ITEM SHOW THAT THE CONDITION A XBF  LAMBDA XBF CAN BE    REWRITTEN AS BEGINBMATRIXAR  AI  AI  AR ENDBMATRIXBEGINBMATRIXXBFR  XBFI ENDBMATRIX  LAMBDABEGINBMATRIX XBFR   XBFIENDBMATRIXITEM LET ABB  LEFTBEGINSMALLMATRIX AR  AI  AI       ARENDSMALLMATRIXRIGHT  SHOW THAT ABB IS SYMMETRICITEM SHOW THAT IF XBFRT XBFITT IS AN EIGENVECTOR OF  ABB CORRESPONDING TO LAMBDA THEN SO IS XBFIT  XBFRTTITEM CONCLUDE THAT EACH EIGENVALUE OF ABB HAS MULTIPLICITY 2 AND  THAT THE EIGENVALUES OF A CAN BE OBTAINED BY SELECTING ONE  EIGENVECTOR CORRESPONDING TO EACH PAIR OF REPEATED EIGENVALUES  ENDENUMERATEITEM CITEGVL LABELEXTRIDIAG1 IN THE HOUSEHOLDER  TRIDIAGONALIZATION ILLUSTRATED IN REFEQTRIDIAG1 THE MATRIX  B1 IS OPERATED ON BY THE HOUSEHOLDER MATRIX HV TO PRODUCE HV  B1 HV FOR A HOUSEHOLDER VECTOR VBF  SHOW THAT HV B1 HV  CAN BE COMPUTED BY HV B1 HV  B VBFWBFT  WBF VBFTWHERE  PBF  FRAC2VBFT VBF B1 VBF QQUADQQUAD WBF  PBF  FRACPBFT VBFVBFT VBF VBFITEM WILKINSON SHIFT IF T  LEFTBEGINSMALLMATRIX AN1  BN1       BN1AN ENDSMALLMATRIXRIGHT SHOW THAT THE EIGENVALUES      ARE OBTAINED BY MU  AN  D PM SQRTD2  BN12WHERE D  AN1  AN2 ITEM SHOW THAT THE EIGENVALUES  OF A  PERMUTATION MATRIX ARE SUCH   THAT LAMBDAI1ENDEXERCISESSECTIONREFERENCESTHE DEFINITIVE HISTORICAL WORK ON EIGENVALUE COMPUTATIONS ISCITEWILKINSONAEP  A MORE RECENT EXCELLENT SOURCE ON THE THEORY OFEIGENVALUES AND EIGENVECTORS IS CITEHORNJOHNSON  THE COURANTMINIMAX THEOREM IS DISCUSSED IN CITECOURANTHILBERT ANDCITEWILKINSONAEP  THE PROOF OF THEOREM REFTHMCOURANT IS DRAWNFROM CITEKEENER  THE DISCUSSION OF CONSTRAINED EIGENVALUE PROBLEMOF SECTION REFSECCONEIG IS DRAWN FROM CITEGOLUB1973 WHEREEFFICIENT NUMERICAL IMPLEMENTATION ISSUES ARE ALSO DISCUSSED  FURTHERRELATED DISCUSSIONS ARE IN CITESPJOTVOLLGANDER1981FORSYTHE1965  OURPRESENTATION OF THE GERSHGORIN CIRCLE THEOREM IS DRAWN FROMCITEHORNJOHNSON IN WHICH EXTENSIVE DISCUSSION OF PERTURBATION OFEIGENVALUE PROBLEMS IS ALSO PRESENTEDTHE DISCUSSION ON LOWRANK APPROXIMATIONS IS DRAWN FROMCITEHAYKIN1996 AND CITESCHARFL1991  AN EXCELLENT COVERAGE OFPRINCIPAL COMPONENT METHODS IS FOUND IN CITEMORRISON1976 WHICHALSO INCLUDES A DISCUSSION ON THE ASYMPTOTIC STATISTICAL DISTRIBUTIONOF THE EIGENVALUES AND EIGENVECTORS OF CORRELATION MATRICES ANDCITEJOLLIFFE1986  EXERCISE REFEXPC1 IS FROM CITEJOLLIFFE1986A RECENTLYPROPOSED ALTERNATIVE TO PRINCIPLECOMPONENT METHODS BEARS MENTIONING HERE  IN EM ARCHETYPAL  ANALYSIS A SMALL SET OF VECTORS KNOWN AS ARCHETYPES IS FOUND THATIS REPRESENTATIVE OF SOME ORIGINAL SET OF DATA IN WHICH THEREPRESENTATIVES ARE FOUND AS CONVEX COMBINATIONS OF THE ORIGINAL DATAUNLIKE PRINCIPAL COMPONENT VECTORS WHICH MAY LOOK NOTHING LIKE THEORIGINAL VECTORS THE ARCHETYPES LOOK LIKE THE ORIGINAL DATA  SEECITECUTLER1994THE EIGENFILTER METHOD FOR RANDOM SIGNALS IS PRESENTED INCITEHAYKIN1996  THE EIGENFILTER METHOD FOR THE DESIGN OF FIR FILTERSWITH SPECTRAL REQUIREMENTS IS PRESENTED IN CITEVAIDYANATHAN1987AADDITIONAL WORK ON EIGENFILTERS IS DESCRIBED INCITEPEISC1988SUNDERSNGUYEN1993NGUYEN1994  IT ALSO POSSIBLE TOINCLUDE OTHER CONSTRAINTS SUCH AS MINIMIZING THE EFFECT OF A KNOWNINTERFERING SIGNAL MAKING THE RESPONSE MAXIMALLY FLAT OR MAKING THERESPONSE ALMOST EQUIRIPPLETHE MUSIC METHOD IS DUE TO SCHMIDT CITESCHMIDT1979 SEECITEKESLERSB  THE PISARENKO HARMONIC DECOMPOSITION APPEARS INCITEPISARENKO1973  CONSIDERABLE WORK HAS BEEN DONE ON MUSICMETHODS SINCE ITS INCEPTION  WE CITECITESTOICA1991STOICA1991ASTOICA1989A AS REPRESENTATIVES  SEEALSOCITESTOICAMOSES1991STOICAMOSESBOOKKAY1988MARPLEESPRIT APPEARS IN CITEROY1987ROY1989ROYKAILATH1989  MUSICESPRIT AND OTHER SPECTRAL ESTIMATION METHODS APPEAR INCITEPROAKISRADERTHE NOISELESS CHANNEL CODING THEOREM IS DISCUSSED INCITEBLAHUT1987 AND WAS ORIGINALLY PROPOSED BY SHANNONCITESHANNON1948  THE BOOK CITELINDMARCUS PROVIDES ATHOROUGH STUDY OF THE DESIGN OF CODES FOR CONSTRAINED CHANNELSINCLUDING AN EXPLANATION OF THE MAGNETIC RECORDING CHANNEL PROBLEMTHE WORKS OF IMMINK CITEIMMINK1990IMMINK1991 PROVIDE ANENGINEERING TREATMENT OF RUNLENGTHLIMITED CODES  OUR DISCUSSION OF CHARACTERISTIC POLYNOMIALS FOLLOWSCITEGANTMACHERI  THIS SOURCE PROVIDES AN EXHAUSTIVE TREATMENT OFJORDAN FORMS AND MINIMAL POLYNOMIALS  ANOTHER EXCELLENT SOURCE OFINFORMATION ABOUT JORDAN FORMS AND MINIMAL POLYNOMIALS ISCITEORTEGA1988  SEE ALSO THE APPENDIX OF CITEKAILATH80 EIGENVALUE PLACEMENT FOR CONTROLS IS BY NOW CLASSICAL SEE EGCITEKAILATH80  OUR DISCUSSION ON LINEAR CONTROLLERS AS WELL ASTHE EXERCISE ON LINEAR OBSERVERS IS DRAWN FROM CITEFRIEDLAND1986THE DISCUSSION HERE ON THE COMPUTATION OF EIGENVALUES AND EIGENVECTORSWAS DRAWN CLOSELY FROM CITECHAPTER 8GVL  THE EIGENVALUE PROBLEMIS ALSO DISCUSSED IN CITEPRESSETALSTOER1993  COMPUTATION OFEIGENVECTORS IS REVIEWED IN CITEIPSEN1997 LOCAL VARIABLES TEXMASTER TEST ENDLEQBEGINTEXTBOX09TEXTWIDTHPOSITIVEDEFINITE MATRICES LABELBOXPDWE WILL ENCOUNTER SEVERAL TIMES IN THE COURSE OF THIS BOOK THE NOTIONOF POSITIVEDEFINITE MATRICES  WE COLLECT TOGETHER HERE SEVERALIMPORTANT FACTS RELATED TO POSITIVEDEFINITE MATRICESINDEXPOSITIVEDEFINITEBEGINDEFINITION  A MATRIX A IS SAID TO BE BF POSITIVEDEFINITE PD IF XBFH A  XBF0 FOR ALL XBF NEQ 0  THIS IS SOMETIMES DENOTED AS A   0  CAUTION THE NOTATION A0 IS ALSO SOMETIMES USED TO INDICATE  THAT ALL THE ELEMENTS OF A ARE GREATER THAN WHICH IS NOT THE SAME  AS BEING PD  IF XBFH A XBF GEQ 0 FOR ALL XBF THEN A IS  BF POSITIVESEMIDEFINITE PSD  IF  IS REPLACED BY  THE  MATRIX IS SAID TO BE BF NEGATIVE DEFINITE ND INDEXNEGATIVE    SEMIDEFINITE AND IF GEQ IS REPLACED BY LEQ THE MATRIX IS  BF NEGATIVE SEMIDEFINITE NSDENDDEFINITIONHERE ARE SOME PROPERTIES OF POSITIVEDEFINITE OR SEMIDEFINITEMATRICESBEGINENUMERATEITEM ALL DIAGONAL ELEMENTS OF A PD PSD MATRIX ARE POSITIVE  NONNEGATIVE  CAUTION THIS DOES NOT MEAN THAT POSITIVE  DIAGONAL ELEMENTS IMPLY THAT A MATRIX IS PDITEM A HERMITIAN MATRIX A IS PD PSD IF AND ONLY IF ALL OF THE  EIGENVALUES ARE POSITIVE NONNEGATIVE  HENCE A PD MATRIX HAS A  POSITIVEDETERMINANT  HENCE A PD MATRIX IS INVERTIBLEITEM A HERMITIAN MATRIX P IS PD IF AND ONLY IF ALL  PRINCIPAL MINORS ARE POSITIVEITEM IF A IS PD THEN THE PIVOTS OBTAINED IN THE LU FACTORIZATION ARE  POSITIVEITEM IF A0 AND B0 THEN AB0  IF A IS PD AND B IS PSD  THEN AB IS PDITEM A HERMITIAN PD MATRIX A CAN BE FACTORED AS A  BHB FOR  INSTANCE USING THE CHOLESKY FACTORIZATION WHERE B IS FULL RANK  THIS IS A MATRIX SQUARE ROOT INDEXSQUARE ROOTOF A MATRIXENDENUMERATEENDTEXTBOX LOCAL VARIABLES TEXMASTER TEST END PAPERS 865 TOEP EIGVECT SPECIAL MATRICESCHAPTERSOME SPECIAL MATRICES AND THEIR APPLICATIONSLABELCHAPSPECIALMATSOME PARTICULAR MATRIX FORMS ARISE FAIRLY OFTEN IN SIGNAL PROCESSINGIN THE DESCRIPTION AND ANALYSIS OF ALGORITHMS SUCH AS IN LINEARPREDICTION FILTERING ETC  THIS CHAPTER PROVIDES AN OVERVIEW OF SOMEOF THE MORE COMMON SPECIAL MATRIX TYPES ALONG WITH SOME APPLICATIONS TOSIGNAL PROCESSINGINPUTLINALGDIRMODALMATRIXSECTIONPERMUTATION MATRICESLABELSECPERMUTEMATINDEXPERMUTATION MATRIX PERMUTATION MATRICES ARE SIMPLE MATRICESTHAT ARE USED TO INTERCHANGE ROWS AND COLUMNS OF A MATRIXBEGINDEFINITION  A PERMUTATION MATRIX P IS AN MATSIZEMM MATRIX WITH ALL  ELEMENTS EITHER 0 OR 1 WITH EXACTLY ONE 1 IN EACH ROW AND COLUMNENDDEFINITIONTHE MATRIX P BEGINBMATRIX 0001  0  1  0  0  1000 0010 ENDBMATRIXIS A PERMUTATION MATRIX  LET A BE A MATRIX  THEN PA IS A BF  ROWPERMUTED VERSION OF A AND AP IS A BF COLUMNPERMUTEDVERSION OF A  PERMUTATION MATRICES ARE ORTHOGONAL IF P IS APERMUTATION THEN P1PT  THE PRODUCT OF PERMUTATION MATRICESIS ANOTHER PERMUTATION MATRIX  THE DETERMINANT OF A PERMUTATION ISPM 1BEGINEXAMPLE  LET  A  BEGINBMATRIX123456789 ENDBMATRIXQQUADTEXTANDQQUADP  BEGINBMATRIX010  001  100 ENDBMATRIXTHEN PA  BEGINBMATRIX456789123 ENDBMATRIXQQUADQQUADAP  BEGINBMATRIX3  1  2 6  4  5 9  7  8 ENDBMATRIXENDEXAMPLEPERMUTATION OPERATIONS ARE BEST IMPLEMENTED WITHOUT AN EXPENSIVEMULTIPLICATION USING AN INDEX VECTOR  FOR EXAMPLE THE PERMUTATION POF THE PREVIOUS EXAMPLE COULD BE REPRESENTED IN COLUMN ORDERING AS THEINDEX IC  231 THEN PA  AIC  IN ROW ORDERING P CANBE REPRESENTED AS IR  312 SO AP  AIRIT CAN BE SHOWN CITEHORNJOHNSON BIRKHOFFS THEOREMINDEXBIRKHOFFS THEOREM THAT EVERY DOUBLY STOCHASTIC MATRIX CAN BEEXPRESSED AS A CONVEX SUM OF PERMUTATION MATRICESBEGINEXERCISESITEM SHOW THAT THE DETERMINANT OF A PERMUTATION MATRIX IS PM 1ITEM THE BITREVERSE SHUFFLE OF THE FFT ALGORITHM IS A  PERMUTATION  TABLE REFTABBRS ILLUSTRATES A BITREVERSE SHUFFLE  FOR AN 86POINT DFT  DETERMINE A PERMUTATION MATRIX WHICH PERMUTES AN  INCOMING VECTOR ACCORDING TO THE BITREVERSE SHUFFLE  BEGINTABLEHTBPBEGINCENTER    BEGINTABULARCCCC HLINEN  BINARY  BIT REVERSE  BITREVERSED N HLINE0  000  000  0 1  001  100  4 2  010  010  2 3  011  110  6 4  100  001  1 5  101  101  5 6  110  011  3 7  111  111  7HLINE    ENDTABULARENDCENTER    CAPTIONBIT REVERSE SHUFFLE    LABELTABBRS  ENDTABLEITEM THE MATSIZEMM PERMUTATION MATRICES FORM A GROUP  DETERMINE THE NUMBER OF MEMBERS IN THE GROUP OF MATSIZEMM  PERMUTATION MATRICES  DETERMINE A POWER K SUCH THAT ALL  MATSIZE33 PERMUTATION MATRICES SATISFY PK  I  ITEM A STOCHASTIC MATRIX INDEXSTOCHASTIC MATRIX HAS NONNEGATIVE  ENTRIES AND THE ROWS SUM TO 1  A MATRIX IS DOUBLE STOCHASTIC IF  ROWS AND COLUMNS SUM TO 1  SHOW THAT EVERY DOUBLY STOCHASTIC MATRIX  M CAN BE WRITTEN AS A CONVEX SUM OF PERMUTATION MATRICES M  LAMBDA1 P1  LAMBDA2 P2  CDOTS  LAMBDAM PMWHERE SUMI1M LAMBDAI  1 LAMBDAI GEQ 0 AND PI IS APERMUTATION MATRIXENDEXERCISESSECTIONTOEPLITZ MATRICES AND SOME APPLICATIONSLABELSECTOEPLITZ DEFINITION WHERE THEY ARISE SOLUTION OF EQUATIONS GRENANDERSZEGO STUFF EIGENVALUES OF TOEPLITZINDEXTOEPLITZ MATRIXBEGINEXAMPLE LABELEXMTOEP1CONSIDER FILTERING A CAUSAL SIGNAL XT USING A FILTER H  1 23 2 1 USING LINEAR AS OPPOSED TO CIRCULAR CONVOLUTION  THEFILTERING RELATIONSHIP CAN BE EXPRESSED AS YBF  BEGINBMATRIXY0  Y1  Y2  VDOTS ENDBMATRIX BEGINBMATRIX 1  2  1 3  2  1 2  3  2  1 1  2  3  2  1  1  2  3  2  1  ENDBMATRIXBEGINBMATRIX X0  X1  X2  VDOTS ENDBMATRIXOBSERVE THAT THE ELEMENTS ON THE DIAGONALS OF THE MATRIX ARE ALL THESAME THE ELEMENTS OF H SHIFTED DOWN AND ACROSS  FINDING XTGIVEN YT WOULD REQUIRE SOLVING A SET OF LINEAR EQUATIONS INVOLVINGTHIS MATRIXENDEXAMPLEBEGINDEFINITION  A MATSIZEMM MATRIX R IS SAID TO BE A BF TOEPLITZ MATRIX  IF THE ENTRIES ARE CONSTANT ALONG EACH DIAGONAL  THAT IS R IS  TOEPLITZ IF THERE ARE SCALARS SM1LDOTSS0LDOTSSM1  SUCH THAT TIJ  SJI FOR ALL I AND J  THE MATRIXBEGINEQUATION R  BEGINBMATRIX S0  S1  S2  S3 S1  S0  S1  S2 S2  S1  S0  S1  S3  S2  S1  S0   ENDBMATRIXLABELEQTOEPDEF1ENDEQUATIONIS TOEPLITZENDDEFINITIONTOEPLITZ MATRICES ARISE IN BOTH MINIMUM MEANSQUARED ERROR ESTIMATIONAND LEASTSQUARES ESTIMATION AS THE GRAMMIAN MATRIX  FOR EXAMPLE INTHE LINEARPREDICTION PROBLEM A SIGNAL XT IS PREDICTED BASED UPONITS M PRIOR VALUES  LETTING XHATT DENOTE THE PREDICTED VALUEWE HAVE XHATT  SUMI1M AFI XTIWHERE AFI ARE THE FORWARD PREDICTION COEFFICIENTS  SEEEXERCISE REFEXLINPRED  THE FORWARD PREDICTION ERROR IS FMT XT  XHATT  THE EQUATIONS FOR THE PREDICTION COEFFICIENTS AREBEGINEQUATION  LABELEQYULEWALKER  BEGINBMATRIXR0  RBAR1 RBAR2 CDOTS  RBARM1 R1  R0  RBAR1  CDOTS  RBARM2 R2  R1  R0  CDOTS  RBARM3 VDOTS RM1  RM2  RM3  CDOTS  R0 ENDBMATRIXBEGINBMATRIX WF1  WF2  WF3  VDOTS  WFMENDBMATRIX  BEGINBMATRIX R1  R2  R3  VDOTS  RMENDBMATRIXENDEQUATIONWHERE WFI  AFI EQUATION REFEQYULEWALKER CAN BEWRITTEN ASBEGINEQUATION R WBF  RBFLABELEQLINPREDTOEPENDEQUATIONWHERE R  EXBFT1XBFHT1 AND RBF  EXTXBFT1 WITH XBFT  BEGINBMATRIX XBART  XBART1  VDOTS   XBFTM1 ENDBMATRIXNOTE THE CONJUGATES AND RJ  EXT XBARTJ  EQUATIONREFEQYULEWALKER IS KNOWN AS THE EM YULEWALKER EQUATIONINDEXYULEWALKER EQUATIONSBEFORE PROCEEDING WITH THE STUDY OF TOEPLITZ MATRICES IT IS USEFUL TOINTRODUCE A RELATED CLASS OF MATRICESBEGINDEFINITION  A MATSIZEMM MATRIX B IS SAID TO BE BF PERSYMMETRIC IF IT  IS SYMMETRIC ABOUT ITS NORTHEASTSOUTHWEST DIAGONAL  THAT IS  INDEXPERSYMMETRIC MATRIX BIJ  BMJ1MI1  THIS IS  EQUIVALENT TO BJBTJ WHERE J IS THE PERMUTATION MATRIX J  BEGINBMATRIX 0  0  CDOTS  0  10  0  CDOTS  1  0 VDOTS 0  1  CDOTS  0  0 1  0  CDOTS  0  0  ENDBMATRIXENDDEFINITIONTHE MATRIX J ALSO DENOTED JK IF THE DIMENSION IS IMPORTANT ISSOMETIMES REFERRED TO AS THE EM COUNTERIDENTITYBEGINEXAMPLE  THE MATRIX B  BEGINBMATRIX 1234 6783 91072 11961 ENDBMATRIXIS PERSYMMETRICENDEXAMPLEPERSYMMETRIC MATRICES HAVE THE PROPERTY THAT THE INVERSE OF APERSYMMETRIC MATRIX IS PERSYMMETRICBEGINEQUATION B1  J B1T JLABELEQPERSYMINVENDEQUATIONTOEPLITZ MATRICES ARE PERSYMMETRICWE WILL APPROACH THE STUDY OF THE SOLUTION TOEPLITZ SYSTEMS OFEQUATIONS FIRST IN THE CONTEXT OF THE LINEAR PREDICTION PROBLEMREFEQLINPREDTOEP WHICH WILL LEAD US TO THE SOLUTION OFHERMITIAN TOEPLITZ EQUATIONS USING AN ALGORITHM KNOWN AS DURBINSALGORITHM  FOLLOWING THE FORMULATION OF DURBINS ALGORITHM WE WILLEXAMINE SOME OF THE IMPLICATIONS OF THIS SOLUTION WITH RESPECT TO THELINEAR PREDICTION PROBLEM  WE WILL DETOUR SLIGHTLY TO INTRODUCE THENOTATION OF LATTICE FORMS OF FILTERS FOLLOWED BY CONNECTIONS BETWEENLATTICE FILTERS AND THE SOLUTION OF THE OPTIMAL LINEAR PREDICTOREQUATION  AFTER THIS DETOUR WE WILL RETURN TO THE STUDY OF TOEPLITZEQUATIONS THIS TIME WITH A GENERAL RHSTO ABBREVIATE THE NOTATION SOMEWHAT LET RM DENOTE THEMATSIZEMM MATRIX RM   BEGINBMATRIXR0  RBAR1 RBAR2 CDOTS  RBARM1 R1  R0  RBAR1  CDOTS  RBARM2 R2  R1  R0  CDOTS  RBARM3 VDOTS RM1  RM2  RM3  CDOTS  R0 ENDBMATRIXAND LET RBFM  BEGINBMATRIX R1  R2  VDOTS  RM ENDBMATRIXOBSERVE THATBEGINEQUATIONRM1  BEGINBMATRIX RM  JM RBFBARM  RBFMT JM R0 ENDBMATRIXLABELEQRTOEP1ENDEQUATIONWHERE JM IS THE MATSIZEMM COUNTERIDENTITYWE ARE MOTIVATED TO LOOK FOR FAST ALGORITHMS TO SOLVE A TOEPLITZ SETOF EQUATIONS RM WBF  RBFM FOR TWO REASONS  FIRST THE TOEPLITZMATRIX ARISES FREQUENTLY IN PRACTICE AND SIGNAL PROCESSING METHODSWHICH LEAD TO TOEPLITZ MATRICES COULD BENEFIT FROM FASTER ALGORITHMSSECOND THE HIGHLYSTRUCTURED NATURE OF THE MATRIX PROVIDES HOPE THATSOMEHOW BY EXPLOITING THE STRUCTURE AN ALGORITHM CAN BE DERIVEDWHICH REQUIRES FEWER COMPUTATIONS THAN FOR A GENERAL MATRIX SUCH ASUSING THE LU FACTORIZATION  IN FACT FAST TOEPLITZ SOLUTIONALGORITHMS CAN SOLVE SYSTEMS OF TOEPLITZ EQUATIONS IN OM2 TIMEAS OPPOSED TO ON3 FOR GENERAL NONSTRUCTURED MATRICES USING THELU FACTORIZATION  THE ALGORITHMS RELY ON THE FACT THAT A TOEPLITZMATRIX IS PERSYMMETRIC AND THAT THE INVERSE OF A TOEPLITZ MATRIXWHEN IT EXISTS IS ALSO PERSYMMETRIC  SUBSECTIONDURBINS ALGORITHMLABELSECDURBINWE ARE SOLVING THE EQUATION RM WBFM  RBFM WHERE RM IS THETOEPLITZ MATRIX FORMED BY ELEMENTS OF RBF AS IN REFEQRTOEP1AND WBFM IS NOW THE VECTOR OF UNKNOWNS  WE PROCEED INDUCTIVELYINDEXPROOFBY INDUCTIONASSUME WE HAVE A SOLUTION FOR RK WBFK  RBFK 1 LEQ K LEQM1  WE WANT TO USE THIS SOLUTION TO FIND RK1  GIVEN THAT WEHAVE SOLVED THE KTH ORDER YULEWALKER SYSTEM RK WBFK  RBFKWHERE RBFK  R1R2LDOTSRKT WE WRITE THE K1STYULEWALKER EQUATION ASBEGINEQUATIONBEGINBMATRIX RK  JK RBFOLK  RBFTK JK  R0  ENDBMATRIX BEGINBMATRIX ZBFK  ALPHAKENDBMATRIX  BEGINBMATRIX RBFK  RK1 ENDBMATRIXLABELEQTOEPT1ENDEQUATIONWHERE JK IS THE MATSIZEKK COUNTERIDENTITY  THE DESIREDSOLUTION IS WBFK1  BEGINBMATRIX ZBFK  ALPHAK ENDBMATRIXMULTIPLYING OUT THE FIRST SET OF EQUATIONS IN REFEQTOEPT1 WESEE THAT ZBFK  RK1RBFK  ALPHAK JK RBFOLK  WBFK  ALPHAKRK1 JK RBFOLKBY THE INDUCTIVE HYPOTHESIS  SINCE RK1 IS PERSYMMETRIC RK1JK  JK RK1AND HENCEBEGINEQUATION ZBFK  WBFK  ALPHAK JK WBFBARKLABELEQTOEPSOL1ENDEQUATIONWE OBSERVE THAT THE FIRST KELEMENTS OF WBFK1 ARE OBTAINED AS A CORRECTION BY ALPHAK JKWBFBARK OF THE ORIGINAL ELEMENTS WBFK  FROM THE SECOND SET OFEQUATIONS IN REFEQTOEPT1BEGINEQUATION ALPHAK  FRAC1R0RK1  RBFTK JK ZBFKLABELEQATENDEQUATIONWHICH BY SUBSTITUTING FOR ZBFK FROM REFEQTOEPSOL1 GIVESBEGINEQUATION ALPHAK  FRACRK1  RBFTK JK WBFKR0  RBFTKWBFBARK FRACRK1  RBFKT JK WBFKBETAKLABELEQALPHATOEPENDEQUATIONWHERE BEGINEQUATION  LABELEQBETADEFBETAK  R0  RBFTKWBFBARK   ENDEQUATIONFOR FUTURE USE OBSERVE THATBEGINEQUATIONALPHAK BETAK  RK1  RBFKT JK WBFKLABELEQTOEPABENDEQUATIONTHE PARAMETER ALPHAK IS KNOWN AS THE KTH EM REFLECTION  COEFFICIENT INDEXREFLECTION COEFFICIENTAT THIS POINT SUFFICIENT INFORMATION IS AVAILABLE TO WRITE ANALGORITHM TO RECURSIVELY SOLVE REFEQTOEPT1  HOWEVER SOMESIMPLIFICATIONS CAN BE MADE IN THE COMPUTATION OF BETAKBEGINALIGN  BETAK  R0  RBFKT WBFBARK  R0  RBFK1T   RKBEGINBMATRIX ZBFBARK1  ALPHABARK1 ENDBMATRIXNONUMBER   R0  RBFK1T RKBEGINBMATRIX WBFBARK1     ALPHABARK1JK1 WBFK1   ALPHABARK1  ENDBMATRIXNONUMBER     R0  RBFK1T WBFBARK1  ALPHABARK1RBFK1T  JK1WBFK1  RK NONUMBER    BETAK1  ALPHABARK1ALPHAK1BETAK1 NONUMBER      BETAK1 1 ALPHAK12 LABELEQBETAD2ENDALIGNWHERE THE PENULTIMATE EQUALITY FOLLOWS FROM REFEQTOEPAB  THISALGORITHM CITEDURBIN1960 CAN BE SUMMARIZED AS SHOWN IN ALGORITHMREFALGDURBINBEGINNEWPROGENVDURBINS ALGORITHM DURBINM DURBINDURBINS ALGORITHMENDNEWPROGENVBEGINPROGTABSSOLUTION OF THE YULEWALKER EQUATIONS USING THE DURBIN ALGORITHM INPUT RBF  R0R1LDOTSRN NOTE  RBF IS INDEXED STARTING AT 0 X1  R1R0 BETA  R0 ALPHA  R1R0 FOR K1N1    BETA  1ALPHA2BETA    ALPHA  RK1  RK11T X1KBETA    FOR I1K       ZI  XI  ALPHA XBARK1I    END    X1K  Z1K    XK1  ALPHAENDENDPROGTABSTHE COMPLEXITY OF THE ALGORITHM IS O2N2SEE DURBINMSUBSECTIONPREDICTORS AND LATTICE FILTERSLABELSECPREDFILTIN THIS SECTION WE EXAMINE SOME SIGNALPROCESSING ORIENTED RESULTS OFTHE DURBIN ALGORITHM  THE REFLECTION COEFFICIENTS ALPHAK THATAROSE IN THE DERIVATION OF DURBINS ALGORITHM HAVE A USEFULINTERPRETATION IN THE CONTEXT OF LINEAR PREDICTION  LET XT BE ASTATIONARY STOCHASTIC PROCESS AND LET XBFMT1 XT1XT2LDOTSXTMT  FOR THE PRESENT DISCUSSION THEVECTORS ARE TAKEN AS REAL FOR CONVENIENCE IT IS STRAIGHTFORWARD TOGENERALIZE TO COMPLEX VECTORS  THE OPTIMUM MSE MTH ORDER FORWARDLINEAR PREDICTOR IS OF THE FORM XHATT  WBFFMT XBFMT1WHEREBEGINEQUATION RMWBFFM  RBFMLABELEQTOEP3ENDEQUATIONAND RM  EXBFMT1 XBFMTT1QQUADTEXTANDQQUADRBFM  EXTXBFMT1EQUATION REFEQTOEP3 IS OF COURSE THE YULEWALKER EQUATIONSOLVED BY THE DURBIN ALGORITHM  WE EXPLORE THE MINIMUM MEANSQUAREDERROR IN LIGHT OF THE DURBIN ALGORITHM PARAMETERS IN THE FOLLOWINGTHEOREMBEGINTHEOREM LABELTHMLATTMIN  THE MINIMUM MEANSQUARE ERROR FOR THEMTH ORDER FORWARD PREDICTOR IS  SIGMAFM2  SIGMAFM121ALPHAM12  BETAMWHERE ALPHAM1 IS THE REFLECTION COEFFICIENT FROM THE DURBINALGORITHMENDTHEOREMBEGINPROOF INDEXPROOFBY INDUCTION  BY INDUCTION  FOR THE 0TH ORDER PREDICTOR THE ERROR IS SIGMAF02  EXTXT  R0  BETA0FOR THE FIRSTORDER PREDICTORBEGINEQUATION SIGMAF12  EXHATT  XT2  R01ALPHA02  BETA1LABELEQTOEPPROOF1ENDEQUATIONSEE EXERCISE REFEXTOEPPROOFEXASSUMING THE THEOREM TO BE TRUE FOR THE K10ST ORDER PREDICTOR WEWRITEBEGINALIGNEDSIGMAFK12  EXHATT  XT2  EWBFTFK1 XBFT1 XBFT2  R0  RBFK1T WBFFK1ENDALIGNEDNOW WRITING WBFFK IN TERMS OF ITS SOLUTION IN THE DURBIN ALGORITHMWE OBTAINBEGINALIGN SIGMAFK2  R0  RBFK1T  RKBEGINBMATRIXZBFK1    ALPHAK1 ENDBMATRIX NONUMBER   R0  RBFK1T  RKBEGINBMATRIX WBFFK1   ALPHAK1 JK1WBFFK1    ALPHAK1 ENDBMATRIX NONUMBER  R0  RBFK1T WBFFK1ALPHAK1RBFK1JK1WBFFK1  RK NONUMBER  R0  RBFK1T WBFFK1LEFT1ALPHAK1FRACRBFK1T JK1 WBFFK1     RKR0  RBFK1T WBFFK1RIGHT NONUMBER  R0  RBFK1T WBFFK11ALPHAK12 LABELEQTP1  BETAK LABELEQTP2ENDALIGNWHERE REFEQTP1 FOLLOWS FROM REFEQALPHATOEP ANDREFEQTP2 FOLLOWS FROM REFEQBETADEF AND REFEQBETAD2ENDPROOFSINCE AS WILL BE SHOWN BELOW ALPHAKLEQ1 THEN AS THE ORDERM GROWS THERE WILL BE LESS ERROR IN THE PREDICTOR AS THE NUMBER OFSTAGES INCREASES UNTIL THE PREDICTOR IS ABLE TO PREDICT EVERYTHINGABOUT THE SIGNAL THAT IS PREDICTABLE  THE PREDICTION ERROR AT THATPOINT WILL BE WHITE NOISETO MOTIVATE THE CONCEPT OF THE LATTICE FILTERS CONSIDER NOW THEPROBLEM OF GROWING A PREDICTOR FROM KTH ORDER TO K1STORDER UP TO A FINAL PREDICTOR OF ORDER M STARTING FROM AFIRSTORDER PREDICTOR  THE FIRSTORDER PREDICTOR IS XHATT  A11 XT1THE SECONDORDER PREDICTOR IS XHATT  A21 XT1  A22 XT2ACCORDING TO THE RECURSION REFEQTOEPSOL1 ALL OF THECOEFFICIENTS IN THE 2NDORDER PREDICTOR A21A22 ARE INGENERAL DIFFERENT FROM THE COEFFICIENTS IN THE 1STORDER PREDICTORA11  IN THE GENERAL CASE IF WE DESIRE TO EXTEND AN KTHORDER FILTER TO AN K1ST ORDER FILTER ALL OF THE COEFFICIENTSWILL HAVE TO CHANGE  WE WILL DEVELOP A FILTER STRUCTURE KNOWN AS AEM LATTICE FILTER INDEXLATTICE FILTER TO WHICH NEW FILTERSTAGES MAY BE ADDED WITHOUT HAVING TO RECOMPUTE THE COEFFICIENTS FORTHE OLD FILTERWE BEGIN BY REVIEWING SOME BASIC NOTATION FOR PREDICTORS  LETBEGINEQUATION FKT  XT  XHATKT  SUMI0M AKI XTI QQUAD AK0  1LABELEQFORPREDENDEQUATIONDENOTE THE EM FORWARD PREDICTION ERROR OF AN KTH ORDER PREDICTORTHE OPTIMAL MMSE FORWARD PREDICTOR COEFFICIENTS SATISFY RABFK RBF WHERE ABFK  AK1AK2LDOTSAKKTIN EXERCISE REFEXLINPRED THE CONCEPT OF A EM BACKWARDPREDICTOR IN WHICH XTK IS PREDICTED USINGXTXT1LDOTSXTK1 WAS PRESENTED  THE BACKWARD PREDICTORIS XHATBTK  SUMI0K1 BKI XTILETBEGINEQUATIONGKT  XTK   XHATBTK  SUMI0K BKI XTI QQUADBKK  1 LABELEQBACKPREDENDEQUATIONDENOTE THE BACKWARD PREDICTION ERROR  AS SHOWN IN EXERCISEREFEXLINPRED THE OPTIMAL MMSE BACKWARD PREDICTION COEFFICIENTSSATISFY R BBFK  JKRBFBARWHERE JK IS THE MATSIZEKK COUNTERIDENTITY HENCE THE OPTIMALFORWARD PREDICTOR COEFFICIENTS ARE RELATED TO THE OPTIMAL BACKWARDPREDICTOR COEFFICIENTS BYBEGINEQUATION ABFK  JBBFBARKLABELEQBACKTOFORENDEQUATIONTHAT IS THE BACKWARD PREDICTION COEFFICIENTS ARE THE FORWARDPREDICTION COEFFICIENTS CONJUGATED AND IN REVERSE ORDERWE WILL NOW DEVELOP THE LATTICE FILTER BY BUILDING UP A SEQUENCE OFSTEPS  THE FIRSTORDER FORWARD AND BACKWARD PREDICTION ERRORS AREBEGINEQUATIONBEGINSPLITF1T  XT  A11 XT1 G1T  XT1  B10 XTENDSPLITLABELEQBACKTOFOR2ENDEQUATIONIN LIGHT OF REFEQBACKTOFOR THE SECOND EQUATION CAN BE WRITTEN AS G1T  XT1  ABAR11 XTNOW CONSIDER THE FILTER STRUCTURE SHOWN IN FIGURE REFFIGLATT1ATHIS STRUCTURE IS KNOWN AS A EM LATTICE FILTER  THE OUTPUTS OFTHAT FILTER STRUCTURE CAN BE WRITTEN ASBEGINEQUATIONBEGINSPLIT  F1T  F0T  KAPPA1 G0T1  XT  KAPPA1 XT1   G1T  G0T1  KAPPABAR1 F0T  XT1  KAPPABAR1 XTENDSPLITLABELEQBACKTOFOR3ENDEQUATIONHENCE BY EQUATING KAPPA1  A11 WE FIND THAT THIS FIRSTORDERLATTICE FILTER COMPUTES BOTH THE FORWARD AND THE BACKWARD PREDICTIONERROR FOR FIRSTORDER PREDICTORSSECONDORDER FORWARD AND BACKWARD PREDICTORS SATISFYBEGINEQUATION  LABELEQBACKTOFOR4BEGINSPLIT  F2T  XT  A21 XT1  A22 XT2   G2T  XT2  B20 XT  B21 XT1           XT2  ABAR22 XT  ABAR21 XT1       ENDSPLIT     ENDEQUATIONFOR THE LATTICE STRUCTURE IN FIGURE REFFIGLATT1B THEOUTPUT CAN BE WRITTEN ASBEGINALIGNF2T  XT  KAPPA1 XT1  KAPPA2 G1T1 NONUMBER    XT  KAPPA1KAPPABAR1 KAPPA2 XT1  KAPPA2 XT2  LABELEQLT2 INTERTEXTAND SIMILARLYG2T  XT2K1KBAR2  KBAR1 XT1  KBAR2 XT LABELEQLT3ENDALIGNBY EQUATING REFEQBACKTOFOR4 AND REFEQLT2 WE OBTAINA21  KAPPA1  KAPPABAR1K2 AND A22  KAPPA2 AGAINWE HAVE THE LATTICE FILTER COMPUTING BOTH THE FORWARD AND BACKWARDPREDICTION ERRORBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    SUBFIGUREFIRST STAGEINPUTPICTUREDIRLATTFILT1    SUBFIGURESECOND STAGEINPUTPICTUREDIRLATTFILT2    CAPTIONFIRST TWO STAGES OF A LATTICE PREDICTION FILTER    LABELFIGLATT1  ENDCENTERENDFIGUREWE NOW GENERALIZE TO PREDICTORS OF ORDER K  THE FORWARD ANDBACKWARD PREDICTORS OF REFEQFORPRED AND REFEQBACKPRED CANBE WRITTEN USING THE ZTRANSFORM ASBEGINEQUATIONBEGINSPLIT FKZ  AKZ XZ  GKZ  BKZ XZENDSPLITLABELEQFBZENDEQUATIONWHERE AKZ  SUMI0K AKI ZI  BECAUSE OF THERELATIONSHIP REFEQBACKTOFOR WE CAN WRITE BKZ  ZK ABARKZ1THAT IS THE POLYNOMIAL WITH THE COEFFICIENTS CONJUGATED AND INREVERSE ORDER  THE KTH ORDER LATTICE FILTER STAGES SHOWN IN FIGUREREFFIGLATT2 SATISFIES THE EQUATIONSBEGINEQUATION  LABELEQLATTFILT  BEGINSPLITFKZ  FK1Z  KAPPAK Z1GK1Z QQUAD K12LDOTSM GKZ  KAPPABARK FK1Z  Z1 GK1Z QQUAD K12LDOTSMENDSPLITENDEQUATIONBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRLATTFILT3    CAPTIONTHE KTH STAGE OF A LATTICE FILTER    LABELFIGLATT2  ENDCENTERENDFIGUREDIVIDING BOTH SIDES OF REFEQLATTFILT BY XZ WE OBTAINBEGINALIGNAKZ  AK1Z  KAPPAK Z1 BK1Z LABELEQLATTFILT2ABKZ  KBARK AK1Z  Z1 BM1Z LABELEQLATTFILT2BENDALIGNEQUATION REFEQLATTFILT2A CAN BE WRITTEN IN TERMS OF ITSCOEFFICIENTS ASBEGINEQUATION  LABELEQLATT2DIRBEGINSPLIT  AK0  1   AKI  AK1I  KAPPAK ABARK1KIQQUAD  I12LDOTSK1   AKK  KAPPAKENDSPLITENDEQUATIONWHEN ITERATED FROM K01LDOTSM REFEQLATT2DIR CONVERTS FROMLATTICE FILTER COEFFICIENTS KAPPA1 KAPPA2 LDOTS KAPPAM TOTHE DIRECTFORM FILTER PREDICTOR COEFFICIENTSAM1AM2LDOTSAMM  SC MATLAB CODE IMPLEMENTING THISCONVERSION IS SHOWN IN ALGORITHM REFALGREFLTODIRBEGINNEWPROGENVCONVERSION OF LATTICE FIR TO    DIRECTFORMREFLTODIRM REFLTODIRCONVERSION OF LATTICE FIR TO DIRECTFORM ENDNEWPROGENVTO CONVERT FROM A DIRECTFORM IMPLEMENTATION TO THE LATTICE FORMIMPLEMENTATION WE WRITE REFEQLATTFILT2A ANDREFEQLATTFILT2B ASBEGINEQUATION AK1Z  FRACAKZ  KAPPAK BKZ1KAPPAK2LABELEQAM1KENDEQUATIONRECALLING THAT KAPPAK  AKK AND WRITING REFEQAM1K IN TERMS OFTHE COEFFICIENTS WE OBTAIN THE FOLLOWING DOWNSTEPPING RECURSION FORFINDING THE REFLECTION COEFFICIENTS FROM THE DIRECTFORM FILTERCOEFFICIENTS  FOR KMM1LDOTS1  BEGINALIGNED  KAPPAK  AKK   AK1I  FRACAKI  KAPPAK ABARKMI1KAPPAK2ENDALIGNEDTHIS RECURSION WORKS PROVIDED THAT KAPPAK NEQ 1BEGINNEWPROGENVCONVERSION OF DIRECTFORM FIR TO LATTICEDIRTOREFLM DIRTOREFLCONVERSION OF DIRECTFORM FIR TO LATTICE ENDNEWPROGENVBEGINEXAMPLE  SUPPOSE WE KNOW THAT KAPPA1  23 KAPPA2  45 AND  KAPPA3  15  THEN   INVOKING TT REFLTODIR WITH THE ARGUMENT KBF  23 45 15 WE  OBTAIN ABF  BEGINBMATRIX1  136  104  02  ENDBMATRIXT CORRESPONDING TO THE FILTER A3Z  1  136 Z1  104 Z2  2 Z3SUPPLYING ABF AS AN ARGUMENT TO TT DIRTOREFL WE OBTAIN KBF  BEGINBMATRIX0666667  08  02 ENDBMATRIXAS EXPECTEDENDEXAMPLESUBSECTIONOPTIMAL PREDICTORS AND TOEPLITZ INVERSESTHE LATTICE REPRESENTATION OF AN FIR FILTER APPLIES TO EM ANY  FIRFILTER THAT IS NORMALIZED SO THAT THE LEADING FILTER COEFFICIENT IS 1HOWEVER FOR THE CASE OF OPTIMAL LINEAR PREDICTORS THERE IS A USEFULRELATIONSHIP BETWEEN THE CONVERSION BETWEEN THE DIRECTFORMREALIZATION AND THE LATTICE REALIZATION  RECALL THAT FOR THE SOLUTIONOF THE YULEWALKER EQUATION THE UPDATESTEP TO GO FROM THE KTHORDER PREDICTOR TO THE K1ST ORDERPREDICTOR IS SEE REFEQTOEPSOL1BEGINEQUATION ZBFK  WBFK  ALPHAK JK WBFBARKLABELEQZB1ENDEQUATIONWHERE WBFK IS THE SOLUTION TO THE K1ST YULEWALKER EQUATIONCONTRAST THIS WITH THE UPDATE EQUATION IN CONVERTING FROM LATTICE TODIRECT FORM FROM REFEQLATTFILT2A AND REFEQLATT2DIRBEGINEQUATION AKI  AK1I  KAPPAK ABARK1KIQQUAD I12LDOTSK1LABELEQAK1ENDEQUATIONTHE COMPARISON BETWEEN REFEQZB1 AND REFEQAK1 MAY BE MADEMORE DIRECT BY WRITING REFEQZB1 IN TERMS OF ITS COMPONENTSRECALLING THAT WBFK1MC K1  ZBFK1  THEN REFEQZB1BECOMESBEGINEQUATION ZKI  ZK1I ALPHAK ZBFBARK1KI QQUAD I12LDOTSK1LABELEQZB2ENDEQUATIONTHEN COMPARISON OF REFEQAK1 AND REFEQZB2 REVEALS THATBEGINEQUATION ALPHAK  KAPPABARKLABELEQALPHAKAPPENDEQUATIONTHUS THE MMSE PREDICTOR ERROR XT  XHATMT IS PRECISELYCOMPUTED BY THE LATTICE FILTER WITH COEFFICIENTS ALPHABARKK12LDOTSM  FURTHERMORE AT EACH STAGE THE FORWARD PREDICTORERROR FKT  XT  XHATKT AND THE BACKWARD PREDICTION ERRORBKT  XTM  XHATMKT ARE PRODUCED BY THE LATTICE FILTERCONSIDER NOW THE PROBLEM OF CHOOSING THE LATTICE COEFFICIENTALPHAK TO MINIMIZE THE MSE AT THE OUTPUT OF THE KTH STAGE OF THELATTICE FILTER INSTEAD OF IN THE DIRECTFORM FILTER IN FKT  FK1T  ALPHAK GK1TMINIMIZING LA FKT FKTRA  EFK2 WITH RESPECT TOALPHAK YIELDS ALPHAK  FRACEFK1T  GBARK1T1EGK1T12BY THE CAUCHYSCHWARZ INEQUALITY 1 LEQ ALPHAK LEQ 1  INLIGHT OF THEOREM REFTHMLATTMIN INCREASING THE ORDER OF THEPREDICTOR CANNOT INCREASE THE PREDICTION ERROR POWERBY THE PROPERTIES OF OPTIMAL LINEAR PREDICTORS  THE ERROR ISORTHOGONAL TO THE DATA WHERE THE DATA IS XTI I12LDOTSMFOR THE FORWARD PREDICTOR AND XTI I01LDOTSM1 FOR THEBACKWARD PREDICTOR  WE CAN OBTAIN IMMEDIATELY THE FOLLOWINGORTHOGONALITY RELATIONSHIPS WHERE LA XYRA  EX YBAR BEGINALIGNEDLA FMT XTI RA  0 QQUAD I12LDOTSM LA GMT XTI RA  0 QQUAD I01LDOTSM1 LA FMT XT RA  SIGMAFM2 LA FIT FJT RA  SIGMAFMAXIJ2 LA GMTGJT RA    BEGINCASES    0  0 LEQ J LEQ M1 SIGMABM2  JM  ENDCASESENDALIGNEDTHUS THE BACKWARD PREDICTION ERROR IS A WHITE SEQUENCE HAVING THE SAMESPAN AS THE INPUT DATASUBSECTIONTOEPLITZ EQUATIONS WITH A GENERAL RHSINDEXLEVINSON ALGORITHM WE NOW GENERALIZE THE SOLUTION OF TOEPLITZSYSTEMS OF EQUATIONS TO EQUATIONS HAVING A RHS WHICH IS NOT FORMEDFROM COMPONENTS OF THE LHS MATRIX  THIS GIVES THE EM LEVINSON ALGORITHMIN THE EQUATIONBEGINEQUATIONRM YBF  BBFLABELEQLEV1ENDEQUATIONRM IS A TOEPLITZ MATRIX AND BBF IS SOME ARBITRARY VECTOR  ASBEFORE THE SOLUTION IS FOUND INDUCTIVELY BUT IN THIS CASE THEUPDATE STEP REQUIRES KEEPING TRACK OF THE SOLUTION BOTH THE SOLUTIONTO THE EQUATION YULEWALKER EQUATION RK WBFK  RBFKUSING THE SAME APPROACH AS FOR THE DURBIN ALGORITHM AND ALSO THEEQUATIONRK YBFK  BBFKWHICH IS THE ONE WE REALLY WANT TO SOLVE  ASSUMING THAT THE WBFK AND YBFK ARE KNOWN FOR STEP K THESOLUTION TO THE K1ST STEP REQUIRES SOLVING BEGINBMATRIXRK  JK RBFOLK RBFTK JK  R0 ENDBMATRIXBEGINBMATRIXVBFK  MUK ENDBMATRIX BEGINBMATRIX BBFK  BK1 ENDBMATRIXWHERE YBFK1  BEGINBMATRIXVBFK  MUK ENDBMATRIXUSING THE SOLUTIONS FROM TIME K VBFK  RK1BBFK  MUK JK RBFOLK  YBFK  MU JKXBFBARKTHEN PROCEEDING AS BEFORE MUK  FRACBK1  RBFKT JK YBFKR0  RBFKT  WBFBARKTHE ALGORITHM THAT SOLVES FOR THE GENERAL RIGHTHAND SIDE ISATTRIBUTED TO LEVINSON CITELEVINSON1947BEGINNEWPROGENVLEVINSONS ALGORITHM LEVINSONM LEVINLEVINSONS ALGORITHMENDNEWPROGENVBEGINPROGTABSSOLUTION OF THE  WIENERHOPF EQUATIONS USING THE LEVINSON ALGORITHM INPUT  RBF  R0R1LDOTSRN BBF B1B2LDOTSBN NOTE RBF IS INDEXED STARTING AT 0 X1  R1R0 Y1  B1R0 BETA  R0 ALPHA  R1R0 FOR K1N1    BETA  1ALPHA2BETA    MU  BK1  RK11T Y1KBETA    V1K  Y1K  MU XBARK11    Y1K  V1K    YK1  MU    IF KN1  QQUADQQUADLAST TIME THROUGH DONT NEED TO COMPUTEIF SOLVING REFEQLEV1       ALPHA  RK1  RK11T X1KBETA        ZI  XI  ALPHA XBARK1I         X1K  Z1K        XK1  ALPHA     END ENDENDPROGTABSBEGINEXERCISESITEM LABELEXTOEPDETAIL SHOW THAT IF R IS PERSYMMETRIC THEN B1J       JB1T WHERE J IS THE COUNTERIDENTITY ITEM IF R IS HERMITIAN THEN OVERLINER1  R1TITEM THE MATSIZEMM MATRICES B  BEGINBMATRIX 010CDOTS0 001CDOTS0 000DDOTS  0 VDOTS 000CDOTS1 000CDOTS0 ENDBMATRIXQQUAD TEXTANDQQUADF  BEGINBMATRIX000CDOTS0 100CDOTS00 010CDOTS00 00DDOTSDDOTS00000CDOTS10 ENDBMATRIXARE CALLED EM BACKWARD SHIFT AND EM FORWARD SHIFT MATRICESRESPECTIVELYBEGINENUMERATEITEM LET ABF  1234T  COMPUTE BABF AND FABF FOR  MATSIZE44 BACKWARD AND FORWARD SHIFT MATRICES  COMPUTE B2  ABF AND F2 ABF  COMMENT ON THE NAME OF THE MATRICESITEM LABELEXTOEPPROOFEX SHOW THAT THE VARIANCE OF THE FIRSTORDER  FORWARD PREDICTION ERROR FILTER REFEQTOEPPROOF1 IS CORRECT  HINT SHOW THAT ALPHA1  R1R0ITEM SHOW THAT AN MATSIZEMM MATRIX OF THE FORM IN  REFEQTOEPDEF1 CAN BE WRITTEN AS R  SUMK1M SK FK  SUMK0M AK BKENDENUMERATEITEM LET R0 RPM 1 RPM 2 LDOTS DENOTE THE  AUTOCORRELATION SEQUENCE OF A STATIONARY STOCHASTIC PROCESS AND LET   SOMEGA  SUMN EJOMEGA N RN    BE ITS POWER SPECTRAL DENSITY  SHOW THAT IF SOMEGA GEQ 0 THEN  THE TOEPLITZ MATRIX R WITH ELEMENTS RIJ  RIJ IS POSITIVE  SEMIDEFINITE  ITEM THE ALGORITHMS TT DURBIN AND TT LEVINSON ARE DESIGNED    FOR A SYMMETRIC TOEPLITZ MATRIX  DEVELOP SIMILAR ALGORITHMS    SUITABLE FOR NONSYMMETRIC MATRICES  ITEM SHOW THAT FOR A SYMMETRIC TOEPLITZ MATRIX RK1 BEGINBMATRIX I  JK WBFK  ZEROBF  1 ENDBMATRIXH RK1BEGINBMATRIXI  JK WBFK  ZEROBF  1 ENDBMATRIX BEGINBMATRIX TK  0  0 WBFKH RBFKR0 ENDBMATRIXHENCE CONCLUDE THAT IF RK1 IS POSITIVE DEFINITE THENR0RBFKT WBFBARK IN REFEQALPHATOEP IS NOT ZEROENDEXERCISESSECTIONVANDERMONDE MATRICESLABELSECVANDERMONDEINDEXVANDERMONDE MATRIXBEGINDEFINITION  AN MATSIZEMM BF VANDERMONDE MATRIX V HAS THE FORMBEGINEQUATION V  BEGINBMATRIX 1  1  CDOTS  1  Z0  Z1  CDOTS ZM1 Z02  Z12  CDOTS ZM12 VDOTS Z0M1  Z1M1  CDOTS  ZM1M1 ENDBMATRIXLABELEQVANDERMONDEENDEQUATIONTHIS MAY BE  WRITTEN AS V  VZ0Z1LDOTSZM1ENDDEFINITIONBEGINEXAMPLE  VANDERMONDE MATRICES ARISE FOR EXAMPLE IN POLYNOMIAL INTERPOLATION  SUPPOSE THAT THE M POINTS X1Y1X2Y2LDOTSXMYM  ARE TO BE FITTED EXACTLY TO A POLYNOMIAL OF DEGREE M1 SO THAT PXI  SUMK0M1 AK XIK  YIQQUAD I12LDOTSNTHIS PROVIDES THE SYSTEM OF EQUATIONS BEGINBMATRIX 1  X1  X12  CDOTS  X1M1 1  X2  X22  CDOTS  X2M1 VDOTS 1  XM  XM2  CDOTS  XMM1 ENDBMATRIX BEGINBMATRIX A0  A1  VDOTS  AM1 ENDBMATRIX  BEGINBMATRIX Y1  Y2  VDOTS  YM ENDBMATRIXOR VT ABF  YBFWHERE V IS OF THE FORM REFEQVANDERMONDEENDEXAMPLETHE DETERMINANT OF A VANDERMONDE MATRIX REFEQVANDERMONDE ISBEGINEQUATION BOXEDDETV  PRODBEGINSUBARRAYC IJ1  I  J    ENDSUBARRAYN ZIZJLABELEQVANDETENDEQUATIONFROM THIS IT IS CLEAR THAT IF ZI NEQ ZJ FOR INEQ J THEN THEDETERMINANT IS NONZERO AND THE MATRIX IS INVERTIBLE  EFFICIENT ALGORITHMS FOR SOLUTION OF VANDERMONDE SYSTEMS OF EQUATIONS V XBF  BBF AND VT XBF  BBF HAVE BEEN DEVELOPED  BECAUSE THESE ALGORITHMS ARE CLOSELY TIED WITHINTERPOLATION THEY ARE PRESENTED IN THAT CONTEXT IN CHAPTERREFCHAPINTERP BEGINEXERCISESITEM SHOW THAT THE FORMULA REFEQVANDET FOR THE DETERMINANT OF  THE VANDERMONDE MATRIX IS CORRECT  HINT USE INDUCTION AND THE  COFACTOR EXPANSIONITEM DETERMINE A POLYNOMIAL INTERPOLATING THE POINTS  12124 41 3227ENDEXERCISESSECTIONCIRCULANT MATRICESLABELSECCIRCULANTINDEXCIRCULANT MATRIXBEGINDEFINITION  A BF CIRCULANT MATRIX C IS OF  THE FORM C  BEGINBMATRIX C1  C2  CDOTS  CM CM  C1  CDOTS  CM1 CM1  CM  CDOTS  CM2 VDOTS C2  C3  CDOTS  C1ENDBMATRIXWHERE EACH ROW IS OBTAINED BY CYCLICALLY SHIFTING TO THE RIGHT THEPREVIOUS ROW  THIS IS ALSO DENOTED AS C  CIRCULANTC1C2LDOTSCMA MATRIX C IS CALLED A GCIRCULANT IF IT IS OF THE FORM C  BEGINBMATRIX C1  C2  CDOTS  CM CMG1  CMG2  CDOTS  CMG CM2G1  CM2G2  CDOTS  CM2G VDOTS CG1  CG2  CDOTS  CG ENDBMATRIXTHAT IS THE ROWS ARE SHIFTED CYCLICALLY SHIFTING BY G  A1CIRCULANT MATRIX IS REFERRED TO SIMPLY AS A CIRCULANT MATRIXENDDEFINITIONBEGINEXAMPLELET H  1234  H0H1H2H3  DENOTE THE IMPULSE RESPONSE THAT IS TO BE CYCLICALLY CONVOLVED  WITH A SEQUENCE X  X0X1X2X3  THE OUTPUT SEQUENCE Y  H CYCC X MAY BE COMPUTED IN MATRIX FORM AS YBF  BEGINBMATRIX1  4  3  2  2  1  4  3 3 2  1  4 4  3  2  1 ENDBMATRIXBEGINBMATRIX X0X1 X2X3ENDBMATRIXEM EVERY CYCLIC CONVOLUTION CORRESPONDS TO MULTIPLICATION BY A CIRCULANTMATRIXENDEXAMPLEIT CAN BE SHOWN THAT A MATRIX A IS CIRCULANT IF AND ONLY IF API PI A WHERE PI  CIRCULANT010LDOTS0 IS A PERMUTATIONMATRIX  IT IS ALSO THE CASE THAT IF C IS A CIRCULANT MATRIX THENCH IS A CIRCULANT MATRIX  A CIRCULANT MATRIXCIRCULANTC1C2LDOTSCM  CIRCULANTCBF CAN BE REPRESENTEDASBEGINEQUATIONCIRCULANTC1C2LDOTSCM  C1 I  C2 PI  CDOTS  CMPIM1LABELEQCIRC1ENDEQUATIONLET PCBFZ  C1  C2 Z  CDOTS  CM ZM1  THE POWER SERIESPCBFZ1 IS THE ZTRANSFORM OF THE SEQUENCE OF CIRCULANTELEMENTS  FROM REFEQCIRC1 THE CIRCULANT MATRIX CAN BE WRITTENAS C  CIRCULANTCBF  PCBFPIBEGINLEMMA IF C1 AND C2 ARE CIRCULANT MATRICES OF THE SAME  SIZE THEN  BEGINENUMERATE  ITEM C1C2C2C1 CIRCULANT MATRICES COMMUTE  ITEM CIRCULANTS ARE NORMAL MATRICES  A NORMAL MATRIX    INDEXNORMAL MATRIX IS A MATRIX C SUCH THAT CCH  CHC  ENDENUMERATEENDLEMMABEGINPROOF  WRITE C1  PCBF1PI AND C2  PCBF2PI  THEN C1C2  PCBF1PIPCBF2PIWHICH IS JUST A POLYNOMIAL IN THE MATRIX PI  BUT POLYNOMIALS INTHE SAME MATRIX COMMUTE PCBF1PIPCBF2PI  PCBF2PIPCBF1PISO C1C2  C2 C1SINCE C AND CH ARE BOTH CIRCULANTS IT FOLLOWS FROM PART 1 THATCCH  CHC OR C IS NORMALENDPROOFDIAGONALIZATION OF CIRCULANT MATRICES IS STRAIGHTFORWARD USING THEFOURIER TRANSFORM MATRIX  LETINDEXCIRCULANT MATRIXEIGENVALUESBEGINEQUATION F  BEGINBMATRIX1  1  1 CDOTS  1 1  OMEGA  OMEGA2  CDOTS  OMEGAM1 1  OMEGA2  OMEGA4  CDOTS  OMEGA2M1 VDOTS 1  OMEGAM1  OMEGA2M1  CDOTS  OMEGAM1M1 ENDBMATRIXLABELEQFMATENDEQUATIONWHERE OMEGA  EJ2PIM  NOTE THAT F IS A VANDERMONDE MATRIXAND THAT  FFH  MIBEGINTHEOREM LABELTHMCIRCDIAG  IF C IS A MATSIZENN CIRCULANT WITH C  PCBFPI  THEN IT IS DIAGONALIZED BY F C  FRAC1N F LAMBDA FHWHERE LAMBDA  DIAGPCBF1PCBFOMEGALDOTSPCBFOMEGAN1CONVERSELY IF LAMBDA  DIAGLAMBDA1LAMBDA2LDOTSLAMBDANTHEN C  FLAMBDA FHIS CIRCULANTENDTHEOREMBEGINPROOF  SEE EXERCISE REFEXCIRC1ENDPROOFBASED UPON THIS THEOREM WE MAKE THE FOLLOWING OBSERVATIONSBEGINENUMERATEITEM THE EIGENVALUES OF CIRCULANTC0C1LDOTSCM1 ARE LAMBDAI  SUMK0M1 CK EXPJ2PI IJMTHAT IS THE EIGENVALUES ARE OBTAINED FROM THE DFT OF THE SEQUENCEC0C1LDOTSCM1ITEM THE NORMALIZED EIGENVECTORS XBFI ARE  XBFI  FRAC1SQRTM BEGINBMATRIX1 EJ2PI IM EJ2PI 2IM VDOTS EJ2PI M1IM  ENDBMATRIXENDENUMERATETHE EIGENVECTORS OF EVERY MATSIZEMM CIRCULANT MATRIX ARE THESAME  THIS FACT MAKES CIRCULANT MATRICES PARTICULARLY EASY TO DEALWITH INVERSES PRODUCTS SUMS AND FACTORS OF CIRCULANT MATRICES AREALSO CIRCULANTTHE DIAGONALIZATION OF C HAS A NATURAL INTERPRETATION IN TERMS OFFAST CONVOLUTION  WRITE THE CYCLICAL CONVOLUTION YBF  CXBFASBEGINEQUATION YBF  FRAC1NFH LAMBDA F XBF  FRAC1NFH LAMBDA FXBFLABELEQFCCIRCENDEQUATIONTHEN FXBF IS THE DISCRETE FOURIER TRANSFORM OF F  THE FILTERINGOPERATION IS ACCOMPLISHED BY MULTIPLICATION OF THE DIAGONAL MATRIXELEMENTBYELEMENT SCALING THEN FRAC1MFH COMPUTES THEINVERSE FOURIER TRANSFORM  IF THE DFT IS COMPUTED USING A FASTALGORITHM AN FFT THEN REFEQFCCIRC REPRESENTS THE FAMILIARFAST CONVOLUTION ALGORITHMTHE DIAGONALIZATION OF C HAS IMPLICATIONS IN THE SOLUTION OFEQUATIONS WITH CIRCULANT MATRICES  TO SOLVE C XBF  BBF WE CANWRITE FRAC1M F LAMBDA FH XBF  BBFWHICH CAN BE WRITTEN AS LAMBDA YBF  M FH BBF  DBFWHERE YBF  FH XBF IS THE DFT OF XBF AND DBF IS THE SCALEDDFT OF BBF  THEN THE SOLUTION IS YI  FRAC1LAMBDAI DIIF THERE ARE FREQUENCY BINS AT WHICH LAMBDAI BECOMES SMALLTHEN THERE MAY BE AMPLIFICATION OF ANY NOISE PRESENT IN THE SIGNALSUBSECTIONVANDERMONDE CIRCULANT AND COMPANION MATRICESRELATIONS BETWEEN VANDERMONDE CIRCULANT AND COMPANION MATRICESINDEXVANDERMONDE MATRIXINDEXCIRCULANT MATRIX INDEXCOMPANION MATRIXCOMPANION MATRICES WERE INTRODUCED IN SECTION REFSECMOVEEIG  THEFOLLOWING THEOREM RELATES VANDERMONDE CIRCULANT AND COMPANIONMATRICESBEGINTHEOREM LET C  BEGINBMATRIX0 1  0  CDOTS  0 0  0  1  CDOTS  0 VDOTS 0  0  0  CDOTS  1 C0  C1  C2  CDOTS  CM1 ENDBMATRIXBE THE COMPANION MATRIX TO THE POLYNOMIAL PX  XM  CM1XM1  CM2XM2  CDOTS  C1 X  C0AND LET X1X2LDOTSXM BE THE ROOTS OF PXLET V  VX1X2LDOTSXM  VXBF BE A VANDERMONDE  MATRIX LET D  DIAGXBF BE A DIAGONAL MATRIX  THEN VD  CVENDTHEOREMBEGINPROOF  THE FIRST M1 ROWS CAN BE VERIFIED BY DIRECT COMPUTATION  THE  MJTH ELEMENT OF VD IS XJM  THE MJTH ELEMENT OF CV  IS C0  C1 XJ  C2 XJ2  CDOTS  CM1XJM1  XJM PXJ  XJMENDPROOFSUBSECTIONASYMPTOTIC EQUIVALENCE OF EIGENVALUESASYMPTOTIC EQUIVALENCE OF EIGENVALUES OF TOEPLITZ AND CIRCULANT  MATRICESINDEXTOEPLITZ MATRIXEIGENVALUESINDEXCIRCULANT MATRIXEIGENVALUESTHERE IS INTEREST IN EXAMINING THE EIGENVALUE STRUCTURE OF TOEPLITZMATRICES FORMED FROM AUTOCORRELATION VALUES BECAUSE THIS PROVIDESINFORMATION ABOUT THE POWER SPECTRUM OF A STOCHASTIC PROCESSOBTAINING EXACT ANALYTICAL EXPRESSIONS FOR EIGENVALUES OF A GENERALTOEPLITZ MATRIX IS DIFFICULT  HOWEVER BECAUSE OF THE SIMILARITYBETWEEN CIRCULANT AND TOEPLITZ MATRICES AND THE SIMPLE EIGENSTRUCTUREOF CIRCULANT MATRICES THERE IS SOME HOPE OF OBTAINING APPROXIMATE ORASYMPTOTIC EIGENVALUE INFORMATION ABOUT A TOEPLITZ MATRIX FROM ACIRCULANT MATRIX WHICH IS CLOSE TO THE TOEPLITZ MATRIXCONSIDER THE AUTOCORRELATION SEQUENCERBF  RMRM1LDOTSR1R0R1LDOTSRM WHERERK  0 FOR K  M OR KM  THE SPECTRUM OF THE SEQUENCE RBF SOMEGA  SUMKMM RK EJKOMEGAIS THE POWER SPECTRUM OF SOME RANDOM PROCESS  THE AUTOCORRELATIONVALUES CAN BE RECOVERED BY THE INVERSE FOURIER TRANSFORM RK  FRAC12PIINT02PI SOMEGA EJKOMEGA DOMEGALET RN BE THE BANDED MATSIZENN TOEPLITZ MATRIX OFAUTOCORRELATION VALUESBEGINEQUATION RN  BEGINBMATRIXR0 R1R2  CDOTS RM R1  R0  R1 R2  CDOTS  RM VDOTS  DDOTS  RM  RM1  RM2  CDOTS  R0  R1  CDOTS  RM  RM   CDOTS  R1  R0  R1  CDOTS  RM   DDOTS DDOTS DDOTS RM  CDOTS  R1R0 R1  CDOTS  RM VDOTS  DDOTS RM  CDOTS  R1  R0  R1  RM  CDOTS  R1  R0 ENDBMATRIXLABELEQRNDEFENDEQUATIONWE SAY THAT RN IS AN MTH ORDER TOEPLITZ MATRIX  EXCEPT FOR THEUPPER RIGHT AND LOWER LEFT CORNERS RN HAS THE STRUCTURE OF ACIRCULANT MATRIX  THE KEY TO OUR RESULT IS THAT AS N GETS LARGETHE CONTRIBUTIONS OF THE ELEMENTS IN THE CORNERS BECOME RELATIVELYNEGLIGIBLE AND THE EIGENVALUES CAN BE APPROXIMATED USING THEEIGENVALUES OF THE RELATED CIRCULANT MATRIX WHICH CAN BE FOUNDFROM THE DFT OF THE AUTOCORRELATION SEQUENCEWE DEFINE A MATSIZENN CIRCULANT MATRIX CNWITH THE SAME ELEMENTS BUT WITH THE PROPER CIRCULANT STRUCTUREBEGINEQUATIONCN CIRCULANTR0R1LDOTSRM0LDOTS0RMRM1LDOTSR1LABELEQCNDEFENDEQUATIONWE NOW WANT TO DETERMINE THE RELATIONSHIP BETWEEN THE EIGENVALUES OFRN AND THE EIGENVALUES OF CN AS N RIGHTARROW INFTY  TO DOTHIS WE NEED TO INTRODUCE THE CONCEPT OF ASYMPTOTIC EQUIVALENCE ANDSHOW THE RELATIONSHIP BETWEEN THE EIGENVALUES OF ASYMPTOTICALLYEQUIVALENT MATRICESBEGINDEFINITION INDEXASYMPTOTIC EQUIVALENCE OF MATRICES  TWO SEQUENCE OF MATRICES AN AND BN ARE SAID TO BY BF  ASYMPTOTICALLY EQUIVALENT IFBEGINENUMERATEITEM THE MATRICES IN EACH SEQUENCE ARE BOUNDED AN2 LEQ M  INFTY QQUAD  BN 2 LEQ M  INFTYFOR SOME FINITE BOUND MITEM FRAC1SQRTNAN  BNF RIGHTARROW 0 AS  NRIGHTARROW INFTYENDENUMERATEENDDEFINITIONNOTE THAT THE BOUNDEDNESS IS STATED USING THE SPECTRAL NORM WHILE THECONVERGENCE IS STATED IN THE FROBENIUS NORM  WE SHALL EMPLOY THEDIFFERENT PROPERTIES OF THESE TWO NORMS BELOWBEGINTHEOREM LABELTHMASYMPTEQUIV CITEGRAY1972  LET AN AND BN BE ASYMPTOTICALLY EQUIVALENT MATRICES WITH  EIGENVALUES LAMBDANK AND MUNK RESPECTIVELY  IF FOR  EVERY POSITIVE INTEGER L LIMNRIGHTARROW INFTY FRAC1N SUMK0N1  LAMBDANKL  INFTY QQUAD TEXTAND QQUADLIMNRIGHTARROW INFTY FRAC1N SUMK0N1  MUNKL  INFTYTHAT IS IF THE SOCALLED EIGENVALUE MOMENTS EXIST THENBEGINEQUATIONLIMNRIGHTARROW INFTY FRAC1N SUMK0N1LAMBDANKL  LIMNRIGHTARROW INFTY FRAC1NSUMK0N1 MUNKLLABELEQASSEIGENDEQUATIONTHAT IS THE EIGENVALUE MOMENTS OF AN AND BN AREASYMPTOTICALLY EQUALENDTHEOREMBEGINPROOF  LET AN  BN  DN  SINCE THE  EIGENVALUES OF ANL ARE LAMBDANKL WE CAN WRITE LIMNRIGHTARROW INFTY FRAC1N SUMK0N1LAMBDANKLAS LIMNRIGHTARROW INFTY FRAC1N TRACE ANLLET DELTAN  ANL  BNL  THEN REFEQASSEIG CAN BE WRITTENAS LIMNRIGHTARROW INFTY FRAC1N TRACE DELTAN  0THE MATRIX DELTAN CAN BE WRITTEN AS A FINITE NUMBER OF TERMS EACHOF WHICH IS A PRODUCT OF DN AND BN EACH TERM CONTAINING ATLEAST ONE DN  FOR A TERM SUCH AS DNALPHA BNBETA ALPHA 0 BETA GEQ 0 USING THE INEQUALITY REFEQFROBINEQ2 WE OBTAIN  DNALPHA BNBETA F LEQ M  DNFWHERE BN2 LEQ M  BUT SINCE AN AND BN AREASYMPTOTICALLY EQUIVALENT DNF RIGHTARROW 0 ESTABLISHING THERESULTENDPROOFWITH THIS DEFINITION AND THEOREM WE ARE NOW READY TO STATE THE MAINRESULT OF THIS SECTIONBEGINTHEOREM LABELTHMRCEQUIV CITEGRAY1972  THE TOEPLITZ MATRIX RN OF REFEQRNDEF AND THE CIRCULANT  MATRIX CN OF  REFEQCNDEF ARE ASYMPTOTICALLY EQUIVALENTENDTHEOREMBEGINPROOF  WE FIRST ESTABLISH THE BOUNDEDNESS OF RN AND CN  BY THE  DEFINITION OF THE 2NORMBEGINALIGNRN22  MAXXBF NEQ 0 FRACXBFH RNH RN XBFXBFH XBFNONUMBER    FRACSUMI0N1SUMK0N1 RIK XI XBARKSUMK0N1 XK2 NONUMBER   LEFTFRAC12PI INT02PI LEFTSUMK0N1 XK EJKOMEGARIGHT2 SOMEGA DOMEGARIGHT LEFT FRAC12PI INT02PI LEFTSUMK0N1 XK EJKOMEGARIGHT2 DOMEGA RIGHT1 LABELEQGRAYPROOF1  LEQ MAXW SOMEGA  LEQ SUMKMM RK INFTY LABELEQGRAYPROOF2ENDALIGNTHE NORM CN DEPENDS UPON THE LARGEST EIGENVALUE OF CN WHICHIS STRAIGHTFORWARD TO SHOW IS BOUNDEDNOW TO COMPUTE  RN  CN F SIMPLY COUNT HOW MANY TIMESELEMENTS APPEAR IN CN THAT DO NOT APPEAR IN RN  THENBEGINALIGNED FRAC1N  RN  CNF2  SUMK0M KRK2  RK2   LEQ FRAC1NM SUMK0M RK2  RK2ENDALIGNEDAS NRIGHTARROW INFTY WITH M BOUNDED FRAC1SQRTN RN CNF RIGHTARROW 0ENDPROOFBY THEOREMS REFTHMRCEQUIV AND REFTHMASYMPTEQUIV THEEIGENVALUES LAMBDANK OF RN AND THE EIGENVALUES MUNKOF CN HAVE THE SAME ASYMPTOTIC MOMENTS   LIMNRIGHTARROW INFTY FRAC1N SUMK0N1LAMBDANKL  LIMNRIGHTARROW INFTY FRAC1NSUMK0N1 MUNKLFOR INTEGER L GEQ 0  THIS LEADS US TO THE FOLLOWING ASYMPTOTICRELATIONSHIP BETWEEN THE EIGENVALUES OF RN AND THE POWER SPECTRUMSOMEGABEGINTHEOREM LABELTHMEIGSPECTEQUIV LET RN BE AN MTH ORDER  TOEPLITZ MATRIX AND LET SOMEGA DENOTE THE FOURIER TRANSFORM OF  THE COEFFICIENTS OF RN  LET CN AND RN BE ASYMPTOTICALLY  EQUIVALENT  IF THE EIGENVALUES OF RN ARE LAMBDANK AND  THE EIGENVALUES OF CN ARE MUNK THEN LIMNRIGHTARROW INFTY FRAC1N SUMK0N1  LAMBDANKL  FRAC12PI INT02PI SOMEGAL DOMEGAFOR EVERY POSITIVE INTEGER LFURTHERMORE IF RN IS HERMITIAN THEN FOR ANY FUNCTION GCONTINUOUS ON THE APPROPRIATE INTERVAL LIMNRIGHTARROW INFTY FRAC1N SUMK0N1GLAMBDANK  FRAC12PI INT02PI GSOMEGADOMEGAENDTHEOREMBEGINPROOF  BY THE DISCUSSION ABOVE THE EIGENVALUES OF CN ARE MUNI   SUMKMM RK EJ2PI IKN  S2PI IN  BY THE  ASYMPTOTIC EQUIVALENCE OF THE MOMENTS OF THE EIGENVALUES BEGINALIGNEDLIMNRIGHTARROW INFTY FRAC1N SUMK0N1LAMBDANKL LIMNRIGHTARROW INFTY FRAC1N SUMK0N1 MUNKL LIMNRIGHTARROW INFTY FRAC1N SUMK0N1 S2PI NLENDALIGNEDNOW LET DELTA OMEGA  2PI N AND OMEGAK  2PI KN  THENBEGINEQUATIONLIMNRIGHTARROW INFTY FRAC1N SUMK0N1LAMBDANKL  LIMNRIGHTARROW INFTY SUMK0N1SOMEGAKL DELTA OMEGA2PI  FRAC12PI INT02PISOMEGAL DOMEGALABELEQLFEQUIVENDEQUATIONFOR ANY POLYNOMIAL P BY REFEQLFEQUIV WE ALSO HAVE LIMNRIGHTARROW INFTY FRAC1N SUMK0N1PLAMBDANK  FRAC12PI INT02PIPSOMEGA DOMEGAIF RN IS HERMITIAN ITS EIGENVALUES ARE REAL  BY THEWEIERSTRASS THEOREM ANY CONTINUOUS FUNCTION G OPERATING ON THEINDEXWEIERSTRASS THEOREMREAL INTERVAL CAN BE UNIFORMLY APPROXIMATED BY A POLYNOMIAL P  SINCE  THE EIGENVALUES OF RN ARE REAL WE CAN APPLY THESTONEWEIERSTRASS THEOREM AND CONCLUDE LIMNRIGHTARROW INFTY FRAC1N SUMK0N1GLAMBDANK  FRAC12PI INT02PIGSOMEGA DOMEGAENDPROOFTHIS THEOREM IS SOMETIMES REFERRED TO AS EM SZEGOS THEOREMINDEXSZEGOS THEOREMSZEGOS THEOREMROUGHLY WHAT THE THEOREM SAYS IS THAT THE EIGENVALUES OF RN HAVETHE SAME DISTRIBUTION AS DOES THE SPECTRUM SOMEGA  THE THEOREMIS SOMEWHAT DIFFICULT TO INTERPRET BECAUSE WHEREAS THERE IS ADEFINITE ORDER TO THE SPECTRUM SOMEGA THE EIGENVALUES OF RNHAVE NO INTRINSIC ORDER THEY ARE OFTEN COMPUTED TO APPEAR IN SORTEDORDER  NEVERTHELESS IT CAN BE OBSERVED THAT IF RN HAS A LARGEEIGENVALUE DISPARITY HIGH CONDITION NUMBER THEN THE SPECTRUMSOMEGA WILL HAVE A LARGE SPECTRAL SPREAD SOME FREQUENCIES WILLHAVE LARGE SOMEGA WHILE OTHER FREQUENCIES HAVE SMALL SPECTRALSPREADBEGINEXAMPLE  LET RI  E2I I87LDOTS8 BE THE AUTOCORRELATION  FUNCTION FOR SOME SEQUENCE  FIGURE REFFIGEIGSPECT SHOWS THE  SPECTRUM SOMEGA AND THE EIGENVALUES OF RN FOR N30 AND  N100  TO MAKE THE PLOT THE EIGENVALUES WERE COMPUTED IN SORTED  ORDER THEN THE BEST MATCH OF THE EIGENVALUES TO SOMEGA WAS  DETERMINED  THE IMPROVEMENT IN MATCH IS APPARENT AS N INCREASES  ALTHOUGH EVEN FOR N30 THERE IS CLOSE AGREEMENT BETWEEN THE  SPECTRUM AND THE EIGENVALUES OF RN  BEGINFIGUREHTBP    CENTERING EIGDISTMEPSFIGFILEPICTUREDIREIGSPECTEPSWIDTH08TEXTWIDTHSUBFIGUREN30EPSFIGFILEPICTUREDIREIGSPECTEPSWIDTH08TEXTWIDTHSUBFIGUREN30EPSFIGFILEPICTUREDIREIGSPECT2EPSWIDTH08TEXTWIDTH    CAPTIONCOMPARISON OF SOMEGA AND THE EIGENVALUES OF    PROTECTRPROTECTNPROTECTCOMPARISON OF SOMEGA AND THE      EIGENVALUES OF PROTECTRPROTECTNPROTECT FOR N30 AND      N100    LABELFIGEIGSPECT  ENDFIGUREENDEXAMPLEBEGINEXERCISESITEM SHOW THAT CIRCULANT1111 IS A HADAMARD MATRIX  SEE  EXERCISE REFEXHADAMARDITEM SHOW THAT THE MATRIX F DEFINED IN REFEQFMAT SATISFIES  FFH  MIITEM LABELEXCIRC1 PROVE THEOREM REFTHMCIRCDIAGITEM SOME PROPERTIES OF CIRCULANT MATRICES    BEGINENUMERATE  ITEM SHOW THAT IF A IS A CIRCULANT MATRIX THEN SUMK0R AK AKIS CIRCULANT WHERE THE AK ARE SCALARSITEM SHOW THAT THE INVERSE OF A CIRCULANT MATRIX A IS A1  N FHLAMBDA1FITEM SHOW THAT THE DETERMINANT OF A CIRCULANT MATRIX IS DETCIRCULANTCBF  PRODJ1N PCBFOMEGAJ1WHERE OMEGAEJ2PINITEM SHOW THAT THE MOOREPENROSE INVERSE OF A CIRCULANT MATRIX C IS CDAGGER  FH LAMBDADAGGER FITEM LET S  BEGINBMATRIX010CDOTS0 001CDOTS0 000DDOTS0 000CDOTS1100CDOTS1 ENDBMATRIX IN RBBMATSIZEMM1S ON THE SUPERDIAGONAL IN IN THE LOWERLEFT CORNER  SHOW THAT C  SUMK0M1 CK1 SKIS CIRCULANTENDENUMERATE  ITEM JUSTIFY REFEQGRAYPROOF1 AND REFEQGRAYPROOF2  ITEM USING THEOREM REFTHMEIGSPECTEQUIV SHOW THAT LIMNRIGHTARROW INFTY DETRN1N EXPLEFTFRAC12PI INT02PI LN SOMEGA DOMEGARIGHTHINT GCDOT  LNITEM SHOW THAT A CIRCULANT MATRIX IS TOEPLITZ BUT A TOEPLITZ    MATRIX IS NOT NECESSARILY CIRCULANTENDEXERCISESSECTIONTRIANGULAR MATRICESLABELSECTRIANGMATA EM UPPER TRIANGULAR MATRIX IS A MATRIX OF THE FORM T  BEGINBMATRIX T11 T12  T13 CDOTS  T1N 0  T22  T23 CDOTS  T2N 0  DDOTS  T33  CDOTS  T3N 000CDOTS  TNN  ENDBMATRIXA LOWER TRIANGULAR MATRIX IS A MATRIX SUCH THAT ITS TRANSPOSE ISTRIANGULAR   TRIANGULAR MATRICES ARISE IN CONJUNCTION WITH THE LUFACTORIZATION SEE SECTION REFSECLUFACT AND THE QR FACTORIZATION SEESECTION REFSECQR  TRIANGULAR MATRICES HAVE THE FOLLOWINGPROPERTIESBEGINENUMERATEITEM THE PRODUCT OF TWO UPPER TRIANGULAR MATRICES IS UPPER  TRIANGULAR  THE PRODUCT OF TWO LOWER TRIANGULAR MATRICES IS LOWER  TRIANGULARITEM THE INVERSE OF AN UPPER TRIANGULAR MATRIX IS UPPER TRIANGULAR  THE INVERSE OF A LOWER TRIANGULAR MATRIX IS LOWER TRIANGULARENDENUMERATETRIANGULAR MATRICES ARE FREQUENTLY SEEN IN SOLVING SYSTEMS OFEQUATIONS THE LU FACTORIZATION  THERE ARE ALSO SYSTEM REALIZATIONSTHAT ARE BUILT ON TRIANGULAR MATRICESSECTIONPROPERTIES PRESERVED IN MATRIX PRODUCTSLABELSECMATPRESOF THE VARIETIES OF MATRICES WITH SPECIAL STRUCTURES THAT WE HAVEENCOUNTERED THROUGHOUT THIS BOOK IT IS VALUABLE TO KNOW WHEN THESEPROPERTIES ARE PRESERVED UNDER MATRIX MULTIPLICATION  THAT IS IF AAND B ARE MATRICES POSSESSING SOME SPECIAL STRUCTURE WHEN DOESCAB ALSO POSSES THIS STRUCTURE  HERE IS A SIMPLE LISTSUBSUBSECTIONMATRIX PROPERTIES PRESERVED UNDER MATRIX  MULTIPLICATIONINDEXMATRIX MULTIPLICATIONMATRIX PROPERTIES PRESERVEDINDEXMATRIX PROPERTIES PRESERVED UNDER MULTIPLICATIONBEGINENUMERATEITEM UNITARY INDEXUNITARYITEM CIRCULANT INDEXCIRCULANTITEM NONSINGULAR INDEXNONSINGULARITEM LOWER OR UPPER TRIANGULAR INDEXTRIANGULAR MATRIXENDENUMERATESUBSUBSECTIONMATRIX PROPERTIES NOT PRESERVED UNDER MATRIX MULTIPLICATIONBEGINENUMERATEITEM HERMITIAN INDEXHERMITIANITEM POSITIVE DEFINITE INDEXPOSITIVE DEFINITEITEM TOEPLITZ INDEXTOEPLITZ MATRIXITEM VANDERMONDE INDEXVANDERMONDE MATRIXITEM NORMAL INDEXNORMAL MATRIXITEM STABILITY EG EIGENVALUES INSIDE UNIT CIRCLEENDENUMERATESETEXSECTREFSECMODALMATBEGINEXERCISESITEM CONSIDER A THIRDORDER EXPONENTIAL SIGNAL WITH REPEATED MODES  R2 M1  2 M21  BEGINENUMERATE  ITEM WRITE DOWN AN EXPLICIT EXPRESSION FOR XT USING    REFEQEXPMOD3  ITEM DETERMINE THE FORM OF V IN EQUATION REFEQXVALPHA FOR    THIS SIGNAL WITH REPEATED MODES  IS IT STILL A VANDERMONDE    MATRIX  ENDENUMERATEITEM LET AZ  1 2Z  3Z2  FIND THE RECIPROCAL  POLYNOMIAL ATILDEZ  WHAT IS THE RELATION OF THE COEFFICIENTS  OF AZ TO THOSE OF ATILDEZITEM LABELEXRECIPOLY2 SHOW THAT IF GAMMANEQ 0 IS A ROOT OF A  POLYNOMIAL AZ THEN 1GAMMA IS A ROOT OF THE RECIPROCAL  POLYNOMIAL ATILDEZITEM SHOW BY FINDING A COUNTEREXAMPLE THAT THE SYMMETRY OF  COEFFICIENTS IS NECESSARY BUT NOT SUFFICIENT FOR THE ROOTS OF A  POLYNOMIAL TO LIE ON THE UNIT CIRCLEITEM COMPUTER EXPERIMENT  USING SC MATLAB GENERATE A SIGNAL  WITH TWO REAL MODES HAVING ROOTS OF THE CHARACTERISTIC EQUATION AT  095EJPM PI5 AND 092 EJPM PI3 AND EXPLORE PRONYS  METHOD  LET THE SIGNAL AMPLITUDES BE ATILDE1  1 ATILDE2   05  BEGINENUMERATE  ITEM GENERATE SUFFICIENT DATA TO USE PRONYS METHOD SOLVE FOR    THE COEFFICIENTS AND PLOT THE POLE LOCATIONS IN THE ZPLANE  ITEM NOW ADD NOISE TO THE SIGNAL AND DETERMINE HOW THE PRONYS    METHOD DETERIORATES AS A FUNCTION OF SNR  TRY SNR10 DB 5 DB 0    DB 3 DB  MEASURE THE SNR RELATIVE TO THE STRONGER SIGNAL  ITEM REPEAT THE PREVIOUS TWO STEPS USING LEASTSQUARES AND TOTAL    LEASTSQUARES PRONYS METHODS VARYING THE NUMBER OF EQUATIONS EMPLOYED  ENDENUMERATEITEM A USEFUL WAY OF INTERPRETING THE EXPONENTIAL MODEL IS AS THE  IMPULSE RESPONSE OF A ARMAPP1 MODEL WITH TRANSFER FUNCTIONBEGINEQUATIONHZ  FRACBZAZ  FRACSUMI0P1 BI ZI 1   SUMI1P AI ZILABELEQHZEXPMODENDEQUATIONIN THE CASE OF SIMPLE MODES THIS CAN BE WRITTEN USING PARTIALFRACTION EXPANSION AS HZ  SUMI1P FRACALPHAI1ZI Z1FROM WHICH THE RELATIONSHIP XT  ZC1HZ  SUMI1P ALPHAI ZIPTIS OBVIOUS  BY WRITING OUT THE DIFFERENCE EQUATION IMPLIED BYREFEQHZEXPMOD DEVELOP A SET OF EQUATIONS ATILDE XBFTILDE  BBFTILDEWHERE ATILDE IS A TOEPLITZ MATRIX WITH COEFFICIENTS FROM AZXBFTILDE HAS TIME SAMPLES AND BBFTILDE HAS COEFFICIENTS FROMBZ  FROM THIS EQUATION THE COEFFICIENTS OF BZ CAN BE FOUNDWITHOUT FINDING THE ROOTS OF AZ BEGINBMATRIX 1   A1  1 A2  A1  1 VDOTS AP1  AP2  CDOTS  A1  1 AP  AP1  CDOTS  A1  1    AP  AP1  CDOTS  A1  1 VDOTS  CDOTS  AP  AP1  CDOTS  A1  1  ENDBMATRIX BEGINBMATRIX X0   X1  VDOTS  XP1  XP  VDOTS  XN1ENDBMATRIX BEGINBMATRIX B0  B1  VDOTS BP1  0  VDOTS  0ENDBMATRIX EXSKIPSETEXSECTREFSECPERMUTEMATITEM SHOW THAT THE DETERMINANT OF A PERMUTATION MATRIX IS PM 1ITEM THE BITREVERSE SHUFFLE OF THE FFT ALGORITHM IS A  PERMUTATION  TABLE REFTABBRS ILLUSTRATES A BITREVERSE SHUFFLE  FOR AN 8POINT DFT  DETERMINE A PERMUTATION MATRIX WHICH PERMUTES AN  INCOMING COLUMN VECTOR ACCORDING TO THE BITREVERSE SHUFFLE  BEGINTABLEHTBPBEGINCENTER    BEGINTABULARCCCC HLINEN  BINARY  BIT REVERSE  BITREVERSED N HLINE0  000  000  0 1  001  100  4 2  010  010  2 3  011  110  6 4  100  001  1 5  101  101  5 6  110  011  3 7  111  111  7HLINE    ENDTABULARENDCENTER    CAPTIONBIT REVERSE SHUFFLE    LABELTABBRS  ENDTABLEITEM THE MATSIZEMM PERMUTATION MATRICES FORM A GROUP  DETERMINE THE NUMBER OF MEMBERS IN THE GROUP OF MATSIZEMM  PERMUTATION MATRICES  DETERMINE A POWER K SUCH THAT ALL  MATSIZE33 PERMUTATION MATRICES P SATISFY PK  I   ITEM A STOCHASTIC MATRIX INDEXSTOCHASTIC MATRIX HAS NONNEGATIVE   ENTRIES AND THE ROWS SUM TO 1  A MATRIX IS DOUBLE STOCHASTIC IF   ROWS AND COLUMNS SUM TO 1  SHOW THAT EVERY DOUBLY STOCHASTIC MATRIX   M CAN BE WRITTEN AS A CONVEX SUM OF PERMUTATION MATRICES  M  LAMBDA1 P1  LAMBDA2 P2  CDOTS  LAMBDAM PM  WHERE SUMI1M LAMBDAI  1 LAMBDAI GEQ 0 AND PI IS A PERMUTATION MATRIXEXSKIPSETEXSECTREFSECTOEPLITZITEM LABELEXTOEPDETAIL SHOW THAT IF R IS PERSYMMETRIC THEN B1J       JB1T WHERE J IS THE COUNTERIDENTITY ITEM IF R IS HERMITIAN THEN OVERLINER1  R1TITEM THE MATSIZEMM MATRICES B  BEGINBMATRIX 010CDOTS0 001CDOTS0 000DDOTS  0 VDOTS 000CDOTS1 000CDOTS0 ENDBMATRIXQQUAD TEXTANDQQUADF  BEGINBMATRIX000CDOTS00 100CDOTS00 010CDOTS00 000DDOTS00000CDOTS10 ENDBMATRIXARE CALLED EM BACKWARD SHIFT AND EM FORWARD SHIFT MATRICESRESPECTIVELYBEGINENUMERATEITEM LET ABF  1234T  COMPUTE BABF AND FABF FOR  MATSIZE44 BACKWARD AND FORWARD SHIFT MATRICES  COMPUTE B2  ABF AND F2 ABF  COMMENT ON THE NAME OF THE MATRICESITEM SHOW THAT AN MATSIZEMM MATRIX OF THE FORM IN  REFEQTOEPDEF1 CAN BE WRITTEN AS R  SUMK1M SK FK  SUMK0M SK BKENDENUMERATEITEM LET R0 RPM 1 RPM 2 LDOTS RPM M DENOTE THE  AUTOCORRELATION SEQUENCE OF A STATIONARY STOCHASTIC PROCESS AND LET   SOMEGA  SUMNMM EJOMEGA N RN    BE ITS POWER SPECTRAL DENSITY  SHOW THAT IF SOMEGA GEQ 0 THEN  THE TOEPLITZ MATRIX R WITH ELEMENTS RIJ  RIJ IS POSITIVE  SEMIDEFINITE  ITEM THE ALGORITHMS TT DURBIN AND TT LEVINSON ARE DESIGNED    FOR A SYMMETRIC TOEPLITZ MATRIX  DEVELOP SIMILAR ALGORITHMS    SUITABLE FOR NONSYMMETRIC MATRICES  HINT PROPAGATE TWO SOLUTIONS  ITEM SHOW THAT FOR A HERMITIAN TOEPLITZ MATRIX RK1 BEGINBMATRIX I  JK WBFK  ZEROBF  1 ENDBMATRIXH RK1BEGINBMATRIXI  JK WBFK  ZEROBF  1 ENDBMATRIX BEGINBMATRIX RK  0  0 R0  WBFKH RBFK ENDBMATRIXHENCE CONCLUDE THAT IF RK1 IS POSITIVE DEFINITE THENR0RBFKT WBFBARK IN REFEQALPHATOEP IS NOT ZEROITEM LABELEXTOEPPROOFEX SHOW THAT THE VARIANCE OF THE FIRSTORDER  FORWARD PREDICTION ERROR FILTER REFEQTOEPPROOF1 IS CORRECT  HINT SHOW THAT ALPHA0  R1R0EXSKIPSETEXSECTREFSECVANDERMONDEITEM SHOW THAT THE FORMULA REFEQVANDET FOR THE DETERMINANT OF  THE VANDERMONDE MATRIX IS CORRECT  HINT USE ROW OPERATIONS  INDUCTION AND THE COFACTOR EXPANSIONITEM DETERMINE A POLYNOMIAL INTERPOLATING THE POINTS  12124 41 3227EXSKIPSETEXSECTREFSECCIRCULANTITEM SHOW THAT CIRCULANT1111 IS A HADAMARD MATRIX THAT IS  THAT IF H  CIRCULANT1111 THEN HHT  4I  SEE SECTION  REFSECKRON2  IT IS BELIEVED CITEDAVIS THAT THIS IS THE  ONLY CIRCULANT HADAMARD MATRIXITEM SHOW THAT THE MATRIX F DEFINED IN REFEQFMAT SATISFIES  FFH  MIITEM LABELEXCIRC1 PROVE THEOREM REFTHMCIRCDIAGITEM SOME PROPERTIES OF CIRCULANT MATRICES    BEGINENUMERATE  ITEM SHOW THAT IF A AND B ARE CIRCULANT MATRICES OF THE SAME    SIZE THEN AB IS CIRCULANT  ITEM SHOW THAT IF A IS A CIRCULANT MATRIX THEN FOR ANY FIXED R     0 SUMK0R AK AKIS CIRCULANT WHERE THE AK ARE SCALARSITEM SHOW THAT THE INVERSE OF A MATSIZEMM CIRCULANT MATRIX A IS A1  M FHLAMBDA1FITEM SHOW THAT THE DETERMINANT OF A MATSIZEMM CIRCULANT MATRIX  A  CIRCULANTCBF IS DETCIRCULANTCBF  PRODJ1M PCBFOMEGAJ1WHERE OMEGAEJ2PIMITEM SHOW THAT THE MOOREPENROSE INVERSE OF A CIRCULANT MATRIX C IS CDAGGER  M FH LAMBDADAGGER FITEM LET S  BEGINBMATRIX010CDOTS0 001CDOTS0 000DDOTS0 000CDOTS1100CDOTS0 ENDBMATRIX IN RBBMATSIZEMM1S ON THE SUPERDIAGONAL AND IN THE LOWERLEFT CORNER  SHOW THAT C  SUMK0M1 CK1 SKIS CIRCULANTENDENUMERATE  ITEM JUSTIFY REFEQGRAYPROOF1 AND REFEQGRAYPROOF2  ITEM USING THEOREM REFTHMEIGSPECTEQUIV SHOW THAT LIMNRIGHTARROW INFTY DETRN1N EXPLEFTFRAC12PI INT02PI LN SOMEGA DOMEGARIGHTHINT GCDOT  LNITEM SHOW THAT A CIRCULANT MATRIX IS TOEPLITZ BUT A TOEPLITZ    MATRIX IS NOT NECESSARILY CIRCULANTEXSKIPSETEXSECTREFSECMATPRESITEM FOR EACH OF THE PROPERTIES LISTED IN SECTION REFSECMATPRES  WHICH EM FAIL TO BE  PRESERVED UNDER MATRIX MULTIPLICATION FIND AN  EXAMPLE TO DEMONSTRATE THIS FAILURE ENDEXERCISESSECTIONREFERENCESLABELSECSPECREF MODALMAT REFERENCESEXPONENTIAL SIGNAL MODELS ARE DISCUSSED IN FOR EXAMPLE CITESCHARFL1991CADZOW1988KAY1988STOICA  PRONYS METHODHAS A CONSIDERABLE HISTORY DATING TO 1795 CITEPRONY1795  THELEASTSQUARES PRONY METHOD APPEARS IN CITEHILDEBRAND1956   THEOBSERVATION ABOUT A REAL UNDAMPED SIGNAL HAVING A SYMMETRICCHARACTERISTIC POLYNOMIAL APPEARS IN CITEKUMARESAN1984THE BASIC ALGORITHMS FOR SOLUTION OF TOEPLITZ EQUATIONS COMES FROMCITEGVL  SIGNAL PROCESSING INTERPRETATIONS OF TOEPLITZ MATRICESARE FOUND FOR EXAMPLE IN CITEHAYKIN1996 AND CITEPROAKISRADERINVERSION FOR BLOCK TOEPLITZ MATRICES IS DISCUSSED INCITEAKAIKE1979  A INTERESTING SURVEY ARTICLE ISCITEKAILATH15  OTHER RELATED ARTICLES DISCUSSING SOLUTIONS OFEQUATIONS WITH HANKEL AND TOEPLITZ MATRICES APPEAR INCITERISSANEN1974RISSANEN1973DIAGONALIZATION OF CIRCULANT MATRICES IS ALSO DISCUSSED INCITEHUNT1971 WHERE IT IS SHOWN THAT A BLOCK CIRCULANT MATRIX AMATRIX WHICH CIRCULATES BLOCKS OF MATRICES CAN BE DIAGONALIZED BY A2DIMENSIONAL DFT  APPLICATION OF THIS TO IMAGE PROCESSING ISDISCUSSED IN THE SURVEY CITEBANHAM1997OUR DISCUSSION OF THE ASYMPTOTIC EQUIVALENCE OF TOEPLITZ AND CIRCULANTMATRICES IS DRAWN CLOSELY FROM CITEGRAY1972 WHICH IN TURN DRAWSFROM CITEGRENANDER AND CITEWIDOM  ASYMPTOTIC EQUIVALENCE OF THEPOWER SPECTRUM AND THE EIGENVALUES OF AUTOCORRELATION FUNCTION BYMEANS OF THE KARHUNENLOEVE REPRESENTATION ARE DISCUSSED INCITECHAPTER 8GALLAGER1968  EIGENVALUES OF TOEPLITZ MATRICES AREALSO DISCUSSED IN CITEBASOR WHILE MAKHOUL1981 ANDCITEREDDI1984 TREAT THE EIGENVECTORS OF SYMMETRIC TOEPLITZMATRICESINFORMATION ON A VARIETY OF SPECIAL MATRICES IS IN CITEHORNJOHNSONTHE LITTLE BOOK CITEDAVIS HAS A GREAT DEAL OF MATERIAL ON CIRCULANTMATRICES TOEPLITZ MATRICES BLOCK MATRICES PERMUTATIONS ANDPSEUDOINVERSES AMONG OTHER THINGS  THE SUMMARY OF MATRIX PROPERTIESPRESERVED UNDER MULTIPLICATION COMES FROM CITEVAIDYANATHAN1993 LOCAL VARIABLES TEXMASTER TEST ENDSUBSECTIONORTHOGONAL WAVELETSLABELSECWAVELETSSINCE ABOUT 1990 A SET OF FUNCTIONS KNOWN AS EM WAVELETSINDEXWAVELETS HAS SPARKED CONSIDERABLE INTEREST IN SIGNALPROCESSING RESEARCH  LIKE THE FOURIER TRANSFORM THE WAVELETTRANSFORM CAN PROVIDE INFORMATION ABOUT THE SPECTRAL CONTENT OF ASIGNAL  HOWEVER UNLIKE A SINUSOIDAL SIGNAL WITH INFINITE SUPPORTWAVELETS ARE PULSES WHICH ARE WELL LOCALIZED IN THE TIME DOMAIN SOTHAT THEY CAN PROVIDE DIFFERENT SPECTRAL INFORMATION AT DIFFERENT TIMELOCATIONS OF A SIGNAL  IN DOING THIS THEY SACRIFICE SOME OF THESPECTRAL RESOLUTION BY THE UNCERTAINTY PRINCIPLE WE CANNOT LOCALIZEPERFECTLY WELL IN BOTH THE TIME DOMAIN AND THE FREQUENCY DOMAINWAVELETS HAVE ANOTHER PROPERTY THAT MAKE THEM PRACTICALLY USEFULWHEN USED TO ANALYZE LOWERFREQUENCY COMPONENTS A WIDE WAVELET SIGNALIS USED TO ANALYZE HIGHERFREQUENCY COMPONENTS A NARROW WAVELETSIGNAL IS USED  THUS WAVELETS CAN IN PRINCIPLE IDENTIFY SHORTBURSTS OF HIGHFREQUENCY SIGNALS IMPOSED ON TOP OF ONGOINGLOWFREQUENCY SIGNALS  ONE OF THE MAJOR PRINCIPLES OF WAVELETANALYSIS IS THAT IT TAKES PLACE ON SEVERAL SCALES USING BASISFUNCTIONS OF DIFFERENT WIDTHSTHERE ARE IN FACT SEVERAL FAMILIES OF WAVELETS EACH WITH ITS OWNPROPERTIES AND ASSOCIATED TRANSFORMS  NOT ALL FAMILIES OF WAVELETSFORM ORTHOGONAL WAVEFORMS  A PARTICULAR FAMILY OF WAVELETS THAT HASPERHAPS ATTRACTED THE MOST ATTENTION IS KNOWN AS THE DAUBECHIESWAVELETS INDEXDAUBECHIESSEEWAVELETS THESE WAVELETS WHICH FORMA COMPLETE SET HAVE SOME VERY SOME VERY NICE ORTHOGONALITY PROPERTIESTHAT LEAD TO FAST COMPUTATIONAL ALGORITHMS  THE DAUBECHIES WAVELETSCAN BE UNDERSTOOD BEST IN THE CONTEXT OF A HILBERT SPACE USING WHATIS KNOWN AS A MULTIRESOLUTION ANALYSIS  THIS INVOLVES PROJECTING AFUNCTION ONTO A WHOLE SERIES OF SPACES WITH DIFFERENT RESOLUTIONS  WENOW PRESENT A BRIEF INTRODUCTION TO THE CONSTRUCTION OF THESEWAVELETS  CONSIDERABLY MORE INFORMATION IS PROVIDED IN THE LITERATURECITED IN THE REFERENCES INCLUDING GENERALIZATION IN A VARIETY OFUSEFUL WAYS OF THE CONCEPTS OUTLINED HERESUBSUBSECTIONCHARACTERIZATION OF WAVELETSTHROUGHOUT THIS SECTION WE WILL ASSUME REAL FUNCTIONS FOR CONVENIENCEMOST OF THESE CONCEPTS CAN BE GENERALIZED TO FUNCTIONS OF COMPLEXNUMBERS  SUPPOSE WE HAVE A SET OF CLOSED SUBSPACES OF THE HILBERTSPACE L2RBB DENOTED BY LDOTSV1 V0 V1 LDOTS WITHTHE FOLLOWING PROPERTIESBEGINENUMERATEITEM NESTING CDOTS V2 SUBSET V1 SUBSET V0 SUBSET V1 SUBSET V2CDOTSITEM CLOSURE CLOSURELEFTBIGCUPJ IN ZBB VJRIGHT  L2RBBTHAT IS THE CLOSURE OF THE SET OF SPACES COVERS ALL OF L2RBBSO THAT EVERY FUNCTION IN L2 HAS A REPRESENTATION USING ELEMENTS INONE OF THESE NESTED SPACES  ITEM SHRINKING BIGCAPJ IN ZBB VJ   0ITEM THE MULTIRESOLUTION PROPERTY  IF FT IN VJ THEN F2J T IN V0ITEM IF FT IN V0 THEN FTN IN V0 FOR ALL N IN ZBBITEM FINALLY THERE IS SOME PHI IN V0 SUCH THAT THE INTEGER  SHIFTS OF PHI FORM AN ORTHONORMAL BASIS FOR V0 V0  LSPANPHITN N IN ZBBENDENUMERATETHE FUNCTION PHIT IS SAID TO BE A BF SCALING FUNCTIONINDEXSCALING FUNCTION THE PROPERTY THAT PHIT PERP PHITNFOR N IN ZBB IS CALLED THE EM SHIFT ORTHOGONALITY PROPERTYWE WILL USE THE NOTATION PJ FT TO DENOTE THE PROJECTION OF THEFUNCTION FT ONTO VJBEGINEXAMPLE  LET   BEGINEQUATION    LABELEQUNITPULSEPHIT  UTUT1ENDEQUATIONA UNIT PULSE AND FORM V0  LSPANPHITNN IN ZBBTHE SET OF FUNCTIONS PHITNN IN ZBB FORMS AN ORTHONORMALSET  THEN FUNCTIONS IN V0 ARE FUNCTIONS THAT ARE EM PIECEWISE  CONSTANT ON THE INTEGERS  FIGURE REFFIGV0A SHOWS A FUNCTIONFT THE PROJECTION P0 FT  THE NEAREST FUNCTION TO FTTHAT IS PIECEWISE CONSTANT IN THE INTEGERS  AND P1 FT WHICH IS PIECEWISE CONSTANT ON THE HALFINTEGERSENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRWAVELET1    CAPTIONA FUNCTION FT AND ITS PROJECTION ONTO    PROTECTVPROTECT0PROTECT AND    PROTECTVPROTECT1PROTECT    LABELFIGV0A  ENDCENTERENDFIGUREAS J DECREASES THE PROJECTION PJFT REPRESENTS FT WITHINCREASING FIDELITYLET US DEFINE THE SCALED AND SHIFTED INDEXSCALING INDEXSHIFTINGVERSION OF THE FUNCTION PHI BY PHIJKT  2J2 PHI2J2 T  KTHE INDEX J CONTROLS THE EM SCALE AND THE INDEX K CONTROLS THELOCATION OF THE FUNCTION PHIJK  IF PHIT IS NORMALIZED SOPHIT  1 THEN SO IS PHIJKT FOR ANY J AND KSINCE PHIT IN V0 SUBSET V1 AND  PHI1KT K INZBB FORMS AN ORTHONORMAL BASIS FOR V1 IT MUST BE POSSIBLETO EXPRESS PHIT AS A LINEAR COMBINATION OF PHI1KTBEGINEQUATION PHIT  SUMK HK PHI1KT  SQRT2SUMK HK PHI2T  KLABELEQTWOSCALE1ENDEQUATIONTHE SET OF COEFFICIENTS IN REFEQTWOSCALE1 DETERMINES THEPARTICULAR PROPERTIES OF THE SCALING FUNCTION AND THE ENTIRE WAVELETDECOMPOSITION  LET N DENOTE THE TOTAL NUMBER OF COEFFICIENTS HNIN REFEQTWOSCALE1  IN GENERAL N COULD BE INFINITE BUT INPRACTICE IT IS ALWAYS A FINITE NUMBER  WE ALSO GENERALLY ASSUME THATTHE COEFFICIENTS HK ARE INDEXED SO THAT HK  0  FOR K0  LETUS DEFINE CK  SQRT2HK  THEN WE CAN WRITEBEGINEQUATION PHIT  SUMK CK PHI2T KLABELEQTWOSCALE2ENDEQUATIONOR GIVEN OUR ASSUMPTIONS WE CAN WRITE THIS MORE PRECISELY ASBEGINEQUATION PHIT  SUMK0N1 CK PHI2TKLABELEQTWOSCALE3ENDEQUATIONAN EQUATION OF THE FORM REFEQTWOSCALE3 IS KNOWN AS A EM  TWOSCALE EQUATION INDEXTWOSCALE EQUATIONBEGINEXAMPLE  IN REFEQTWOSCALE3 LET US HAVE TWO COEFFICIENTS C0  1 AND  C1  1  THEN THE TWOSCALE EQUATION BECOMES PHIT  PHI2T  PHI2T1IT IS STRAIGHTFORWARD TO VERIFY THAT THE PULSE IN REFEQUNITPULSESATISFIES THIS EQUATIONENDEXAMPLEBEGINLEMMA LABELLEMCCOND IF PHIT SATISFIES A TWOSCALE EQUATION REFEQTWOSCALE2 ANDPHIT PERP PHITN FOR ALL N IN ZBB WITH N NEQ 0 THENBEGINEQUATIONSUMK CK CK2P  2DELTA0PLABELEQWAVEORTHOG1ENDEQUATIONENDLEMMABEGINPROOF  USING REFEQTWOSCALE2 WE HAVE BEGINALIGNEDINT PHIT PHITNDT  INT SUMK CK PHI2TK SUMJ CJPHI2TN JDT  FRAC12 SUMJ LEFTSUMK CK CKJ2NRIGHT INT PHITPHITJ DTENDALIGNEDIN ORDER FOR THIS TO BE ZERO BECAUSE OF THE ORTHOGONALITY THEBRACKETED TERM MUST BE ZERO WHEN J0 AND WHEN 2NNEQ 0  THENSUMK CK CK2N  2DELTA0NENDPROOFIN GOING FROM A PROJECTION PJ1 FT TO A LOWERRESOLUTIONPROJECTION PJFT THERE IS SOME DETAIL INFORMATION THAT IS LOSTIN THE ORTHOGONAL COMPLEMENT OF VJ  WE CAN REPRESENT THIS DETAILBY SAYING THATBEGINEQUATION VJ1  VJ OPLUS WJLABELEQMULTRESENDEQUATIONWHERE WJ  VJPERP IN VJ1  THE DIRECT SUM ISINTERPRETED IN THE ISOMORPHIC SENSE  THUS WJ CONTAINS THE DETAILLOST IN GOING FROM VJ1 TO VJ  ALSO AS WE SHALL SEE THEWJ SPACES ARE ORTHOGONAL SO WJ PERP WJ IF J NEQ JNOW WE INTRODUCE THE SET OF FUNCTIONS PSIJKT  2J2PSI2J T  K AS AN ORTHONORMAL BASIS SET FOR WJ WITHPSIT IN W0  THE FUNCTION PSIT IS KNOWN AS A BF WAVELETFUNCTION OR SOMETIMES AS THE BF MOTHER WAVELET SINCE THEFUNCTIONS PSIJKT ARE DERIVED FROM IT  SINCE V1  V0OPLUS W0 AND PSIT IN V1 WE HAVEBEGINEQUATION PSIT  SUMK GK PHI1KT  SQRT2SUMK GKPHI2TKLABELEQTWOSCALE4ENDEQUATIONWE DESIRE TO CHOOSE THE GN COEFFICIENTS TO ENFORCE THEORTHOGONALITY OF THE SPACES  IT WILL BE CONVENIENT TO WRITE DK  SQRT2 GKBEGINTHEOREM LABELTHMWAVEORTHOGIF  PHITN NIN ZBB FORMS AN ORTHOGONAL SET AND DK  1K C2M1KFOR ANY M IN ZBB THEN  PSIJKT FORMS AN ORTHOGONAL SETFOR ALL J K IN ZBB  FURTHERMORE PSIJKT PERPPHILMT FOR L LEQ JENDTHEOREMBEGINPROOF  WE BEGIN BY SHOWING THAT  PSIJKT FORMS AN  ORTHOGONAL SET FOR FIXED J  BEGINALIGNINT 2J PSI2JT PSI2JT  K DT  INT SUML DLPHI2UL SUMM DM PHI2UKM DU NONUMBER INTERTEXTHFILL WHERE U2JT INT LEFTFRAC12 SUML 1L C2M1L PHIXRIGHTNONUMBER  QQUADQQUAD LEFTSUMM 1M C2M1MPHIXL2KMRIGHTDX NONUMBER INTERTEXTHFILLWHERE  X  2UL FRAC12 SUMJ CJCJ2K INT PHI2XDX NONUMBER INTERTEXTHFILLBY ORTHOGONALITY WITH J2M1L DELTA0K NONUMBER  DELTA0K NONUMBER INTERTEXTHFILLUSING REFEQWAVEORTHOG1ENDALIGNWHERE THE FINAL EQUALITY FOLLOWS FROM REFEQWAVEORTHOG1NOW WE SHOW THAT PHIJKT PERP PSIJMT FOR ALL KM INZBB FOR FIXED J  WE HAVEBEGINALIGNINT PSIJKT PHIJMTDT  INT 2J PSI2J T  KPHI2J T M DT NONUMBER  INT PSIUK PHIUM DT NONUMBER  INTERTEXTHFILLWHERE U 2JT INT SUML 1L C2M 1L PHI2UKL SUMJ CJ  PHI2UMJ DU NONUMBER  FRAC12 INT SUML 1L C2M1L PHIX SUMJ CJ  PHIXL2K2MJ DX  NONUMBER INTERTEXTHFILLWHERE  X2UL2K FRAC12 SUML 1L C2M1L CL2K2M INT PHI2TDT LABELEQORTHOWAV2  INTERTEXTHFILLBY ORTHOGONALITY NONUMBERENDALIGNIN THE SUMMATION IN REFEQORTHOWAV2  LET P  MK SO THESUMMATION IS S  SUML1L C2M1LCL2PNOW LETTING J2M1L2P WE CAN WRITE S  SUMJ 11J CJ2PC2M1J  SUMJ 1JC2M1JCJ2P  SSINCE SS WE MUST HAVE BEGINEQUATION  LABELEQSEQ0  0  S  SUML1L C2M1LCL2PENDEQUATIONESTABLISHING THE DESIRED ORTHOGONALITYFINALLY WE SHOW THAT PSIJK PERP PSILM FOR ALL JKLM INZBB IF J NEQ L AND K NEQ M  WE HAVE ALREADY ESTABLISHED THISFOR JL  BY THE MULTISCALE RELATIONSHIP PSIJKT IN WJLET J  J SO THAT WJ SUBSET VJ  BUT VJ PERPWJ SO THAT PSIJKT WHICH IS IN WJ MUST BEORTHOGONAL TO PSIJKTENDPROOFBEGINEXAMPLE  WE HAVE SEEN THAT A SCALING FUNCTION PHIT CAN BE FORMED WHEN  C0  C1  1  THE WAVELET PSIT CORRESPONDING TO THIS  SCALING FUNCTION IS PSIT  PHI2T  PHI2T1A PLOT OF PHIT AND PSIT IS SHOWN IN FIGUREREFFIGWAVELET1  THE FUNCTION PSIT IS ALSO KNOWN AS THE EM  HAAR BASIS FUNCTION INDEXHAAR FUNCTIONENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRWAVELET2    CAPTIONTHE SIMPLEST SCALING AND WAVELET FUNCTIONS    LABELFIGWAVELET1  ENDCENTERENDFIGURETHERE ARE SEVERAL FAMILIES OF ORTHONORMAL COMPACTLY SUPPORTEDWAVELETS  ALGORITHM REFALGWAVCOEF PROVIDES COEFFICIENTS FORSEVERAL DAUBECHIES WAVELETS THERE EXIST WAVELETS IN THIS FAMILY WITHCOEFFICIENTS OF EVERY POSITIVE EVEN LENGTH  THE TRANSFORM FOR THESECOEFFICIENTS IS CALLED THE DN WAVELET TRANSFORM WHERE THERE AREN COEFFICIENTS  INDEXDNDN WAVELET FAMILY PLOTS OF SOME OFTHE CORRESPONDING SCALING AND WAVELET FUNCTIONS ARE SHOWN IN FIGUREREFFIGWAVESCALE  WE OBSERVE THAT THE FUNCTIONS BECOME SMOOTHER ASTHE NUMBER OF COEFFICIENTS INCREASESBEGINNEWPROGENVSOME WAVELET COEFFICIENTS CITEPAGE 195DAUBECHIES1992 WAVCOEFSOME WAVELET COEFFICIENTSWAVECOEFFMENDNEWPROGENVBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE PLOTWAVELETMSUBFIGUREEPSFIGFILEPICTUREDIRWAVE4EPSHEIGHT015TEXTHEIGHTSUBFIGUREEPSFIGFILEPICTUREDIRWAVE6EPSHEIGHT015TEXTHEIGHTSUBFIGUREEPSFIGFILEPICTUREDIRWAVE8EPSHEIGHT015TEXTHEIGHTSUBFIGUREEPSFIGFILEPICTUREDIRWAVE10EPSHEIGHT015TEXTHEIGHTSUBFIGUREEPSFIGFILEPICTUREDIRWAVE12EPSHEIGHT015TEXTHEIGHT    CAPTIONILLUSTRATION OF SCALING AND WAVELET FUNCTIONS    LABELFIGWAVESCALE  ENDCENTERENDFIGURESUBSUBSECTIONWAVELET TRANSFORMSINDEXWAVELET TRANSFORMIN THE WAVELET TRANSFORM A FUNCTION FT IS EXPRESSED AS A LINEARCOMBINATION OF SCALING AND WAVELET FUNCTIONS  BOTH THE SCALINGFUNCTIONS AND THE WAVELET FUNCTIONS ARE COMPLETE SETS  HOWEVER IT ISCOMMON TO EMPLOY BOTH WAVELET AND SCALING FUNCTIONS IN THE TRANSFORMREPRESENTATIONSUPPOSE THAT WE HAVE A PROJECTION OF FT ONTO SOME SPACE VJOF SUFFICIENT RESOLUTION THAT IT PROVIDES AN ADEQUATE REPRESENTATION OFTHE DATA  THEN WE HAVE FT APPROX PJ FT  SUMK LA FT PHIJKT RA PHIJKTCOMMONLY WE ASSUME THAT THE DATA HAS BEEN SCALED SO THAT THE INITIAL SCALEIS J0 SO THAT OUR STARTING POINT IS P0 FT  LET US CALL THISSTARTING FUNCTION F0T SO THAT F0T  SUMN LA FTPHI0NTRA PHI0NTFOR THE PURPOSES OF THE TRANSFORM WE REGARD THE EM  COEFFICIENTS OF THISREPRESENTATION AS THE REPRESENTATION OF FT  IN PRACTICE THE SETOF INITIAL COEFFICIENTS ARE SIMPLY EM SAMPLES OF FTOBTAINED BY SAMPLING EVERY T SECONDS  THAT IS WE ASSUME THAT LAFT PHI0NTRA APPROX FNT FOR SOME SAMPLING INTERVAL TUNDER THIS APPROXIMATION THE WAVELET TRANSFORM DEALS WITHDISCRETETIME SEQUENCES  FURTHER DISCUSSION OF THIS POINT ISPROVIDED IN CITEPAGE 166DAUBECHIES1992  FOR CONVENIENCE OFNOTATION LET US DENOTE THE SEQUENCE  LA F0 PHI0NTRAAS  C0N AND LET US DENOTE THE VECTOR OF THESE VALUES ASCBF0 CBF0  BEGINBMATRIX C00 C01  C02  LDOTSENDBMATRIXTIN THE WAVELET TRANSFORM WE EXPRESS F0T IN TERMS OF WAVELETS ONLONGER SCALES  FOR EXAMPLE USING REFEQMULTRES WE HAVEV0  V1 OPLUS W1 SO THAT F0T IN V0 CAN BE REPRESENTED ASF0T  SUMN LA F0T PSI1NTRA PSI1NT SUMN LA F0T PHI1NTRA PHI1NT LET C1N  LA F0TPHI1NTRA AND D1N  LA F0TPSI1NTRA ANDLET US DENOTE F1T  SUMN LA F0T PHI1NTRA PHI1NT SUMN C1N PHI1N AND DELTA1T  SUMN LA F0TPSI1NTRA PSI1NT SUMN D1N PSI1NWHERE F1 IN V1 AND DELTA1 IN W1  THEN BEGINEQUATION F0T  F1T  DELTA1TLABELEQMULTSCALE3ENDEQUATIONSINCE F1 IN V1 AND V1  V2  W2 WE CAN SPLIT F1 INTO ITSPROJECTION ONTO V2 AND W2 ASBEGINALIGNF1T  SUMN LA F1TPHI2NT RA PHI2NT SUMN LA F1TPSI2NTRA PSI2NT NONUMBER  SUMN C2N  PHI2N  SUMN D2N PSI2N NONUMBER  F2T  DELTA2T LABELEQMULTSCALE2ENDALIGNWHERE F2T IN V2 AND DELTA2T IN W2 AND CN2  LAF1T PHI2NRA AND DN2  LA F1TPSI2NRASUBSTITUTING REFEQMULTSCALE2 INTO REFEQMULTSCALE3 WE HAVE F0T  DELTA1T  DELTA2T  F2TWE WILL USE THE NOTATION CBFJ AND DBFJ TO REPRESENT THECOEFFICIENTS CJN AND DJN RESPECTIVELY  WE CAN REPEAT THISDECOMPOSITION FOR UP TO J SCALES WRITING FJT IN VJON EACH SCALE J12LDOTSJ ASBEGINEQUATION FJT  FJ1T  DELTAJ1TLABELEQMULTSCALE6ENDEQUATIONSO F0T  SUMJ1J DELTAJT   FJTTHE SET OF COEFFICIENTS DBF1 DBF2 LDOTSDBFJCBFJCOLLECTIVELY ARE THE BF WAVELET TRANSFORM OF THE FUNCTION F0TTHE COEFFICIENTS AT SCALE DBFJ REPRESENT THE SIGNAL ON LONGERSCALES LOWER FREQUENCY BAND THAN THE COEFFICIENTS AT SCALEDBFJ1  THE COEFFICIENTS CBFJ REPRESENTS AN AVERAGE OF THEORIGINAL DATATHE COMPUTATIONS JUST DESCRIBED ARE OUTLINED IN FIGUREREFFIGWAVELET3  STARTING FROM THE INITIAL SET OF COEFFICIENTSCBF0 THE ALGORITHM SUCCESSIVELY PRODUCES CBFJ1 ANDDBFJ1 UNTIL THE JTH LEVEL IS REACHED  BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRWAVELET3    CAPTIONILLUSTRATION OF A WAVELET TRANSFORM    LABELFIGWAVELET3  ENDCENTERENDFIGUREWHILE IT IS CONCEIVABLE TO COMPUTE THE TRANSFORM BY DIRECTLYEVALUATING THE INDICATED INNER PRODUCTS A SIGNIFICANTLY FASTERALGORITHM EXISTS  WE NOTE THAT BY REFEQTWOSCALE3BEGINALIGNPSIJKT  2J2 PSI2JT  K  2J2 SQRT2 SUMGN PHI22JT  KN NONUMBER  SUMN GN PHIJ12KNT NONUMBER  SUMN GN2KPHIJ1NTLABELEQMULTSCALE9ENDALIGNWHEN WE COMPUTE THE WAVELET TRANSFORM COEFFICIENT LA F0TPSI1KTRA WE GETBEGINEQUATION LA F0TPSI1KTRA  SUMN GN2K LAF0T PHI0NTRA  SUMN GN2K C0NLABELEQWAVETRANS1ENDEQUATIONTO UNDERSTAND THIS SUM BETTER LET US WRITE  XN  GNAND FORM THE VECTOR XBF  X0X1LDOTSXN1  LET YBF XBF  CBF0 CONVOLUTION THEN YJ  SUMN XJN C0N  SUMN GNJ C0NFROM THIS WE OBSERVE THAT THE SUMMATION IN REFEQWAVETRANS1 ISTHE CONVOLUTION OF THE SEQUENCE GN WITH THE SEQUENCEC0N IN WHICH WE RETAIN ONLY THE EVENNUMBERED OUTPUTS  AT A GENERAL SCALE J WE COMPUTE THE WAVELET COEFFICIENTS ASBEGINEQUATION LA F0 PSIJKTRA  SUMN GN2K LA F0PHIJ1N RALABELEQWAVETRANS2ENDEQUATIONWHICH IS A CONVOLUTION OF THE SEQUENCE  GN WITH THESEQUENCE  LA F0 PHIJ1NRA RETAINING EVEN SAMPLES  TOCOMPUTE THE COEFFICIENTS IN REFEQWAVETRANS2 WE NEED TO KNOW LAF0 PHIJ1NRA  HOWEVER THESE CAN ALSO BE OBTAINEDEFFICIENTLY SINCEBEGINALIGNPHIJKT  2J2 PHI2JT K NONUMBER  SUMN HN2K PHIJ1NT LABELEQMULTSCALE10ENDALIGNSO THAT LA F0 PHIJKRA  SUMN  HN2K LA F0PHIJ1NRA WHICH IS AGAIN A CONVOLUTION FOLLOWED BY DECIMATION BY 2PUTTING ALL THE PIECES TOGETHER THE WAVELET TRANSFORM IS OUTLINED ASFOLLOWSBEGINENUMERATEITEM LET C0K  LA F0 PHI0KRA BE THE GIVEN INITIAL  DATA NORMALLY A SEQUENCE OF SAMPLES OF FTITEM COMPUTE THE SET OF WAVELET COEFFICIENTS ON SCALE 1 D1K  LA F0  PSI1KRA USINGBEGINEQUATIOND1K  SUMN GN2K CN0LABELEQWAVETRANS5ENDEQUATIONALSO COMPUTE THE SCALING COEFFICIENTS ON THIS SCALE C1K  LA F0PHI1KRA USINGBEGINEQUATION C1K  SUMN HN2K CN0LABELEQWAVETRANS4ENDEQUATIONITEM NOW PROCEED UP THROUGH LEVEL J SIMILARLYBEGINALIGNDJK  SUMN GN2K CNJ1 LABELEQDJK CJK  SUMN HN2K CNJ1QQUAD J12LDOTSJLABELEQCJK ENDALIGNENDENUMERATETHE WAVELET TRANSFORM COMPUTATIONS CAN BE REPRESENTED IN MATRIXNOTATION  THE OPERATION REFEQCJK CAN BE REPRESENTED AS A MATRIXL WHERE LIJ  HJ2I FOR I AND J IN SOME SUITABLE RANGETHE OPERATION REFEQDJK CAN BE REPRESENTED AS A MATRIX H WHEREHIJ  GJ2IBEGINEXAMPLEWE WILL DEMONSTRATE THIS MATRIX NOTATION FOR A WAVELET WITH FOURCOEFFICIENTS H0H1H2 H3  WE CHOOSE M SO THATG0G1G2G3  H3H2H1H0  ALSO FOR THE SAKE OF ASPECIFIC REPRESENTATION WE ASSUME THAT C0N HAS SIX ELEMENTSIN IT  FROM REFEQWAVETRANS4 CBF1  BEGINBMATRIXC11  C10  C11  C12 ENDBMATRIX BEGINBMATRIX H2  H3 H0 H1  H2  H3  H0 H1  H2  H3  H0H1 ENDBMATRIXBEGINBMATRIXC00  C01  C02  C03  C04  C05ENDBMATRIX L CBF0THE TRUNCATION EVIDENT IN THE FIRST AND LAST ROWS OF THE MATRIXCORRESPONDS TO AN ASSUMPTION THAT DATA OUTSIDE THE SAMPLES ARE EQUAL TOZERO  AS DISCUSSED BELOW THERE IS ANOTHER ASSUMPTION THAT CAN BEMADEFROM REFEQWAVETRANS5BEGINALIGNED DBF1  BEGINBMATRIXD11 D10  D11  D12 ENDBMATRIX BEGINBMATRIX G2  G3 G0 G1  G2  G3  G0 G1  G2  G3  G0G1 ENDBMATRIXBEGINBMATRIXC00  C01  C02  C03  C04  C05ENDBMATRIX  BEGINBMATRIX H1  H0 H3  H2  H1  H0  H3  H2  H1  H0  H3 H2 ENDBMATRIXBEGINBMATRIXC00  C01  C02  C03  C04  C05ENDBMATRIX  H CBF0ENDALIGNEDTHE TRANSFORM DATA AT THE NEXT RESOLUTION DBF2 AND THE DATACBF2 CAN BE OBTAINED USING THE SAME INDEXING CONVENTION ASBEFORE AS CBF2  BEGINBMATRIX H3  H1   H2  H3   H0 H1   H2  ENDBMATRIXCBF1 DBF2  BEGINBMATRIXG3  G1  G2  G3    G0  G1  G2  ENDBMATRIXCBF1IT IS PERHAPS WORTHWHILE TO POINT OUT THAT THE INDEXING CONVENTION ONCBF1 COULD BE CHANGED WITH A CORRESPONDING CHANGE INREFEQCJK SO THAT WE INTERPRET CBF1 AS THE VECTOR CBF1  BEGINBMATRIX C10C11 C12 C13 ENDBMATRIXMAKING THIS CHANGE THE MATRIX FOR THE SECOND STAGE TRANSFORMATIONWOULD BE WRITTEN AS CBF2  BEGINBMATRIX H1  H0 H3  H2  H1  H0  H3 H2 ENDBMATRIX CBF1WITH SIMILAR CHANGES FOR DBF2 AND ITS ASSOCIATED TRANSFORMATIONMATRIX  PROVIDED THAT THE SAME INDEXING CONVENTION IS USED FOR THEFORWARD TRANSFORMATION AS THE INVERSE TRANSFORMATION THE TRANSFORM ISSTILL FULLY REVERSIBLEENDEXAMPLETHE NOTATION L AND H FOR THE MATRIX OPERATORS IS DELIBERATELYSUGGESTIVE  THE L MATRIX IS A LOWPASS OPERATOR AND THE DATASEQUENCE CBF1 IS A LOWPASS SEQUENCE  IT CORRESPONDS TO ABLURRING OF THE ORIGINAL DATA CBF0  THE G MATRIX IS A HIGHPASS OPERATOR AND THE DATA DBF1 ISHIGHPASS OR BANDPASS DATA  THE FILTERINGSUBSAMPLING OPERATION REPRESENTED BY THESE MATRICES CANCONTINUE THROUGH SEVERAL STAGES  THE TRANSFORM COEFFICIENTS AT THEEND OF THE PROCESS IS THE COLLECTION OF DATA DBF1 DBF2 LDOTSDBFJ AND CBFJ WHERE CBFJ IS A FINAL COURSE APPROXIMATIONOF THE ORIGINAL STARTING DATA CBF0  THE WAVELET TRANSFORMCOMPUTATIONS CAN ALSO BE REPRESENTED AS A FILTERINGDECIMATIONOPERATION AS SHOWN IN FIGURE REFFIGMULTIRATE1  THE SIGNALCBF0 PASSES THROUGH A LOWPASS AND HIGHPASS FILTER WHOSE OUTPUTSARE DECIMATED AS INDICATED BY BOXEDDOWNARROW 2 TAKING EVERYOTHER SAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRWAVELET4    CAPTIONMULTIRATE INTERPRETATION OF WAVELET TRANSFORM    LABELFIGMULTIRATE1  ENDCENTERENDFIGURESUBSUBSECTIONINVERSE WAVELET TRANSFORMTHE INVERSE WAVELET TRANSFORM CAN BE OBTAINED BY WORKING BACKWARDSGIVEN DBFJ AND CBFJ WE WISH TO FIND CBFJ1  WE NOTE FROM REFEQMULTSCALE6 THATBEGINALIGNFJ1   FJ  DELTAJNONUMBER   SUMK CJK PHIJK  SUMK DJKPSIJK LABELEQMULTSCALE7 ENDALIGNTHEN USING THE FACT THAT CNJ1  LA FJ1PHIJ1NRA ANDTAKING INNER PRODUCTS OF BOTH SIDES OF REFEQMULTSCALE7 WE HAVEBEGINALIGN  CJ1N  LA FJ1PHIJ1NRA NONUMBER  SUMK CJK LA PHIJKPHIJ1K RA  SUMK DJK LAPSIJK PHIJ1K RA LABELEQMULTSCALE11ENDALIGNTAKING INNER PRODUCTS ON BOTH SIDES OF REFEQMULTSCALE10 WITHPHIJ1M WE OBSERVE THATLA PHIJK PHIJ1MRA  SUMN HN2K LAPHIJ1NPHIJ1MRA  HM2KBY THE ORTHOGONALITY OF THE PHI FUNCTION  SIMILARLY FROMREFEQMULTSCALE9 LA PSIJKPHIJ1M RA   GM2KSUBSTITUTING THESE INTO REFEQMULTSCALE11 WE FIND THATBEGINEQUATION  LABELEQMULTSCALE12  CJ1N  SUMK CKJ HN2K  SUMK DJK GN2KENDEQUATIONTHIS TELLS US HOW TO GO UPSTREAM FROM CBFJ AND DBFJ TOCBFJ1   THE PROCESS IS OUTLINED IN FIGURE REFFIGWAVELET5BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRWAVELET5    CAPTIONILLUSTRATION OF THE INVERSE WAVELET TRANSFORM    LABELFIGWAVELET5  ENDCENTERENDFIGUREAS BEFORE THE RECONSTRUCTION CAN BE EXPRESSED IN  MATRIX FORM CBFJ1  L CBFJ  H DBFJWHERE L IS THE ADJOINT CONJUGATE TRANSPOSE OF L AND H ISTHE ADJOINT OF H SEE SECTION REFSECADJOINTBEGINEXAMPLELET US CONSIDER A SPECIFIC NUMERIC EXAMPLE  USING THE WAVELET WITH FOURCOEFFICIENTS THE CODE IN ALGORITHM REFALGWAVETEST1 FINDS THETWOSCALE WAVELET TRANSFORM DATA DBF1DBF2 CBF2 FOR THE DATASET CBF  123456T  ALSO THE INVERSE TRANSFORM IS FOUNDTHE PERTINENT VARIABLES OF THE EXECUTION ARE CBF0  BEGINBMATRIX1  2  3  4  5  6 ENDBMATRIXQQUADDBF1  BEGINBMATRIXHFILL 0482963 HFILL 762188E09 HFILL 228656E08 HFILL 338074 HFILL 0776457 ENDBMATRIX QQUADDBF2  BEGINBMATRIXHFILL 0541266 HFILL 0670753 HFILL 0270032 HFILL 0375 ENDBMATRIX QQUADCBF2  BEGINBMATRIXHFILL 0145032 HFILL 0557132 HFILL 868838 HFILL 139952 ENDBMATRIXOBSERVE THAT THERE ARE SIX POINTS IN THE ORIGINAL DATA AND THIRTEENPOINTS IN THIS TRANSFORM  THE RECONSTRUCTED SIGNAL TT C0NEW ISEQUAL TO THE ORIGINAL SIGNAL CBF0BEGINNEWPROGENVDEMONSTRATION OF WAVELET    DECOMPOSITIONWAVETESTM WAVETEST1DEMONSTRATION OF WAVELET DECOMPOSITIONENDNEWPROGENVFOR COMPARISON ALGORITHM REFALGWAVETEST2 SHOWS A DECOMPOSITIONAND RECONSTRUCTION WITH A DIFFERENT INDEXING CONVENTION  IN THISCASE THE TRANSFORM DATA IS DBF1   BEGINBMATRIXHFILL 012941 HFILL 0 HFILL 0 HFILL 199191 ENDBMATRIXQQUAD DBF2  BEGINBMATRIXHFILL 114503 HFILL 0195272 HFILL 233133 ENDBMATRIX QQUAD CBF2  BEGINBMATRIXHFILL 030681 HFILL 210617 HFILL 870064 ENDBMATRIXTHE RECONSTRUCTED SIGNAL TT C0NEW IS EQUAL TO THE ORIGINAL SIGNALTHIS TRANSFORM HAS TEN POINTS IN ITBEGINNEWPROGENVDEMONSTRATION OF WAVELET    DECOMPOSITION ALTERNATIVE INDEXINGWAVETESTOMWAVETEST2DEMONSTRATION OF WAVELET DECOMPOSITION ALTERNATIVE INDEXINGENDNEWPROGENVENDEXAMPLETHE L AND H MATRICES HAVE SOME INTERESTING PROPERTIES  IN THEFOLLOWING THEOREM THE L AND H MATRICES ARE ASSUMED TO BEINFINITE SO THAT PARTIAL SEQUENCES OF COEFFICIENTS DO NOT APPEAR ONANY ROWSBEGINTHEOREM  THE L AND H OPERATORS DEFINED BY THE OPERATIONS L CBF  SUMN HN2K CN QQUAD H CBF  SUMNGN2K CNHAVE THE FOLLOWING PROPERTIESBEGINENUMERATEITEM HLH  0ITEM LLH  I AND HHH  I ANDITEM LHL AND HHH ARE MUTUALLY ORTHOGONAL PROJECTIONSENDENUMERATEENDTHEOREMBEGINPROOF  LET HN2K DENOTE THE KTH COLUMN OF LH AND LET  GN2L DENOTE THE LTH ROW OF H  THE INNER  PRODUCT OF THESE CAN BE WRITTEN  SUMN HN2K GN2L  SUMN HN2K 1N H2ML  1NWHICH IS ZERO BY REFEQSEQ0  SINCE THIS IS TRUE FOR ANY L ANDK IT FOLLOWS THAT HLH  0THE FACT THAT  LLH  I AND HHH  I  IS SHOWN BYMULTIPLICATION USING REFEQWAVEORTHOG1  THEN WE NOTE THAT LHLLHL  LHLLHLL  LHL SO LHL IS APROJECTION AND SIMILARLY FOR HHH  BY THE FACT THAT HLH0 ITFOLLOWS THAT LHL AND HHH ARE ORTHOGONAL  NOW NOTE THAT HLHL  HHH  HHH  H AND LLHL  HHH  LTHUS LHL  HHH ACTS AS AN IDENTITY ON THE RANGES OF BOTH H ANDL SO IT IS AN IDENTITYENDPROOFTHE FILTERING INTERPRETATION FOR THE RECONSTRUCTION IS SHOWN IN FIGUREREFFIGMULTIRATE2 THE SAMPLES ARE EXPANDED BY INSERTING A ZERO BETWEENEVERY SAMPLE THEN FILTERING  WHEN THE FORWARD OPERATION AND THEBACKWARD OPERATION ARE PLACED TOGETHER AS SHOWN IN FIGUREREFFIGMULTIRATE3 AN IDENTITY OPERATION FROM END TO END RESULTSONE FAMILY OF SUCH FILTERING CONFIGURATIONS IS KNOWN AS A QUADRATUREMIRROR FILTER IT IS AN EXAMPLE OF A PERFECT RECONSTRUCTION FILTERTHIS MULTIRATE CONFIGURATION IS USED IN DATA COMPRESSION INWHICH THE LOWPASS AND HIGHPASS SIGNALS ARE QUANTIZED USING QUANTIZERSSPECIALIZED FOR THE FREQUENCY RANGE OF THE SIGNALSBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRWAVELET6    CAPTIONFILTERING INTERPRETATION OF AN INVERSE WAVELET TRANSFORM    LABELFIGMULTIRATE2  ENDCENTERENDFIGUREBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRWAVELET7    CAPTIONPERFECT RECONSTRUCTION FILTER BANK    LABELFIGMULTIRATE3  ENDCENTERENDFIGURESUBSUBSECTIONPERIODIC WAVELET TRANSFORMTHE WAVELET TRANSFORM PRODUCES MORE OUTPUT COEFFICIENTS THAN INPUTCOEFFICIENTS DUE TO THE CONVOLUTION  IF THERE ARE N INPUT POINTSAND THE FILTERS ARE M POINTS LONG THEN THE CONVOLUTIONDECIMATIONOPERATION PRODUCES LFLOOR NM2 RFLOOR POINTS OR ONE LESSDEPENDING ON HOW THE INDEXING IS INTERPRETED SO EACH STAGE OF THETRANSFORM PRODUCES MORE THAN HALF THE NUMBER OF POINTS FROM THEPREVIOUS STAGE  HAVING MORE TRANSFORM DATA THAN ORIGINAL DATA ISTROUBLING AN MANY CIRCUMSTANCES SUCH AS DATA COMPRESSION  IT ISCOMMON TO ASSUME THAT THE DATA IS PERIODIC AND TO PERFORM A PERIODIZEDTRANSFORM  SUPPOSE THAT THERE ARE L POINTS IN CBF0 CBF0  C00C01LDOTSC0L1TTHEN PERIODIZED DATA CBFTILDE0 IS FORMED CONCEPTUALLY BYSTACKING CBF0 CBFTILDE0  LDOTS CBF0TCBF0TCBF0TLDOTSTTHEN AN LPOINT WAVELET TRANSFORM IS COMPUTED ON THE PERIODIZEDDATA  THE EFFECT IS THAT THE WAVELET TRANSFORM COEFFICIENTS APPEARCYCLICALLY SHIFTED AROUND THE L AND H MATRICES  FOR EXAMPLE WITHFOUR COEFFICIENTS AND EIGHT DATA POINTS THE L AND H MATRICES WOULDLOOK LIKE THE FOLLOWING L  BEGINBMATRIX H0H1H2H3 H0H1H2H3 H0H1H2H3 H2H3H0H1 ENDBMATRIX H  BEGINBMATRIX G0G1G2G3 G0G1G2G3 G0G1G2G3 G2G3G0G1 ENDBMATRIXTHE SAME EQUATIONS USED TO REPRESENT THE NONPERIODIZED TRANSFORMSREFEQDJK AND REFEQCJK AND THE INVERSETRANSFORM REFEQMULTSCALE12 ALSO APPLY FOR THE PERIODIZEDTRANSFORM AND ITS INVERSE PROVIDED THAT THE INDICES ARE TAKEN MODULOTHE APPROPRIATE DATA SIZESUBSUBSECTIONWAVELET TRANSFORM IMPLEMENTATIONSALGORITHM REFALGWAVETRANS PERFORMS A NONPERIODIC WAVELETTRANSFORM  THE FIRST FUNCTION TT WAVETRANS SETS UP SOME DATATHAT IS USED BY THE RECURSIVELYCALLED FUNCTION TT WAVEIMPLEMENTATION OF TT WAVE IS STRAIGHTFORWARD WITH SOME CAUTIONNEEDED TO GET THE INDEXING STARTED CORRECTLY  SINCE DIFFERENT LEVELSHAVE DIFFERENT LENGTHS OF COEFFICIENTS AN ARRAY IS ALSO RETURNEDINDEXING THE TRANSFORM COEFFICIENTS FOR EACH LEVELBEGINNEWPROGENVNONPERIODIC WAVELET TRANSFORMWAVETRANSMWAVETRANSNONPERIODIC WAVELET TRANSFORMENDNEWPROGENVAN INVERSE NONPERIODIC WAVELET TRANSFORM IS SHOWN IN ALGORITHMREFALGINVWAVEBEGINNEWPROGENVNONPERIODIC INVERSE WAVELET TRANSFORMINVWAVETRANSMINVWAVENONPERIODIC INVERSE WAVELET TRANSFORMENDNEWPROGENVBEGINEXAMPLE  THE TWOLEVEL NONPERIODIC WAVELET TRANSFORM CBF  12345T  USING THE D4 COEFFICIENTS IS COMPUTED USING TT CAP   WAVETRANSCD4COEFF2 WHICH GIVESBEGINALIGNED C  BEGINBMATRIX01294  0  28978 0647   1145   32688  13068  03068 29297  48771 ENDBMATRIX TT AP  BEGINBMATRIX5158 ENDBMATRIXENDALIGNEDFROM WHICH WE INTERPRETBEGINALIGNEDDBF1  BEGINBMATRIX01294  00000 28978 06470ENDBMATRIX DBF2  BEGINBMATRIX11450 32688 13068 ENDBMATRIX CBF2  BEGINBMATRIX03068 29297 48771 ENDBMATRIXENDALIGNEDTHE INVERSE TRANSFORM COMPUTED BY TT INVWAVECAPD4COEFF RETURNSTHE ORIGINAL DATA VECTORENDEXAMPLECODE FOR THE PERIODIZED WAVELET TRANSFORM APPEARS IN ALGORITHMREFALGPERWAVE AND THE PERIODIZED INVERSE WAVELET TRANSFORM IS INALGORITHM REFALGINVPERWAVEBEGINNEWPROGENVPERIODIC WAVELET TRANSFORMWAVETRANSPERMPERWAVEPERIODIC WAVELET TRANSFORMENDNEWPROGENVBEGINNEWPROGENVINVERSE PERIODIC WAVELET TRANSFORMINVWAVETRANSPERMINVPERWAVEINVERSE PERIODIC WAVELET TRANSFORMENDNEWPROGENVSUBSUBSECTIONAPPLICATIONS OF WAVELETSWAVELETS HAVE BEEN USED IN A VARIETY OF APPLICATIONS OF WHICH WEMENTION ONLY A FEWBEGINDESCRIPTIONITEMDATA COMPRESSION INDEXDATA COMPRESSION ONE OF THE MOST  COMMON APPLICATIONS OF WAVELETS IS TO DATA COMPRESSION  A SET OF  DATA FBF IS TRANSFORMED USING A WAVELET TRANSFORM  THE WAVELET  TRANSFORM COEFFICIENTS SMALLER THAN SOME PRESCRIBED THRESHOLD ARE  SET TO ZERO AND THE REMAINING COEFFICIENTS ARE QUANTIZED USING SOME  UNIFORM QUANTIZER  IT IS A MATTER OF EMPIRICAL FACT THAT IN MOST  DATA SETS A LARGE PROPORTION OF THE COEFFICIENTS ARE ZEROED OUT  THE TRUNCATEDQUANTIZED COEFFICIENTS ARE THEN PASSED THROUGH A  RUNLENGTH ENCODER AND PERHAPS OTHER LOSSLESS ENCODING TECHNIQUES  WHICH REPRESENTS RUNS OF ZEROS BY A SINGLE DIGIT INDICATING HOW MANY  ZEROS ARE IN THE RUN  A MORE SOPHISTICATED VERSION OF THIS ALGORITHM IS EMPLOYED FOR IMAGE  COMPRESSION IN WHICH A TWODIMENSIONAL WAVELET TRANSFORM IS  EMPLOYED  IN THIS CASE THE HIERARCHICAL STRUCTURE OF THE WAVELET  TRANSFORM IS EXPLOITED SO THAT IF COEFFICIENTS ON ONE STAGE ARE  SMALL THERE IS A HIGH PROBABILITY THAT COEFFICIENTS UNDERNEATH ARE  ALSO SMALL  DETAILS OF  AN ALGORITHM OF THIS SORT ARE GIVEN IN  CITESHAPIRO1993ITEMTIMEFREQUENCY ANALYSIS WAVELETS ARE NATURALLY EMPLOYED IN THE  ANALYSIS OF SIGNALS WHICH HAVE A TIMEVARYING FREQUENCY CONTENT  SUCH AS SPEECH OR GEOPHYSICAL SIGNALSENDDESCRIPTIONBEGINEXERCISESITEM SHOW THAT IF PHIT IS NORMALIZED THEN 2J2PHI2J  T IS NORMALIZEDITEM IN REFEQTWOSCALE2 SHOW THAT THE COEFFICIENTS CN MUST  SATISFY SUMN CN  2ITEM USING REFLEMCCOND SHOW THAT   BEGINENUMERATE  ITEM THE SET OF FUNCTIONS  2J2PHI2J T  N  FORM AN ORTHOGONAL SET FOR EACH  FIXED JITEM THE SET OF FUNCTIONS  2J2 PSI2JT  N FORM AN  ORTHOGONAL SET FOR EACH FIXED J    ENDENUMERATEITEM SHOW THAT THERE IS NO ORTHOGONAL SCALING FUNCTION DEFINED BY A  TWOSCALE EQUATION REFEQTWOSCALE2 WITH EXACTLY THREE NONZERO  COEFFICIENTS C0 C1 AND C2  ITEM FOR THE MULTIRESOLUTION ANALYSIS OF THIS SECTION    BEGINENUMERATE    ITEM SHOW THAT WJ PERP WJ    ITEM SHOW THAT FOR J  J VJ   VJ OPLUS BIGOPLUSK0JJ1 WJK    ENDENUMERATE  ITEM SHOW THAT IF PHIT OBEYS THE TWOSCALE RELATIONSHIP IN    REFEQTWOSCALE1 AND IF PHIHATOMEGA REPRESENTS THE    FOURIER TRANSFORM OF PHIT THEN PHIHATOMEGA  M0OMEGA2PHIHATOMEGA2WHERE BEGINEQUATIONM0OMEGA  FRAC1SQRT2 SUMN HN EJNOMEGALABELEQM0ENDEQUATIONIS THE SCALED DISCRETETIME FOURIER TRANSFORM OF THE COEFFICIENTSEQUENCEITEM SHOW THAT THE ORTHOGONALITY CONDITION REFEQWAVEORTHOG1  IS EQUIVALENT TO  M0OMEGA22  M0OMEGA2PI2  1HINT RECOGNIZE THAT REFEQWAVEORTHOG1 IS A DECIMATEDCONVOLUTION AND USE THE FACT THAT IF THE FOURIER TRANSFORM OF ASEQUENCE ZN IS ZOMEGA THEN THE FOURIER TRANSFORM OF Z2NIS FRAC12ZOMEGA2  ZOMEGA2PIITEM DECIMATION INDEXDECIMATION BECAUSE OF THE CONNECTION OF  WAVELET TRANSFORMS WITH MULTIRATE SIGNALING IT IS WORTHWHILE TO  EXAMINE THE TRANSFORM OF DECIMATED SIGNALS  YOU WILL SHOW THAT IF  YN IS A DECIMATION OF XN YN  XNDTHENBEGINEQUATIONYZ  FRAC1D SUMK0D1 XEJ2PI KDZ1DLABELEQDECIMATEENDEQUATIONBEGINENUMERATEITEM LET PN BE THE PERIODIC SAMPLING SEQUENCE PN  BEGINCASES  1  N  0 PM D PM 2D LDOTS   0  TEXTOTHERWISEENDCASESSHOW THAT PN  FRAC1DSUMK0D1 EJ2PI KNDITEM LET ZN  XNPN  THEN YN  ZND  SHOW THAT YZ  SUMM YM ZM  SUMM ZMITEM FINALLY SHOW THAT REFEQDECIMATE IS TRUEENDENUMERATEITEM COMPUTER EXERCISE  IN THIS EXAMPLE YOU WILL BE INTRODUCED TO A  RUDIMENTARY APPROACH TO DATA COMPRESSION USING WAVELETS  WRITE A  PROGRAM WHICH WAVELET TRANSFORMS DATA THEN TRUNCATES THE DATA USING  A PRESET THRESHOLD THEN INVERSE TRANSFORMS THE DATA  USING  SAMPLED SPEECH OR MUSIC DATA EXPLORE THE QUALITY OF THE  INVERSETRANSFORMED DATA AS A FUNCTION OF THE THRESHOLD  DETERMINE  HOW MANY COEFFICIENTS ARE SET TO ZERO AS A FUNCTION OF THE THRESHOLDENDEXERCISES LOCAL VARIABLES TEXMASTER TEST END THE VEC OPERATOR AND ITS USE TO REPRESENT MATRIX OPERATIONSSECTIONTHE VEC OPERATORLABELSECVECOPINDEXVEC OPERATORBEGINDEFINITION  FOR A MATSIZEMN MATRIX A  ABF1ABF2LDOTSABFN  THE VEC OPERATOR CONVERTS THE MATRIX TO A COLUMN VECTOR BY STACKING  THE COLUMNS OF A VECOPA  BEGINBMATRIX ABF1  ABF2  VDOTS  ABFN  ENDBMATRIXTO OBTAIN A VECTOR OF MN ELEMENTSENDDEFINITIONTHE VEC OPERATION CAN BE COMPUTED IN SC MATLAB BY INDEXING WITH ASINGLE COLON VECOPA  TT AMC  A VECTOR CAN BE RESHAPED USINGTHE TT RESHAPE FUNCTIONBEGINEXAMPLE  LET  A  BEGINBMATRIX123  456 ENDBMATRIXTHEN VECOPA  BEGINBMATRIX1 4 2 5 3 6 ENDBMATRIXENDEXAMPLETHE VEC REPRESENTATION CAN BE USED TO REWRITE A VARIETY OFOPERATIONS  FOR EXAMPLEBEGINEQUATIONTRACEAB  VECOPATT VECOPBLABELEQVECTRACEENDEQUATIONSEE EXERCISE REFEXVECTRACE  BEGINTHEOREM LABELTHMPTOVBEGINEQUATIONVECOPAYB  BT OTIMES A VECOP YLABELEQVECPRODENDEQUATIONENDTHEOREMBEGINPROOF   GRAHAM P 25LET B BE MATSIZEMNOBSERVE THAT THE KTH COLUMN OF AYB CAN BE WRITTEN AS SEESECTION REFSECMATRIXNOT FOR NOTATION AYBK  SUMJ1M BJK AYJ  B1KA  B2KA LDOTS   BNKABEGINBMATRIX Y1  Y2  VDOTS  YN ENDBMATRIXQQUAD K12LDOTSNTHIS IN TURN CAN BE WRITTEN AS AYBK  BKOTIMES A VECOPYSTACKING THE COLUMNS TOGETHER WE OBTAIN THE DESIRED RESULT VECOPAYB  BEGINBMATRIX AYB1  AYB2  VDOTS AYBNENDBMATRIX BEGINBMATRIX B1OTIMES AVECOPY B2OTIMES AVECOPY VDOTS BNOTIMES AVECOPY  ENDBMATRIX BEGINBMATRIX B1OTIMES A B2OTIMES A VDOTS BNOTIMES A ENDBMATRIXVECOPY BTOTIMES A VECOPYENDPROOFBEGINEXAMPLE  THE VEC OPERATOR CAN BE USED TO CONVERT MATRIX EQUATIONS TO VECTOR  EQUATIONS  THE EQUATION BEGINBMATRIX A11  A12  A21  A22 ENDBMATRIXBEGINBMATRIX X11  X12  X21  X22 ENDBMATRIX BEGINBMATRIX C11  C12  C21  C22 ENDBMATRIXCAN BE VECTORIZED BY WRITING AXI  CSO THAT VECOPAXI  VECOPCOR BY REFEQVECPROD IOTIMES A VECOPX  VECOPCTHIS IS EQUIVALENT TO BEGINBMATRIX A11  A12 00 A21  A22 00 00A11A12 00 A21  A22 ENDBMATRIXBEGINBMATRIX X1  X2  X3  X4 ENDBMATRIX BEGINBMATRIXC11  C21  C12  C22ENDBMATRIXENDEXAMPLEINDEXAXBCBEGINEXAMPLE  SUPPOSE IT IS DESIRED TO SOLVE THE EQUATIONBEGINEQUATIONAXB  CLABELEQVECEX1ENDEQUATIONFOR THE MATRIX X  IF A AND B ARE INVERTIBLE ONE METHOD OFSOLUTION IS SIMPLYBEGINEQUATIONX  A1C B1LABELEQVECEX2ENDEQUATIONANOTHER APPROACH IS TO REWRITE REFEQVECEX1 AS YXBF  CBF WHERE Y  BT OTIMES A XBF  VECOPX AND CBF  VECOPCGENERALIZING THE PROBLEM SUPPOSE IT IS DESIRED TO SOLVE A1 X B1  A2 X B2  CDOTS  AS X BS  CFOR X  IN THIS CASE SIMPLE MATRIX INVERSION AS INREFEQVECEX2 WILL NOT SUFFICE  HOWEVER IT CAN BE VECTORIZED ASBEFORE WHERE Y  B1T OTIMES A1  B2T OTIMES A2  CDOTS  BST OTIMESASENDEXAMPLEBEGINDEFINITION  A LINEAR OPERATOR A IS SAID TO BE EM SEPARABLE IF A  A1  OTIMES A2 FOR SOME A1 AND A2 INDEXSEPARABLE OPERATORENDDEFINITIONOPERATIONS INVOLVING SEPARABLE LINEAR OPERATORS CAN BE REDUCED INCOMPLEXITY BY USE OF REFEQVECPROD  FOR EXAMPLE SUPPOSE THATA IS MATSIZEM2M2  COMPUTATION OF THE PRODUCTBEGINEQUATION BBF  A XBFLABELEQVECOPEXENDEQUATIONWILL REQUIRE ON4 OPERATIONS  IF A  A1 OTIMES A2 WHEREEACH AI IS MATSIZEMM THEN REFEQVECOPEX CAN BE WRITTENASBEGINEQUATION B  A2XA1TLABELEQVECOPEX2ENDEQUATIONWHERE B AND X ARE MATSIZEMM  THE TWO MATRIXMULTIPLICATIONS IN REFEQVECOPEX2 REQUIRE A TOTAL OF 2ON3OPERATIONSBEGINEXAMPLE  THE MATRIX A  BEGINBMATRIX2  3  4  6 5  6  10  12 10  15  14  21 25  30  35  42 ENDBMATRIXIS SEPARABLE A  BEGINBMATRIX1  2 5  7 ENDBMATRIX OTIMES BEGINBMATRIX2  3 5  6 ENDBMATRIXENDEXAMPLEANOTHER VECTORIZING PROBLEM THAT OCCURS IN SOME MINIMIZATION PROBLEMSIS GIVEN A MATSIZEMN MATRIX X DETERMINE VECOPXT INTERMS OF VECOPX  THE TRANSPOSE SHUFFLES THE COLUMNS AROUND SOIT MAY BE ANTICIPATED THAT VECOPXT  P VECOPXWHERE P IS A PERMUTATION MATRIX BEGINEXAMPLE  LET X  BEGINBMATRIX X11  X12 X21  X22 ENDBMATRIXTHEN VECOPX  BEGINBMATRIX X11  X21  X12  X22ENDBMATRIX QQUAD TEXTAND QQUADVECOPXT  BEGINBMATRIXX11  X12  X12  X22ENDBMATRIX  BEGINBMATRIX 1  0  0  0 0  0  1  0 0  1  0  0 0  0  0  1 ENDBMATRIXVECOPX ENDEXAMPLE THE PERMUTATION MATRIX CAN BE DETERMINED USING ELEMENT MATRICES OBSERVE THAT X CAN BE WRITTEN IN TERMS OF UNIT ELEMENT MATRICES SEE SECTION REFSECMATRIXNOT AS X  SUMR1M SUMS1N XRS ERSWHERE THE UNIT ELEMENT MATRIX ERS IS MATSIZEMN  THEN XTCAN BE WRITTEN AS  XT  SUMR1M SUMS1N XRS ESRWHERE IN THIS CASE ESR IS MATSIZENM  IT IS STRAIGHTFORWARDTO SHOW SEE EXERCISE REFEXELMATPROD THAT THE RHS CAN BE WRITTEN AS XT  SUMR1M SUMS1N ESR X ESRWITH ESR OF SIZE MATSIZENM  THEN USINGREFEQVECPRODBEGINALIGNED VECOPXT   VECOP SUMR1M SUMS1N ESR X ESR  SUMR1M SUMS1N ERS OTIMES ESR VECOPXENDALIGNEDSO THATBEGINEQUATIONP  SUMR1M SUMS1N ERS OTIMES ESRLABELEQVECTPOSEENDEQUATIONWITH THE UNIT ELEMENT MATRICES SUITABLY SIZEDBEGINEXERCISESITEM LABELEXVECTRACE  SHOW THAT TRACEAB  VECOPATT VECOPBITEM SHOW THAT FOR MATSIZENN MATRICES A AND BBEGINALIGNEDVECOPAB  I OTIMES A VECOP B VECOPAB  BT OTIMES A VECOP I VECOPAB  SUMK1 BTK OTIMES AKENDALIGNEDITEM FIND THE SOLUTION X TO THE EQUATION A1 X B1  A2 X B2  CWHERE A1  BEGINBMATRIX 4  2  1  2 ENDBMATRIXQQUAD B1 BEGINBMATRIX 0  1  1  1 ENDBMATRIX A2  BEGINBMATRIX 5  2  1  0 ENDBMATRIX QQUAD B2 BEGINBMATRIX 2  0  0  2 ENDBMATRIX C  BEGINBMATRIX 1  2  3  4 ENDBMATRIXITEM LABELEXELMATPROD SHOW THAT SUMR1M SUMS1N XRS ESR  SUMR1M SUMS1NESR X ESRITEM THE MATRIX A IS SEPARABLE A  BEGINBMATRIX24  8  12  4 4  12  2  6 12  4  6  2 2  6  1  3 ENDBMATRIXBEGINENUMERATEITEM DETERMINE A1 AND A2 SO THAT A  A1 A2ITEM LET XBF  1234T  COMPUTE THE PRODUCT A XBF BOTH  DIRECTLY AND USING REFEQVECPRODENDENUMERATEENDEXERCISES LOCAL VARIABLES TEXMASTER TEST ENDSECTIONITERATIVE REWEIGHTED LS IRLS FOR PROTECT  LPROTECTPPROTECT OPTIMIZATION LABELSECRWLSINDEXWEIGHTED LEASTSQUARESINDEXITERATIVE REWEIGHTED LEASTSQUARESTHIS CHAPTER HAS FOCUSED LARGELY ON L2 OPTIMIZATION BECAUSE THEPOWER OF THE ORTHOGONALITY THEOREM ALLOWS ANALYTICAL EXPRESSIONS TO BEDETERMINED IN THIS CASE  IN THIS SECTION WE EXAMINE AN ALGORITHM FORDETERMINING SOLUTIONS TO LP OPTIMIZATION PROBLEMS FOR P NEQ 2THE METHOD RELIES UPON WEIGHTED LEASTSQUARES TECHNIQUES BUT USING ADIFFERENT WEIGHTING FOR EACH ITERATIONWE BEGIN BY EXAMINING A WEIGHTED LEASTSQUARES PROBLEM  SUPPOSE ASIN SECTION REFSECHILBAPPROX WE WISH TO DETERMINE A COEFFICIENTVECTOR CBF IN RBBM TO MINIMIZE THE WEIGHTED NORM OF THE ERROREBF IN XBF  A CBF  EBFLET W  STS BE A WEIGHTING MATRIX  THEN TO FIND MINCBF EBFT W EBF  MINCBF EBFT ST S EBFWE USE REFEQWLS2 TO OBTAINBEGINEQUATION CBF  AT ST SA1 AT ST S XBFLABELEQRWLS1ENDEQUATIONNOW CONSIDER THE LP OPTIMIZATION PROBLEMBEGINEQUATION MINCBF XBF  A CBFPP  MINCBF SUMI1M XI  ACBFIPLABELEQRWLS2ENDEQUATIONLET CBF BE THE SOLUTION TO THIS OPTIMIZATION PROBLEM  THEPROBLEM REFEQRWLS2 CAN BE WRITTEN USING A WEIGHTING AS SUMI1M WIXI  A CBFI2WHERE WI  XI  ACBFIP2 PRODUCING A WEIGHTEDLEASTSQUARES PROBLEM WHICH HAS A TRACTABLE SOLUTION  HOWEVER THESOLUTION CANNOT BE FOUND IN ONE STEP BECAUSE CBF IS NEEDED TOCOMPUTE THE APPROPRIATE WEIGHT  IN ITERATIVE REWEIGHTEDLEASTSQUARES THE CURRENT SOLUTION IS USED TO COMPUTE A WEIGHT WHICHIS USED FOR THE NEXT ITERATIONTO THIS END LET SK BE THE WEIGHT MATRIX FOR THE KTHITERATION AND LET CK BE THE CORRESPONDING WEIGHTEDLEASTSQUARES SOLUTION OBTAINED VIA REFEQRWLS1  THE ERROR ATTHE KTH ITERATION IS EBFK  XBF  A CBFKTHEN A NEW WEIGHT MATRIX SK1 IS CREATED ACCORDING TO SK1  DIAGBEGINBMATRIXEK1P22   EK2P22  CDOTS  EKMP22 ENDBMATRIXUSING THIS WEIGHT THE WEIGHTED ERROR MEASURE AT THE K1STITERATION IS EBFK1SK1T SK1 EBFK1  SUMI1M XI A CBFK1IPIF THIS ALGORITHM CONVERGES THEN THE WEIGHTED LEASTSQUARES SOLUTIONPROVIDES A SOLUTION TO THE LP APPROXIMATION PROBLEMHOWEVER IT IS KNOWN THAT THE ALGORITHM AS DESCRIBED HAS SLOWCONVERGENCE CITEBYRDPYNE  A MORE STABLE APPROACH HAS BEEN FOUNDLET CBFHATK1  AT SK1T SK1 A1 ATSK1T SK1XBFAND CBFK1  LAMBDA CBFHATK1  1LAMBDA CBFKFOR SOME LAMBDA IN 01  IT HAS BEEN FOUNDCITEFLETCHER1971KAHNG1972 THAT CHOOSING LAMBDA  FRAC1P1LEADS TO CONVERGENCE PROPERTIES OF THE ALGORITHM SIMILAR TO NEWTONSMETHOD SEE SECTION REFSECNEWTONONE FINAL ENHANCEMENT HAS BEEN SUGGESTED CITEBURRUS1994  ATIMEVARYING VALUE OF P IS CHOSEN SUCH THAT NEAR THE BEGINNING OFTHE ITERATIVE PROCESS P IS CHOSEN TO BE SMALL THEN GRADUALLYINCREASED UNTIL THE DESIRED P IS OBTAINED  THUS PK  MINPGAMMA PK1IS USED FOR SOME SMALL GAMMA1 A TYPICAL VALUE IS GAMMA15ALGORITHM REFALGWLS INCORPORATES THESE IDEASBEGINNEWPROGENVITERATIVE REWEIGHTED LEASTSQUARESIRWLSM   WLSITERATIVE REWEIGHTED LEASTSQUARES ENDNEWPROGENV BEGINEXAMPLE   LP OPTIMIZATION METHODS HAVE BEEN USED FOR FILTER DESIGN   CITEBURRUS1994  IN THIS EXAMPLE WE CONSIDER AN ODD TAPLENGTH   FILTER HZ  SUMN0N HN ZNWITH N EVEN  THE FILTER FREQUENCY RESPONSE CAN BE WRITTEN SEESECTION REFSECEIGFSR AS HEJOMEGA  EJNOMEGA2 HROMEGAWHERE HROMEGA  SUMN0N2 BN COSOMEGA N  BBFTCBFOMEGALET HDOMEGA BE THE MAGNITUDE RESPONSE OF THE DESIRED FILTERWE DESIRE TO MINIMIZE INT0PI HROMEGA  HDOMEGAPDOMEGATHIS CAN BE CLOSELY APPROXIMATED BY SAMPLING THE FREQUENCY RANGE ATLF FREQUENCIES OMEGA0OMEGA1LDOTS OMEGALF1 ANDMINIMIZING SUMK0LF1 HROMEGA  HDOMEGAPTHIS IS NOW EXPRESSED AS A FINITEDIMENSIONAL LP OPTIMIZATIONPROBLEM AND THE METHODS OF THIS SECTION APPLY  SAMPLE CODE THAT SETSUP THE MATRICES FINDS THE SOLUTION THEN PLOTS THE SOLUTION IS SHOWNIN ALGORITHM REFALGWLSFILT  RESULTS OF THIS FOR P4 AND P10ARE SHOWN IN FIGURES REFFIGWLSFILTA AND B RESPECTIVELY  THEP10 RESULT SHOWN CLOSELY APPROXIMATES LINFTY EQUIRIPPLEDESIGNENDEXAMPLEBEGINNEWPROGENVFILTER DESIGN USING IRLSTESTIRWLSM   WLSFILTFILTER DESIGN USING IRLS ENDNEWPROGENV BEGINFIGUREHTBPCENTERINGMBOXSUBFIGUREP4EPSFIGFILEPICTUREDIRIRWLSP4EPSWIDTH045TEXTWIDTH QQUAD SUBFIGUREP10EPSFIGFILEPICTUREDIRIRWLSP10EPSWIDTH045TEXTWIDTH     CAPTIONMAGNITUDE RESPONSE FOR FILTERS DESIGNED USING IRLS     LABELFIGWLSFILT ENDFIGURE LOCAL VARIABLES TEXMASTER TEST ENDLEQSETCOUNTERPAGE1SETCOUNTERFIGURE0SETCOUNTEREQUATION0SETCOUNTERLEMMA0SETCOUNTERTHEOREM0SETCOUNTERDEFINITION0SETCOUNTEREXAMPLE0SETCOUNTEREXERCISE0CHAPTERINTRODUCTION AND MATHEMATICAL NOTATIONLABELCHAPFORMALISMBEGINQUOTESOURCEALBERT CAMUSEM THE MYTH OF SISYPHUSTHE MINDS DEEPEST DESIRE EVEN IN ITS MOST ELABORATE OPERATIONSPARALLELS MANS UNCONSCIOUS FEELING IN THE FACE OF HIS UNIVERSE IT ISAN INSISTENCE UPON FAMILIARITY AN APPETITE FOR CLARITYUNDERSTANDING THE WORLD FOR A MAN IS REDUCING IT TO THE HUMANSTAMPING IT WITH HIS SEALENDQUOTESOURCEIN THIS CHAPTER WE INTRODUCE STATISTICAL DECISION MAKING AS ANINSTANCE OF A GAME IN THE MATHEMATICAL SENSE THEN FORMALIZE THEELEMENTS OF THE PROBLEM  WE WILL THEN PRESENT SOME BASIC NOTATION ANDCONCEPTS RELATED TO RANDOM SAMPLES  WE THEN PRESENT SOME BASIC THEORYWHICH WILL BE OF USE IN OUR STUDYBEGINENUMERATEITEM CONDITIONAL EXPECTATIONSITEM TRANSFORMATIONS OF RANDOM VARIABLESITEM SUFFICIENT STATISTICSITEM EXPONENTIAL FAMILIESENDENUMERATETHE CHAPTER CONCLUDES WITH AINVESTIGATION OF EM SUFFICIENCY HOW MUCH INFORMATION NEEDS TO BERETAINED TO PERFORM THE DESIRED DETECTION OR ESTIMATIONTHIS INTRODUCTION IS FOLLOWED BY ANEXAMINATION OF CONCEPTS FROM PROBABILITY THAT WILL SUPPORT OURUNDERSTANDING  THIS WILL INCLUDE A CAREFUL DEFINITION OF PROBABILITYSPACES PROBABILITY MEASURES RANDOM VARIABLES EXPECTATION ANDCONDITIONAL EXPECTATION  IN SOME REGARDS THE LEVEL OF THE MATERIALPRESENTED PROVIDES A BIGGER HAMMER THAN IS NECESSARY TO ANUNDERSTANDING OF THE APPLICATION MATERIAL  HOWEVER IN THE INTERESTOF LAYING A FOUNDATION SUITABLE FOR ADVANCED STUDY AND UNDERSTANDINGOF RESEARCH RESULTS THE STUDENT IS ENCOURAGED TO BECOME FAMILIAR WITHTHE NOTATION  EQUIPPED WITH THESE TOOLS WE ADDRESS THE IMPORTANTTOPIC OF SUFFICIENCY  HOW MUCH INFORMATION NEEDS TO BE RETAINED INORDER TO MAKE VALID ESTIMATES OF PARAMETERS OF A RANDOM VARIABLETHE NOTION OF SUFFICIENCY WILL ARISE FREQUENTLY IN THE WORK THAT FOLLOWSSECTIONINTRODUCTION TO DETECTION AND ESTIMATION THEORYOBSERVATIONS OF SIGNALS IN PHYSICAL SYSTEMS ARE FREQUENTLY MADE IN THEPRESENCE OF NOISE AND EFFECTIVE PROCESSING OF THESE SIGNALS OFTENRELIES UPON TECHNIQUES DRAWN FROM THE STATISTICAL LITERATURE  THESESTATISTICAL TECHNIQUES ARE GENERALLY CATEGORIZED INTO TWO DIFFERENTKINDS OF PROBLEMS ILLUSTRATED IN THE FOLLOWING EXAMPLESBEGINEXAMPLE  BEGINDESCRIPTION  ITEMDETECTION LET XT  A COS2PI FC T QQUAD T IN 0TWHERE A TAKES ON ONE OF TWO VALUES A IN 11  THE SIGNALXT IS OBSERVED IN NOISE YT  XTNTWHERE NT IS A RANDOM PROCESS  A EM DETECTION PROBLEM IS TOCHOOSE BETWEEN THE TWO VALUES OF A THE SIGNAL AMPLITUDE GIVEN THE OBSERVATION YT T IN 0T  THIS PROBLEM ARISES IN THE TRANSMISSION OF BINARY DATA OVER A NOISY CHANNELITEMESTIMATION THE SIGNAL YT  XT COS2PI FC T  THETA   NT IS MEASURED AT A RECEIVER WHERE THETA IS AN UNKNOWN PHASE  AN EM ESTIMATION PROBLEM IS TO DETERMINE THE PHASE BASED UPON  OBSERVATION OF THE SIGNAL OVER SOME INTERVAL OF TIMEENDDESCRIPTIONENDEXAMPLEDETECTION THEORY INVOLVES MAKING A CHOICE OVER SOME COUNTABLE USUALLYFINITE SET OF OPTIONS WHILE ESTIMATION INVOLVES MAKING A CHOICE OVERA CONTINUUM OF OPTIONSSUBSECTIONGAME THEORY AND DECISION THEORY LABELSECGAMEINTROTAKING A BROADER PERSPECTIVE THE COMPONENT OF STATISTICALTHEORY THAT WE WILL BE CONCERNED WITH FITS IN AN EVEN LARGERMATHEMATICAL CONSTRUCT THAT OF GAME THEORY  THEREFORE TO ESTABLISHTHESE CONNECTIONS TO INTRODUCE SOME NOTATION AND TO PROVIDE A USEFULCONTEXT FOR FUTURE DEVELOPMENT WE WILL BEGIN OUR DISCUSSION OF THISTOPIC WITH A BRIEF DETOUR INTO THE GENERAL AREA OF MATHEMATICAL GAMESIN A TWOPERSON GAME EACH PERSON EITHER OF WHICH MAY BE NATUREHAS OPTIONS OPEN TO THEM AND EACH ATTEMPTS TO MAKE A CHOICE THATAPPEARS TO HELP THEM ACHIEVE ITS GOAL EG OF WINNING  IN A EM  ZEROSUM GAME ONE PERSONS LOSS IS ANOTHER PERSONS GAIN  MOREFORMALLY WE HAVE THE FOLLOWINGBEGINDEFINITIONLABELDEFGAME1  A TWOPERSON ZEROSUM MATHEMATICAL GAME INDEXGAME DEFINITION  WHICH WE WILL REFER TO FROM NOW ON SIMPLY AS A BF GAME CONSISTS  OF THREE BASIC COMPONENTSBEGINENUMERATEITEM A NONEMPTY SET THETA1 OF POSSIBLE ACTIONS AVAILABLE TOPLAYER 1ITEM A NONEMPTY SET THETA2 OF POSSIBLE ACTIONS AVAILABLE TOPLAYER 2ITEM A LOSS FUNCTION LMC  THETA1 TIMES THETA2 MAPSTO RBBREPRESENTING THE LOSS INCURRED BY PLAYER 1 WHICH UNDER THE ZEROSUMCONDITION CORRESPONDS TO THE GAIN OBTAINED BY PLAYER 2 ENDENUMERATEANY SUCH TRIPLE THETA1 THETA2 L DEFINES A GAME  THE LOSSES ARE EXPRESSED WITH RESPECT TO PLAYER 1 A NEGATIVE LOSS ISINTERPRETED AS A GAIN FOR PLAYER 2ENDDEFINITIONHERE IS A SIMPLE EXAMPLE CITEPAGE 2FERGUSON67BEGINEXAMPLE  LABELEXMEVENODD1SC ODD OR EVEN  TWO CONTESTANTS SIMULTANEOUSLY  PUT UP EITHER ONE OR TWO FINGERS  PLAYER 1 WINS IF THE SUM OF THE  DIGITS SHOWING IS ODD AND PLAYER 2 WINS IF THE SUM OF THE DIGITS  SHOWING IS EVEN  THE WINNER IN ALL CASES RECEIVES IN DOLLARS THE  SUM OF THE DIGITS SHOWING THIS BEING PAID TO HIM BY THE LOSER    TO CREATE A TRIPLE THETA1THETA2 L FOR THIS GAME WE DEFINE  THETA1  THETA2  1 2 AND DEFINE A LOSS FUNCTION BY BEGINALIGNEDL11    2L12    3L21    3L22    4ENDALIGNEDIT IS CUSTOMARY TO ARRANGE THE LOSS FUNCTION INTO A EM LOSS MATRIXAS DEPICTED IN FIGURE REFGAME11BEGINFIGUREHTBBEGINCENTERINPUTPICTUREDIRLOSS1LATEXENDCENTERCAPTIONLOSS FUNCTION OR MATRIX FOR ODD OR EVEN GAMELABELGAME11ENDFIGUREENDEXAMPLEAN IMPORTANT CLASS OF GAMES ARE THOSE IN WHICH ONE PLAYER IS ABLE TOOBTAIN INFORMATION RELATING TO THE CHOICE MADE BY THE OPPONENT BEFORECOMMITTING TO A CHOICE  TO ILLUSTRATE SUPPOSE WITH THE ODD OREVEN GAME THAT PLAYER 2 IS ABLE TO OBSERVE DATA REGARDING THEACTION TO BE TAKEN BY PLAYER 1 BUT THAT THESE DATA ARE SUBJECT TOERROR  THIS MODIFICATION IS A SIGNIFCANT COMPLICATION TO THE ORIGINALGAME WHICH MUST NOW BE EXPANDED TO ACCOUNT FOR THIS ADDITIONALSTRUCTURE  ONE WAY TO INCORPORATE THIS ADDITIONAL INFORMATION IS FORPLAYER 2 TO MODEL THE OBSERVATION IN TERMS OF PROBABILITY THEORYTHE CHARACTERIZATION OF UNCERTAIN INFORMATION IN TERMS OF PROBABILITYTHEORY PROVIDES A POWERFUL ADDITION TO THE BASIC GAMETHEORETICSTRUTURE PROVIDED BY DEFINITION REFDEFGAME1  THIS ADDITION IS OFGREAT VALUE IN THE CONTEXT WE INTEND TO CONCENTRATE OURATTENTIONTHAT OF DECISION AND ESTIMATION THEORY  WE WILL VIEWDECISION AND ESTIMATION THEORY AS A TWOPERSON GAME BETWEEN NATURE INTHE ROLE OF PLAYER 1 AND A DECISIONMAKING OR COMPUTATIONAL AGENT INTHE ROLE OF PLAYER 2  THE CHOICES AVAILABLE TO NATURE AREREPRESENTED AS ELEMENTS OF A SET THETA  THE DECISIONS THAT THEAGENT MAKES ARE REPRESENTED AS ELEMENTS OF A SET DELTA  INADDITION THE AGENT WILL HAVE AT ITS DISPOSAL SAMPLES OF A RANDOMVARIABLE OR VECTOR X  AS WITH THE ORIGINAL TWOPERSON GAME THEREIS A LOSS FUNCTION L  A BF STATISTICAL GAME IS A GAME REPRESENTED BY THE TRIPLETHETADELTAL COUPLED WITH A RANDOM OBSERVABLE X DEFINEDOVER A BF SAMPLE SPACE OR BF OBSERVATION SPACE XC WHOSEDISTRIBUTION DEPENDS ON THE STATE THETA IN THETA CHOSEN BYNATURE  ASSOCIATED WITH THIS RANDOM VARIABLE IS A DECISION  FUNCTION PHI THAT MAPS THE OBSERVED VALUES OF X INTO THEDECISION SPACEBEGINENUMERATEITEM THETASUBSET RBBK  IS A NONEMPTY SET OF POSSIBLE STATES OF  NATURE OR PARAMETERS  THETA IS SOMETIMES REFERRED TO AS  THE BF PARAMETER SPACE  AN ELEMENT OF THETA IS CALLED  THETA FOR A SCALAR PARAMETER OR THETABF FOR A VECTOR  PARAMETERITEM DELTA IS A NONEMPTY SET OF POSSIBLE DECISIONS AVAILABLE TO  THE AGENT SOMETIMES CALLED THE BF DECISION SPACE  AN ELEMENT OF  DELTA IS REPRESENTED AS DELTAITEM LMC THETA TIMES DELTA MAPSTO RBB IS A BF LOSS    FUNCTION OR BF COST FUNCTION  ITEM XMC XC MAPSTO RBBN  NGEQ 1 IS A RANDOM VARIBLE OR  VECTOR WHOSE CUMULATIVE DISTRIBUTION FUNCTION IS FX MC XC  TIMES THETA MAPSTO 01  WE SHALL REPRESENT THIS CUMULATIVE  DISTRIBUTION FUNCTION AS FXXTHETA  THAT IS THE DISTRIBUTION  OF X IS GOVERNED BY A PARAMETERS THETA IN THETAITEM PHI MC XC MAPSTO DELTA IS A BF DECISION RULE  ALTERNATIVELY TERMED A BF STRATEGY OR BF DECISION FUNCTION  OR BF TEST THAT PROVIDES THE COUPLING BETWEEN THE OBSERVATIONS  AND THEREFORE THE STATE OF NATURE THROUGH FXCDOT  THETA AND  THE DECISIONSENDENUMERATEIN THE DETECTION OR ESTIMATION STATISTICAL GAME NATURE CHOOSES APOINT THETA IN THETA AND AN OBSERVATION XXINXC IS GENERATEDAT RANDOM ACCORDING TO THE DISTRIBUTION FXXTHETA  THE AGENTUSING THE OBSERVATION X BUT WITHOUT OTHER EXPLICIT KNOWLEDGE OFNATURES CHOICE CHOOSES AN ACTION PHIX  DELTA IN DELTA  ASA CONSEQUENCE OF THESE CHOICES THE AGENT EXPERIENCES A LOSSLTHETADELTA  THE ELEMENTS OF THIS STRUCTURE ARE REPRESENTED INFIGURE REFFIGDETEST1BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRDETEST1    CAPTIONELEMENTS OF THE STATISTICAL DECISION GAME    LABELFIGDETEST1  ENDCENTERENDFIGUREBEGINEXAMPLE LABELEXMBINCHANNEL1  IN THIS EXAMPLE WE MODIFY THE CONCEPTS FROM THE GAME IN EXAMPLE  REFEXMEVENODD1 TO APPLY TO A COMMUNICATION CHANNEL  CONSIDER  THE BINARY CHANNEL SHOWN IN FIGURE REFFIGBINCHAN  THE BITS ZERO  OR ONE CAN BE CHOSEN WHERE THE TRANSMITTER TAKES THE ROLE OF PLAYER  1 OR NATURE  THE PARAMETER SPACE IS THUS THETA  01  AS THE  TRANSMITTED BITS PASS THROUGH THE CHANNEL THEY ARE CORRUPTED  THE  RECEIVER IS TO DECIDE WHETHER A 0 OR A 1 WAS SENT  THUS THE  DECISION SPACE IS DELTA  01  IN A COMMUNICATION PROBLEM A  COMMON COST STRUCTURE IS TO IMPOSE A COST OF 1 ON INCORRECT  DECISIONS AND A COST OF 0 ON CORRECT DECISIONS  THUS LTHETADELTA  BEGINCASES  0  TEXTIF  THETA  DELTA   1  TEXTIF  THETA NEQ DELTAENDCASESENDEXAMPLEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRBSC1    CAPTIONA SIMPLE BINARY COMMUNICATIONS CHANNEL    LABELFIGBINCHAN  ENDCENTERENDFIGURESOME OF THE ISSUES THAT WILL ARISE IN OUR EXPLORATION ARE THEFOLLOWINGBEGINENUMERATEITEM DETERMINATION OF THE DECISION RULE PHI BY WHICH THE AGENT  MAKES A DECISION  A COMMON APPROACH IS TO CHOOSE PHI SUCH THAT  THE AVERAGE LOSS IS AS SMALL AS POSSIBLE  THIS DECISION RULE IS  FUNDAMENTAL TO THE DETECTION OR ESTIMATION PROBLEM AS IT INDICATES  GIVEN AN OBSERVATION WHICH ACTION ESTIMATE DECISION SHOULD BE  MADE  WE SHALL DENOTE THE SPACE OF ALL POSSIBLE DECISION RULES AS  D  THUS THE DESIGN PROBLEM IS TO SELECT SOME PHI IN D IN  SUCH A WAY THAT THE GOALS OF THE AGENT ARE METITEM EVALUATION OF THE QUALITY OF THE DECISION RULE  FOR A DETECTION  PROBLEM THE QUALITY MIGHT BE MEASURED FOR EXAMPLE IN TERMS OF  PROBABILITY OF ERROR PROBABILITY OF CONDITIONAL ERROR OR COST OF  FALSE ALARM  FOR AN ESTIMATION PROBLEM THE QUALITY OF THE DECISION  RULE AND ITS RESULTING ESTIMATE IS EXAMINED IN TERMS OF THE  BIAS AND VARIANCE OF THE ESTIMATEITEM IN SOME PROBLEMS THE QUESTION OF INVARIANCE ARISES HOW MAY THE  DETECTOR OR ESTIMATOR BE DEVELOPED IN SUCH A WAY THAT IT IS  INSENSITIVE INVARIANT TO TRANSFORMATIONS ON THE DATAENDENUMERATESUBSECTIONRANDOMIZATIONWE INTRODUCED THE DECISON RULE OR STRAGETY PHI AS A SINGLEFUNCTION MAPPING OBSERVATIONS INTO THE DECISION SPACE  SUCH AFUNCTION IS TERMED A BF PURE STRATEGY OR BF NONRANDOMIZED  DECISION RULE  WE MAY GENERALIZE THE NOTION OF A DECISION RULEHOWEVER BY SPECIFYING A EM PROBABILITY DISTRIBUTION VARPHIOVER THE SPACE OF ALL POSSIBLE NONRANDOMIZED RULES  SUCH A DECISIONRULE IS CALLED A BF MIXED STRATEGY OR BF RANDOMIZED DECISION  RULE INDEXRANDOMIZED RULE  LET D DENOTE THE SPACE OF ALLNONRANDOMIZED DECISION RULES AND LET D DENOTE THE SPACE OF ALLRANDOMIZED DECISION RULES  THEN VARPHI D RIGHTARROW 01 IS APROBABILITY DISTRIBUTION THAT SPECIFIES THE PROBABILITY OF SELECTINGTHE ELEMENTS OF D  IF D CONTAINS COUNTABLY MANY ELEMENTS PHI1 PHI2 LDOTSLET VARPHI  PI1 PI2 LDOTS  PII GEQ 0 I12LDOTSSUMI PII 1 WITH THE UNDERSTANDING THAT WE INVOKE DECISIONRULE PHII WITH PROBABILITY PII  FOR EXAMPLE SUPPOSE THEREARE TWO DISTINCT PURE STRATEGIES SO D  PHI1PHI2  DEFINEBEGINDISPLAYMATHVARPHIPI   PI 1PI 0LEQ PILEQ 1ENDDISPLAYMATHWITH PI  PPHI1  1PPHI2 WHERE PPHII IS THEPROBABILITY THAT RULE PHII I12 IS INVOKEDCLEARLY D  VARPHIPI PI IN 01  APPLYING THE RANDOMIZED RULEVARPHIPI MEANS THAT THE NONRANDOMIZED RULE PHI1 WILL BESELECTED WITH PROBABILITY PI AND PHI2 WILL BE SELECTED WITHPROBABILITY 1PI  NONRANDOMIZED RULES MAY BE VIEWED AS ADEGENERATE RANDOMIZED RULES SUCH THAT ALL OF THE PROBABILITY MASS OFTHE RANDOMIZED RULE IS APPLIED TO A SINGLE PURE STRATEGY  FOREXAMPLE LETTING PI  1 MEANS THAT PHI1 WILL BE SELECTED WITHPROBBILITY ONERANDOMIZED DECISION RULES CONSTITUTE AN IMPORTANT MATHEMATICAL CONCEPTTHAT IS NECESSARY TO ESTABLISH FUNDAMENTAL RESULTS SUCH AS THENEYMANPEARSON LEMMA SEE SECTION REFSECNP AND THEMINIMAX THEOREM SEE SECTION REFSECMINIMAXTHEOREM  ALTHOUGH THE MATHEMATICAL TREATMENT OF RANDOMIZED RULES IS ABOVEREPROACH THE INTERPRETATION OF ACTUALLY APPLYING RANDOMIZEDRULES IS A TOPIC WORTHY OF CONSIDERABLE DEBATE  FOR AN INTERSTINGDISCUSSION OF THIS CONCEPT SEE FOR EXAMPLE CITELUCERAIFFA57SUBSECTIONSPECIAL CASESTHE ABOVE FRAMEWORK PROVIDES A FORMALISM FOR MUCH OF THE STATISTICALANALYSIS WE WILL DO IN THIS TEXT  HOWEVER ONLY A PART OF STATISTICSIS REPRESENTED BY THIS FORMALISM  WE WILL NOT DISCUSS SUCH TOPICS ASTHE CHOICE OF EXPERIMENTS THE DESIGN OF EXPERIMENTS OR SEQUENTIALANALYSIS  IN EACH CASE HOWEVER ADDITIONAL STRUCTURE COULD BE ADDEDTO THE BASIC FRAMEWORK TO INCLUDE THESE TOPICS AND THE PROBLEM COULDBE REDUCED AGAIN TO A SIMPLE GAME  MOST OF THE BODY OF STATISTICALDECISIONMAKING INVOLVES THREE SPECIAL CASES OF THE GENERAL GAMEFORMULATION PRESENTED ABOVEBEGINENUMERATEITEM EM DELTA CONSISTS OF TWO POINTS DELTA   DELTA0    DELTA1  CORRESPONDING TO EACH DECISION IS A HYPOTHESIS  BY  CHOOSING DECISION DELTA0 THE AGENT ACCEPTS HYPOTHESIS H0  THEREBY REJECTING HYPOTHESIS H1 AND BY CHOOSING DECISION  DELTA1 THE AGENTS ACCEPTS HYPOTHESIS H1 THEREBY REJECTING  HYPOTHESIS H0  IN THIS CASE WITH ONLY TWO DECISION THE PROBLEM  IS CALLED A EM BINARY HYPOTHESIS TESTING    PROBLEM INDEXHYPOTHESIS TESTINGFOR EXAMPLE CONSIDER THE FOLLOWING SCENARIO  SUPPOSE THETA  RBBAND THE LOSS FUNCTION ISBEGINEQUATIONBEGINSPLITLTHETA DELTA1   LEFTBEGINARRAYCCC ELL1  RM IF  THETA  THETA0 0  RM IF   THETALEQ THETA0ENDARRAYRIGHTLTHETA DELTA2   LEFTBEGINARRAYCCC 0  RM IF  THETA  THETA0 ELL2  RM IF   THETALEQTHETA0 ENDARRAYRIGHTENDSPLITLABELEQL1COSTENDEQUATIONWHERE THETA0 IS SOME FIXED NUMBER AND ELL1 AND ELL2 AREPOSITIVE NUMBERS  WITH THIS EXAMPLE WE WOULD LIKE TO TAKE ACTIONDELTA1 IF THETA LEQ THETA0 AND ACTION DELTA2 IF THETA THETA0AS A SPECIFIC EXAMPLE OF A HYPOTHESIS TESTING PROBLEM SUPPOSE THAT ARADAR SIGNAL IS EXAMINED AT A RECEIVER TO DETERMINE WHETHER A TARGETIS PRESENT  SUPPOSE THE OBSERVED RETURN IS OF THE FORMBEGINDISPLAYMATHX  THETA  NENDDISPLAYMATHWHERE THETA REPRESENTS THE REFLECTED ENERGY OF A RADAR SIGNAL ANDN IS RECEIVER NOISE  IF THETA IS SUFFICIENTLY SMALL THEN WECONCLUDE THAT THERE IS NO REFLECTED SIGNAL AND HENCE NO TARGETALTHOUGH THETA MAY TAKE ON A CONTINUUM OF VALUES THESE REPRESENTONLY TWO STATES OF NATURE WHICH ARE SUMMARIZED IN THE FOLLOWING TWOHYPOTHESES  BEGINALIGNEDH0MC  TEXT NO TARGET PRESENT  THETA LEQ THETA0 H1MC  TEXT TARGET PRESENT  THETA  THETA0ENDALIGNEDIN STATISTICAL PARLANCE H0 IS TERMED THE BF NULL HYPOTHESISINDEXNULL HYPOTHESIS AND H1 THE BF ALTERNATIVE HYPOTHESISTHE AGENT MAKES ON THE BASIS OF OBSERVING X A DECISION AND ITSASSOCIATED ACTION ABOUT WHAT THE STATE OF NATURE IS  WITH THISSIMPLE PROBLEM FOUR OUTCOMES ARE POSSIBLEBEGINDESCRIPTIONITEM H0 TRUE CHOOSE DELTA0 TARGET NOT PRESENT DECIDE  TARGET NOT PRESENT CORRECT DECISIONITEM H1 TRUE CHOOSE DELTA1 TARGET PRESENT DECIDE TARGET  PRESENT CORRECT DECISIONITEM H1 TRUE CHOOSE DELTA0 TARGET PRESENT DECIDE TARGET  NOT PRESENT BF MISSED DETECTION  THIS TYPE OF ERROR IS ALSO  KNOWN AS A BF TYPE I ERROR INDEXTYPE I ERRORITEM H0 TRUE CHOOSE DELTA1 TARGET NOT PRESENT DECIDE  TARGET PRESENT BF FALSE ALARM  THIS TYPE OF ERROR IS ALSO KNOWN  AS A BF TYPE II ERROR INDEXTYPE II ERRORENDDESCRIPTIONWITH THIS STRUCTURE IN PLACE THE PROBLEM IS TO DETERMINE THE DECISIONFUNCTION PHIX WHICH MAKES A SELECTION OUT OF DELTA BASED ONTHE OBSERVED VALUE OF X  IN CHAPTER REFCHAPDETECTION WE WILLPRESENT TWO WAYS OF DEVELOPING THE DECISION FUNCTIONBEGINITEMIZEITEM THE NEYMANPEARSON TEST IN WHICH THE TEST IS DESIGNED FOR  MAXIMUM PROBABILITY OF DETECTION FOR A FIXED PROBABILITY OF FALSE  ALARM  ITEM THE BAYES TEST IN WHICH AN AVERAGE COST IS MINIMIZED  BY  APPROPRIATE SELECTION OF COSTS THIS IS EQUIVALENT TO MINIMIZING THE  PROBABILITY OF ERROR BUT OTHER MORE GENERAL COSTS AND DECISION  STRUCTURES CAN BE DEVELOPEDENDITEMIZEIN EACH CASE THE TEST CAN BE EXPRESSED IN TERMS OF A EM LIKELIHOOD  RATIO FUNCTIONITEM EM DELTA CONSISTS OF M POINTS DELTA  DELTA1DELTA2 LDOTS DELTAM M GEQ 3  THESE PROBLEMS ARE CALLEDEM MULTIPLE DECISION PROBLEMS OR EM MULTIPLE HYPOTHESIS TESTINGPROBLEMS  MULTIPLE HYPOTHESIS TESTING PROBLEMS ARISE IN DIGITAL COMMUNICATIONIN WHICH THE SIGNAL CONSTELLATIONS HAVE MORE THAN 2 POINTS AND INPATTERN RECOGNITION PROBLEMS IN WHICH ONE OF M CLASSES OF DATA ARETO BE DISTINGUISHEDITEM EM DELTA CONSISTS OF THE REAL LINE DELTA  RBB  SUCH  DECISION PROBLEMS ARE REFERRED TO AS EM POINT ESTIMATION OF A REAL    PARAMETER  ESTIMATION PROBLEMS APPEAR IN A VARIETY OF CONTEXTS  TARGET BEARING ESTIMATION FREQUENCY ESTIMATION MODEL PARAMETER  ESTIMATION STATE ESTIMATION PHASE ESTIMATION AND SYMBOL TIMING  TO NAME BUT A FEW  CONSIDER THE CASE WHERE THETA  RBB AND THE LOSS FUNCTION IS  GIVEN BYBEGINDISPLAYMATHLTHETA DELTA  CTHETA  DELTA2ENDDISPLAYMATHWHERE C IS SOME POSITIVE CONSTANT  A DECISION FUNCTION D INTHIS CASE IS A REALVALUED FUNCTION DEFINED ON THE SAMPLE SPACE ANDIS OFTEN CALLED AN EM ESTIMATE OF THE TRUE UNKNOWN STATE OF NATURETHETA  IT IS THE AGENTS DESIRE TO CHOOSE THE FUNCTION DTO MINIMIZE AVERAGE LOSSENDENUMERATEBEGINEXERCISESITEM THOUGHT QUESTION CONSIDER THE WELLKNOWN GAME OF PRISONERS  DILEMMA  TWO AGENTS DENOTED X1 AND X2 ARE ACCUSED OF A  CRIME  THEY ARE INTERROGATED SEPARATELY BUT THE SENTENCES THAT ARE  PASSED ARE BASED UPON THE JOINT OUTCOME  IF THEY BOTH CONFESS THEY  ARE BOTH SENTENCED TO A JAIL TERM OF THREE YEARS  IF NEITHER  CONFESSES THEY ARE BOTH SENTENCED TO A JAIL TERM OF ONE YEAR  IF  ONE CONFESSES AND THE OTHER REFUSES TO CONFESS THEN THE ONE WHO  CONFESSES IS SET FREE AND THE ONE WHO REFUSES TO CONFESS IS  SENTENCED TO A JAIL TERM OF FIVE YEARS  THIS PAYOFF MATRIX IS  ILLUSTRATED IN FIGURE REFFIGPAYOFF THE FIRST ENTRY IN EACH QUADRANT  OF THE PAYOFF MATRIX CORRESPONDS TO X1S PAYOFF AND THE SECOND  ENTRY CORRESPONDS TO X2S PAYOFF  THIS PARTICULAR GAME  REPRESENTS AN SLIGHT EXTENSION TO OUR ORIGINAL DEFINITION SINCE IT  IS NOT A ZEROSUM GAME    WHEN PLAYING SUCH A GAME A REASONABLE STRATEGY IS FOR EACH AGENT TO  MAKE A CHOICE SUCH THAT ONCE CHOSEN NEITHER PLAYER WOULD HAVE AN  INCENTIVE TO DEPART UNILATERALLY FROM THE OUTCOME  SUCH A DECISION  PAIR IS CALLED A EM NASH EQUILIBRIUM INDEXNASH EQUILIBRIUM  POINT  IN OTHER WORDS AT THE NASH EQUILIBRIUM POINT BOTH PLAYERS  CAN ONLY HURT THEMSELVES BY DEPARTING FROM THEIR DECISION  WHAT IS  THE NASH EQUILIBRIUM POINT FOR THE PRISONERS DILEMMA GAME  EXPLAIN  WHY THIS PROBLEM IS CONSIDERED A DILEMMA  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODEBEGINTABULARCCC HLINE  MULTICOLUMN2CX2  CLINE23 X1  SILENT  CONFESSES  HLINE SILENT  11  50  HLINE CONFESSES  05  33  HLINEENDTABULAR      CAPTIONA TYPICAL PAYOFF MATRIX FOR THE PRISONERS DILEMMA      LABELFIGPAYOFF    ENDCENTER  ENDFIGUREITEM VERIFY THE CONTENTS OF THE RISK MATRIX FOR THE STATISTICAL SC    ODD OR EVEN GAMEITEM SHOW THAT THE RISK FUNCTION FOR THE HYPOTHESIS TESTING PROBLEM  WITH THE COSTS IN REFEQL1COST ISBEGINDISPLAYMATHRTHETA D  LEFT BEGINARRAYCCC ELL1 PXTHETAX IN XCDX DELTA1  RM IF   THETA  THETA0ELL2PXTHETA X IN XC DX  DELTA2  RM IF   THETALEQ THETA0ENDARRAYRIGHTENDDISPLAYMATHENDEXERCISESINPUTDETESTDIRPROBSPACEINPUTDETESTDIRINTROESTINPUTDETESTDIRCONDEXPINPUTDETESTDIRSUFFICIENTSETEXSECTREFSECGAMEINTROBEGINEXERCISESITEM CONSIDER THE WELLKNOWN GAME OF PRISONERS DILEMMA  TWO AGENTS  DENOTED X1 AND X2 ARE ACCUSED OF A CRIME  THEY ARE  INTERROGATED SEPARATELY BUT THE SENTENCES THAT ARE PASSED ARE BASED  UPON THE JOINT OUTCOME  IF THEY BOTH CONFESS THEY ARE BOTH  SENTENCED TO A JAIL TERM OF THREE YEARS  IF NEITHER CONFESSES THEY  ARE BOTH SENTENCED TO A JAIL TERM OF ONE YEAR  IF ONE CONFESSES AND  THE OTHER REFUSES TO CONFESS THEN THE ONE WHO CONFESSES IS SET FREE  AND THE ONE WHO REFUSES TO CONFESS IS SENTENCED TO A JAIL TERM OF  FIVE YEARS  THIS PAYOFF MATRIX IS ILLUSTRATED IN FIGURE  REFFIGPAYOFF THE FIRST ENTRY IN EACH QUADRANT OF THE PAYOFF  MATRIX CORRESPONDS TO X1S PAYOFF AND THE SECOND ENTRY  CORRESPONDS TO X2S PAYOFF  THIS PARTICULAR GAME REPRESENTS AN  SLIGHT EXTENSION TO OUR ORIGINAL DEFINITION SINCE IT IS NOT A  ZEROSUM GAME    WHEN PLAYING SUCH A GAME A REASONABLE STRATEGY IS FOR EACH AGENT TO  MAKE A CHOICE SUCH THAT ONCE CHOSEN NEITHER PLAYER WOULD HAVE AN  INCENTIVE TO DEPART UNILATERALLY FROM THE OUTCOME  SUCH A DECISION  PAIR IS CALLED A EM NASH EQUILIBRIUM INDEXNASH EQUILIBRIUM  POINT  IN OTHER WORDS AT THE NASH EQUILIBRIUM POINT BOTH PLAYERS  CAN ONLY HURT THEMSELVES BY DEPARTING FROM THEIR DECISION  WHAT IS  THE NASH EQUILIBRIUM POINT FOR THE PRISONERS DILEMMA GAME  EXPLAIN  WHY THIS PROBLEM IS CONSIDERED A DILEMMA  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODEBEGINTABULARCCC HLINE  MULTICOLUMN2CX2  CLINE23 X1  SILENT  CONFESSES  HLINE SILENT  11  50  HLINE CONFESSES  05  33  HLINEENDTABULAR      CAPTIONA TYPICAL PAYOFF MATRIX FOR THE PRISONERS DILEMMA      LABELFIGPAYOFF    ENDCENTER  ENDFIGUREITEM VERIFY THE CONTENTS OF THE RISK MATRIX FOR THE STATISTICAL SC    ODD OR EVEN GAMEITEM SHOW THAT THE RISK FUNCTION FOR THE HYPOTHESIS TESTING PROBLEM  WITH THE COSTS IN REFEQL1COST ISBEGINDISPLAYMATHRTHETA D  LEFT BEGINARRAYCCC ELL1 PXTHETAX IN XCDX DELTA1  RM IF   THETA  THETA0ELL2PXTHETA X IN XC DX  DELTA2  RM IF   THETALEQ THETA0ENDARRAYRIGHTENDDISPLAYMATHEXSKIPSETEXSECTREFSECSUFFSTAT  ITEM LET XBF  XBF1XBF2LDOTSXBFN DENOTE A RANDOM    SAMPLE OF AN MDIMENSIONAL GAUSSIAN RANDOM VECTOR XI WHERE    XBFI SIM NCMUBFR  SHOW THAT THE STATISTICS MBF  FRAC1N SUMI1N XBFIAND S2  SUMI1N  XBFI  MBFXBFI MBFTARE SUFFICIENT FOR MUBFR  THE MATRIX S2 IS CALLED THE EM  SCATTER MATRIX OF THE DATA  HINT SUMI XBFI  MUBFT R1 XBFI  MUBF  TRACELEFTR1  SUMI XBFI  MUBF XBFI  MUBFTRIGHTITEM A BF POISSON RANDOM VARIABLE HAS PMF INDEXPOISSON RANDOM VARIABLEINDEXRANDOM VARIABLEPOISSON FXXTHETA  ETHETA FRAC1X THETAXWHERE THETA0 IS THE PARAMETER OF THE DISTRIBUTION  WE WRITE X SIM PCTHETATHE POISSON DISTRIBUTION MODELS THE DISTRIBUTION OF THE NUMBER OFEVENTS THAT OCCUR IN THE UNIT INTERVAL 01 WHEN THE EVENTS AREOCCURRING AT AN AVERAGE RATE OF THETA EVENTS PER UNIT TIMELET XBF X1X2LDOTSXNT BE A RANDOM SAMPLE WHERE EACH XI ISPOISSON DISTRIBUTED BEGINENUMERATEITEM SHOW THAT IF X SIM PCTHETA THEN EX  THETA AND  VARX  THETAITEM SHOW THAT  K  SUMI1N XIIS SUFFICIENT FOR THETAITEM DETERMINE THE DISTRIBUTION OF KENDENUMERATEITEM LET XBF BE A RANDOM VECTOR WITH DENSITY  FXBFXBFTHETA AND LET  YBF  WBFXBFBE AN INVERTIBLE TRANSFORMATION  SUPPOSE THAT SBFYBF IS ASUFFICIENT STATISTIC FOR THETA IN FYBFYBFTHETASHOW THAT  TXBF  SBFWBFXBFIS A SUFFICIENT STATISTIC FOR THETA IN FXBFXBFTHETAITEM LABELEXBINOM THE BF BINOMIAL DISTRIBUTION HAS PMF  INDEXBINOMIAL RANDOM VARIABLE INDEXRANDOM VARIABLEBINOMIAL FXP  N CHOOSE X PX1PNXWHERE N IS A POSITIVE INTEGER  THE NOTATION X SIM BCNP MEANSTHAT X HAS A BINOMIAL DISTRIBUTION WITH PARAMETERS N AND P  THEBINOMIAL BCNP REPRESENTS THE DISTRIBUTION OF THE TOTALNUMBER OF SUCCESSES IN N INDEPENDENT BERNOULLI BCP TRIALSWHERE THE PROBABILITY OF SUCCESS IN EACH TRIAL IS PBEGINENUMERATEITEM SHOW THAT THE MEAN OF BCNP IS NP AND THE  VARIANCE IS NP1PITEM LET X1X2LDOTSXN BE N INDEPENDENT BERNOULLI RANDOM  VARIABLES WITH PXI1P  SHOW THAT X  X1X2CDOTS  XNIS BCNPITEM IF X1X2LDOTSXN ARE BCNITHETAI12LDOTSN  SHOW THAT  BEGINENUMERATE  ITEM SUMI1N XI IS SUFFICIENT FOR THETA  ITEM SHOW THAT SUMI1N XI SIM BCSUMI1N    NITHETA  THAT IS THE DISTRIBUTION OF THE SUFFICIENT    STATISTIC IS ITSELF BINOMIALLY DISTRIBUTED  ENDENUMERATEENDENUMERATEITEM IT IS INTERESTING TO CONTEMPLATE THE USE OF SUFFICIENT  STATISTICS FOR DATA COMPRESSION  BEGINENUMERATE  ITEM LET XI I12LDOTSN BE BERNOULLI RANDOM VARIABLES    COMPARE THE NUMBER OF BITS REQUIRED TO REPRESENT THE SUFFICIENT    STATISTIC T  SUMI1N XIWITH THE NUMBER OF BITS REQUIRED TO CODE THE SEQUENCE X1X2LDOTSXN  ENDENUMERATEITEM LET XBF SIM NCHTHETAR WHERE H IS MATSIZEMP  AND THETA IS MATSIZEP1  SHOW THAT IF H AND R ARE  KNOWN THAT HT R1 XBF IS SUFFICIENT FOR THETA  DETERMINE  THE DISTRIBUTION OF THE RANDOM VARIABLE HT R1XBF SCHARF P 91ITEM CITEBICKELDOKSUM1977 LET X1X2LDOTSXN BE A SAMPLE FROM A  POPULATION WITH DENSITY FXXTHETA  BEGINCASESFRAC1SIGMAEXPLEFTXMUSIGMARIGHT  X GEQ MU 0  TEXTOTHERWISE  ENDCASESTHE PARAMETERS ARE THETA  MUSIGMA WHERE MU IN RBB ANDSIGMA  0BEGINENUMERATEITEM SHOW THAT MINX1X2LDOTSXN IS SUFFICIENT FOR MU  WHEN SIGMA IS KNOWNITEM FIND A ONEDIMENSIONAL SUFFICIENT STATISTIC FOR SIGMA WHEN  MU IS KNOWNITEM FIND A TWODIMENSIONAL SUFFICIENT STATISTIC FOR THETAENDENUMERATEEXSKIPSETEXSECTREFSECEXPFAMITEM LET T BE A SUFFICIENT STATISTIC THAT IS DISTRIBUTED AS T SIM  BC2THETA  SHOW THAT T IS A COMPLETE SUFFICIENT STATISTIC  ITEM SHOW THAT EACH OF THE FOLLOWING STATISTICS IS NOT COMPLETE BY  FINDING A NONZERO FUNCTION W SUCH THAT EWT  0  BEGINENUMERATE  ITEM T SIM UCTHETATHETA  T IS UNIFORMLY DISTRIBUTED    FROM THETA TO THETA  ITEM T SIM NC0THETA  ENDENUMERATEITEM LET XI HAVE PMF FXXTHETA     THETAX1THETA1X X01 FOR I12LDOTSN  SHOW    THAT T  SUMI1N XI IS A COMPLETE SUFFICIENT STATISTIC FOR    THETA   ALSO FIND A FUNCTION OF T THAT IS AN UNBIASED    ESTIMATOR OF THETA HOGGCRAIG P 356  ITEM EXPRESS THE FOLLOWING PDFS OR PMFS AS MEMBERS OF THE    EXPONENTIAL FAMILY AND DETERMINE THE SUFFICIENT STATISTICS    BEGINENUMERATE    ITEM EXPONENTIAL PDF FXXTHETA  THETA ETHETA X X      GEQ 0INDEXEXPONENTIAL RANDOM VARIABLEINDEXRANDOM      VARIABLEEXPONENTIAL    ITEM RAYLEIGH PDF FXXTHETA  2THETA ETHETA X2 X      GEQ 0 INDEXRAYLEIGH RANDOM VARIABLEINDEXRANDOM VARIABLERAYLEIGH    ITEM GAMMA PDF FXXTHETA1THETA2       FRACTHETA2THETA1  1GAMMATHETA11 XTHETA1      ETHETA2 X X GEQ 0 INDEXGAMMA RANDOM VARIABLEINDEXRANDOM VARIABLEGAMMA    ITEM POISSON PMF FXXTHETA  THETAXXETHETA      X012LDOTS INDEXPOISSON RANDOM VARIABLEINDEXRANDOM      VARIABLEPOISSON    ITEM MULTINOMIAL PMF      FXXBFTHETA1THETA2LDOTSTHETAD  LEFTPRODI1D      THETAIXI RIGHT MPRODI1D XI XI012LDOTS      AND SUMI1D XI  M AND SUMI1D THETAI  1 WITH      THETAI  0 INDEXMULTINOMIAL RANDOM VARIABLEINDEXRANDOM      VARIABLEMULTINOMIAL    ITEM GEOMETRIC PMF FXX THETA  1THETAX      THETA INDEXGEOMETRIC RANDOM VARIABLEINDEXRANDOM      VARIABLEGEOMETRIC    ENDENUMERATEEXSKIPSETEXSECTREFSECMVUB  ITEM LET XI SIM BCP I12LDOTSN AND LET T     SUMI1N XI  SHOW THAT T IS A COMPLETE MINIMAL    STATISTIC  SCHARF P 88  ITEM LET X1 LDOTS XN BE A SAMPLE FROM THE EXPONENTIAL FAMILYREFEXPONENTIAL EITHER CONTINUOUS OR DISCRETE  SHOW THAT THE DISTRIBUTION OF THE SUFFICIENT STATISTIC TBF T1LDOTS TKT HAS THE FORM BEGINEQUATIONLABELMARGINALTEXFRM TTBF THETA CTHETA A0TBFEXPLEFTSUMI1KPIITHETATIRIGHTENDEQUATIONWHERE TBF  T1 LDOTS TKT  ITEM CITESCHARFL1991 LET XBF0 XBF1 LDOTS  XBFM1 DENOTE A RANDOM SAMPLE OF NDIMENSIONAL RANDOM  VECTORS XBFN EACH OF WHICH HAS MEAN VALUE MBF AND COVARIANCE  MATRIX R SHOW THAT THE SAMPLE MEANBEGINDISPLAYMATHHATMBFT  FRAC1T1 SUMN0T XBFNENDDISPLAYMATHAND THE SAMPLE COVARIANCEBEGINDISPLAYMATHSTHATMBFT  FRAC1T1SUMN0T XBFNHATMBFTXBFN HATMBFTTENDDISPLAYMATHMAY BE WRITTEN RECURSIVELY ASBEGINDISPLAYMATHHATMBFT  FRACTT1HATMBFT1  FRAC1T1XBFTQQUAD HATMBF0  XBF0ENDDISPLAYMATHANDBEGINDISPLAYMATHSTHATMBFT  QT  HATMBFTHATMBFTTENDDISPLAYMATHWHEREBEGINDISPLAYMATHQT  FRACTT1QT1  FRAC1T1XBFTXBFTTENDDISPLAYMATHITEM CITESCHARFL1991 EXTEND THE RAOBLACKWELL THEOREM BYSHOWING THATBEGINDISPLAYMATHRBF  QBF  PBFENDDISPLAYMATHWHERE RBF  EYBF THETABFYBF THETABFT ANDQBF  EGBFZBF THETABFGBFZBF THETABFT ARE THECOVARIANCE MATRICES FOR YBF AND GBFZBF RESPECTIVELY ANDPBF IS THE NONNEGATIVE DEFINITE MATRIX PBF YBFGBFZBFYBFGBFZBFT  USE THIS RESULT TO SHOW BEGINDISPLAYMATHELEFTABFTHATTHETABF1 THETABFRIGHT2 GEQ ELEFTABFTHATTHETABF2 THETABFRIGHT2 ENDDISPLAYMATHFOR HATTHETABF1 AN UNBIASED ESTIMATOR OF THETABF ANDHATTHETABF2 ELEFTHATTHETABF1TBFXBFRIGHT A RAOBLACKWELLIZED VERSION OF HATTHETABF1  INTERPRET THERESULT ENDEXERCISESBEGINEXAMPLE FROM CITEFERGUSON67SECTIONREFERENCESTHE VIEWPOINT OF DECISION MAKING IN TERMS OF GAMES AND THE SPECIALCASES PRESENTED HERE ARE PROMOTED IN CITEFERGUSON67  THE BOOKCITEBILLINGSLEY PROVIDES A SOLID ANALYTICAL COVERAGE OF MEASURETHEORY AND CONDITIONAL EXPECTATION  FOR THOSE WHO MAY BE INTERESTEDIN GENERAL GAME THEORY CITELUCERAIFFA57 IS A REASONABLEINTRODUCTION  ANOTHER WORK ON GAMES WITH CONNECTIONS TO LINEARPROGRAMMING IS CITEKARLIN1992THE BOOKS CITEBICKELDOKSUM1977 AND CITEHOGGCRAIG1978 PROVIDE A GOODBACKGROUND ON THE INTRODUCTORY MATERIAL ON TRANSFORMATIONS OFVARIABLES CONDITIONAL EXPECTATIONS EXPONENTIAL FAMILIES ANDSUFFICIENT STATISTICS  MATERIAL AND INSIGHT HAS ALSO BEEN DRAWN FROMCITESCHARFL1991 LOCAL VARIABLES TEXMASTER TEST ENDCHAPTERDETECTION THEORYLABELCHAPDETECTIONBEGINQUOTESOURCE  WILLIAM THOMPSON LORD KELVINI OFTEN SAY THAT WHEN YOU CAN MEASURE WHAT YOU ARE SPEAKING ABOUT ANDEXPRESS IT IN NUMBERS YOU KNOW SOMETHING ABOUT IT BUT WHEN YOUCANNOT MEASURE IT WHEN YOU CANNOT EXPRESS IT IN NUMBERS YOURKNOWLEDGE OF IT IS OF A  MEAGRE AND UNSATISFACTORY  KINDENDQUOTESOURCESECTIONINTRODUCTION TO HYPOTHESIS TESTINGBEGINQUOTESOURCELIFE IS A CONDITIONAL PROBABILITYENDQUOTESOURCEIN THE DETECTION PROBLEM AN OBSERVATION OF A RANDOM VARIABLE ORSIGNAL X IS USED TO MAKE DECISIONS ABOUT A FINITE NUMBER OFOUTCOMES  MORE SPECIFICALLY IN A MARY HYPOTHESIS TESTING PROBLEMIT IS ASSUMED THAT THE PARAMETER SPACE THETA  THETA0 CUPTHETA1 CUP CDOTS CUP THETAM1 WHERE THE THETAI AREMUTUALLY DISJOINT  CORRESPONDING TO EACH OF THESE SETS ARE CHOICES OR HYPOTHESES  DENOTED ASBEGINDISPLAYMATHBEGINARRAYCCH0MC   THETAIN THETA0H1MC   THETAIN THETA1 VDOTS HM1MC   THETA IN THETAM1ENDARRAYENDDISPLAYMATHTHE PARAMETER THETA DETERMINES THE DISTRIBUTION OF A RANDOMVARIABLE X TAKING VALUES IN A SPACE XC ACCORDING TO THEDISTRIBUTION FUNCTION FXXTHETA  BASED ON THE OBSERVATIONXX A DECISION IS MADE BY A DECISIONMAKING AGENT  IN THE SIMPLESTCASE THE DECISION SPACE IS DELTA DELTA0DELTA2LDOTSDELTAM1 WITH ONE CHOICECORRESPONDING TO EACH HYPOTHESIS SUCH THAT DELTAI REPRESENTS THEDECISION TO ACCEPT HYPOTHESIS HI THEREBY REJECTING THE OTHERSBEGINEXAMPLE  IN AN MARY COMMUNICATION PROBLEM ONE OF M SYMBOLS IS SENT OVER  THE CHANNEL TO A RECEIVER THAT OBSERVES A RANDOM VARIABLE X WHERE  THE DISTRIBUTION OF X DEPENDS UPON WHICH OF THE M SYMBOLS  WERE SENT  THE RECEIVER MAKES A DECISION BASED UPON ITS MEASUREMENTSENDEXAMPLETHERE ARE TWO MAJOR APPROACHES TO DETECTIONBEGINDESCRIPTIONITEMTHE BAYESIAN APPROACH IN THE BAYESIAN APPROACH THE EMPHASIS IS  ON EM MINIMIZING LOSS  WITH THE BAYESIAN APPROACH WE ASSUME  THAT THE PARAMETERS ARE ACTUALLY RANDOM VARIABLES GOVERNED BY A  PRIOR PROBABILITY  A LOSS FUNCTION IS ESTABLISHED FOR EACH POSSIBLE  OUTCOME AND EACH POSSIBLE DECISION AND DECISIONS ARE MADE TO  MINIMIZE THE AVERAGE LOSS  THE BAYESIAN APPROACH CAN BE APPLIED  WELL TO THE MARY DETECTION PROBLEM FOR M GEQ 2ITEMNEYMANPEARSON APPROACH THE NEYMANPEARSON APPROACH IS USED  PRIMARILY FOR THE BINARY DETECTION PROBLEM  IN THIS APPROACH THE  PROBABILITY OF FALSE ALARM IS FIXED AT SOME VALUE AND THE DECISION  FUNCTION IS FOUND WHICH MAXIMIZES THE PROBABILITY OF DETECTIONENDDESCRIPTIONIN EACH CASE THE TESTS ARE REDUCED DOWN TO COMPARISONS OF RATIOS OFPROBABILITY DENSITY OR PROBABILITY MASS FORMING WHAT IS CALLED A EM  LIKELIHOOD RATIO TESTTHE DETECTION THEORY PRESENTED HERE HAS HAD GREAT UTILITY FORDETECTION USING RADAR SIGNALS AND SOME OF THE TERMINOLOGY USED INTHAT CONTEXT HAVE PERMEATED THE GENERAL FIELD  NOTIONS SUCH AS FALSEALARM MISSED DETECTION RECEIVER OPERATING CHARACTERISTIC ETC OWETHEIR ORIGINS TO RADAR  STATISTICS HAS COINED THEIR OWN VOCABULARYFOR THESE CONCEPTS HOWEVER AND WE WILL FIND IT DESIRABLE TO BECOMEFAMILIAR WITH BOTH THE ENGINEERING AND STATISTICS TERMINOLOGY  THEFACT THAT MORE THAN ONE DISCIPLINE HAS EMBRACED THESE CONCEPTS IS ATESTIMONY TO THEIR GREAT UTILITYWE WILL BEGIN OUR INVESTIGATION OF THE HYPOTHESIS TESTING PROBLEM THEEM BINARY DETECTION PROBLEM  DESPITE ITS APPARENT SIMPLICITY THEREIS A CONSIDERABLE BODY OF THEORY ASSOCIATED WITH THE PROBLEM  THISCLASSICAL BINARY DECISION PROBLEM GIVES RISE TO FOLLOWING TERMINOLOGYH0 IS CALLED THE EM NULL HYPOTHESIS AND H1 IS THE EM  ALTERNATIVE HYPOTHESIS  ONLY ONE OF THESE DISJOINT HYPOTHESES ISTRUE AND THE JOB OF THE DECISIONMAKER IS TO DETECT GUESS WHICHHYPOTHESIS IS TRUE  THE DECISION SPACE IS DELTA  DELTA0 TEXT  ACCEPT H0 DELTA1 TEXT REJECT H0BEGINEXAMPLE DIGITAL COMMUNICATIONS  ONE OF TWO SIGNALS IS SENT  WE CAN TAKE THE PARAMETER SPACE AS  THETA  11  THE RECEIVER DECIDES BETWEEN H0MC  THETA  1  AND H1MC   THETA  1 BASED UPON THE OBSERVATION OF A RANDOM  VARIABLE    IN A COMMON SIGNAL MODEL ADDITIVE GAUSSIAN NOISE CHANNEL THE  RECEIVED SIGNAL IS MODELED AS R  S  NWHERE S IS THE TRANSMITTED SIGNAL AND N IS A RANDOM VARIABLE  IFN SIM NC0SIGMA2 AND S  THETA A FOR SOME AMPLITUDE ATHEN THE DISTRIBUTION FOR R CONDITIONED UPON KNOWING THE TRANSMITTEDSIGNAL THETA FRRTHETA  FRAC1SQRT2PI SIGMA ER   ATHETA22SIGMA2ENDEXAMPLEBEGINEXAMPLELABELEXPOISSON OPTICAL COMMUNICATIONS  ANOTHER SIGNAL MODEL MORE APPROPRIATE FOR AN OPTICAL CHANNEL IS TO  ASSUME THAT R  XWHERE X IS A POISSON RANDOM VARIABLE WHOSE RATE DEPENDS UPONTHETA FXXTHETA  BEGINCASES  EXPLAMBDA0 FRACLAMBDA0XX  THETA  LAMBDA0 EXMATSP  EXPLAMBDA1 FRACLAMBDA1XX  THETA  LAMBDA1ENDCASES X  0 1 LDOTSTHIS MODELS FOR EXAMPLE THE RATE OF RECEIVED PHOTONS WHEREDIFFERENT PHOTON INTENSITIES ARE USED TO REPRESENT THE TWO POSSIBLEVALUES OF THETAENDEXAMPLEBEGINEXAMPLE  RADAR DETECTION  ASSUME THAT A RECEIVED SIGNAL IS R  THETANWHERE N IS A RANDOM VARIABLE REPRESENTING THE NOISE AND THETA IS ARANDOM VARIABLE INDICATING THE PRESENCE OF ABSENCE OF SOME TARGETTHE TWO HYPOTHESES CAN NOW BE DESCRIBED AS BEGINALIGNEDH0MC  TEXT TARGET IS ABSENTMC  THETA LEQ THETA0 H1MC   TEXT TARGET IS PRESENTMC   THETA  THETA0ENDALIGNEDENDEXAMPLEBASED UPON THE OBSERVATION X WE MUST MAKE A DECISION REGARDINGWHICH HYPOTHESIS TO ACCEPT  WE DIVIDE THE SPACE XC INTO TWODISJOINT REGIONS RC AND AC WITH XC  RC CUP AC WEFORMULATE OUR DECISION FUNCTION PHIX ASBEGINEQUATION  LABELEQPHXPHIX  BEGINCASES1  TEXTIF   XIN RC10PT0  TEXTIF   XIN ACENDCASESENDEQUATIONWE INTERPRET THIS DECISION RULE AS FOLLOWS IF XIN RC REJECTWE TAKE ACTION DELTA1 ACCEPTING H1 REJECTING H0 AND IFXIN AC ACCEPT WE TAKE ACTION DELTA0 ACCEPTING H0REJECTING H1  THE DECISION REGIONS THAT ARE CHOSEN DEPEND UPONTHE PARTICULAR STRUCTURE PRESENT IN THE PROBLEMSECTIONNEYMANPEARSON THEORYLABELSECNPIN THE NEYMANPEARSON APPROACH TO DETECTION THE FOCUS IS ON THECONDITIONAL PROBABILITIES  IN PARTICULAR IT IS DESIRED TO MAXIMIZETHE PROBABILITY OF CHOOSING H1 WHEN IN FACT H1 IS TRUE WHILEAT THE SAME TIME NOT EXCEEDING A FIXED PROBABILITY OF CHOOSING H1WHEN IT IS NOT TRUE  THAT IS WE WANT TO MAXIMIZE THE PROBABILITY OFDETECTION WHILE NOT EXCEEDING A STANDARD FOR THE PROBABILITY OF FALSEALARMSUBSECTIONSIMPLE BINARY HYPOTHESIS TESTINGWE FIRST LOOK AT THE CASE WHERE H0 AND H1 ARE EM SIMPLEBEGINDEFINITION LABELDEFSIMPCOMPINDEXHYPOTHESIS TESTINGSIMPLEINDEXHYPOTHESIS TESTINGCOMPOSITE  A TEST FOR THETA IN THETAI I01LDOTSK1 IS SAID TO BE  BF SIMPLE IF EACH THETAI CONSISTS OF EXACTLY ONE ELEMENT  IF  ANY THETAI HAS MORE THAN ONE POINT A TEST IS SAID TO BE BF  COMPOSITEENDDEFINITIONBEGINEXAMPLE  LET THETA  01  THE TEST BEGINALIGNEDH0MC   THETA  0 H1MC    THETA  1ENDALIGNEDIS A SIMPLE TEST  NOW LET THETA  RBB  THE TEST BEGINALIGNEDH0MC    THETA  0 H1MC    THETA  0ENDALIGNEDIS A COMPOSITE TESTENDEXAMPLEFOR A BINARY HYPOTHESIS TEST THE DECISION SPACE CONSISTS OF TWOPOINTS DELTA   DELTA0DELTA1 CORRESPONDING TO ACCEPTINGH0 AND H1 THEN IF THETA  THETA0 IS THE TRUE VALUE OF THEPARAMETER WE PREFER TO TAKE ACTION DELTA0 WHEREAS IF THETA1IS THE TRUE VALUE WE PREFER DELTA1BEGINDEFINITION  THE PROBABILITY OF REJECTING THE NULL HYPOTHESIS H0 WHEN IT IS  TRUE IS CALLED THE BF SIZE OF THE RULE PHI AND IS DENOTED  ALPHA  THIS IS CALLED A BF TYPE I ERROR OR BF FALSE    ALARM INDEXSIZE OF A TEST FALSE ALARM INDEXFALSE ALARMENDDEFINITIONFOR THE SIMPLE BINARY HYPOTHESIS TEST BEGINALIGNEDALPHA    PTEXTDECIDE H1 TEXTH0 IS TRUE  PPHIX  1  THETA0     ETHETA0PHIX      PFAENDALIGNEDTHE NOTATION PFA IS STANDARD FOR THE PROBABILITY OF A FALSEALARM  THIS LATTER TERMINOLOGY STEMS FROM RADAR APPLICATIONS WHERE APULSED ELECTROMAGNETIC SIGNAL IS TRANSMITTED  IF A RETURN SIGNAL ISREFLECTED FROM THE TARGET WE SAY A TARGET IS DETECTED  BUT DUE TORECEIVER NOISE ATMOSPHERIC DISTURBANCES SPURIOUS REFLECTIONS FROMTHE GROUND AND OTHER OBJECTS AND OTHER SIGNAL DISTORTIONS IT IS NOTPOSSIBLE TO DETERMINE WITH ABSOLUTE CERTAINTY WHETHER OR NOT A TARGETIS PRESENTBEGINDEFINITIONINDEXPOWER OF A TEST DETECTIONINDEXDETECTION PROBABILITY  THE BF POWER OR BF DETECTION PROBABILITY OF A DECISION RULE  PHI IS THE PROBABILITY OF CORRECTLY ACCEPTING THE ALTERNATIVE  HYPOTHESIS H1 WHEN IT IS TRUE AND IS DENOTED BY BETA  ONE  MINUS THE POWER IS THE PROBABILITY OF ACCEPTING H0 WHEN H1 IS  TRUE RESULTING IN A BF TYPE II ERROR OR BF MISSED DETECTIONENDDEFINITIONWE THUS HAVE BEGINALIGNEDBETA     PTEXTDECIDE H1TEXTH1 IS TRUE  PPHIX  1  THETA1     ETHETA1PHIX      PDENDALIGNEDTHE NOTATION PD IS STANDARD FOR THE PROBABILITY OF A DETECTIONAND BEGINDISPLAYMATH PMD  1PDENDDISPLAYMATHIS THE PROBABILITY OF A MISSED DETECTIONBEGINDEFINITION  A TEST PHI IS SAID TO BE BF BEST OF SIZE ALPHA FOR TESTING  H0 AGAINST H1 IF ETHETA0PHIX  ALPHA AND IF FOR  EVERY TEST PHIPRIME FOR WHICH  ETHETA0PHIPRIMEXLEQ ALPHA WE HAVEBEGINDISPLAYMATH BETA  ETHETA1PHIX  GEQETHETA1PHIPRIMEX  BETAPRIMEENDDISPLAYMATHTHAT IS A TEST PHI IS BEST OF SIZE ALPHA IF OUT OF ALL TESTS WITHPFA  NOT GREATER THAN ALPHA PHI HAS THE LARGEST PROBABILITY OFDETECTIONENDDEFINITIONSUBSECTIONTHE NEYMANPEARSON LEMMAWE NOW GIVE A GENERAL METHOD FOR FINDING THE BEST TESTS OF A SIMPLEHYPOTHESIS AGAINST A SIMPLE ALTERNATIVE  THE TEST WILL TAKE THEFOLLOWING FORM PHIX  BEGINCASES 0  TEXTCONDITION 1  GAMMA  TEXTCONDITION 2  1  TEXTCONDITION 3ENDCASESWHERE THE THREE CONDITIONS ARE MUTUALLY EXCLUSIVE  IF CONDITION 1 ISSATISFIED THEN THE TEST CHOOSES DECISION 0 SELECTS H0  IFCONDITION 3 IS SATISFIED THEN THE TEST CHOOSES DECISION 1 SELECTSH1  HOWEVER IF CONDITION 2 IS SATISFIED AND GAMMA IS CHOSENWHAT THIS MEANS IS THAT A RANDOM SELECTION TAKES PLACE  DECISION 1 ISCHOSEN WITH PROBABILITY GAMMA AND DECISION 0 IS CHOSEN WITHPROBABILITY 1GAMMA  THE INSTANTIATION OF CONDITION 3 IS ANEXAMPLE OF A RANDOMIZED DECISION RULETHE BEST TEST OF SIZE ALPHA IS PROVIDED BY THE FOLLOWING IMPORTANT LEMMABEGINLEMMA NEYMANPEARSON LEMMA  INDEXNEYMANPEARSON LEMMA  SUPPOSE THAT THETA  THETA0 THETA1 AND THAT THE  DISTRIBUTIONS OF X HAVE DENSITIES OR MASS FUNCTIONS FXX    THETA  LET NU0 BE A THRESHOLDBEGINENUMERATEITEM ANY TEST PHIX OF THE FORMBEGINEQUATIONLABELNEYMANPEARSONPHIX  BEGINCASES1  TEXTIF  FXX THETA1  NU FXXTHETA0 10PT GAMMA  TEXTIF   FXX THETA1  NU FXXTHETA0 10PT  0  TEXTIF  FXX THETA1  NU FXX THETA0 ENDCASESENDEQUATIONFOR SOME 0 LEQ GAMMAXLEQ 1 IS EM BEST OF ITS SIZE FOR TESTINGFOR SOME 0 LEQ GAMMALEQ 1 IS EM BEST OF ITS SIZE FOR TESTINGH0MC  THETATHETA0 AGAINST H1MC  THETATHETA1CORRESPONDING TO NU  INFINITY THE TEST BEGINEQUATIONLABELNEYMANPEARSON0PHIX  BEGINCASES1  TEXTIF   FXX THETA0 010PT0  TEXTIF   FXX THETA0  0 ENDCASESENDEQUATIONIS BEST OF SIZE ZERO FOR TESTING H0 AGAINST H1ITEM EM EXISTENCE  FOR EVERY ALPHA 0LEQ ALPHALEQ 1  THERE EXISTS A TEST OF THE FORM ABOVE WITH GAMMA A  CONSTANT FOR WHICH ETHETA0PHIX  ALPHAITEM EM UNIQUENESS  IF PHIPRIME IS A BEST TESTOF SIZE ALPHA FOR TESTING H0 AGAINST H1 THEN IT HAS THE FORMGIVEN BY REFNEYMANPEARSON EXCEPT PERHAPS FOR A SET OF X WITHPROBABILITY ZERO UNDER H0 AND H1ENDENUMERATEENDLEMMABEGINPROOF  THE PROOF THAT FOLLOWS IS FOR THE CONTINUOUS CASE THE DISCRETE CASE  IS LEFT TO THE READER AND MAY BE PROVEN BY REPLACING INTEGRALS  EVERYWHERE WITH SUMMATIONSBEGINENUMERATEITEM CHOOSE ANY PHIX OF THE FORMREFNEYMANPEARSON AND LET PHIPRIMEX 0LEQPHIPRIMEX LEQ 1 BE ANY TEST WHOSE SIZE IS NOT GREATER THANTHE SIZE OF PHIX THAT IS FOR WHICHBEGINDISPLAYMATHETHETA0PHIPRIMEX LEQ ETHETA0PHIXENDDISPLAYMATHWE ARE TO SHOW THAT ETHETA1PHIPRIMEX LEQETHETA1PHIX IE THAT THE POWER OF PHIPRIMEX ISNOT GREATER THAN THE POWER OF PHIX  NOTE THAT BEGINALIGNEDLEFTEQNINT PHIXPHIPRIMEXFXX THETA1 NU FXXTHETA0DX    10PT QQUADQQUAD INTA 1PHIPRIMEXFXX THETA1 NU FXXTHETA0DX 10PT   QQUADQQUAD MBOX INTA0 PHIPRIMEXFXX THETA1 NU FXXTHETA0DX 10PT   QQUADQQUAD MBOX INTA0GAMMAX PHIPRIMEXFXX THETA1INTA0GAMMA PHIPRIMEXFXX THETA1NU FXXTHETA0DX ENDALIGNEDWHERE BEGINALIGNEDA    XMC  FXX THETA1 NU FXXTHETA0  0A    XMC   FXX THETA1 NU FXXTHETA0  0A0    XMC   FXX THETA1 NU FXXTHETA0  0ENDALIGNEDSINCE PHIPRIMEXLEQ 1 THE FIRST INTEGRAL IS NONNEGATIVEALSO THE SECOND INTEGRAL IS NONNEGATIVE BY INSPECTION AND THE THIRDINTEGRAL IS IDENTICALLY ZERO  THUS BEGINEQUATIONLABELBAR1INT PHIXPHIPRIMEXFXX THETA1 NU FXXTHETA0DX  GEQ 0ENDEQUATIONTHIS IMPLIES THATBEGINDISPLAYMATHETHETA1PHIX ETHETA1PHIPRIMEX GEQNU ETHETA0PHIX NU ETHETA0PHIPRIMEX GEQ 0ENDDISPLAYMATHWHERE THE LAST INEQUALITY IS A CONSEQUENCE OF THE HYPOTHESIS THAT ETHETA0PHIPRIMEX LEQ ETHETA0PHIXTHIS PROVES THAT PHIX IS MORE POWERFUL THAN PHIPRIMEXIEBEGINDISPLAYMATHBETA BETAPRIME GEQ NUALPHAALPHAPRIMEENDDISPLAYMATHFOR THE CASE NU INFINITY ANY TEST PHIPRIME OF SIZE ALPHA 0MUST SATISFYBEGINEQUATIONLABELALPHACONSTRAINTALPHA  INT PHIPRIMEXFXX THETA0DX  0ENDEQUATIONHENCE PHIPRIMEX MUSTBE ZERO ALMOST EVERYWHERE ON THE SET XMC   FXX THETA0 0  THUS USING THIS RESULT AND REFNEYMANPEARSON0 BEGINALIGNEDETHETA1PHIX PHIPRIMEX    UNDERBRACEINTXMC  FXXTHETA00PHIXPHIPRIMEXFXXTHETA1DX 010PT  MBOX INTXMC  FXXTHETA00PHIXPHIPRIMEXFXXTHETA1DX10PT   INTXMC FXXTHETA001PHIPRIMEXFXXTHETA1DX GEQ 0ENDALIGNEDSINCE PHIX  1 WHENEVER THE DENSITY FXXTHETA0 0 BYREFNEYMANPEARSON0 ANDPHIPRIMEX LEQ 1  THIS COMPLETES THE PROOF OF THE FIRST PARTITEM SINCE A BEST TEST OF SIZE ALPHA  0 IS GIVEN BYREFNEYMANPEARSON0 WE MAY RESTRICT ATTENTION TO 0  ALPHA LEQ 1  THE SIZE OF THE TEST REFNEYMANPEARSON WHEN GAMMAX  GAMMATHE SIZE OF THE TEST REFNEYMANPEARSONISBEGINEQUATIONBEGINSPLITETHETA0PHIX    PTHETA0FXXTHETA1 NU FXXTHETA0  GAMMA PTHETA0FXXTHETA1 NU FXXTHETA010PT    1  PTHETA0FXXTHETA1 LEQ NU FXXTHETA0 GAMMA PTHETA0FXXTHETA1  NU FXXTHETA010PT NULL ENDSPLITLABELFOOENDEQUATIONFOR FIXED ALPHA 0  ALPHA LEQ 1 WE ARE TO FIND NU ANDGAMMA SO THAT ETHETA0PHIX  ALPHA OR EQUIVALENTLYUSING THE REPRESENTATION REFFOOBEGINDISPLAYMATH1  PTHETA0FXXTHETA1 LEQ NU FXXTHETA0 GAMMA PTHETA0FXXTHETA1  NU FXXTHETA0  ALPHAENDDISPLAYMATHORBEGINEQUATIONLABELBARPTHETA0FXXTHETA1 LEQ NU FXXTHETA0 GAMMA PTHETA0FXXTHETA1  NU FXXTHETA0 1ALPHA ENDEQUATIONIF THERE EXISTS A NU0 FOR WHICHPTHETA0FXXTHETA1 LEQ NU0 FXXTHETA01ALPHA WE TAKE GAMMA  0 AND NU  NU0  IF NOT THEN THERE ISA DISCONTINUITY IN PTHETA0FXXTHETA1 LEQ NU FXXTHETA0 WHENVIEWED AS A FUNCTION OF NU THAT BRACKETS THE PARTICULAR VALUE1ALPHA THAT IS THERE EXISTS A NU0 SUCH THAT BEGINEQUATIONLABELFOO1PTHETA0FXXTHETA1  NU0FXXTHETA0 1ALPHA LEQ PTHETA0FXXTHETA1 LEQ NU0FXXTHETA0 ENDEQUATIONFIGURE REFTHRESHOLD ILLUSTRATES THIS SITUATION  USING REFBAR FOR 1ALPHA IN REFFOO1 AND SOLVING THE EQUATIONBEGINDISPLAYMATH1ALPHA LEQ PTHETA0FXXTHETA1 LEQ NU0FXXTHETA0 ENDDISPLAYMATHFOR GAMMA YIELDSBEGINEQUATIONLABELEQNGAMMAGAMMA  FRACPTHETA0FXXTHETA1 LEQ NU0FXXTHETA0 1ALPHAPTHETA0FXXTHETA1  NU0FXXTHETA0ENDEQUATIONSINCE THIS SATISFIES REFBAR AND 0 LEQ GAMMALEQ 1 LETTING NUNU0 THE SECOND PART IS PROVEDBEGINFIGUREHTBBEGINCENTERINPUTPICTUREDIRNPTHRESHLATEXENDCENTERCAPTIONILLUSTRATION OF THRESHOLD FOR NEYMANPEARSON TESTLABELTHRESHOLDENDFIGUREITEM IF ALPHA  0 THE ARGUMENT IN A SHOWS THATPHIX0 ALMOST EVERYWHERE ON THE SET XMC   FTHETA0X  0IF PHIPRIME HAS A MINIMUM PROBABILITY OF THE SECOND KIND OF ERROR THEN1PHIPRIMEX  0 ALMOST EVERYWHERE ON THE SET XMC  FTHETA1X  0 SIM XMC  FTHETA0X  0  THUSPHIPRIME DIFFERS FROM THE PHI OF REFNEYMANPEARSON0 BY ASET OF PROBABILITY ZERO UNDER EITHER HYPOTHESISIF ALPHA  0 LET PHI BE THE BEST TEST OF SIZE ALPHA OF THEFORM REFNEYMANPEARSON  THEN BECAUSE ETHETAIPHIX ETHETAIPHIPRIMEX I0 1 THEINTEGRAL REFBAR1 MUST BE EQUAL TO ZERO  BUT BECAUSE THISINTEGRAL IS NONNEGATIVE IT MUST BE ZERO ALMOST EVERYWHERE THAT IS TOSAY ON THE SET FOR WHICH FXX THETA1 NOT  FXX THETA0WE HAVE PHIX  PHIPRIMEX ALMOST EVERYWHERE  THUS EXCEPTFOR A SET OF PROBABILITY ZERO PHIPRIMEX HAS THE SAME FORM ASREFNEYMANPEARSON WITH THE SAME VALUE FOR NU AS PHIX THUSTHE FUNCTION PHIX SATISFIES THE UNIQUENESS REQUIREMENT ENDENUMERATEENDPROOFSUBSECTIONAPPLICATION OF THE NEYMANPEARSON LEMMATHE NEYMANPEARSON LEMMA PROVIDES A GENERAL DECISION RULE FOR A SIMPLEHYPOTHESIS VERSUS A SIMPLE ALTERNATIVE  WE WOULD APPLY IT AS FOLLOWSBEGINENUMERATEITEM FOR A GIVEN BINARY DECISION PROBLEM DETERMINE WHICH HYPOTHESISIS TO BE THE NULL AND WHICH IS TO BE THE ALTERNATIVE  THIS CHOICE ISAT THE DISCRETION OF THE ANALYST  AS A PRACTICAL ISSUE IT WOULD BEWISE TO CHOOSE AS THE NULL HYPOTHESIS THE ONE THAT HAS THE MOSTSERIOUS CONSEQUENCES IF REJECTED BECAUSE THE ANALYST IS ABLE TO CHOOSETHE SIZE OF THE TEST WHICH ENABLES CONTROL OF PROBABILITY OFREJECTING THE NULL HYPOTHESIS WHEN IT IS TRUE  ITEM SELECT THE SIZE OF THE TEST  IT SEEMS TO BE THE TRADITION FORMANY APPLICATIONS TO SET ALPHA  005 OR ALPHA  001 WHICHCORRESPOND TO COMMON SIGNIFICANCE LEVELS USED IN STATISTICS  THEMAIN ISSUE HOWEVER IS TO CHOOSE THE SIZE RELEVANT TO THE PROBLEM ATHAND  FOR EXAMPLE IN A RADAR TARGET DETECTION PROBLEM IF THE NULLHYPOTHESIS IS NO TARGET PRESENT SETTING ALPHA  005 MEANSTHAT WE ARE WILLING TO ACCEPT A 5 CHANCE THAT A TARGET WILL NOT BETHERE WHEN OUR TEST TELL US THAT A TARGET IS PRESENT  THE SMALLER THESIZE IN GENERAL THE SMALLER ALSO IS THE POWER AS WILL BE MADE MOREEVIDENT IN THE DISCUSSION OF THE RECEIVER OPERATOR CHARACTERISTICITEM CALCULATE THE THRESHOLD NU  THE WAY TO DO THIS IS NOT  OBVIOUS FROM THE THEOREM  CLEARLY NU MUST BE A FUNCTION OF THE  SIZE ALPHA BUT UNTIL SPECIFIC DISTRIBUTIONS ARE USED THERE IS  NO OBVIOUS FORMULA FOR DETERMINING NU  THAT WILL BE ONE OF THE  TASKS EXAMINED IN THE EXAMPLES TO FOLLOWENDENUMERATETHE STRUCTURE OF THE TEST WHEN GAMMA NOT  0 DESERIVES SOMEDISCUSSION  IF THIS EQUALITY CONDITION OBTAINS THEN THERE IS A NONZEROPROBABILITY THAT FXXTHETA1  NU FXXTHETA0  THEPARAMETER GAMMA DEFINED IN THE PROOF OF THE NEYMANPEARSON LEMMAHAS A NATURAL INTERPRETATION AS THE PROBABILITY OF SETTING PHIX 1 WHEN THE EQUALITY CONDITION OBTAINS  ACCORDINGLY WE MAY DEFINETHE RANDOMIZED DECISION RULE VARPHIGAMMA  GAMMA 1GAMMAWHERE GAMMA  PPHI1 1PPHI2 THE PROBABILITY OF CHOOSINGRULE PHI1 ANDBEGINEQUATIONPHI1X  BEGINCASES1  TEXTIF  FXX THETA1 GEQ NU FXXTHETA0 10PT 0  TEXTIF  FXX THETA1  NU FXX THETA0 ENDCASESENDEQUATIONANDBEGINEQUATIONPHI2X  BEGINCASES1  TEXTIF  FXX THETA1  NU FXXTHETA0 10PT 0  TEXTIF  FXX THETA1 LEQ NU FXX THETA0 ENDCASESENDEQUATIONSUBSECTIONTHE LIKELIHOOD RATIO AND THE ROCTHE KEY QUANTITIES IN THE NEYMANPEARSON THEORY ARE THE DENSITYFUNCTIONS FXXTHETA1 AND FXXTHETA0  THESEQUANTITIES ARE SOMETIMES VIEWED AS THE CONDITIONAL PDFS OR PMFS OFX GIVEN THETA  THE CONCEPT OF CONDITIONING HOWEVER REQUIRESTHAT THE QUANTITY THETA BE A RANDOM VARIABLE  BUT NOTHING IN THENEYMANPEARSON THEORY REQUIRES THETA TO BE SO VIEWED IN FACT THENEYMANPEARSON APPROACH IS OFTEN CONSIDERED TO BE AN ALTERNATIVE TOTHE BAYESIAN APPROACH IN WHICH THETA EM IS VIEWED AS A RANDOMVARIABLE  SINCE THE PURISTS INSIST THAT THE NEYMANPEARSON NOT BECONFUSED WITH THE BAYESIAN APPROACH THEY HAVE COINED THE TERM EM  LIKELIHOOD FUNCTION FOR FXXTHETA1 ANDFXXTHETA0  TO KEEP WITH TRADITION AND WE WILL RESPECTTHIS CONVENTION AND CALL THESE THINGS LIKELIHOOD FUNCTIONS ORLIKELIHOODS WHEN REQUIRED OR WHEN WE THINK ABOUT ITENGINEERS DONTUSUALLY GET TOO WORKED UP OVER THESE TYPES OF ISSUES BUT PERHAPS THEYSHOULDTHE INEQUALITY  FXXTHETA1 THREECOMP NU FXXTHETA0HAS EMERGED AS A NATURAL EXPRESSION IN THESTATEMENT AND PROOF OF THE NEYMANPEARSON LEMMA  USING THE RATIOBEGINEQUATION  LABELEQELLX  ELLX  FRACFXXTHETA1 FXXTHETA0ENDEQUATIONKNOWN AS THE BF LIKELIHOOD RATIO INDEXLIKELIHOOD RATIO THENEYMANPEARSON TEST CAN BE EXPRESSED AS ONE OF THE THREE COMPARISONSINBEGINDISPLAYMATHELLX THREECOMP NUENDDISPLAYMATHTHE TEST REFNEYMANPEARSON MAY BE REWRITTEN AS A BF LIKELIHOOD  RATIO TEST LRTBEGINEQUATIONLABELLIKELIHOODRATIOPHIX  BEGINCASES1  TEXTIF   ELLX  NU10PTGAMMA  TEXTIF   ELLX  NU10PT0  TEXTIF   ELLX  NUENDCASESENDEQUATIONFOR MANY DISTRIBUTIONS IT IS CONVENIENT TO USE THE LOGARITHM OF THELIKELIHOOD FUNCTION  ACCORDINGLY WE DEFINE WHERE APPROPRIATEBEGINEQUATION  LABELEQLOGLIKE  LAMBDAX  LOG ELLX  LOGFRACFXXTHETA1  FXXTHETA0ENDEQUATIONTHE FUNCTION LAMBDAX OR SOME MULTIPLE OF IT AS CONVENIENT ISKNOWN AS THE EM LOGLIKELIHOOD RATIO  INDEXLOGLIKELIHOOD RATIOSINCE THE LOG FUNCTION ISMONOTONICALLY INCREASING WE CAN REWRITE THE TESTREFNEYMANPEARSON ASBEGINEQUATIONLABELLOGLIKELIHOODRATIOPHIX  BEGINCASES1  TEXTIF  LAMBDAX  LOG NU10PTGAMMA  TEXTIF  LAMBDAX  LOG NU10PT0  TEXTIF  LAMBDAX  LOG NUENDCASESENDEQUATIONSINCE LOGLIKELIHOOD FUNCTIONS ARE COMMON WE WILL FIND IT CONVENIENTTO INTRODUCE A NEW THRESHOLD VARIABLE FOR OUR  TEST ETA  LOG NUYOU MAY HAVE NOTICED IN THE PROOF OF THE LEMMA THAT WE HAVE USEDEXPRESSIONS SUCH AS FXXTHETA1 WHERE WE HAVE USED THERANDOM VARIABLE X AS AN ARGUMENT OF THE DENSITY FUNCTION  WHEN WEDO THIS THE FUNCTION FXXTHETA1 IS OF COURSE A RANDOMVARIABLE SINCE IT BECOMES A FUNCTION OF A RANDOM VARIABLE THE LIKELIHOOD RATIO ELLX IS ALSO A RANDOM VARIABLE AS IS THE LOGLIKELIHOOD RATIO LAMBDAXA FALSE ALARM OCCURS ACCEPTING H1 WHEN H0 IS TRUE IF ELLX NU WHEN THETA  THETA0 AND XX  LETFELLL THETA0 DENOTE THE DENSITY OF ELLGIVEN THETA  THETA0 THENBEGINDISPLAYMATHALPHA  PFA  PTHETA0ELLX  NU  INTNUINFINITY FELLL THETA0 DLENDDISPLAYMATHTHUS IF WE COULD COMPUTE THE DENSITY OF ELL GIVENTHETATHETA0 WE WOULD HAVE A METHOD OF COMPUTING THE VALUE OFTHE THRESHOLD NU  OR IN TERMS OF THE LOG LIKELIHOOD WE CANWRITEALPHA  PFA  PTHETA0LAMBDAX  ETA INTETAINFINITY  FLAMBDAL THETA0 DL WHERE FLAMBDAL THETA0 IS THE DENSITY OF THE RANDOMVARIABLE LAMBDAXTHE PROBABILITY OF DETECTION CAN SIMILARLY BE FOUND BEGINALIGNEDBETA   PD  PTHETA1ELLX  NU  INTNUINFTYFELLL THETA1DL  PTHETA1LAMBDAX  ETA  INTETAINFTYFLAMBDAL THETA1 DLENDALIGNEDIN PRACTICE WE ARE OFTEN INTERESTING IN COMPARING HOW PFA VARIESWITH PD  FOR A NEYMANPEARSON TEST THE SIZE AND POWER ASSPECIFIED BY PFA AND PD COMPLETELY SPECIFY THE TESTPERFORMANCE  WE CAN GAIN SOME VALUABLE INSIGHT BY CROSSPLOTTINGTHESE PARAMETERS FOR A GIVEN TEST THE RESULTING PLOT IS CALLED THEEM RECEIVER OPERATING CHARACTERISTIC INDEXRECEIVER OPERATING  CHARACTERISTIC ROC OR ROC CURVE BORROWING FROM RADARTERMINOLOGY  ROC CURVES ARE PERHAPS THE MOST USEFUL SINGLE METHOD OFEVALUATION OF PERFORMANCE OF A BINARY DETECTION SYSTEM  WE WILL SEESOME EXAMPLES BELOW OR ROCSSUBSECTIONA POISSON EXAMPLEWE WISH TO DESIGN A NEYMANPEARSON DETECTOR FOR THE POISSON RANDOM VARIABLE INTRODUCED IN EXAMPLE REFEXPOISSON  THE TWO HYPOTHESES ARE BEGINALIGNEDH0MC X SIM    EXPLAMBDA0 FRACLAMBDA0XX H1MC X SIM    EXPLAMBDA1 FRACLAMBDA1XX ENDALIGNEDTHE LIKELIHOOD RATIO FOR THE PROBLEM ISBEGINDISPLAYMATHELLX EXPLAMBDA0LAMBDA1LEFTFRACLAMBDA1LAMBDA0RIGHTXENDDISPLAYMATHAND REFNEYMANPEARSON BECOMES AFTER SIMPLIFICATIONBEGINEQUATIONPHIX  BEGINCASES1  TEXTIF  X  FRACLOG NU  LAMBDA1LAMBDA0LOGLAMBDA1LOGLAMBDA010PT GAMMA  TEXTIF   X  FRACLOG NU  LAMBDA1LAMBDA0LOGLAMBDA1LOGLAMBDA010PT 0  TEXTIF  X  FRACLOG NU  LAMBDA1LAMBDA0LOGLAMBDA1LOGLAMBDA0ENDCASESENDEQUATIONFOR A FIXED SIZE ALPHA WE MUST COMPUTE THE THRESHOLD NUTHE PROBABILITY OF A FALSE ALARMDECIDING THAT THETA  LAMBDA1 WHEN THETA LAMBDA0 IS TRUE IS EQUAL TO THE PROBABILITY UNDER THE NULLHYPOTHESIS THAT ELLX  NU THAT ISPFA  PTHETA0ELLX  NU  GAMMA PTHETA0ELLX NULET QNU  FRACLOGNU  LAMBDA1LAMBDA0LOGLAMBDA1LOGLAMBDA0BE SUCH THAT IT ALWAYS TAKES INTEGER VALUES BY APPROPRIATE SELECTION  OF NU  THEN BEGINALIGNEDPFA   PTHETA0X  QNU  GAMMA PTHETA0X  QNU SUMK QNU 1INFTYFRACEXPLAMBDA0LAMBDA0KKGAMMA PX  QNU    1 SUMK0 QNUFRACEXPLAMBDA0LAMBDA0KK  GAMMA EXPLAMBDA0FRAC LAMBDA0QNUQNUENDALIGNEDIF ALPHA IS SUCH THAT THERE EXISTS AN INTEGER QPRIME THAT SATISFIESBEGINEQUATIONLABELEQNPOISSONEQ1ALPHA  SUMK0QPRIMEFRACEXPLAMBDA0LAMBDA0KKENDEQUATIONTHEN WE MAY TAKE  GAMMA  0 ANDBEGINEQUATIONLABELEQNNUVALUENU  EXPQPRIME LOGFRACLAMBDA1LAMBDA0LAMBDA1  LAMBDA0ENDEQUATIONIN GENERAL HOWEVER THERE WILL NOT BE AN Q THATSOLVEDREFEQNPOISSONEQ WITH GAMMA  0 AND WE MUST SETQPRIME IN REFEQNNUVALUE EQUAL TOBEGINDISPLAYMATHQPRIME  ARGMINN IN BBBZLEFT SUMK0NFRACEXPLAMBDA0LAMBDA0KK  1ALPHARIGHTENDDISPLAYMATHAND APPLY REFEQNGAMMA TO YIELDBEGINDISPLAYMATHGAMMA  FRACSUMK0QPRIME FRACEXPLAMBDA0LAMBDA0KK  1ALPHAFRACEXPLAMBDA0LAMBDA0QPRIMEQPRIMEENDDISPLAYMATHAS A SIMPLE NUMERICAL EXAMPLE LET LAMBDA0  1 LAMBDA1  EAND ALPHA  01  STRAIGHTFORWARD CALCULATION YIELDSQPRIME2 NU  1325 AND GAMMA  01071  THUS IF THEOBSERVED VALUE XX IS GREATER THAN 2 DECIDE THAT THETA E IF THE VALUE IS LESS THAN 2 DECIDE THAT THETA 1 AND IFTHE OBSERVED VALUE EQUALS 2 MAKE A RANDOM SELECTION WITH THEPROBABILITY OF CHOOSING THETA  1 BEING EQUAL TO 01071  THISDECISION RULE ASSURES THAT THE PROBABILITY OF DETECTION WILL BEMAXIMIZED WHILE HOLDING THE PROBABILITY OF A FALSE ALARM TO EXACTLY01SUBSECTIONSOME GAUSSIAN EXAMPLESIN THIS SECTION WE PRESENT SEVERAL EXAMPLES AND IMPLICATIONS OFNEYMANPEARSON DETECTION WHERE THE OBSERVATIONS ARE GOVERNED BY RANDOMVARIABLES  NOT ONLY DO THESE EXAMPLES ILLUSTRATE SEVERAL IMPORTANTASPECTS OF THE THEORY BUT THEY ARISE FREQUENTLY IN PRACTICE  WE WILLPRESENT A SEQUENCE OF PROBLEMS ORDERED MOREORLESS IN ORDER OFINCREASING DIFFICULTYBEGINENUMERATEITEM SCALAR GAUSSIAN DETECTION WITH DIFFERENT MEANS AND COMMON  VARIANCES BEGINALIGNEDH0MC   X SIM NCMU0SIGMA2 H1MC X SIM NCMU1SIGMA2ENDALIGNEDWE WILL COMPUTE PFA AND PD BY INTRODUCING THE QFUNCTIONITEM VECTOR GAUSSIAN DETECTION WITH DIFFERENT MEANS AND COMMON  COVARIANCES BEGINALIGNEDH0MC X SIM NCMBF0 R H1MC X SIM NCMBF1RENDALIGNEDWE WILL DEMONSTRATE DETECTOR ARCHITECTURES AND PERFORMANCEITEM VECTOR GAUSSIAN DETECTION WITH COMMON MEANS AND DIFFERENT  COVARIANCES  WITHOUT LOSS OF GENERALITY WE WILL ASSUME THE MEANS  TO BE ZERO BEGINALIGNEDH0MC X SIM NCZEROBF SIGMA02 I H1MC X SIM NCZEROBF SIGMA12 IENDALIGNEDANALYSIS OF PERFORMANCE IN THIS CASE WILL REQUIRE INTRODUCTION OF THECHI2 DISTRIBUTIONENDENUMERATESUBSUBSECTIONSCALAR GAUSSIAN DETECTION DIFFERENT MEANS COMMON  VARIANCE    AS A PHYSICAL MOTIVATION FOR THIS PROBLEM LET US ASSUME THAT UNDER  HYPOTHESIS H1 A SOURCE OUTPUT IS A CONSTANT VOLTAGE MU1 AND  UNDER H0 THE SOURCE OUTPUT IS A CONSTANT VOLTAGE MU0  BEFORE  OBSERVATION THE VOLTAGE IS CORRUPTED BY AN ADDITIVE NOISE THE  SAMPLE RANDOM VARIABLES AREBEGINEQUATIONLABELMODEL4X  THETA  ZENDEQUATIONWHERE THETA IN THETA0 THETA1 WITH THETA0  MU0 ANDTHETA1  MU1  THE RANDOM VARIABLES Z ARE ZEROMEAN GAUSSIANRANDOM VARIABLES WITH KNOWN VARIANCE SIGMA2 AND ARE ALSOINDEPENDENT OF THE SOURCE OUTPUT THETA  WE DESIRE TO FORMULATE ATEST TO DISCRIMINATE BETWEEN THE TWO HYPOTHESES  WE HAVE H0MC  X    Z  MU0 QQUADQQUADH1MC  X    Z  MU1WITH BEGINDISPLAYMATHFZZ FRAC1SQRT2PISIGMAEXPLEFTFRACZ22SIGMA2RIGHTENDDISPLAYMATHTHE PROBABILITY DENSITIES OF X UNDER EACH HYPOTHESIS ARE BEGINALIGNEDFXXTHETA0    FRAC1SQRT2PISIGMAEXPLEFTFRACXMU022SIGMA2RIGHT10PTFXXTHETA1  FRAC1SQRT2PISIGMAEXPLEFTFRACXMU122SIGMA2RIGHTENDALIGNEDTHE PROBLEM CAN THUS ALSO BE STATED AS H0MC  X SIMNCMU0SIGMA2 QQUADQQUADH1MC  X  SIMNCMU1SIGMA2THE LIKELIHOOD RATIO ISELLX  FRACFXX THETA1FXXTHETA0    FRACFRAC1SQRT2PISIGMAEXPLEFTFRACXMU122SIGMA2RIGHTFRAC1SQRT2PISIGMAEXPLEFTFRACXMU022SIGMA2RIGHT AFTER CANCELING COMMON TERMS AND TAKING THE LOGARITHM WE HAVEBEGINEQUATIONLABELLLRTLAMBDAX  LOG ELLX  FRAC1SIGMA2 XMU1  MU0 FRAC12SIGMA2MU02  MU12ENDEQUATIONTHE LOG LIKELIHOOD RATIO TEST THEN BECOMESBEGINEQUATIONLABELLOGLIKELIHOODRATIOAPHIX  BEGINCASES1  TEXTIF   LAMBDAX  ETA10PTGAMMA  TEXTIF   LAMBDAX  ETA10PT0  TEXTIF  LAMBDAX  ETAENDCASESENDEQUATIONWHERE ETA  LOG NU  SINCE LAMBDAX  ETA WITH PROBABILITYZERO BECAUSE THE PDF IS CONTINUOUS THE MIDDLE CHOICE IN THE TESTCAN BE REMOVED WITH NO EFFECT ON THE PROBABILITY OF ERRORALSO LETTING TAU  FRACMU1  MU02  SIGMA2 FRACETAMU1   MU0WE SEE THAT THE TEST CAN BE WRITTEN ASBEGINEQUATIONLABELLOGLIKELIHOODRATIO2PHIX  BEGINCASES    1  TEXTIF  X GEQ TAU 0  TEXTIF    X  TAUENDCASESENDEQUATIONFIGURE REFFIGNP1 ILLUSTRATES A BLOCK DIAGRAM OF THIS TEST  THETEST SIMPLY BECOMES A MATTER OF TESTING AGAINST A THRESHOLDBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRHTEST1    CAPTIONSCALAR GAUSSIAN DETECTION OF THE MEAN    LABELFIGNP1  ENDCENTERENDFIGUREIN ORDER TO QUANTIFY THE PERFORMANCE OF THIS TEST WE NEED TODETERMINE THE DISTRIBUTION OF THE LOGLIKELIHOOD FUNCTION  WE OBSERVETHAT LAMBDAX IS A LINEAR FUNCTION OF THE RANDOM VARIABLE X SOTHAT LAMBDAX IS ITSELF A GAUSSIAN RANDOM VARIABLE WITH MEAN ANDCOVARIANCE UNDER HYPOTHESES H0 AND H1BEGINXALIGNAT2MUTHETA0  ETHETA0 LAMBDAX   FRAC12SIGMA2MU0MU12  QQUAD   SIGMATHETA02  VARTHETA0 LAMBDAX   FRAC1SIGMA2MU1  MU02 LABELEQGT0  MUTHETA1    ETHETA1 LAMBDAX   FRAC12SIGMA2MU0 MU12   QQUAD SIGMATHETA12  VARTHETA1 LAMBDAX   FRAC1SIGMA2MU1MU02 LABELEQGT1ENDXALIGNATTHUS THE LOGLIKELIHOOD FUNCTION HAS THE DISTRIBUTIONS FLLTHETA0 SIM NCMUTHETA0SIGMATHETA02 QQUADFLLTHETA1 SIM NCMUTHETA1SIGMATHETA12THENBEGINEQUATIONLABELEQGAUSSPROBNP1ALPHA  PFA  PTHETA0LAMBDAX  ETA PNCMUTHETA0SIGMATHETA02  ETA  INTETAINFTY  FRAC1SQRT2PI SIGMATHETA0   EYMUTHETA022SIGMATHETA02  DYENDEQUATIONBEGINTEXTBOX09TEXTWIDTHTHE Q FUNCTION LABELBOXQFINDEXQ FUNCTIONTHE QFUNCTION IS FREQUENTLY USED IN PROBABILITY OF ERROR ANALYSISIN COMMUNICATIONS PROBLEMS  IF Z SIM NC01 THAT IS Z IS AUNIT GAUSSIAN RANDOM VARIABLE THENPARBOX05LINEWIDTH QX  PZX  INTXINFTY  FRAC1SQRT2PI EY22 DYQQUAD PARBOX05LINEWIDTHINPUTPICTUREDIRQFUN1NOINDENT IF WSIM NCMUSIGMA2 IT IS STRAIGHTFORWARD TO SHOWBY A CHANGE OF VARIABLES THAT PW  X  QLEFTFRACX  MUSIGMARIGHTIT IS ALSO STRAIGHTFORWARD TO SHOW THAT QX  1QXTHE PLOT BELOW ILLUSTRATES THE QFUNCTION FOR X GEQ 0BEGINFIGUREHBEGINCENTEREPSFIGFILEPICTUREDIRQFEPSWIDTH04TEXTWIDTHENDCENTER QFM  CAPTION  LABELFIGQFENDFIGURESEE ALSO THE BOUNDS IN EXERCISE REFEXQTHE Q FUNCTION IS RELATED TO THE COMPLEMENTARY ERROR FUNCTION COMMONIN STATISTICS  IT MAY BE COMPUTED USING SC MATLAB USING THEFOLLOWINGSMALL BEGINBOXEDVERBATIMFUNCTION P  QFX FUNCTION P  QFX COMPUTE THE Q FUNCTION P  1SQRT2PIINTXINFTY EXPT22DTP  05ERFCXSQRT2ENDBOXEDVERBATIMVERBATIMINPUTMATLABDIRQFMENDTEXTBOXFIGURE REFERRORPROB ILLUSTRATES THE NORMAL CURVES FOR THE TWOHYPOTHESES UNDER QUESTION SHOWING THE PFA AS THE AREA UNDER THECURVE FXXTHETA0 TO THE RIGHT OF THE THRESHOLD TAU ANDPD AS THE AREA UNDER FXXTHETA1 TO THE RIGHT OF THETHRESHOLDBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    SUBFIGUREPFAINPUTPICTUREDIRDETEST2    SUBFIGUREPDINPUTPICTUREDIRDETEST3    CAPTIONERROR PROBABILITIES FOR GAUSSIAN VARIABLESERROR PROBABILITIES FOR GAUSSIAN VARIABLES WITH DIFFERENT MEANS AND EQUAL VARIANCES    LABELERRORPROB  ENDCENTERENDFIGUREBEGINFIGUREHTBUNITLENGTH 2INBEGINCENTERBEGINPICTURE1020PUT2645EPSFIGFILEDETESTPICTDIRPFAPSHEIGHT45IN  HSCALE50VSCALE50HOFFSET0PUT52SPECIALPSFILEPE1USERSWYNNTEXCLASSEE540PFAPS   HSCALE 50 VSCALE  50 HOFFSET 0PUT5115MAKEBOX00APUT265EPSFIGFILEDETESTPICTDIRPDPSHEIGHT45INHSCALE50VSCALE50HOFFSET0PUT58SPECIALPSFILEPE1USERSWYNNTEXCLASSEE540PDPS   HSCALE 50 VSCALE  50 HOFFSET 0PUT52MAKEBOX00BPUT017MAKEBOX00SCRIPTSTYLE FELLLTHETA0PUT1117MAKEBOX00SCRIPTSTYLE FELLLTHETA1PUT55175MAKEBOX00SCRIPTSTYLE DPUT5175VECTOR102PUT6175VECTOR102PUT65125MAKEBOX00SCRIPTSTYLE FRACETA D22DPUT15155VECTOR418PUT16155MAKEBOX00PFAPUT1536VECTOR416PUT166MAKEBOX00PDPUT00MAKEBOX00PUT55MAKEBOX00ENDPICTUREENDCENTERCAPTIONERROR PROBABILITIES FOR NORMAL VARIABLES WITH DIFFERENT MEANS ANDEQUAL VARIANCES A PFA CALCULATION B PD CALCULATION  LABELERRORPROBAENDFIGUREBASED ON THE DEFINITION OF THE QFUNCTION SEE BOX REFBOXQFREFEQGAUSSPROBNP1 CAN BE WRITTEN ALPHA  QLEFTFRACETA  FRAC12SIGMA2MU0     MU12  FRAC1SIGMAMU1  MU0RIGHT  QSIGMA ETAMU1 MU0  MU0  MU12SIGMAIF WE LET D  MU0  MU1 BE THE DISTANCE BETWEEN THE MEANS WEHAVEBEGINEQUATION ALPHA  QSIGMA ETAD  D2SIGMALABELEQALPHAGAUSSENDEQUATIONWE CAN ALSO WRITE THIS ASBEGINEQUATION  LABELEQALPHAGAUSS2   ALPHA  QZENDEQUATIONWHERE BEGINEQUATION  LABELEQALPHAZZ  SIGMA ETAD  D2SIGMA  ENDEQUATIONSIMILARLY THE PROBABILITY OF DETECTION IS OBTAINED FROMBEGINEQUATION  LABELEQBETAGAUSSBEGINSPLITBETA  PD  QLEFTFRACETA  FRAC12SIGMA2MU0     MU12 FRAC1SIGMAMU1  MU0RIGHT   QSIGMA ETAD  D2SIGMA  QZDSIGMAENDSPLITENDEQUATIONA PLOT OF THE ROC IS SHOWN IN FIGURE REFFIGROC1  THE PLOT SHOWSPERFORMANCE FOR VARIOUS VALUES OF THE SIGNALTONOISE RATIOSNR WHICH IS DEFINED HERE ASBEGINEQUATION SNR  FRACDSIGMA  FRACMU0  MU1SIGMALABELEQSNDROCDEFENDEQUATIONAS THE SNR INCREASES IT IS POSSIBLE TO OBTAIN GREATER POWER FOR AGIVEN SIZEBEGINFIGUREHTBPCENTERLINEINPUTPICTUREDIRROC1  CAPTIONROC FOR TEST OF MEANS OF GAUSSIAN  LABELFIGROC1ENDFIGURESUBSUBSECTIONVECTOR GAUSSIAN DETECTION DIFFERENT MEANS COMMON  VARIANCELET MITHETA I12LDOTS N BE SAMPLES OF SIGNAL A SIGNALPARAMETERIZED BY SOME PARAMETER THETA  SUPPOSE THAT THE SIGNAL ISOBSERVED IN NOISE PRODUCING A MEASUREMENT XI  MITHETA  ZI QQUAD I12LDOTSNWHERE THE ZI ARE NC0SIGMA2 AND ARE INDEPENDENT  THENBECAUSE THE ZI ARE INDEPENDENT THE JOINT PDF OF XBF  X1X2LDOTSXN IS SIMPLY THE PRODUCT OF THE INDIVIDUAL PDFSBEGINALIGNED  FX1X2LDOTSXNX1X2LDOTSXNTHETA   FXBFXBFTHETA    PRODI1N  FRAC1SQRT2PISIGMAEXPLEFTFRACXIMITHETA22SIGMA2RIGHT     FRAC12PIN2SIGMAN EXPLEFTFRAC12SIGMA2    XBF  MBFTHETATXBFMBFRIGHTENDALIGNEDWHERE MBFTHETA  M1THETAM2THETALDOTSMNTHETATUNDER THIS MODEL WE CAN CONSIDER A DETECTION PROBLEM SUCH ASDETERMINATION OF WHICH SIGNAL WAS SENTBEGINEXAMPLE ONOFF SIGNALINGSUPPOSE THAT THERE ARE TWO POSSIBLE SIGNALS THETA  01CORRESPONDING TO THE HYPOTHESES BEGINALIGNEDH0MC  MBF  ZEROBFQQUAD THETA  0  H1MC MBF  MBF1QQUAD THETA  1ENDALIGNEDTHAT IS THE SIGNAL IS EITHER ABSENT OR IT IS PRESENT AND THEOBSERVED VECTOR XBF HAS MEAN MBF1ENDEXAMPLEWE CAN GENERALIZE THE DETECTION PROBLEM TO SAMPLES THAT ARE NOTINDEPENDENT  CONSIDER THE SIMPLY BINARY GAUSSIAN DETECTION PROBLEM BEGINALIGNEDH0MC XBF SIM NCMBF0R H1MC XBF SIM NCMBF1RENDALIGNEDTHENBEGINALIGNEDFXBFXBFTHETA0  FRAC12PIN2 R12EXPLEFT FRAC12XBF  MBF0T R1XBF  MBF0RIGHTFXBFXBFTHETA1  FRAC12PIN2 R12EXPLEFT FRAC12XBF  MBF1T R1XBF  MBF1RIGHTENDALIGNEDAS WE DID FOR THE SCALAR GAUSSIAN DETECTION CASE WE DETERMINE THELIKELIHOOD RATIOBEGINALIGNEDELLXBF FRACFXXBFTHETA1FXXBFTHETA0   EXPMBF1  MBF0T R1 XBF  FRAC12MBF0MBF1TR1 MBF0 MBF1ENDALIGNEDAND LOGLIKELIHOOD RATIOLAMBDAXBF  MBF1  MBF0TR1 XBF  XBF0WHERE XBF0  FRAC12MBF1   MBF0LETTING WBF  R1MBF1  MBF0 WE CAN WRITE LAMBDAXBF  WBFTXBFXBF0THE SET OF POINTS WHERE LAMBDAXBF0 FORMS A PLANE ORTHOGONAL TOWBF PASSING THROUGH XBF0THE DECISION BASED UPON THE LOGLIKELIHOOD RATIO IS LAMBDAXBF  BEGINCASES  1  WBFTXBFXBF0  ETA   0  WBFTXBFXBF0  ETAENDCASESFIGURE REFFIGNP2 ILLUSTRATES THE BLOCK DIAGRAM FOR THIS TESTBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRHTEST2    CAPTIONTEST FOR VECTOR GAUSSIAN RANDOM VARIABLE WITH DIFFERENT MEANS    LABELFIGNP2  ENDCENTERENDFIGURETHE PERFORMANCE FOR THIS VECTOR GAUSSIAN CASE IS STRAIGHTFORWARD TO DETERMINE  WE OBSERVE THAT LAMBDAXBF IS ASCALAR GAUSSIAN RANDOM VARIABLE WITHBEGINEQUATIONMUTHETA0  ETHETA0 LAMBDAXBF  FRAC12MBF1 MBF0TR1MBF1 MBF0  FRAC12 WBFT R WBFLABELEQGM2ENDEQUATIONBEGINEQUATIONVARTHETA0 LAMBDAXBF  MBF1  MBF0T R1MBF1  MBF0  WBFT R WBFLABELEQGV2ENDEQUATIONAND SIMILARLYMUTHETA1 FRAC12 WBFT R WBF QQUADQQUADVARTHETA1  WBFT R WBFLET BEGINEQUATIONS2  WBFT R WBFLABELEQDEFFNPENDEQUATIONTHEN UNDER H0 LAMBDAX SIM NCS22S2AND UNDER H1 LAMBDAX SIM NCS22S2THE PERFORMANCE OF THE DETECTOR IS PFA  PTHETA0LAMBDAX GEQ ETA  QLEFTFRAC    ETA  S22SRIGHT  QZAND PD  PTHETA1LAMBDAX GEQ ETA  QLEFTFRAC    ETA  S22SRIGHT  QZSWHERE Z  ETAS  S2  BY COMPARISON WITH REFEQALPHAGAUSSTHE QUANTITY S IS DIRECTLY ANALOGOUS TO DSIGMA WHICH WE DEFINEDAS THE SIGNALTONOISE RATIO SNR  THUS ROC FOR THE VECTOR GAUSSIANCASE IS IDENTICAL TO THAT OF THE SCALAR GAUSSIAN CASE WHEN PLOTTED ASA FUNCTION OF SNRSSUBSUBSECTIONSIMPLIFICATIONS WHEN RIIT IS INTERESTING TO EXAMINE SOME DETECTOR STRUCTURES UNDER THEFREQUNTLYENCOUNTEREDCIRCUMSTANCE THAT R  I THAT IS THAT THE SAMPLES OF THE SIGNAL AREINDEPENDENT  THEN LAMBDAX  MBF1  MBF0TXBF  XBF0THE QUANTITY S2 DEFINED IN REFEQDEFFNP IS SIMPLY S2  FRAC1SIGMA2 MBF1  MBF02  FRACD2SIGMA2WHERE D   MBF1  MBF0  NOTE THE L2  EUCLIDEAN NORM IS USED HERE AND THROUGHOUT ALL THE DISCUSSION OF GAUSSIANDETECTION  IT IS A NATURAL NORM TO USE FOR PROBLEMS ASSOCIATED WITHGAUSSIAN PROBLEMS  AN ADDITIONAL SIMPLIFICATION OCCURS WHEN MBF1  MBF0  THEN THE LOGLIKELIHOOD RATIO IS LAMBDAXBF  FRAC1SIGMA2MBF1T  MBF0T XBF  CWHERE C IS A CONSTANT THAT DOES NOT DEPEND UPON XBF  ABSORBINGTHE CONSTANT AND THE FACTOR SIGMA2 INTO THE THRESHOLD THEDETECTOR COMPUTES MBF1MBF0T XBF AND COMPARES THIS INNERPRODUCT TO THE MODIFIED THRESHOLD  IN THIS CASE THE DETECTORDETERMINES ON THE BASIS OF THE ANGLE BETWEEN THE SIGNALS TO WHICHSIGNAL THE RECEIVED VECTOR IS MOST SIMILARBEGINEXAMPLE LABELEXMDIGCOMDET    THE DETECTION PROBLEM APPLIES DIRECTLY TO DIGITAL COMMUNICATIONS  WHERE SIGNAL MBF0 AND MBF1 ARE SENT AND WE DESIRE TO  DISTINGUISH BETWEEN THEM AT THE RECEIVER  SUPPOSE THAT MBF0 OR  MBF1 ARE SENT WITH EQUAL PROBABILITY  MOST COMMONLY WE CHOOSE  THE THRESHOLD SO THAT THERE IS THE SAME PROBABILITY OF ERROR GIVEN  THAT A ZERO IS SENT AS THERE IS GIVEN THAT A ONE IS SENT  THAT IS  WE SET ALPHA  1BETAWHICH CORRESPONDS TO THE CASE THEN ETA  0  LET PECMBFI BETHE PROBABILITY OF ERROR GIVEN THAT MBFI WAS SENT  THEN THEPROBABILITY OF ERROR DENOTED PEC IS PEC  PMBF0PECMBF0  PMBF1PECMBF1 FRAC12ALPHA  FRAC121BETA  QF2THIS CAN BE WRITTEN AS BOXEDPEC  QLEFTFRACD2SIGMARIGHTWHERE D   MBF1  MBF0  IN A DIGITAL COMMUNICATIONSSETTING ULTIMATELY THE PROBABILITY OF ERROR FOR BINARY COMMUNICATIONSEM DEPENDS UPON THE DISTANCE BETWEEN SIGNALS D RELATIVE TO THE NOISEENERGY  THIS IS WHY THE SNR IS SUCH AN IMPORTANT MEASURE INCOMMUNICATIONS  FIGURE REFFIGSNCCOMP ILLUSTRATES THE PROBABILITYOF ERROR AS A FUNCTION OF SNR IN DB  THE SNR IS GIVEN FOR REASONSWHICH WILL BECOME MORE CLEAR LATER AS SNR  EBN0BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    EPSFIGFILEPICTUREDIRPLOTBPSKEPS PLOTBPSKM  ENDCENTER    LABELFIGSNCCOMPCAPTIONPROBABILITY OF ERROR FOR BPSK SIGNALLINGENDFIGURECONSIDER NOW THE TWO BINARY SIGNAL CONSTELLATIONS SHOWN IN FIGUREREFFIGBINCONST  IN EACH CONSTELLATION THE SIGNALS HAVE EQUALENERGY MBF0  MBF11IN THE EM ORTHOGONAL SIGNAL CONSTELLATION INDEXORTHOGONAL SIGNAL  CONSTELLATION IN WHICHMBF0TMBF1  0 THE DISTANCE BETWEEN THE SIGNALS IS D  SQRT2 EIN THE EM ANTIPODAL SIGNAL CONSTELLATION INDEXANTIPODAL SIGNAL  CONSTELLATION THE DISTANCE BETWEEN THE SIGNALS IS D  2EIN COMPARING THE TWO DISTANCES THE ANTIPODAL SIGNALING HAS A 3 DBADVANTAGE IN SNR OVER ORTHOGONAL SIGNALINGBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRBINCONST1    CAPTIONAN ORTHOGONAL AND ANTIPODAL BINARY SIGNAL CONSTELLATION    LABELFIGBINCONST  ENDCENTERENDFIGUREENDEXAMPLESUBSUBSECTIONVECTOR GAUSSIAN SAME MEANS DIFFERENT COVARIANCELET US NOW CONSIDER A DIFFERENT KIND OF DETECTION PROBLEM IN WHICHTHE MEANS ARE THE SAME BUT THE COVARIANCES ARE DIFFERENT  WE WILLASSUME FOR CONVENIENCE THAT THE MEANS ARE EQUAL TO ZERO  WE WISH TOEXAMINE THE DETECTION PROBLEM BEGINALIGNEDH0MC XBF SIM NCZEROBFSIGMA02 I H1MC XBF SIM NCZEROBFSIGMA12 IENDALIGNEDIN WHICH XBF IS AN NDIMENSIONAL RANDOM VECTOR  THE LOG LIKELIHOODIS LAMBDAXBF  LOGFRACSIGMA0SIGMA1  XBFTXBFLEFTFRAC12SIGMA02  FRAC12SIGMA12RIGHTSINCE THE FIRST TERM DOES NOT DEPEND ON THE DATA WE WILL DISCARD ITAND WRITEBEGINEQUATION LAMBDAXBF  XBFT XBFLEFTFRAC12SIGMA02   FRAC12SIGMA12RIGHTLABELEQXI2DETECTENDEQUATIONLET US DENOTE GAMMA2  FRAC12SIGMA02   FRAC12SIGMA12 SO THAT LAMBDAXBF  GAMMA2XBFT  XBFTHE NEYMANPEARSON TEST BECOMESBEGINEQUATION PHIXBF  BEGINCASES  1  GAMMA2 XBFT XBF GEQ ETA   0  GAMMA2 XBFT XBF  ETAENDCASESLABELEQNPSMDVENDEQUATIONFOR SOME THRESHOLD ETAIN THE EVALUATION OF THE PERFORMANCE OF THIS TEST IT MUST BERECOGNIZED THE LAMBDAXBF BEING A EM QUADRATIC FUNCTION OF AGAUSSIAN VECTOR IS NO LONGER GAUSSIAN DISTRIBUTED  WE MUST EXAMINE ANEW DISTRIBUTION TO DETERMINE THE POWER AND SIZE OF THIS TESTSUBSUBSECTIONCHI2 RANDOM VARIABLESINDEXCHISQUARED RANDOM VARIABLESINDEXCHI2 RANDOM VARIABLESINDEXRANDOM VARIABLESCHI2TO ANALYZE THE PERFORMANCE OF THE DETECTOR IN REFEQXI2DETECT WENEED TO INTRODUCE A NEW DISTRIBUTION  SUPPOSE THAT Z  SUMI1N YI2WHERE THE RANDOM VARIABLES YI I1 LDOTS N ARE INDEPENDENT ANDNC01  THE RANDOM VARIABLE Z IS SAID TO BECENTRAL CHISQUARED WITH N DEGREES OF FREEDOM DENOTED AS Z SIM CHIN2BEGINTHEOREM LABELTHMXI2  IF Z SIM CHIN2 THENBEGINEQUATIONFZZ  FRAC1GAMMAN22N2 ZN21EZ2LABELEQCHI2ENDEQUATIONENDTHEOREMTHE FUNCTION GAMMACDOT IS THE GAMMA GAMMA FUNCTIONDESCRIBED IN BOX REFBOXGAMMA FIGURE REFFIGCHI2 ILLUSTRATES THIS DENSITY FUNCTIONBEGINPROOF  LET Y1 SIM NC01 AND LET Z1  Y12  THEN BEGINALIGNEDPZ1 LEQ Z  PSQRTZ LEQ Y1 LEQ SQRTZ  2INT0SQRTZ FRAC1SQRT2PI EX22DX  INT0Z FRAC1SQRT2PI EX2 X12DXENDALIGNEDBY TAKING THE DERIVATIVE WITH RESPECT TO Z WE OBTAIN FZ1Z  FRAC1SQRT2PI EZ2Z12 QQUAD Z GEQ 0INDEXCHARACTERISTIC FUNCTIONCHI2THE CHARACTERISTIC FUNCTION OF Z1 IS INDEXCHARACTERISTIC FUNCTIONBEGINEQUATION PHIZ1OMEGA  EEJOMEGA Z1  FRAC112JOMEGA12LABELEQCHICHARENDEQUATIONNOW LET Z  SUMI1N YI2WHERE EACH YI SIM NC01 INDEPENDENTLY  THEN PHIZOMEGAIS THE NFOLD PRODUCT OF PHIZ1OMEGA BEGINEQUATION  LABELEQCHI2CHARPHIZOMEGA  FRAC112JOMEGAN2ENDEQUATIONTHE INVERSE FOURIER TRANSFORM OF THIS FUNCTION IS SEE EXERCISE REFEXCHI2PDFBEGINEQUATIONLABELEQCHI2PDF FZZ  FRAC1GAMMAN2 2N2 ZN21EZ2 QQUAD Z GEQ 0ENDEQUATIONENDPROOFA RESULT THAT WE WILL NEED SHORTLY RELATES TO QUADRATIC FORMS OFGAUSSIAN RANDOM VARIABLES A GENERALIZATION OF CHI2N RANDOM VARIABLESBEGINTHEOREM LABELTHMPRCHI2 CITESCHARFL1991  LET XBF SIM NC0R BE NDIMENSIONAL AND LET Q  XBFT P XBFWHERE P IS SYMMETRICIF PR  RP THEN THEN THE CHARACTERISTIC FUNCTION OF Q IS PHIQOMEGA  FRAC1I 2JOMEGA RP12HENCE IF RP IS A PROJECTION MATRIX WITH R NONZERO EIGENVALUESTHEN Q IS A CHIR2 RANDOM VARIABLEENDTHEOREMBEGINPROOF   BEGINALIGNEDPHIQOMEGABF  E EJOMEGA Q FRAC12PIN2R12 INT EXPFRAC12 XBFTR1XBF EXPJOMEGA XBFT P XBF DXBF  FRAC12PIN2R12 INT FRACI2JOMEGA  RP12I 2JOMEGA RP12EXPFRAC12XBFTR1I2JOMEGA RP XBF DXBF  FRAC1I2JOMEGA RP12  FRAC12PIN2  RI2JOMEGA RP112 10PT QQUAD QQUAD TIMES INT EXPFRAC12XBFTR1I2JOMEGA RPXBF DXBF FRAC1I2JOMEGA RP12ENDALIGNEDNOW SUPPOSE RP IS A RANKR PROJECTION MATRIX  SINCE THEEIGENVALUES OF RP ARE EITHER 0 OR 1 THE DIAGONALIZATION OF RPUSING ORTHOGONAL THE EIGENVECTOR MATRIX U IS UT RPU   DIAG11LDOTS100LDOTS0WHERE THERE ARE R 1S ON THE DIAGONAL  IN THIS CASE PHIQOMEGA  FRAC112JOMEGARWHICH IS THE CHARACTERISTIC FUNCTION FOR A CHIR2 RANDOM VARIABLEENDPROOFBEGINTEXTBOX09TEXTWIDTHTHE GAMMA FUNCTIONLABELBOXGAMMAINDEXGAMMA FUNCTION INDEXGAMMA FUNCTIONTHE GAMMA FUNCTION IS DEFINED BY THE INTEGRAL GAMMAX  INT0INFTY TX1 ETDTUSING INTEGRATION BY PARTS IT IS STRAIGHTFORWARD TOSHOW FOR THAT FOR X0 LAMBDAX1  XGAMMAXSO THAT FOR AN INTEGER K GAMMAK  K1SOME USEFUL SPECIAL VALUES OF THE GAMMA FUNCTION ARE GAMMA12  SQRTPI  GAMMAM12  FRAC1CDOT 3 CDOT 5 CDOTS 2M12MSQRTPI QQUAD M123LDOTSENDTEXTBOXSUBSUBSECTIONPERFORMANCE OF DIFFERENTCOVARIANCE DETECTORSWE RETURN NOW TO ANALYZING THE PERFORMANCE OF THE DETECTORREFEQNPSMDV  UNDER H0 FRACLAMBDAXBFGAMMA2 SIGMA02 SIM CHIN2AND UNDER H1 FRACLAMBDAXBFGAMMA2 SIGMA12 SIM CHIN2THENBEGINEQUATION  LABELEQCHI2PFABEGINSPLITPFA  PTHETA0LAMBDAXBF  ETA  PLAMBDAXBFGAMMA2 SIGMA02  ETA GAMMA2 SIGMA02 PCHIN2  TAUGAMMA2 SIGMA02  INTETAGAMMA2 SIGMA02INFTY FRAC1GAMMAN2 2N2ZN21EZ2 DZENDSPLITENDEQUATIONSIMILARLY BEGINEQUATIONPD  INTETAGAMMA2 SIGMA12INFTY FRAC1GAMMAN2 2N2ZN21EZ2 DZLABELEQCHI2PDENDEQUATIONIN CASE OF GENERAL N REFEQCHI2PFA AND REFEQCHI2PD MUSTBE COMPUTED NUMERICALLY  HOWEVER AS THE NEXT EXAMPLE ILLUSTRATESTHE ROC IS READILY OBTAINED WHEN N2BEGINEXAMPLE  THE COMPUTATIONS IN REFEQCHI2PFA AND REFEQCHI2PD ARE  READILY ACCOMPLISHED WHEN N2 SINCE THE DENSITY OF A CHI22  RANDOM VARIABLE Y IS FYZ  FRAC12 EY2LETTING  EPSILON  ETAGAMMA2 WE HAVE PFA  EEPSILONSIGMA02QQUADTEXTAND QQUAD PD EEPSILONSIGMA12GIVEN A SIZE ALPHA THE THRESHOLD FOR THE TEST USING XBFT XBFAS THE STATISTIC CAN BE DETERMINED FROM EPSILON  SIGMA02 LOG PFAFURTHERMORE THE ROC CAN BE READILY OBTAINED SINCE PD  PFASIGMA02SIGMA12FIGURE REFFIGROC2 ILLUSTRATES THIS ROC FOR RHO  FRACSIGMA12SIGMA02  12345AS EXPECTED THERE IS IMPROVED PERFORMANCE AS THE RATIO BETWEEN THEVARIANCES INCREASESENDEXAMPLEWE CONSIDER BRIEFLY THE PROBLEM BEGINALIGNEDH0MC XBF SIM NCZEROBFR0 H1MC XBF SIM NCZEROBFR1ENDALIGNEDDEVELOPING THE LIKELIHOOD RATIO TEST IS STRAIGHTFORWARD SEE EXERCISEREFEXLLT2  HOWEVER QUANTIFYING THE PERFORMANCE IS MOREDIFFICULT BECAUSE THE PDF OF LAMBDAXBF CAN ONLY BE OBTAINED BYNUMERICAL INTEGRATIONBEGINFIGUREHTBPBEGINCENTERINPUTPICTUREDIRROC2ENDCENTER EPSFIGFILEPICTUREDIRROC2PS  CAPTIONROC NORMAL VARIABLES WITH EQUAL MEANS AND UNEQUAL VARIANCES  LABELFIGROC2ENDFIGURESUBSECTIONPROPERTIES OF THE ROCLABELSECROCPROP BEGINDESCRIPTION ITEMPROPERTY 1 SUBSUBSECTIONPARAGRAPHPROPERTY 1EM ALL LIKELIHOOD RATIO TESTS HAVE ROC CURVES THAT ARE CONCAVEBEGINPROOF SUPPOSE THE ROC HAS A SEGMENT THAT ISCONVEX  TO BE SPECIFIC SUPPOSE PFAA PDA AND PFABPDB ARE POINTS ON THE ROC CURVE BUT THE CURVE IS CONVEX BETWEENTHESE TWO POINTS AS ILLUSTRATED IN FIGURE REFFIGCONCAVEROC  LETPHIAX AND PHIBX BE THE DECISION RULES OBTAINED FOR THECORRESPONDING SIZES AND POWERS AS GIVEN BY THE NEYMANPEARSON LEMMABEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRROCPROOF    CAPTIONDEMONSTRATION OF CONCAVE PROPERTY OF THE ROC  ENDCENTERLABELFIGCONCAVEROCENDFIGURENOW FORM A NEW RULE BY CHOOSING PHIA WITH PROBABILITY Q ANDPHIB WITH PROBABILITY 1Q FOR ANY 0  Q  1  IEBEGINDISPLAYMATHPHIX  LEFT BEGINARRAYCL PHIAX  MBOX WITH PROBABILITY  Q10PTPHIBX  MBOX WITH PROBABILITY  1Q ENDARRAYRIGHT ENDDISPLAYMATHTHIS IS A RANDOMIZED RULE UNDER WHICH THE DECISION MAKER WOULD TAKEACTION CORRESPONDING TO PHIA WITH PROBABILITY Q OTHERWISE HEWOULD TAKE ACTION CORRESPONDING TO RULE PHIB  THE PROBABILITY OFDETECTION PD FOR THIS RANDOMIZED RULE ISBEGINDISPLAYMATHPD  Q PDA  1Q PDBENDDISPLAYMATHA EM CONVEX COMBINATION OF PDA AND PDB  THE SET OF ALLSUCH CONVEX COMBINATIONS MUST LIE ON THE LINE CONNECTING PDA ANDPDB HENCE THE RULE PHIX OF SIZE PFA HAS GREATERPOWER THAN THE RULE PROVIDED BY THE NEYMANPEARSON TEST THUSCONTRADICTING THE OPTIMALITY OF THE NEYMANPEARSON TEST  THUS THEROC CURVE CANNOT BE CONCAVE ENDPROOFITEMPROPERTY 2 PARAGRAPHPROPERTY 2EM ALL CONTINUOUS LIKELIHOOD RATIO TESTS HAVE ROC CURVES THAT ARE  ABOVE THE PD  PFA LINE THIS PROPERTY IS JUST A SPECIAL CASE OF PROPERTY 1 BECAUSE THE POINTS00 AND 11 ARE CONTAINED ON ALL ROC CURVESITEMPROPERTY 3 PARAGRAPHPROPERTY 3EM THE SLOPE OF THE ROC CURVE AT ANY DIFFERENTIABLE POINT IS EQUAL    TO THE VALUE OF THE THRESHOLD NU     REQUIRED TO ACHIEVE THE PD AND PFA OF THAT POINT USING THE    ORIGINAL LIKELIHOOD RATIO NOT THE LOGLIKELIHOOD RATIOBEGINPROOF  LET ELL BE THE LIKELIHOOD RATIOAND SUPPOSE NU IS A GIVEN  THRESHOLD  THEN BEGINALIGNEDPD   INTNUINFINITY FELL L THETA1DL10PTPFA    INTNUINFINITY FELL L THETA0DLENDALIGNEDLET DELTA BE A SMALL PERTURBATION IN THE THRESHOLD THEN BEGINALIGNEDDELTA PD   INTNUNUDELTA FELL L THETA1DL10PTDELTA PFA   INTNUNUDELTA FELL L THETA0DLENDALIGNEDREPRESENT THE CHANGES IN PD AND PFA RESPECTIVELY AS A RESULTOF THE CHANGE IN THRESHOLDTHEN THE SLOPE OF THE ROC CURVE IS GIVEN BYBEGINEQUATIONLABELTEMPLIMDELTA RIGHTARROW 0 FRACDELTA PDDELTA PFA LIMDELTA RIGHTARROW 0 FRACDELTA FELLNU THETA1DELTA FELLNUTHETA0  FRAC FELLNU THETA1FELLNUTHETA0ENDEQUATION TO ESTABLISH THAT THIS RATIO EQUALS NU WE WE OBSERVE THAT INGENERAL BEGINALIGNEDETHETA1ELLNX   INT ELLNX FXX THETA1DX10PT   INT FRACFXNXTHETA1FXNXTHETA0 FXXTHETA1DX10PT   INT FRACFXN1XTHETA1FXN1XTHETA0 FXX THETA0DX10PT   INT ELLN1X FXX THETA0DX10PT   ETHETA0ELLN1X ENDALIGNEDBUT THE CONDITION ETHETA1ELLN  ETHETA0ELLN1REQUIRES THATBEGINDISPLAYMATHINT  LNFELLL THETA1DL  INT LN1 FELLL THETA0DLENDDISPLAYMATHMUST HOLD FOR ALL N WHICH IMPLIES THATBEGINEQUATIONLABELIDENTITY1FELLL THETA1   L FELLL THETA0ENDEQUATIONMUST HOLD FOR ALL VALUES OF L  THUS APPLYING REFIDENTITY1 TO REFTEMP WE OBTAIN THE DESIREDRESULTBEGINDISPLAYMATHFRACDPDDPFA FRAC FELLNU THETA1FELLNUTHETA0 NUENDDISPLAYMATHENDPROOFENDDESCRIPTIONSECTIONNEYMANPEARSON TESTING WITH COMPOSITE BINARY HYPOTHESESLABELSECCOMPBININDEXHYPOTHESIS TESTINGCOMPOSITETHUS FAR WE HAVE DEALT WITH THE SIMPLEST FORM OF BINARY HYPOTHESISTESTING A SIMPLE HYPOTHESIS VERSUS A SIMPLE ALTERNATIVE  WE NOWGENERALIZE OUR THINKING TO COMPOSITE HYPOTHESES  AS MENTIONED INDEFINITION REFDEFSIMPCOMP A HYPOTHESIS H0MC THETAINTHETA0 ISSAID TO BE EM COMPOSITE IF THETA0 CONSISTS OF AT LEAST TWOELEMENTS  WE ARE INTERESTED IN TESTING A COMPOSITE HYPOTHESISH0MC THETAINTHETA0 AGAINST A COMPOSITE ALTERNATIVE H1MC THETAINTHETA1  BEFORE PURSUING THE DEVELOPMENT OF A THEORY FORCOMPOSITE HYPOTHESES WE NEED TO GENERALIZE THE NOTIONS OF SIZE ANDPOWER FOR THIS SITUATIONBEGINDEFINITIONA TEST PHI OFH0MC THETAINTHETA0 AGAINST H1MC  THETAINTHETA1 IS SAID TOHAVE BF SIZE ALPHA IFBEGINDISPLAYMATHSUPTHETAINTHETA0 ETHETAPHIX ALPHAENDDISPLAYMATHENDDEFINITIONBEGINDEFINITION  A TEST PHI0 IS SAID TO BE BF UNIFORMLY MOST POWERFUL UMP OF    SIZE ALPHA INDEXUNIFORMLY MOST POWERFUL TEST FOR TESTING  H0MC THETAINTHETA0 AGAINST H1MC THETAINTHETA1 IF  PHI0 IS OF SIZE ALPHA AND IF FOR ANY OTHER TEST PHI OF  SIZE AT MOST ALPHABEGINDISPLAYMATHETHETAPHI0X GEQ ETHETAPHIX ENDDISPLAYMATHFOR EACH THETAINTHETA1ENDDEFINITIONFOR A TEST TO BE UMP IT MUST MAXIMIZE THE POWER ETHETAPHIXFOR EACH THETAINTHETA1  THIS IS A VERY STRINGENT CONDITION ANDTHE EXISTENCE OF A UNIFORMLY MOST POWERFUL TEST IS NOT GUARANTEED INALL CASES  FOR EXAMPLE ALTHOUGH THE NEYMANPEARSON LEMMA TELLS USTHAT THERE EXISTS A MOST POWERFUL TEST OF SIZE ALPHA FOR FIXEDTHETA1INTHETA1 THERE IS NO REASON WHY THIS SAME TEST SHOULD ALSO BEMOST POWERFUL OF SIZE ALPHA FOR THETA2NOTTHETA1 WITHTHETA2INTHETA1  OUR GOAL IN THIS SECTION IS TO ARRIVE ATCONDITIONS FOR WHICH THE EXISTENCE OF A UMP CAN INDEED BE GUARANTEEDTHAT IS WE WANT TO ESTABLISH CONDITIONS UNDER WHICH THERE EXISTS ATEST SUCH THAT THE PROBABILITY OF FALSE ALARM IS LESS THAN A GIVENALPHA FOR ALL THETAINTHETA0 BUT AT THE SAME TIME HAS MAXIMUMPROBABILITY OF DETECTION FOR ALL THETAINTHETA1  WE WILL APPROACH THIS DEVELOPMENT THROUGH AN EXAMPLE THIS RESULT WILLMOTIVATE THE CHARACTERIZATION OF THE CONDITIONS FOR THE EXISTENCE OF AUMP TESTBEGINEXAMPLE  LET X SIM NCTHETA1  LET THETA0  INFINITY  THETA0 AND LET THETA1  THETA0 INFINITY  WE WISH TO  TEST H0MC  THETAIN THETA0 AGAINST H1MC    THETAINTHETA1  WE DESIRE THE TEST TO BE UNIFORMLY MOST  POWERFUL OUT OF THE CLASS OF ALL TESTS PHI FOR WHICHBEGINEQUATIONLABELCLASSETHETAPHIX LEQ ALPHA QUAD FORALLTHETALEQ THETA0ENDEQUATIONTO SOLVE THIS PROBLEM WE FIRST SOLVE A RELATED PROBLEM AND SEEK THEBEST TEST PHI0 OF SIZE ALPHA FOR TESTING THE SIMPLE HYPOTHESISH0PRIMEMC  THETA  THETA0 AGAINST THE SIMPLE ALTERNATIVEH1PRIMEMC  THETA  THETA1 WHERE THETA1  THETA0  BYTHE NEYMANPEARSON LEMMA THIS TEST IS OF THE FORMBEGINDISPLAYMATHPHI0X  BEGINCASES1  TEXTIF   FRAC1SQRT2PIEXPXTHETA122            FRACNUSQRT2PIEXPXTHETA022 10PTGAMMA  TEXTIF   FRAC1SQRT2PIEXPXTHETA122            FRACNUSQRT2PIEXPXTHETA022 10PT0  TEXTIF   FRAC1SQRT2PIEXPXTHETA122            FRACNUSQRT2PIEXPXTHETA022         ENDCASESENDDISPLAYMATHAFTER TAKING LOGARITHMS AND REARRANGING THIS TEST ASSUMES ANEQUIVALENT FORMBEGINEQUATIONLABELUMPPHI0X  BEGINCASES1  TEXTIF   X  NUPRIME10PT0   TEXTOTHERWISEENDCASESENDEQUATIONWHEREBEGINDISPLAYMATHNUPRIME  FRACTHETA122 THETA022  ETATHETA1THETA0ENDDISPLAYMATHWE MAY SET GAMMA  0 SINCE THE PROBABILITY THATXNUPRIME IS ZERO   WITH THIS TEST WE SEE THAT BEGINALIGNEDPTHETA0X  NUPRIME   INTNUPRIMEINFINITY FRAC1SQRT2PIEXPXTHETA022 DX10PT   INTNUPRIMETHETA0INFINITY FRAC1SQRT2PIEXPX22 DX10PT   QNU  THETA0  ALPHAENDALIGNEDIMPLIES THAT BEGINEQUATIONLABELTHRESHNUPRIME  THETA0 Q1ALPHAENDEQUATIONIT IS IMPORTANT TO NOTE THAT NUPRIME DEPENDS ONLY ON THETA0AND ALPHA BUT EM NOT OTHERWISE ON THETA1  IN FACT EXACTLYTHE SAME TEST AS GIVEN BY REFUMP WITH NUPRIME DETERMINEDBY REFTHRESH IS BEST ACCORDING TO THE NEYMANPEARSON LEMMA FOREM ALL THETA1IN THETA0 INFINITY  THUS PHI0 GIVENBY REFUMP IS UMP OUT OF THE CLASS OF ALL TESTS FOR WHICHBEGINDISPLAYMATHETHETA0PHIX LEQ ALPHA ENDDISPLAYMATHWE HAVE THUS ESTABLISHED THAT PHI0 IS UMP FOR H0MC THETA  THETA0 SIMPLE ANDH1MC THETATHETA0 COMPOSITE  TO COMPLETE THE DEVELOPMENT WE NEED TO EXTEND THEDISCUSSION TO PERMIT H0MC  THETALEQ THETA0 COMPOSITE  WE MAYDO THIS BY ESTABLISHING THAT PHI0 SATISFIES THE CONDITION GIVEN BYREFCLASS  FIX NUPRIME BY REFTHRESH FOR THE GIVENALPHA  NOW EXAMINE BEGINALIGNEDETHETAPHI0X    PTHETAX  NUPRIME10PT    INTNUPRIMEINFINITY  FRAC1SQRT2PIEXPXTHETA22 DX  QNUTHETA0 ENDALIGNEDAND NOTE THAT THIS QUANTITY IS AN INCREASING FUNCTION OF THETANUPRIME BEING FIXED  HENCE BEGINDISPLAYMATHETHETAPHI0X  ETHETA0PHI0X LEQ ALPHA QUADFORALLTHETALEQTHETA0ENDDISPLAYMATHAND CONSEQUENTLYBEGINDISPLAYMATHSUPTHETAIN INFINITY THETA0ETHETAPHI0X LEQALPHAENDDISPLAYMATHHENCE PHI0 IS UNIFORMLY BEST OUT OF ALL TESTS SATISFYINGREFCLASS IE IT IS UMPENDEXAMPLESUMMARIZING WE HAVE ESTABLISHED THAT THERE DOES INDEED EXIST AUNIFORMLY MOST POWERFUL TEST FOR TESTING THE HYPOTHESISH0MC  THETALEQ THETA0 AGAINST THE ALTERNATIVES H1MC THETA THETA0 FOR ANY THETA0 WHERE THETA0 IS THE MEAN OF A NORMALRANDOM VARIABLE X WITH KNOWN VARIANCE  SUCH A TEST IS SAID TO BEEM ONESIDED AND HAS VERY SIMPLEFORM REJECT H0 IF X  NUPRIME AND ACCEPT H0 IF XLEQNUPRIME WHERE NUPRIME IS CHOSEN TO MAKE THE SIZE OF THE TESTEQUAL TO ALPHA  WE NOW TURN ATTENTION TO THE ISSUE OF DETERMINING WHAT CONDITIONS ONTHE DISTRIBUTION ARE SUFFICIENT TO GUARANTEE THE EXISTENCE OF A UMPBEGINDEFINITION  A REAL PARAMETER FAMILY OF DISTRIBUTIONS IS SAID TO HAVE BF    MONOTONE LIKELIHOOD RATIO INDEXMONOTONE LIKELIHOOD RATIO IF  DENSITIES OR PROBABILITY MASS FUNCTIONS FXTHETA EXIST  SUCH THAT WHENEVER THETA1  THETA2 THE LIKELIHOOD RATIOBEGINDISPLAYMATHELLX  FRACFXTHETA2FXTHETA1ENDDISPLAYMATHIS A NONDECREASING FUNCTION OF X IN THE SET OF ITS EXISTENCE THAT ISFOR X IN THE SET OF POINTS FOR WHICH AT LEAST ONE OFFXTHETA1 AND FXTHETA2 IS POSITIVE  IF FXTHETA1  0 AND FXTHETA2  0 THE LIKELIHOODRATIO IS DEFINED AS INFINITY  ENDDEFINITIONTHUS IF THE DISTRIBUTION HAS MONOTONE LIKELIHOOD RATIO THE LARGERX THE MORE LIKELY THE ALTERNATIVE H1 IS TO BE TRUEBEGINTHEOREM LABELTHMKARLRUB KARLIN AND RUBIN IF THE  DISTRIBUTION OF X HAS MONOTONE   LIKELIHOOD RATIO THEN ANY TEST OF THE FORMBEGINEQUATIONLABELMONOTONEPHIX  BEGINCASES1  TEXTIF   X  X010PTGAMMA  TEXTIF   X  X010PT0  TEXTIF   X  X0ENDCASESENDEQUATIONHAS NONDECREASING POWER  ANY TEST OF THE FORM REFMONOTONE IS UMPOF ITS SIZE FOR TESTING H0MC THETALEQ THETA0 AGAINSTH1MC THETA THETA0 FOR ANY THETA0INTHETA PROVIDED ITS SIZEIS NOT ZERO  FOR EVERY 0  ALPHALEQ 1 AND EVERYTHETA0INTHETA THERE EXIST NUMBERS INFINITY  X0 INFINITY AND 0 LEQ GAMMA LEQ 1 SUCH THAT THE TESTREFMONOTONE IS UMP OF SIZE ALPHA FOR TESTINGH0MC THETALEQ THETA0 AGAINST H1MC THETA THETA0ENDTHEOREMBEGINPROOFLET THETA1 AND THETA2 BE ANYPOINTS OF THETA WITH THETA1  THETA2  BY THE NEYMANPEARSONLEMMA ANY TEST OF THE FORMBEGINEQUATIONLABELTHRESHHOLDPHIX  BEGINCASES1  TEXTIF   FXXTHETA2  NU FXXTHETA110PTGAMMA  TEXTIF   FXXTHETA2  NU FXXTHETA110PT0  TEXTIF    FXXTHETA2  NU FXXTHETA1ENDCASESENDEQUATIONFOR 0LEQ NU  INFINITY IS BEST OF ITS SIZE FOR TESTINGTHETATHETA1 AGAINST THETATHETA2  BECAUSE THE DISTRIBUTIONHAS MONOTONE LIKELIHOOD RATIO ANY TEST OF THE FORM REFMONOTONEIS ALSO OF THE FORM REFTHRESHHOLD  TO SEE THIS NOTE THAT IFXPRIME  X0 THEN ELLXPRIMELEQ ELLX0 FOR ANY NU IN THE RANGE OF ELL THERE EXISTS A X0SUCH THAT IF ELLX  NU THEN X  X0  THUS REFMONOTONE ISBEST OF SIZE ALPHA 0 FOR TESTING THETA THETA1 AGAINSTTHETA  THETA2THE REMAINDER OF THE PROOF IS ESSENTIALLY THESAME AS THE PROOF FOR THE NORMAL DISTRIBUTION AND WILL BE OMITTEDENDPROOFBEGINEXAMPLETHE ONEPARAMETER EXPONENTIAL FAMILY OF DISTRIBUTIONS WITH DENSITY ORPROBABILITY MASS FUNCTIONBEGINDISPLAYMATHFXBF THETA  CTHETAAXBFEXPPITHETATXBFENDDISPLAYMATHHAS A MONOTONE LIKELIHOOD RATIO PROVIDED THAT BOTH PI AND T ARE NONDECREASING  TO SEE THISSIMPLY WRITE WITH THETA1  THETA2BEGINDISPLAYMATHFRACFXBFTHETA2FXBFTHETA1 FRACCTHETA2CTHETA1 EXPLEFTPITHETA2 PITHETA1TXBFRIGHTENDDISPLAYMATHWHICH IS NONDECREASING IN XENDEXAMPLEBEGINEXERCISESITEM CITEBARKAT1991  FOR THE HYPOTHESIS TESTING PROBLEM BEGINALIGNEDH0MC Y SIM UC02 H1MC FYYH0  EY Y  0ENDALIGNEDBEGINENUMERATEITEM SET UP THE LIKELIHOOD RATIO TEST AND DETERMINE THE DECISION  REGIONS AS A FUNCTION OF THE THRESHOLDITEM FIND THE MINIMUM PROBABILITY OF ERROR WHEN I P0  12  II  P0  23  III P013ENDENUMERATEITEM FOR THE TEST BEGINALIGNEDH0MC Y SIM UC01 H1MC Y SIM UC02ENDALIGNEDBEGINENUMERATEITEM SET UP THE LIKELIHOOD RATIO TEST AND DETERMINE THE DECISION  REGIONSITEM FIND PF AND PDENDENUMERATEITEM CITEBARKAT1991 FOR THE TEST BEGINALIGNEDH0MC  Y  N H1MC  Y  SNENDALIGNEDWHERE S SIM UC11 AND N SIM UC22 AND S AND N ARESTATISTICALLY INDEPENDENTBEGINENUMERATEITEM SET UP THE LIKELIHOOD RATIO TEST AND DETERMINE THE DECISION  REGIONS WHEN I NU  14  II NU  1 III NU  2ITEM FIND PFA AND PD FOR EACH OF THE VALUES OF NUITEM SKETCH THE ROCENDENUMERATEITEM SHOW THAT THE MEANS AND VARIANCES IN REFEQGT0 AND  REFEQGT1 ARE CORRECTITEM SHOW THAT THE MEAN AND VARIANCE IN REFEQGM2 AND  REFEQGV2 ARE CORRECTITEM SHOW THAT THE INVERSE FOURIER TRANSFORM OF THE CHARACTERISTIC  FUNCTION IN REFEQCHI2CHAR IS REFEQCHI2PDFITEM BY INTEGRATION BY PARTS SHOW THAT THE GAMMA FUNCTION  GAMMAX  INT0INFTY TX1 ETDTSATISFIES GAMMAX1  X GAMMAX FOR X0ITEM FINALLY INTEGRATE OUT U TO DERIVE THE DENSITY REFEQTDISTITEM CONSIDER TWO HYPOTHESES BEGINALIGNEDH0MC  FRR   FRAC12 EXPRH1MC  FRR    FRAC1SQRT2 PI EXPFRAC12 R2ENDALIGNEDBEGINENUMERATEITEM PLOT THE DENSITY FUNCTIONSITEM FIND THE LIKELIHOOD RATIOITEM COMPUTE THE DECISION REGIONS AS A FUNCTION OF NUITEM DETERMINE EXPRESSIONS FOR ALPHA AND BETAENDENUMERATEITEM THE RANDOM VARIABLE X IS NORMAL ZEROMEAN AND UNIT VARIANCE  IT IS PASSED THROUGH ONE OF TWO NONLINEAR TRANSFORMATIONS BEGINALIGNEDH0MC Y    X2 H1MC Y    X3ENDALIGNEDFIND THE LRTITEM DETECTION OF POISSON RANDOM VARIABLES  LET XI  I12LDOTS N BE INDEPENDENT POISSON RANDOM VARIABLES WITH RATE  LAMBDA  WE WILL SAY THAT XI SIM PLAMBDA  BEGINENUMERATE  ITEM SHOW THAT X  SUMI1N XI IS PNLAMBDA  ITEM FIND THE LIKELIHOOD RATIO OF OF A TEST FOR LAMBDA1     LAMBDA0  ITEM DETERMINE HOW TO FIND THE THRESHOLD FOR A TEST OF SIZE    ALPHA IN A NEYMANPEARSON TEST OF H0MC LAMBDA LEQ  2 VERSUS    H1MC LAMBDA2  THE INTERMEDIATE TEST GAMMA WILL NEED TO    BE USED  ENDENUMERATEITEM IN AN OPTICAL COMMUNICATION CHANNEL BITS ARE REPRESENTED BY  ONOFF PULSES  LET XT BE NUMBER OF PHOTONS RECEIVED  ACCORDING TO THE MODEL PX  K  ELAMBDA TLAMBDA TKKIN THE ABSENCE OF PULSES THE RECEIVER DETECTS PULSES DUE TOBACKGROUND RADIATION  THE DETECTION PROBLEM IS H0MC LAMBDA LAMBDA0 VERSUS H1MC LAMBDA  LAMBDA1 WHERE LAMBDA1 LAMBDA0  DETERMINE A NEYMANPEARSON TEST FOR THE DETECTORDETERMINE THE SIZE AND POWER OF THE DETECTORITEM CITESCHARFL1991 IN THE OPTICAL COMMUNICATION CHANNEL NOW SUPPOSE  THAT A LEAKY DETECTOR IS USED  THE PHOTONS ARRIVE AT THE RECEIVER  ACCORDING THE THE PROBABILITY LAW PXK  ELAMBDA TLAMBDA  TKK BUT EACH PHOTON ARRIVING AT THE RECEIVER IS DETECTED WITH  PROBABILITY P  LET THE OUTPUT OF THE DETECTOR BE YT  BEGINENUMERATE  ITEM SHOW THAT PYKXN IS BCPN  ITEM SHOW THAT PYT  M IS PP LAMBDA T  ITEM FIND THE NEYMANPEARSON DETECTOR FOR THIS DETECTOR  ITEM COMPUTE AND PLOT THE ROC  ENDENUMERATEITEM FSK  A SIGNAL VECTOR SBFI  SI0SI1LDOTSSIN1TIS OBTAINED BY SIJ  COS 2PI FI JN FOR AN INTEGER FREQUENCYFI  THE RECEIVED SIGNAL IS YBF  SBF  NBF WHERE NBF SIMNC0SIGMA2 I  BEGINENUMERATEITEM DETERMINE AN OPTIMAL NEYMANPEARSON DETECTORITEM DRAW A BLOCK DIAGRAM OF THE DETECTOR STRUCTUREENDENUMERATEITEM PSK  SUPPOSE THAT WE HAVE THE DETECTION PROBLEM BEGINALIGNEDH0MC SI  COS 2PI FIN QQUAD I012LDOTSN1 H1MC SI  COS 2PI FIN  THETA QQUAD I012LDOTSN1ENDALIGNEDTHIS CAN BE USED FOR TRANSMISSION OF INFORMATION USING PHASEINFORMATION  BEGINENUMERATEITEM DETERMINE A NEYMANPEARSON DETECTOR AND DRAW THE BLOCK DIAGRAMITEM DETERMINE A MEANS OF FINDING THE THRESHOLD TO OBTAIN A GIVEN  SIZE ALPHAENDENUMERATEITEM SHOW THAT THE NEGATIVE EXPONENTIAL DISTRIBUTION WITH DENSITYBEGINDISPLAYMATHFXXTHETA  EXTHETAITHETA INFTYXENDDISPLAYMATHHAS A MONOTONE LIKELIHOOD RATIO ITEM FOR THE GAMMA FUNCTION DEFINED IN BOX REFBOXGAMMA SHOW  THAT GAMMAX1  XGAMMAXITEM LABELEXLLT2 FOR THE DETECTION PROBLEM BEGINALIGNEDH0MC XBF SIM NCZEROBFR0 H1MC XBF SIM NCZEROBFR1ENDALIGNEDDEVELOP A LIKELIHOOD RATIO TEST  SIMPLIFY AS MUCH AS POSSIBLEENDEXERCISESINPUTDETESTDIRBAYESDEC WHICH INCLUDES   INVARIANT   CONTDECSETEXSECTREFSECNPBEGINEXERCISESITEM FOR THE TEST H0MC Y SIM UC01 QQUADQQUAD H1MC Y SIM UC02BEGINENUMERATEITEM SET UP THE LIKELIHOOD RATIO TEST AND DETERMINE THE DECISION  REGIONS AS A FUNCTION OF THE THRESHOLDITEM FIND PF AND PDENDENUMERATEITEM CITEBARKAT1991 FOR THE TEST BEGINALIGNEDH0MC  Y  N H1MC  Y  SNENDALIGNEDWHERE S SIM UC11 AND N SIM UC22 AND S AND N ARESTATISTICALLY INDEPENDENTBEGINENUMERATEITEM SET UP THE LIKELIHOOD RATIO TEST AND DETERMINE THE DECISION  REGIONS WHEN I NU  14  II NU  2 III NU  1  CLEARLY INDICATE WHAT IS HAPPENING WHEN NU  1ITEM FIND PFA AND PD FOR EACH OF THESE VALUES OF NUITEM SKETCH THE ROCENDENUMERATEITEM SHOW THAT THE MEANS AND VARIANCES IN REFEQGT0 AND  REFEQGT1 ARE CORRECTITEM SHOW THAT THE MEAN AND VARIANCE IN REFEQGM2 AND  REFEQGV2 ARE CORRECTITEM LABELEXCHI2PDF SHOW THAT THE INVERSE FOURIER TRANSFORM OF THE CHARACTERISTIC  FUNCTION IN REFEQCHI2CHAR IS REFEQCHI2PDF  HINT USE  THE FACT THAT INT0INFTY XNU1 EMU XDX  GAMMANUMUNUITEM CITEBARKAT1991 CONSIDER TWO HYPOTHESES BEGINALIGNEDH0MC   FRRH0   FRAC1SQRT2 PI EXPFRAC12 R2 H1MC   FRRH1    FRAC12 EXPR ENDALIGNEDBEGINENUMERATEITEM PLOT THE DENSITY FUNCTIONSITEM FIND THE LIKELIHOOD RATIO AND THE LOGLIKELIHOOD RATIO  PLOT  THE LOGLIKELIHOOD RATIO AS A FUNCTION OF R FOR NU  SQRTPI2ITEM COMPUTE THE DECISION REGIONS AS A FUNCTION OF NUITEM DETERMINE EXPRESSIONS FOR ALPHA AND BETAENDENUMERATEITEM THE RANDOM VARIABLE X IS NORMAL ZEROMEAN AND UNIT VARIANCE  IT IS PASSED THROUGH ONE OF TWO NONLINEAR TRANSFORMATIONS BEGINALIGNEDH0MC Y    X2 H1MC Y    X3ENDALIGNEDFIND THE LRTITEM BF POISSON CHARACTERISTIC FUNCTION  LET X SIM PLAMBDA  THAT IS X IS POISSONDISTRIBUTED WITH PARAMETER LAMBDA  THEN PXK  FRACLAMBDAKK ELAMBDA QQUAD K GEQ 0SHOW THAT THE CHARACTERISTIC FUNCTION OF X IS PHIXOMEGA  SUMK0INFTY PXK EJOMEGA K ELAMBDAEJOMEGA  1ITEM BF DETECTION OF POISSON RANDOM VARIABLES   LET XI  I12LDOTS N  BE INDEPENDENT POISSON RANDOM VARIABLES WITH RATE  LAMBDA  WE WILL SAY THAT XI SIM PLAMBDA  BEGINENUMERATE  ITEM SHOW THAT X  SUMI1N XI IS PNLAMBDA  ITEM FIND THE LIKELIHOOD RATIO OF OF A TEST FOR LAMBDA1     LAMBDA0  ITEM DETERMINE HOW TO FIND THE THRESHOLD FOR A TEST OF SIZE    ALPHA001 IN A NEYMANPEARSON TEST OF H0MC LAMBDA  2 VERSUS    H1MC LAMBDA4   THE INTERMEDIATE TEST GAMMA WILL NEED TO    BE USED  ENDENUMERATE ITEM IN AN OPTICAL COMMUNICATION CHANNEL BITS ARE REPRESENTED BY   ONOFF PULSES  LET XT BE NUMBER OF PHOTONS RECEIVED   ACCORDING TO THE MODEL  PX  K  ELAMBDA TLAMBDA TKK  IN THE ABSENCE OF PULSES THE RECEIVER DETECTS PULSES DUE TO BACKGROUND RADIATION  THE DETECTION PROBLEM IS H0MC LAMBDA  LAMBDA0 VERSUS H1MC LAMBDA  LAMBDA1 WHERE LAMBDA1  LAMBDA0  DETERMINE A NEYMANPEARSON TEST FOR THE DETECTOR DETERMINE THE SIZE AND POWER OF THE DETECTORITEM CITESCHARFL1991 IN AN OPTICAL COMMUNICATION CHANNEL USING  ONOFF SIGNALLING SUPPOSE THAT A LEAKY DETECTOR IS USED  WHEN A  PULSE IS SENT PHOTONS ARRIVE AT THE DETECTOR AT A RATE LAMBDA1  AND WHEN NO PULSE IS SENT ONLY BACKGROUND PHOTONS ARRIVE AT A RATE  LAMBDA0  LAMBDA1  IN THE LEAKY DETECTOR PHOTONS ARRIVE AT  THE RECEIVER ACCORDING THE THE PROBABILITY LAW PXTK   ELAMBDA TLAMBDA TKK SIM PLAMBDA T BUT EACH PHOTON  IS DETECTED WITH PROBABILITY P  LET THE OUTPUT OF THE DETECTOR BE  YT  BEGINENUMERATE  ITEM SHOW THAT PYTKXTN IS BCNP  SEE EXERCISE REFEXBINOM  ITEM SHOW THAT PYT  K IS PP LAMBDA T  ITEM FIND THE NEYMANPEARSON DETECTOR FOR THIS DETECTOR  ITEM COMPUTE AND PLOT THE ROC WHEN LAMBDA1  2 LAMBDA0     1 AND P099  ENDENUMERATEITEM COHERENT FSK  A SIGNAL VECTOR SBFI  SI0SI1LDOTSSIN1TIS OBTAINED BY SIJ  COS 2PI FI JN J01LDOTSN1 FOR ANINTEGER FREQUENCY FI I01  THE RECEIVED SIGNAL IS YBF  SBF NBF WHERE NBF SIM NC0SIGMA2 IBEGINENUMERATEITEM DETERMINE AN OPTIMAL NEYMANPEARSON DETECTORITEM DRAW A BLOCK DIAGRAM OF THE DETECTOR STRUCTUREENDENUMERATE ITEM PSK  SUPPOSE THAT WE HAVE THE DETECTION PROBLEM  BEGINALIGNED H0MC SI  COS 2PI FIN QQUAD I012LDOTSN1  H1MC SI  COS 2PI FIN  THETA QQUAD I012LDOTSN1 ENDALIGNED  THIS CAN BE USED FOR TRANSMISSION OF INFORMATION USING PHASE INFORMATION   BEGINENUMERATE ITEM DETERMINE A NEYMANPEARSON DETECTOR AND DRAW THE BLOCK DIAGRAM ITEM DETERMINE A MEANS OF FINDING THE THRESHOLD TO OBTAIN A GIVEN   SIZE ALPHA ENDENUMERATEITEM BY INTEGRATION BY PARTS SHOW THAT THE GAMMA FUNCTION  INTRODUCED IN BOX REFBOXGAMMA AS GAMMAX  INT0INFTY TX1 ETDTSATISFIES GAMMAX1  X GAMMAX FOR X0ITEM LABELEXLLT2 FOR THE DETECTION PROBLEM BEGINALIGNEDH0MC XBF SIM NCZEROBFR0 H1MC XBF SIM NCZEROBFR1ENDALIGNEDDEVELOP A LIKELIHOOD RATIO TEST  EXPRESS THE TEST IN TERMS OF THESIGNALTONOISE RATIO S  R012 R1 R012SIMPLIFY AS MUCH AS POSSIBLEITEM LABELEXQ BF BOUNDS AND APPROXIMATIONS TO THE Q FUNCTION  BEGINENUMERATE  ITEM SHOW THAT  SQRT2PI QX  FRAC1X EX22  INTXINFTYFRAC1Y2 EY22 DY QQUAD X0HINT INTEGRATE BY PARTSITEM SHOW THAT 0  INTXINFTY FRAC1Y2EY22DY  FRAC1X3EX22ITEM HENCE CONCLUDE THAT FRAC1SQRT2PIX EX2211X2  QX FRAC1SQRT2PIX EX22 QQUAD X0ITEM PLOT THESE LOWER AND UPPER BOUNDS ON A PLOT WITH QX USE A  LOG SCALEITEM ANOTHER USEFUL BOUND IS QX LEQ FRAC12EX22  DERIVE THIS BOUND  HINT IDENTIFY QALPHA2 AS THE  PROBABILITY THAT THE ZEROMEAN UNITGAUSSIAN RANDOM VARIABLES LIE IN  THE SHADED REGION SHOWN ON THE LEFT IN FIGURE REFFIGQFBOUND THE REGION  ALPHAINFTYTIMES ALPHAINFTY THIS PROBABILITY IS  EXCEEDED BY THE PROBABILITY THAT XY LIES IN THE SHADED REGION SHOWN ON THE RIGHT EXTENDED OUT TO INFTYEVALUATE THIS PROBABILITY  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODE INPUTPICTUREDIRQFUN2      CAPTIONREGIONS FOR BOUNDING THE Q FUNCTION      LABELFIGQFBOUND    ENDCENTER  ENDFIGURE  ENDENUMERATEEXSKIPSETEXSECTREFSECBAYESDEC   14ITEM CITEBARKAT1991  FOR THE HYPOTHESIS TESTING PROBLEM BEGINALIGNEDH0MC FYYH1  EYQUAD Y  0QQUADQQUADH1MC Y SIM UC02 ENDALIGNEDBEGINENUMERATEITEM SET UP THE LIKELIHOOD RATIO TEST AND DETERMINE THE DECISION  REGIONS AS A FUNCTION OF THE THRESHOLDITEM FIND THE MINIMUM PROBABILITY OF ERROR WHEN I P0  12  II  P0  23  III P013ENDENUMERATEITEM CITEWOZENCRAFT ONE OF TWO SIGNALS S01 OR S11 IS  TRANSMITTED OVER THE CHANNEL SHOWN IN FIGURE REFFIGLAP1A  WHERE THE NOISES N1 AND N2 ARE INDEPENDENT LAPLACIAN NOISE  INDEXLAPLACIAN RANDOM VARIABLE INDEXRANDOM VARIABLELAPLACIAN  WITH PDF FNALPHA  FRAC12 EALPHABEGINENUMERATEITEM SHOW THAT THE OPTIMUM DECISION REGIONS FOR EQUALLY LIKELY  MESSAGES ARE AS SHOWN IN FIGURE REFFIGLAP1BITEM DETERMINE THE PROBABILITY OF ERROR FOR THIS DETECTORENDENUMERATEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    SUBFIGURECHANNEL MODELINPUTPICTUREDIRLAP1QQUAD    SUBFIGUREDECISION REGIONSINPUTPICTUREDIRLAP2    CAPTIONCHANNEL WITH LAPLACIAN NOISE AND DECISION REGION    LABELFIGLAP1  ENDCENTERENDFIGUREITEM BF COMPUTING EXERCISE SIGNAL SPACE SIMULATION  IN THIS EXERCISE  YOU SIMULATE SEVERAL DIFFERENT DIGITAL COMMUNICATIONS SIGNAL  CONSTELLATIONS AND THEIR DETECTION  SUPPOSE THAT AN MARY  TRANSMISSION SCHEME IS TO BE SIMULATED WHERE M2K  THE  FOLLOWING IS THE GENERAL ALGORITHM TO ESTIMATE THE PROBABILITY OF  ERROR SMALLBEGINPROGTABSQUAD  QUAD  QUAD  QUAD QUAD  QUAD  QUAD KILLGENERATE K RANDOM BITS MAP THE BITS INTO THE MARY CONSTELLATION TO PRODUCE THESIGNAL SBF  THIS IS ONE SYMBOL GENERATE A GAUSSIAN RANDOM NUMBER NOISE WITH VARIANCE SIGMA2 N02  IN EACH SIGNAL COMPONENT DIRECTION ADD THE NOISE TO THE SIGNAL CONSTELLATION POINT RBF  SBF  NBF PERFORM A DETECTION ON THE RECEIVED SIGNAL RBF MAP THE DETECTED POINT XBFHAT BACK TO BITS COMPARE THE DETECTED BITS WITH THE TRANSMITTED BITS AND COUNT BITS INERRORENDPROGTABSREPEAT THIS UNTIL MANY PREFERABLY AT LEAST 100 BITS IN ERROR HAVEBEEN COUNTED  THE ESTIMATED EM BIT ERROR PROBABILITY IS PB APPROX FRACTEXTNUMBER OF BITS IN ERRORTEXTTOTAL NUMBER OF BITS GENERATEDTHE ESTIMATED EM SYMBOL ERROR PROBABILITY IS PE APPROX FRACTEXTNUMBER OF SYMBOLS IN ERRORTEXTTOTAL NUMBER OF SYMBOLS GENERATEDIN GENERAL PB NEQ PE SINCE A SYMBOL IN ERROR MAY ACTUALLY HAVESEVERAL BITS IN ERRORTHE PROCESS ABOVE SHOULD BE REPEATED FOR VALUES OF SNR EBN0 INTHE RANGE FROM 0 TO 10 DBTHE ASSIGNMENTBEGINENUMERATEITEM PLOT THE THEORETICAL PROBABILITY OF ERROR FOR BPSK DETECTION WITH EQUAL  PROBABILITIES AS A FUNCTION OF SNR IN DB EM VS PB ON A LOG  SCALE  YOUR PLOT SHOULD LOOK LIKE FIGURE REFFIGBPSKITEM BY SIMULATION ESTIMATE THE PROBABILITY OF ERROR FOR BPSK  TRANSMISSION USING THE METHOD OUTLINED ABOVE  PLOT THE RESULTS ON  THE SAME AXES AS THE THEORETICAL PLOT  THEY SHOULD BE VERY  SIMILARITEM PLOT THE THEORETICAL PROBABILITY OF EM SYMBOL ERROR FOR QPSK  SIMULATE USING QPSK AND PLOT THE ESTIMATED SYMBOL ERROR PROBABILITYITEM PLOT THE UPPER BOUND FOR THE PROBABILITY OF 8PSK    SIMULATE USING 8PSK AND PLOT THE ESTIMATED ERROR PROBABILITYITEM REPEAT PARTS A AND B USING UNEQUAL PRIOR PROBABILITIES PMBF0  08 QQUAD PMBF1  02ITEM COMPARE THE THEORETICAL AND EXPERIMENTAL PLOTS AND COMMENTENDENUMERATEEXSKIPSETEXSECTREFSECMARY    15ITEM FOR SOME DISTRIBUTIONS OF MEANS THE PROBABILITY OF  CLASSIFICATION ERROR IS STRAIGHTFORWARD TO COMPUTE  FOR THE SET OF  POINTS REPRESENTING MEANS SHOWN IN FIGURE REFFIGDETPROB  COMPUTE THE PROBABILITY OF ERROR ASSUMING THAT EACH HYPOTHESIS  OCCURS WITH EQUAL PROBABILITY AND THAT THE NOISE IS  NC0SIGMA2I  THESE SETS OF MEANS COULD REPRESENT SIGNAL  CONSTELLATIONS IN A DIGITAL COMMUNICATIONS SETTING  INDEXSIGNAL CONSTELLATION  IN EACH CONSTELLATION THE DISTANCEBETWEEN NEAREST SIGNAL POINTS IS D     ALSO COMPUTE AVERAGE ENERGY ES OF THE SIGNAL CONSTELLATION AS A  FUNCTION OF D  IF THE MEANS ARE AT MBFI THEN THE AVERAGE  ENERGY IS E  FRAC1M SUMI1M MBFI2FOR EXAMPLE FOR THE 4PSK CONSTELLATION E  FRAC144LEFTD22  D22RIGHT  D22FOR EACH CONSTELLATION EXPRESS THE PROBABILITY OF ERROR AS A FUNCTIONOF ES  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODESUBFIGURE4PSKINPUTPICTUREDIRDETPROBA QQUADSUBFIGURE8QAMINPUTPICTUREDIRDETPROBB SUBFIGURE16QAMINPUTPICTUREDIRDETPROBC      CAPTIONSOME SIGNAL CONSTELLATIONS      LABELFIGDETPROB    ENDCENTER  ENDFIGUREITEM LET M  2K WHERE K IS AN EVEN NUMBER  DETERMINE THE  PROBABILITY OF ERROR FOR A SIGNAL CONSTELLATION WITH M POINTS  ARRANGED IN A SQUARE CENTERED AT THE ORIGIN WITH MINIMUM DISTANCE  BETWEEN POINTS EQUAL TO D AND NOISE VARIANCE SIGMA2  ASSUME  THE NOISE IS GAUSSIAN  EXPRESS THIS AS A FUNCTION OF ES THE  AVERAGE SIGNAL ENERGY FOR THE CONSTELLATION  INDEXQUADRATUREAMPLITUDE MODULATION QAM INDEXPHASESHIFT    KEYING PSKEXSKIPSETEXSECTREFSECUB   17  UNION BOUND  ITEM IN AN MDIMENSIONAL ORTHOGONAL DETECTION PROBLEM THERE ARE    M HYPOTHESES HIMC XBFSIM NCMBFISIGMA2I WHERE MBFI PERP MBFJ QQUAD I NEQ JASSUME THAT ES  MBFI2 FOR I12LDOTSMLET M  2K AND ASSUME THAT THESE M ORTHOGONAL SIGNALS ARE USEDTO SEND K BITS OF INFORMATIONBEGINENUMERATEITEM SHOW THAT THE MINIMUM DISTANCE BETWEEN SIGNALS IS D   SQRT2ES  ALSO SHOW THAT EB THE ENERGY PER BIT IS EB   ESKITEM BY THE UNION BOUND SHOW THAT THE PROBABILITY OF SYMBOL ERROR IS  BOUNDED BY PEC LEQ M1 QD2SIGMAITEM USING THE UPPER BOUND ON THE Q FUNCTION QX LEQ FRAC12  EX22 SHOW THAT THE ERROR APPROACHES ZERO AS  KRIGHTARROWINFTY PROVIDED THAT EBSIGMA2  4 LN 2ENDENUMERATEITEM FOR POINTS IN THE SIGNAL CONSTELLATION SHOWN IN FIGURE  REFFIGUCONST WHERE THE BASIS FUNCTIONS ARE ORTHONORMAL  DETERMINE AN UPPER BOUND ON THE PROBABILITY OF ERROR USING THE UNION  BOUND  ASSUME THAT THE NOISE IS AWGN WITH VARIANCE SIGMA2   01  EXPRESS YOUR ANSWER IN TERMS OF THE Q FUNCTION  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODE      INPUTPICTUREDIRUCONST      CAPTIONSIGNAL CONSTELLATION WITH THREE POINTS      LABELFIGUCONST    ENDCENTER  ENDFIGUREEXSKIPSETEXSECTREFSECINVTESTITEM CITESCHARFL1991 SUPPOSE THAT A SIGNAL IS OF THE FORM SBF  H THETABFWHERE H IS A KNOWN MATSIZEMP MATRIX BUT THETABF IS NOTKNOWN  THAT IS THE SIGNAL IS KNOWN TO LIE IN RANGEH BUT THEPARTICULAR POINT IN THAT SPACE IS NOT KNOWN  LET XBF  MU HTHETABF  N WHERE N SIM NCZEROBFSIGMA2 I  IT IS DESIREDTO DISTINGUISH BETWEEN H0MC MU0 SIGNAL ABSENT EM VSH1MC MU0 SIGNAL PRESENT  HOWEVER IT IS NOT POSSIBLE TOOBSERVE XBF DIRECTLY  INSTEAD WE OBSERVE THE OUTPUT YBF OF ACHANNEL WHICH INTRODUCES SOME BIAS VBF PERP RANGEH AND ALSOROTATES XBF IN RANGEH  LET Q INDICATE THE ROTATION INRANGEHBEGINENUMERATEITEM SHOW THAT Q  UH QTILDE UH   PHPERP WHERE  PHPERP  IS A PROJECTION ONTO RANGEHPERP ANDPH  UHUHT IS A PROJECTION ONTO RANGEH AND QTILDE IS ANORTHOGONAL MATRIXITEM SHOW THAT THE ROTATION OF MU H THETABF IS MU H THETABF  FOR SOME THETABFITEM SHOW THAT THE STATISTIC Z  YBFT PH YBFIS INVARIANT TO THE OFFSET VBF AND ANY ROTATION QITEM SHOW THAT UNDER H0 ZSIGMA2 IS DISTRIBUTED AS  CHI2MENDENUMERATEITEM CITESCHARFL1991 LET XBF SIM NCMU H THETABF  SIGMA2I WHERE H IS A KNOWN MATSIZEMP MATRIX BUT  SIGMA2 IS NOT KNOWN  ASSUME THAT THE SIGNAL IS BIASED BY A  VECTOR VBF PERP RANGEH AND ROTATED IN RANGEH TO PRODUCE  THE MEASUREMENT YBF  BEGINENUMERATE  ITEM SHOW THAT THE STATISTIC F  FRACYBFT PH YBFSIGMA2 P YBFTIPHYBFSIGMA2MPIS INVARIANT TO VBF AND Q AND INDEPENDENT OF SIGMA2ITEM EXPLAIN WHY F IS THE RATIO OF INDEPENDENT CHI2 RANDOM  VARIABLESITEM THE DISTRIBUTION OF F IS CALLED THE  FDISTRIBUTION  IT IS KNOWN TO HAVE A MONOTONE LIKELIHOOD RATIO  BASED ON THIS FACT WRITE DOWN A UNIFORMLY MOST POWERFUL TEST  ENDENUMERATE ITEM SHOW THAT THE NEGATIVE EXPONENTIAL DISTRIBUTION WITH DENSITY BEGINDISPLAYMATH FXXTHETA  EXTHETAITHETA INFTYX ENDDISPLAYMATH HAS A MONOTONE LIKELIHOOD RATIO ITEM LABELEXTDIST BF THE T DISTRIBUTION LET T   ZSQRTYR WHERE Z SIM NC01 AND Y SIM CHI2R  LET  TU  WZY BE AN INVERTIBLE TRANSFORMATION WHERE T  ZSQRTYRQQUAD U  YBEGINENUMERATEITEM SHOW THAT THE JACOBIAN OF THE TRANSFORMATION IS J  DET BEGINBMATRIXPARTIALDZT  PARTIALDZU   EXMATSPPARTIALDYT  PARTIALDYU ENDBMATRIX  SQRTURITEM HENCE SHOW THAT THE JOINT DENSITY FTUTU IS FTUTU  SQRTURFRAC1SQRT2PIGAMMAR22R2UR21 EU2ITEM FINALLY INTEGRATE OUT U TO DERIVE THE DENSITY  REFEQTDIST  USE INT0INFTY XNU1 EMU XDX   GAMMANUMUN ENDENUMERATEEXSKIPSETEXSECTREFSECCONTTIMEDETECTITEM LABELEXCD1 SHOW THAT LIMNRIGHTARROW INFTY SUMI1N  XI SJI2  INT0T XT  SJT2DTITEM LABELEXCBF OR THE BINARY DETECTOR IN GAUSSIAN NOISE OF EXAMPLE  REFEXMCBG VERIFY THAT PFA  PMD  QD2SIGMAITEM LABELEXMTH IN THE PROOF OF THEOREM REFTHMKARHUNEN WE USED THE FACT THAT ESUMI1N SUMJ1N ZI ZJ PSIITPSIJT  SUMI1NLAMBDAI PSII2TSHOW THAT THIS IS TRUEITEM SHOW THAT REFEQRNPHI IS TRUEITEM DRAW THE BLOCK DIAGRAM FOR A NONCOHERENT DETECTOR FOR THEPROBLEM BEGINALIGNEDH0MC XT  S0T  NT H1MC XT  S1T  NTENDALIGNEDWHERE SIT  A SINOMEGAI T  THETA QQUAD 0 LEQ T LEQ TAND WHERE THETA IS UNIFORMLY DISTRIBUTED AS UC02PI ANDNT IS GAUSSIAN WHITE NOISEEXSKIPSETEXSECTREFSECMINIMAXBAYESITEM FOR THE BINARY CHANNEL REPRESENTED BY    BEGINCENTER      INPUTPICTUREDIRBSCLAMBDA    ENDCENTER  BEGINENUMERATE  ITEM DETERMINE THE LIKELIHOOD RATIO TEST  ITEM DETERMINE THE THRESHOLD NU TO OBTAIN A TEST OF SIZE    ALPHA WHEN LAMBDA0  LAMBDA1  LAMBDA AS A FUNCTION OF    LAMBDA  ITEM IF LAMBDA0LAMBDA1 LAMBDA DETERMINE AND PLOT THE ROC    FOR A NEYMANPEARSON TEST ON THE CHANNEL FOR LAMBDA18    LAMBDA14 LAMBDA38 AND LAMBDA12  ITEM DETERMINE THE BAYES DECISION RULE WHEN THE PRIOR PROBABILITIES    P0  PTHETA0 AND P1  PTHETA1 ARE EQUAL AND THE    COSTS ARE UNIFORM  ITEM PLOT THE BAYES ENVELOPE FUNCTION WHEN LAMBDA0 01 AND    LAMBDA1  02  ENDENUMERATEITEM CONSIDER TWO BOXES A AND B EACH OF WHICH CONTAINS BOTH RED BALLSAND GREEN BALLS  IT IS KNOWN THAT IN ONE OF THE BOXES FRAC12OF THE BALLS ARE RED AND FRAC12 ARE GREEN AND THAT IN THEOTHER BOX FRAC14 OF THE BALLS ARE RED AND FRAC34 AREGREEN  LET THE BOX IN WHICH FRAC12 ARE RED BE DENOTED BOX WAND SUPPOSE PW  A  XI AND PW  B  1XI  SUPPOSE YOU MAYSELECT ONE BALL AT RANDOM FROM EITHER BOX A OR BOX B AND THATAFTER OBSERVING ITS COLOR MUST DECIDE WHETHER WA OR WB  PROVETHAT IF FRAC12  XI  FRAC23 THEN IN ORDER TO MAXIMIZETHE PROBABILITY OF MAKING A CORRECT DECISION HE SHOULD SELECT THEBALL FROM BOX B  PROVE ALSO THAT IF FRAC23LEQ XI LEQ 1 THEN ITDOES NOT MATTER FROM WHICH BOX THE BALL IS SELECTEDITEM A WILDCAT OILMAN MUST DECIDE HOW TO FINANCE THE DRILLING OF A WELLIT COSTS 100000 TO DRILL THE WELL  THE OILMAN HAS AVAILABLE THREEOPTIONS BEGINDESCRIPTIONITEMH0 FINANCE THE DRILLING HIMSELF AND RETAIN ALL THE PROFITSITEMH1 ACCEPT 70000 FROM INVESTORS IN RETURN FOR PAYING THEM  50 OF THE OIL PROFITSITEMH2 ACCEPT 120000 FROM INVESTORS IN RETURN FOR PAYING THEM  90 OF THE OIL PROFITSENDDESCRIPTIONTHE OIL PROFITS WILL BE 3THETA WHERE THETA IS THE NUMBER OFBARRELS OF OIL IN THE WELL  FROM PAST DATA IT IS BELIEVED THAT THETA  0 WITH PROBABILITY09 AND THE DENSITY FOR THETA  0 ISBEGINDISPLAYMATHGVARTHETA  FRAC01300000EVARTHETA300000I0    INFTYVARTHETAENDDISPLAYMATHA SEISMIC TEST IS PERFORMED TO DETERMINE THE LIKELIHOOD OF OIL IN THEGIVEN AREA  THE TEST TELLS WHICH TYPE OF GEOLOGICAL STRUCTURE X1X2 OR X3 IS PRESENT  IT IS KNOWN THAT THE PROBABILITIES OFTHE XI GIVEN THETA AREBEGINALIGNEDFXTHETAX1VARTHETA    08EVARTHETA100000FXTHETAX2VARTHETA    02FXTHETAX3VARTHETA    081 EVARTHETA100000ENDALIGNEDBEGINITEMIZEITEM FOR MONETARY LOSS WHAT IS THE BAYES ACTION IF X  X1 IS  OBSERVEDITEM FOR MONETARY LOSS WHAT IS THE BAYES ACTION IF X  X2 IS  OBSERVEDITEM FOR MONETARY LOSS WHAT IS THE BAYES ACTION IF X  X3 IS  OBSERVEDENDITEMIZEITEM A DEVICE HAS BEEN CREATED WHICH CAN SUPPOSEDLY CLASSIFY BLOOD AS TYPEA B AB OR O  THE DEVICE MEASURES A QUANTITY X WHICH HASDENSITYBEGINDISPLAYMATHFXTHETAXVARTHETA  EX  VARTHETAIVARTHETA INFTYXENDDISPLAYMATHIF 0  THETA  1 THE BLOOD IS OF TYPE AB IF 1  THETA  2 THEBLOOD IS OF TYPE A IF 2  THETA  3 THE BLOOD IS OF TYPE B ANDIF THETA  3 THE BLOOD IS OF TYPE O  IN THE POPULATION AS A WHOLETHETA IS DISTRIBUTED ACCORDING TO THE DENSITYBEGINDISPLAYMATHFTHETAVARTHETA  EVARTHETAI0 INFTYVARTHETAENDDISPLAYMATHTHE LOSS IN MISCLASSIFYING THE BLOOD IS GIVEN BY THE FOLLOWING TABLEVSPACE2INDEFLIMMSHATADEFSPANNHATBDEFRANKHATCDEFKERHATDDEFCOVHATEDEFVARHATFDEFTRHATGDEFDIAGHATHBEGINCENTERBEGINTABULARCCCCCCMULTICOLUMN6CCLASSIFICATION   AB  A  B  OCLINE26  AB  0  1  1  2 CLINE26TRUE  A  1  0  2  2CLINE26TYPE  B  1  2  0  2CLINE26  O  3  3  3  0CLINE26ENDTABULARENDCENTERIF X  4 IS OBSERVED WHAT IS THE BAYES ACTIONITEM FOR THE BINARY CHANNEL TAKE LAMBDA0  13 AND LAMBDA1   14  DETERMINE  BEGINENUMERATE  ITEM THE RISK SET  ITEM THE MINIMAX BAYES RISK  ITEM THE OPTIMUM DECISION RULE  ITEM THE LEAST FAVORABLE PRIOR  ENDENUMERATE  ITEM LABELEXDETECTCHANGE1 INDEXDETECTION OF CHANGE  INDEXCHANGE DETECTION IN THESE LAST TWO EXERCISES WE INTRODUCE  BRIEFLY SOME OTHER TOPICS IN DETECTION THEORY  THIS PROBLEM DEALS  WITH BF DETECTION OF CHANGE  SUPPOSE THAT A SIGNAL CHANGES ITS  MEAN AT SOME UNKNOWN TIME N0 AND THE PROBLEM IS TO DETECT THE  CHANGE  WE SET UP THE FOLLOWING HYPOTHESIS TEST BEGINALIGNEDH0MC  XI SIM NCM0SIGMA2QQUAD I12LDOTSN H1MC   XI SIM NCM0SIGMA2 QQUAD I12LDOTSN01           XI SIM NCM1SIGMA2 QQUAD IN0N01LDOTSNENDALIGNEDWHERE WE ASSUME M1  M0 AND ARE ASSUMED TO BE KNOWN AS ISSIGMA2  ASSUME THAT N0 IS KNOWN  BEGINENUMERATEITEM BASED UPON A LIKELIHOODRATIO TEST SHOW THAT A TEST FOR THE CHANGE IS DECIDE H1 IF TXBF  FRAC1NN01 SUMIN0N XI  M0  ETAFOR SOME THRESHOLD ETAITEM DETERMINE THE DISTRIBUTION OF TXBF UNDER THE TWO  HYPOTHESES AND DETERMINE AN EXPRESSION FOR PFA AS A FUNCTION  OF THE THRESHOLD ETAENDENUMERATEITEM LABELEXDETECTCHANGE2 IN THE DETECTION OF CHANGE PROBLEM  PRESENT PREVIOUSLY ASSUME NOW THAT WE DONT KNOW N0  FORMING  THE LIKELIHOOD RATIO ELLN0XBF  FRACFXBFXBFH1FXBFXBFH0WE CHOOSE THE MAXIMUM LIKELIHOOD ESTIMATE OF N0 TO BE THAT VALUEWHICH MAXIMIZES ELLN0XBF  SHOW THAT THIS REDUCES TO  MAXN0 SUMIN0N1 XI  M0  FRACM1M02ENDEXERCISESSECTIONREFERENCESTHE RESULTS ON NEYMANPEARSON DETECTION IN THIS DEVELOPMENT WE HAVEDRAWN HEAVILY ON CITECHAPTER 4SCHARFL1991 AND CITECHAPTER5FERGUSON67 AND ON CITEVANTREES68  ALSO CITEPOOR1988BOOK ISUSEFUL READING  DISCUSSION OF THE PHILOSOPHY OF BAYESIAN DECISIONMAKING IS IS FOUND IN CITEHOWSON89  FOR THIS DEVELOPMENT WE RELYON CITESCHARFL1991FERGUSON67DEGROOT70DECISION THEORY TO THE DETECTION OF SIGNALS IS A MAINSTAY OF DIGITALCOMMUNICATIONS IN WHICH SEVERAL DIFFERENT SIGNAL SETS ARECHARACTERIZED BY THEIR DETECTOR STRUCTURES AND THEIR PROBABILITY OFERROR PERFORMANCE  MANY EXCELLENT BOOKS ON COMMUNICATIONS EXIST OFWHICH WE CITE CITEPROAKIS3RDEDBENEDETTO1987LEEANDMESSERSCHMITTTHE GAME THEORY TOUCHED ON IN THE EXAMPLES IS BUT THE TIP OF A VERYLARGE BODY OF RESEARCH FIRST FORMALIZED INCITEVONNEUMANNMORGANSTERN  A BOOK WHICH EXPLORES THE CONNECTIONSBETWEEN GAMES AND LINEAR PROGRAMMING IS CITEKARLIN1992  AN INTERESTINGDISCUSSION OF THE PRISONERS DILEMMA GAME MENTIONED IN THE HOMEWORKAPPEARS IN CITEHOFSTADTER1 AND CITEAXELROD  THECONCEPT OF A MINIMAX POINT  MINIMIZING THE MAXIMUM LOSS  HASSEEN APPLICATION IN A VARIETY OF AREAS BESIDES GAME THEORY AMONG THEMTHE MINIMAX FILTER APPROXIMATION APPROACH CITEPARKSMCCLELLANPARKSMCCLELLANBA DISCUSSION OF DETECTION IN CONTINUOUS TIME IS PROVIDED INCITEVANTREES68 AND CITEPOOR1988BOOK  AN EXCELLENT DISCUSSION OFDETECTION IN NONWHITE GAUSSIAN NOISE ALSO APPEARS INCITESIMON1995  SEE ALSO THE SURVEY ARTICLE CITEKAILATHPOOR1998THE DETECTION OF CHANGE PROBLEMS INDEXDETECTION OF CHANGEINTRODUCED IN EXERCISES REFEXDETECTCHANGE1 ANDREFEXDETECTCHANGE2 ARE THOROUGHLY DISCUSSED INCITEBASSEVILLE1981ABASSEVILLE1981B SEE ALSO CITECHAPTER12KAY1998CITEBASSEVILLE1983HINKLEY1BANSALWILLSKY2WE INTRODUCED IN SECTION REFSECCOMPBIN THE NOTION OF A UNIFORMLYMOST POWERFUL TEST FOR COMPOSITE HYPOTHESES  A SIGNIFICANTLY MORETHOROUGH COVERAGE OF TESTS FOR COMPOSITE HYPOTHESES APPEARS INCITECHAPTER 6KAY1998 LOCAL VARIABLES TEXMASTER TEST ENDSECTIONBAYES DECISION THEORYLABELSECBAYESDECTHUS FAR OUR TREATMENT OF DECISION THEORY HAS BEEN TO CONSIDER THEPARAMETER AS AN UNKNOWN QUANTITY BUT NOT A RANDOM VARIABLE ANDFORMULATE A DECISION RULE ON THE BASIS OF MAXIMIZING THE PROBABILITY OFCORRECT DETECTION THE POWER WHILE AT THE SAME TIME ATTEMPTING TOKEEP THE PROBABILITY OF FALSE ALARM THE SIZE TO AN ACCEPTABLY LOWLEVEL  THE RESULT WAS THE LIKELIHOOD RATIO TEST AND RECEIVEROPERATING CHARACTERISTIC  DECISION THEORY IS NOTHING MORE THAN THE ART OF GUESSING AND AS WITHANY ART THERE IS NO ABSOLUTE OR OBJECTIVE MEASURE OF QUALITY  INFACT WE ARE FREE TO INVENT ANY PRINCIPLE WE LIKE BY WHICH TO ACT INMAKING OUR CHOICE OF DECISION RULE  IN OUR STUDY OF NEYMANPEARSONTHEORY WE HAVE SEEN ONE ATTEMPT AT THE INVENTION OF A PRINCIPLE BYWHICH TO ORDER DECISION RULES NAMELY THE NOTIONS OF POWER AND SIZETHE BAYESIAN APPROACH CONSTITUTES ANOTHER APPROACHSUBSECTIONTHE BAYES PRINCIPLEINDEXBAYES PRINCIPLELET THETA TC TAU BE A PROBABILITY SPACE WHERE THETA ISTHE BY NOW FAMILIAR PARAMETER SET TC IS A SIGMAFIELD OVERTHETA AND TAU IS A PROBABILITY DEFINED OVER THISSIGMAFIELD  LET THETA DELTA L BE A STATISTICAL GAMELET THETA BE THE SET OF POSSIBLE VALUES FOR THETA AND DELTABE THE SET OF POSSIBLE CHOICES  LET X BE A RANDOM VARIABLE ORVECTOR TAKING VALUES IN XC XC MAY BE A SUBSET OF RBB OR OFRBBK FOR CONTINUOUS RANDOM VARIABLES OR IT MAY BE A COUNTABLESET FOR DISCRETE RANDOM VARIABLES  LET PHIX BE A DECISIONFUNCTION AS FOR THE NEYMANPEARSON CASE  WE INTRODUCE A EM LOSS  FUNCTION LVARTHETA PHI REPRESENTING THE LOSS TO THE AGENTIF IT SELECTS PHI WHEN THE TRUE STATE OF NATURE IS VARTHETA  WEINTRODUCE R AS THE THE RISK FUNCTION DEFINED AS THE EM EXPECTED VALUEOF THE LOSS FUNCTIONBEGINDISPLAYMATHRVARTHETA PHI   INTXCLVARTHETA PHIXFXTHETAXVARTHETADXENDDISPLAYMATHWHEN FXTHETAXVARTHETA IS A DENSITY FUNCTION ANDBEGINDISPLAYMATHRVARTHETA PHI  SUMXIN XC LVARTHETAPHIXFXTHETAXVARTHETAENDDISPLAYMATHWHEN FXTHETAXVARTHETA IS A PROBABILITY MASS FUNCTIONTHE BAYES THEORY REQUIRES THAT THE PARAMETER THETA BE VIEWED AS ARANDOM VARIABLE OR RANDOM VECTOR RATHER THAN JUST AN UNKNOWNQUANTITY  THIS ASSUMPTION IS A MAJOR LEAP AND SHOULD NOT BE GLOSSEDOVER LIGHTLY  MAKING IT REQUIRES US TO ACCEPT THE PREMISE THAT NATUREHAS SPECIFIED A PARTICULAR PROBABILITY DISTRIBUTION CALLED THE EM  PRIOR OR EM A PRIORI INDEXPRIOR DISTRIBUTION FOR BAYESDISTRIBUTION OF THETA  FURTHERMORE STRICTLY SPEAKING BAYESIANISMREQUIRES THAT WE KNOW WHAT THIS DISTRIBUTION IS  THESE ARE LARGEPILLS FOR SOME PEOPLE TO SWALLOW PARTICULARLY FOR THOSE OF THESOCALLED OBJECTIVISTS SCHOOL WHICH INCLUDES THOSE OF THENEYMANPEARSON PERSUASION  BAYESIANISM HAS BEEN SUBJECTED TO MUCHCRITICISM FROM THIS QUARTER OVER THE YEARS  BUT THE MORE MODERNSCHOOL OF SUBJECTIVE PROBABILITY HAS GONE A LONG WAY TOWARDS THEDEVELOPMENT OF A RATIONALE FOR BAYESIANISMBRIEFLY SUBJECTIVISTS ARGUE THAT IT IS NOT NECESSARY TO BELIEVE THATNATURE ACTUALLY CHOOSES A STATE ACCORDING TO A PRIOR DISTRIBUTION BUTRATHER THE PRIOR DISTRIBUTION IS VIEWED MERELY AS A REFLECTION OF THEBELIEF OF THE DECISIONMAKING AGENT ABOUT WHERE THE TRUE STATE OFNATURE LIES AND THE ACQUISITION OF NEW INFORMATION USUALLY IN THEFORM OF OBSERVATIONS ACTS TO CHANGE THE AGENTS BELIEF ABOUT THESTATE OF NATURE  IN FACT IT CAN BE SHOWN THAT IN GENERAL EVERYREALLY GOOD DECISION RULE IS ESSENTIALLY A BAYES RULE WITH RESPECT TOSOME PRIOR DISTRIBUTIONIN THE INTEREST OF DISTINGUISHING THE RANDOM VARIABLE FROM THE VALUESIT ASSUMES WE WILL ADOPT THE NOTATIONAL CONVENTION THAT THETADENOTES THE STATE OF NATURE VIEWED AS A RANDOM VARIABLE ANDVARTHETA DENOTES THE VALUES ASSUMED BY THETA THAT ISVARTHETAIN THETA WHERE THETA IS THE PARAMETER SPACE  THUSWE WRITE THETA  VARTHETA TO MEAN THE EVENT THAT THE RANDOMVARIABLE THETA TAKES ON THE PARAMETER VALUE VARTHETA SIMILAR TOTHE WAY WE WRITE X  X TO MEAN THE EVENT THAT THE THE RANDOMVARIABLE X TAKES ON THE VALUE XTO CHARACTERIZE THETA AS A RANDOM VARIABLE WE MUST BE ABLE TODEFINE THE JOINT DISTRIBUTION OF X AND THETA  LET THISDISTRIBUTION BE REPRESENTED BYBEGINDISPLAYMATHFXTHETAX VARTHETA ENDDISPLAYMATHWE WILL ASSUME FOR OUR TREATMENT THAT SUCH A JOINT DISTRIBUTIONEXISTS AND RECALL THATBEGINDISPLAYMATHFXTHETAX VARTHETA FXTHETAXVARTHETAFTHETAVARTHETAFTHETAXVARTHETAXFXXENDDISPLAYMATHNOTE A SLIGHT NOTATIONAL CHANGE HERE  BEFORE WITH THE NEYMANPEARSONAPPROACH WE DID NOT EXPLICITLY INCLUDE THE THETA IN THE SUBSCRIPTOF THE DISTRIBUTION FUNCTION WE MERELY CARRIED IT ALONG AS APARAMETER IN THE ARGUMENT LIST OF THE FUNCTION  WHILE THAT NOTATIONWAS SUGGESTIVE OF CONDITIONING IT WAS NOT REQUIRED THAT WE INTERPRETIT IN THAT LIGHT  WITHIN THE BAYESIAN CONTEXT HOWEVER WE WISH TOEMPHASIZE THAT THE PARAMETER IS VIEWED AS A RANDOM VARIABLE ANDFXTHETA IS A CONDITIONAL DISTRIBUTION SO WEWILL BE CAREFUL TO CARRY IT IN SUBSCRIPT OF THE DISTRIBUTION FUNCTIONAS WELL AS IN ITS ARGUMENT LISTBEGINDEFINITION  THE DISTRIBUTION OF THE THE RANDOM VARIABLE THETA IS CALLED THE  BF PRIOR OR BF A PRIORI DISTRIBUTION  THE SET OF ALL  POSSIBLE PRIOR DISTRIBUTIONS IS DENOTED BY THE SET THETA  WE  WILL ASSUME THAT THIS SET OF PRIOR DISTRIBUTIONS A CONTAINS ALL  FINITE DISTRIBUTIONS IE ALL DISTRIBUTIONS THAT GIVE ALL THEIR  MASS TO A FINITE NUMBER OF POINTS OF THETA AND B IS CONVEX  IE IF TAU1 INTHETA AND TAU2 INTHETA THEN A  TAU1  1ATAU2 INTHETA FOR ALL 0 LEQ ALEQ 1 THIS IS  THE SET OF SOCALLED CONVEX COMBINATIONSENDDEFINITIONSUBSECTIONTHE RISK FUNCTIONAS WE HAVE SEEN A NONRANDOMIZED DECISION FUNCTION PHIMC XCRIGHTARROW DELTA IS A RULE FOR DECIDING DELTA  PHIX AFTERHAVING OBSERVED XX  IN THE NEYMANPEARSON APPROACH THE DECISIONFUNCTION WAS CHOSEN IN LIGHT OF THE CONDITIONAL PROBABILITIES ALPHAAND BETA  IN THE BAYES APPROACH A COST IS ASSOCIATED WITH EACHDECISION FOR EACH STATE OF NATURE AND AN ATTEMPT IS MADE TO MAKE ACHOICE WHICH MINIMIZES THE COST  RECALL THAT IN SECTION REFSECGAMEINTRO WE INTRODUCED THE CONCEPTOF STATISTICAL GAMES  AS PART OF THE GAME WE INTRODUCED THE COSTFUNCTION LMC THETA TIMES DELTA RIGHTARROW RBB SO THATLVARTHETADELTA IS THE COST OF MAKING DECISION DELTA WHENTHETA IS THE TRUE STATE OF NATURE  IF THE AGENT USES DECISIONFUNCTION DELTA  PHIX THEN HIS LOSS BECOMES LVARTHETA PHIX WHICH FOR FIXED VARTHETAINTHETA IS ARANDOM VARIABLE IE IT IS A FUNCTION OF THE RANDOM VARIABLEXBEGINDEFINITION  THE EXPECTATION OF THE LOSS LVARTHETAPHIX WHERE THE  EXPECTATION IS WITH RESPECT TO X IS CALLED THE BF RISK    FUNCTION RMC THETA TIMES D RIGHTARROW RBB INDEXRISK    FUNCTION R  DENOTED RVARTHETAPHI RVARTHETAPHI  E LVARTHETAPHIXENDDEFINITIONTO ENSURE THAT RISK IS WELL DEFINED WE MUST RESTRICT THE SET OFNONRANDOMIZED DECISION RULES D TO ONLY THOSE FUNCTIONS PHIMCXC RIGHTARROW RBB FOR WHICHRVARTHETA PHI EXISTS AND IS FINITE FOR ALL VARTHETA IN THETAIF A PROBABILITY DENSITY FUNCTION PDF FXTHETAXVARTHETAEXISTS THEN THE RISK FUNCTION MAY BE WRITTEN ASBEGINDISPLAYMATHRVARTHETA PHI   INTINFINITYINFINITY LVARTHETA PHIXFXTHETAXVARTHETADXENDDISPLAYMATHIF THE PROBABILITY IS PURELY DISCRETE WITH PROBABILITY MASS FUNCTIONPMF FXTHETAXK VARTHETA THEN THE RISK FUNCTION MAY BEEXPRESSED ASBEGINDISPLAYMATHRVARTHETA PHI   SUMK1N LVARTHETAPHIXKFXTHETAXKVARTHETA ENDDISPLAYMATHTHE RISK REPRESENTS THE AVERAGE LOSS TO THE AGENT WHEN THE TRUESTATE OF NATURE IS VARTHETA AND THE AGENT USES THE DECISIONRULE PHI  APPLICATION OF THE BAYES PRINCIPLE HOWEVER PERMITS US TO VIEWRTHETA PHI AS A RANDOM VARIABLE SINCE IT IS A FUNCTION OF THERANDOM VARIABLE THETA  BEGINEXAMPLE LABELEXMEVENODD2  ODD OR EVEN  THE GAME OF ODD OR EVEN MENTIONED  IN EXAMPLE REFEXMEVENODD1 MAY BE EXTENDED TO A STATISTICAL  DECISION PROBLEM  SUPPOSE THAT BEFORE THE GAME IS PLAYED THE AGENT  IS ALLOWED TO ASK NATURE HOW MANY FINGERS IT INTENDS TO PUT UP AND  THAT NATURE MUST ANSWER TRUTHFULLY WITH PROBABILITY 34 HENCE  UNTRUTHFULLY WITH PROBABILITY 14  THIS MODELS FOR EXAMPLE A  NOISY OBSERVATION  THE AGENT OBSERVES A RANDOM VARIABLE X THE  ANSWER NATURE GIVES TAKING THE VALUES OF 1 OR 2  IF THETA  1  IS THE TRUE STATE OF NATURE THE PROBABILITY THAT X1 IS 34  THAT IS PX1THETA1  34  SIMILARLY PX1THETA2   14  THE OBSERVATION SPACE IN THIS CASE IS XC  12  THE  CHOICE OF NATURE AND THE OBSERVATION PRODUCED CAN BE REPRESENTED  AS SHOWN IN FIGURE REFFIGEVENODDCH  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODE      INPUTPICTUREDIREVENODDCH      CAPTIONILLUSTRATION OF EVENODD OBSERVATIONS      LABELFIGEVENODDCH    ENDCENTER  ENDFIGURE  THE DECISION SPACE IS DELTA  12  RECALL THAT THE LOSS  FUNCTION IS BEGINALIGNEDL11    2L12    3L21    3L22    4ENDALIGNEDWE WILL FIRST EXAMINE THE POSSIBLE DECISION FUNCTIONS  FOR SMALLDECISION PROBLEMS SUCH AS THIS ONE IT IS POSSIBLE TO EXHAUSTIVELYENUMERATE ALL DECISION FUNCTIONS  THERE ARE EXACTLY FOUR POSSIBLEFUNCTIONS FROM XC INTO DELTA SO D PHI1PHI2PHI3PHI4 WHEREBEGINEQUATIONLABELEQDSETBEGINSPLITPHI11  1 QQUAD PHI12  1 PHI21  1 QQUAD PHI22  2 PHI31  2 QQUAD PHI32  1 PHI41  2 QQUAD PHI42  2ENDSPLITENDEQUATIONRULES PHI1 AND PHI4 IGNORE THE VALUE OF X  RULE PHI2 REFLECTSTHE AGENTS BELIEF THAT NATURE IS TELLING THE TRUTH ANDRULE PHI3 THAT NATURE IS NOT TELLING THE TRUTH  LET US NOW EXAMINE THE RISK FUNCTION RTHETAPHI FOR THIS GAMEFOR EXAMPLERTHETAPHI1  SUMX12 LTHETAPHI1XFXXTHETAWHEN THETA  1 WE HAVE R1PHI1  SUMX12 L1PHI11 FX1 1 L1PHI12 FX21  234  214  2THE RISK MATRIX GIVEN IN FIGURE REFRISKMATRIX CHARACTERIZES THISSTATISTICAL GAMEBEGINFIGUREHTBBEGINCENTERINPUTPICTUREDIRODDEVENRISKLATEXENDCENTERCAPTIONRISK FUNCTION FOR STATISTICAL ODD OR EVEN GAMELABELRISKMATRIXENDFIGUREENDEXAMPLEBEGINEXAMPLE LABELEXMBINCHANNEL2 BINARY CHANNELCONSIDER NOW THE PROBLEM OF TRANSMISSION IN A BINARY CHANNEL WITH  CROSSOVER PROBABILITIES LAMBDA0 AND LAMBDA1 AS SHOWN IN  FIGURE REFFIGBAYESBIN  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODE      INPUTPICTUREDIRBSCLAMBDA      CAPTIONA BINARY CHANNEL      LABELFIGBAYESBIN    ENDCENTER  ENDFIGURE  AS FOR THE EVENODD GAME FOUR POSSIBLE DECISION FUNCTIONS EXISTBEGINEQUATIONLABELEQDSET2BEGINSPLITPHI10  0 QQUAD PHI11  0 PHI20  0 QQUAD PHI21  1 PHI30  1 QQUAD PHI31  0 PHI40  1 QQUAD PHI41  1ENDSPLITENDEQUATIONWHERE THE FIRST AND LAST DECISION FUNCTIONS IGNORE THE MEASURED VALUE AND THETHIRD DECISION FUNCTION REFLECTS A BELIEF THAT THE OBSERVED VALUE ISINCORRECT  IF WE ASSUME THE COST STRUCTURE LXY  DELTAXYTHAT IS THE COST OF MAKING BIT ERRORS THEN THE RISK FUNCTION FORTHIS GAME IS SHOWN IN FIGURE REFFIGRISKMATRIXBINCHANBEGINFIGUREHTBBEGINCENTERINPUTPICTUREDIRBSCLLATEXENDCENTERCAPTIONRISK FUNCTION FOR THE BINARY CHANNELLABELFIGRISKMATRIXBINCHANENDFIGUREENDEXAMPLEWITH THE INTRODUCTION OF THE RISK FUNCTION R AND THE CLASS OFDECISION FUNCTIONS D WE MAY REPLACE THE ORIGINAL GAME THETADELTA L BY A NEW GAME WHICH WE WILL DENOTE BY THE TRIPLE THETAD R IN WHICH THE SPACE D AND THE FUNCTION R HAVE HAVE ANUNDERLYING STRUCTURE DEPENDING ON DELTA AND L AND THEDISTRIBUTION OF X WHOSE EXPLOITATION IS THE MAIN OBJECTIVE OFDECISION THEORY  SOMETIMES THE TRIPLE THETA D R IS ALSO CALLED ASTATISTICAL GAMESUBSECTIONBAYES RISKLABELSECBAYESRISKWE MIGHT SUPPOSE THAT A REASONABLE DECISION CRITERION WOULD BE TOCHOOSE THE DECISION RULE PHI SUCH THAT THE RISK IS MINIMIZED BUTTHIS IS NOT GENERALLY POSSIBLE SINCE THE VALUE THETA ASSUMES ISEM UNKNOWN SO WE CANNOT UNILATERALLY MINIMIZE THE RISK AS LONG ASTHE LOSS FUNCTION DEPENDS ON THETA AND THAT TAKES IN JUST ABOUTALL INTERESTING CASES  A NATURAL WAY TO DEAL WITH THIS SITUATION INTHE BAYESIAN CONTEXT IS TO COMPUTE THE AVERAGE RISK AND THEN FIND ADECISION RULE THAT MINIMIZES THIS AVERAGE RISK  UNDER THE ASSUMPTIONTHAT THETA IS A RANDOM VARIABLE WE CAN NOW INTRODUCE THE CONCEPTOF BAYES RISKBEGINDEFINITIONINDEXBAYES RISK FUNCTION R  THE BF BAYES RISK FUNCTION WITH RESPECT TO A PRIOR DISTRIBUTION  FTHETA DENOTED RFTHETA PHI IS GIVEN BY  RFTHETA PHI  ERTHETA PHI WHERE THE EXPECTATION IS  TAKEN OVER THE SPACE THETA OF VALUES THAT THETA MAY ASSUMEBEGINDISPLAYMATHRFTHETA PHI   INTTHETARVARTHETAPHIFTHETAVARTHETA DVARTHETAENDDISPLAYMATHWHEN FTHETA HAS A DENSITY FUNCTION FTHETAVARTHETA AND BEGINDISPLAYMATHRFTHETA PHI  SUMVARTHETAINTHETA RVARTHETAIPHIFTHETAVARTHETA ENDDISPLAYMATHWHEN FTHETA HAS A PROBABILITY MASS FUNCTION FTHETAVARTHETAENDDEFINITIONWE NOTE THAT WHEREAS THE RISK R IS DEFINED AS THE AVERAGEOF THE LOSS FUNCTION OBTAINED BY AVERAGING OVER ALL VALUES XX FOR A FIXED THETA THE BAYES RISK R IS THEAVERAGE VALUE OF THE LOSS FUNCTION OBTAINED BY AVERAGING OVER ALLVALUES XX AND THETA  VARTHETA  FOR EXAMPLE WHEN BOTH X ANDTHETA ARE CONTINUOUSBEGINALIGNRFTHETA PHI    ELTHETA PHIXNONUMBER 10PT     INTTHETA            RVARTHETAPHIFTHETAVARTHETA DVARTHETANONUMBER 10PT     INTTHETAINTXC            LVARTHETA PHIXFXTHETAXVARTHETA              FTHETAVARTHETA DX DVARTHETA  LABELRISKCONTINUOUSENDALIGNIF X IS CONTINUOUS AND THETA IS DISCRETE THENBEGINALIGNRFTHETA PHI    ELTHETA PHIXNONUMBER 10PT     SUMVARTHETAINTHETA            RVARTHETAPHIFTHETAVARTHETANONUMBER 10PT     SUMVARTHETAINTHETA INTXC            LVARTHETA PHIXFXTHETAXVARTHETA              FTHETAVARTHETA DX  LABELRISKDISCRETE            ENDALIGNTHE REMAINING CONSTRUCTIONS WHEN X IS DISCRETE ARE ALSO EASILY OBTAINEDWE MAY EXTEND THE DEFINITION OF BAYES RISK TO RANDOMIZED DECISIONRULES BY TAKING THE EXPECTATION OF BAYES RISK WITH RESPECT TO THERANDOMIZED DECISION RULE  LET VARPHI IN D BE A RANDOMIZED RULEOVER THE SET D OF NONRANDOMIZED RULES  THEN THE BAYES RISK WITHRESPECT TO THE PRIOR THETA AND THE RANDOMIZED DECISION RULEVARPHI ISBEGINEQUATIONLABELEQNRANDOMRISKRFTHETAVARPHI  EVARPHIRFTHETAPHIENDEQUATIONFOR EXAMPLE IF D   PHI1 LDOTS PHIK AND VARPHI PI1 LDOTS PIK THEN BEGINEQUATIONLABELEQNRANDOMRISKDRFTHETA VARPHI  SUMI1K RFTHETAPHIIENDEQUATIONSUBSECTIONBAYES TESTS OF SIMPLE BINARY HYPOTHESESLET THETA VARTHETA0VARTHETA1 CORRESPONDING TO THEHYPOTHESES H0 AND H1 RESPECTIVELY AND LET DELTA DELTA0 DELTA1 CORRESPOND RESPECTIVELY  WE DESIRE TOFASHION A DECISION RULE PHIMC XC RIGHTARROWRBB SUCH THAT WHEN X  X IS OBSERVEDBEGINEQUATIONPHIX  BEGINCASES1  TEXTIF  XIN RCTEXT REJECT H00  TEXTIF  XIN AC TEXT ACCEPT H0 ENDCASESLABELEQBAYESPHIENDEQUATIONWHERE RC AND AC ARE DISJOINT SUBSETS OF XC AND XC  RCCUP AC  WE INTERPRET THIS DECISION RULE AS FOLLOWS IF XIN RCWE TAKE ACTION DELTA1 THAT IS CHOOSE H1 AND IF XIN ACWE TAKE ACTION DELTA0 CHOOSE H0  IN ORDER TO ESTABLISHPHI WE MUST DETERMINE THE SETS RC AND AC  THE RISK FUNCTIONFOR THE RULE REFEQBAYESPHI IS BEGINALIGNEDRTHETA PHI   INTAC LTHETAPHIX FXXTHETADX  INTRC LTHETAPHIX FXXTHETADX  EXMATSP LTHETADELTA0PACTHETA  LTHETADELTA1PRCTHETA1PRC THETALTHETA DELTA0 PRC THETALTHETA DELTA1 EXMATSP   LTHETA DELTA0  PRC THETALTHETA DELTA1LTHETA DELTA0 ENDALIGNEDWHERE BY PRC THETA WE MEAN THE CONDITIONAL PROBABILITY THAT X WILLTAKE VALUES IN RC GIVEN THETAFOR OUR PARTICULAR CHOICE OF DECISION RULE WE OBSERVE THAT THECONDITIONAL EXPECTATION OF PHIX GIVEN THETA ISBEGINALIGNEDEPHIXTHETA    1CDOT PRC THETA  0CDOT 1PRC THETA   PRC THETAENDALIGNEDSO WE MAY WRITERTHETA PHI   LTHETA DELTA0  EPHIXTHETALTHETA DELTA1LTHETA DELTA0FOR THE CASE OF BINARY ALTERNATIVES AND SIMPLE BINARY HYPOTHESESTHERE ARE FOUR TYPES OF COST THAT WE MIGHT INCURBEGINENUMERATEITEM THE COST OF DECIDING H0 GIVEN THAT H0 IS CORRECT DENOTED  L00ITEM THE COST OF DECIDING H1 GIVEN THAT H1 IS CORRECT DENOTED  L11ITEM THE COST OF DECIDING H0 GIVEN THAT H1 IS CORRECT DENOTED  L10ITEM THE COST OF DECIDING H1 GIVEN THAT H0 IS CORRECT DENOTED  L01ENDENUMERATEMORE GENERALLY LIJ INDICATES THE COST CHOOSING HJ GIVEN THATHI IS CORRECTFOR THE LAST TWO CASES WE WILL DENOTEBEGINEQUATIONLABELLOSSBEGINARRAYLLLTHETA DELTA0    AIVARTHETA1THETA  LEFTBEGINARRAYCCC A  MBOXIF   THETA VARTHETA1 0  MBOXIF   THETAVARTHETA0 ENDARRAYRIGHT LTHETA DELTA1   BIVARTHETA0THETA ENDARRAY ENDEQUATIONWHERE A AND B ARE ARBITRARY POSITIVE CONSTANTS  THUS IF THETA VARTHETA1 BUT WE WRONGLY GUESS THETAVARTHETA0 WE INCUR APENALTY OR LOSS OF A UNITS AND IF THETA  VARTHETA0 AND WEGUESS THAT THETA VARTHETA1 WE LOSE B UNITSTHE RISK FUNCTION BECOMES BEGINALIGNRTHETA PHI     AIVARTHETA1THETA  EPHIXTHETABIVARTHETA0THETAAIVARTHETA1THETANONUMBER 10PT   LEFT BEGINARRAYLLLB EPHIXTHETA  VARTHETA0  MBOX FOR   THETA VARTHETA0 10PTA 1 EPHIXTHETA  VARTHETA1  MBOX FOR   THETA VARTHETA1 ENDARRAYRIGHT LABELRISKENDALIGNBEGINALIGNRTHETA PHI     BEGINCASES  L00PACTHETA0  L01PRCTHETA1  THETA  VARTHETA0   L10PACTHETA0  L11PRCTHETA1  THETA  VARTHETA1ENDCASESENDALIGNWE WILL ALSO INTRODUCE THE PROBABILITY NOTATION BEGINALIGNEDPACTHETA0  TEXT PROBABILITY OF CORRECT ACCEPTANCE  1ALPHA  P00PRCTHETA0  TEXT PROBABILITY OF FALSE ALARM   ALPHA P01 PACTHETA1  TEXT PROBABILITY OF MISSED DETECTION  1BETA P10 PRCTHETA1  TEXT PROBABILITY OF DETECTION  BETA  P11ENDALIGNEDON THIS BASIS WE CAN WRITEBEGINALIGNRTHETA PHI     BEGINCASES  L00P00   L01P01  THETA  VARTHETA0   L10P10  L11P11  THETA  VARTHETA1ENDCASESNONUMBER 10PT  BEGINCASES  L01  L00P01  L00  THETA  VARTHETA0   L10  L11P10  L11  THETA  VARTHETA1ENDCASESLABELEQRISKENDALIGNFROM REFEQRISK WE OBSERVE THAT NO MATTER WHAT DECISION WEMAKE THERE IS A CONSTANT COST L00 ASSOCIATED WITH THE CASETHETA  VARTHETA0 AND SIMILARLY A CONSTANT COST L11ASSOCIATED WITH THETA VARTHETA1  IT IS CUSTOMARY TO ASSUME THATL00  L11  0 MAKING ADJUSTMENTS TO L01 AND L10 ASNECESSARY  WE THEN HAVEBEGINEQUATION  LABELEQRISK2  RTHETAPHI    BEGINCASES    L01 P01  L01ALPHA  THETA  VARTHETA0     L10 P10  L101BETA  THETA  VARTHETA1  ENDCASESENDEQUATIONWE NOW INTRODUCE THE NUMBER P TO BE THE PRIOR PROBABILITYBEGINEQUATIONLABELAPRIORIBEGINSPLITP   FTHETAVARTHETA1 PTHETA  VARTHETA110PT1 P  FTHETAVARTHETA0 PTHETA  VARTHETA0 ENDSPLITENDEQUATIONALTHOUGH THE ABOVE DEVELOPMENT INVOLVED ONLY NONRANDOMIZED RULES WEMAY EASILY EXTEND TO RANDOMIZED RULES BY REPLACING PHI WITHVARPHI IN ALL CASES RECALL THAT NONRANDOMIZED RULES MAY BE VIEWEDAS DEGENERATE RANDOMIZED RULES WHERE ALL OF THE PROBABILITY MASS ISPLACEDON ONE NONRANDOMIZED RULE  AS P REPRESENTS THE DISTRIBUTIONFTHETA WE WILL WRITE THE BAYES RISK RFTHETAVARPHI ASRPVARPHI SEE REFEQNRANDOMRISK  THE BAYES RISK ISBEGINEQUATIONLABELBAYESRISKRP VARPHI      1P RVARTHETA0VARPHI P RVARTHETA1VARPHIENDEQUATIONANY RANDOMIZED DECISION FUNCTION THAT FOR FIXED P MINIMIZES THEBAYES RISK IS SAID TO BE BF BAYES WITH RESPECT TO P AND WILL BE DENOTED VARPHIP WHICH SATISFIES BEGINEQUATIONLABELBAYESWRTTAUVARPHIP  ARGMINVARPHIIN D RP PHIENDEQUATIONTHE USUAL INTUITIVE MEANING ASSOCIATED WITH REFBAYESRISK IS THEFOLLOWING  SUPPOSE THAT YOU KNOW OR BELIEVE THAT THE UNKNOWNPARAMETER THETA IS IN FACT A RANDOM VARIABLE WITH SPECIFIED PRIORPROBABILITIES OF P AND 1P OF TAKING VALUES VARTHETA1AND VARTHETA0 RESPECTIVELY  THEN FOR ANY DECISION FUNCTION VARPHITHE GLOBAL EXPECTED LOSS WILL BE GIVEN BY REFBAYESRISK ANDHENCE IT WILL BE REASONABLE TO USE THE DECISION FUNCTION VARPHIPWHICH MINIMIZES RP VARPHIWE NOW PROCEED TO FIND THE DECISION FUNCTION VARPHIP WHICHMINIMIZES THE BAYES RISK  WE WILL ASSUME THAT THE TWO CONDITIONALDISTRIBUTIONS OF X FOR THETA  VARTHETA0 AND THETA VARTHETA1 ARE GIVEN IN TERMS OF DENSITY FUNCTIONSFXTHETAXVARTHETA0 ANDFXTHETAXVARTHETA1  THEN FROM REFEQRISK2 ANDREFBAYESRISK WE HAVEBEGINALIGNRP PHI    P L101EPHIXTHETA VARTHETA1     1PL01 EPHIXTHETA  VARTHETA0 NONUMBER 10PT   P L10 LEFT 1INTXC FXTHETAXVARTHETA1 PHIX DX RIGHT 1PL01 INTXC FXTHETAXVARTHETA0 PHIX DXNONUMBER 10PT    P L10 INTXC LEFT P L10 FXTHETAXVARTHETA1  1PL01 FXTHETAXVARTHETA0 RIGHTPHIX DX  LABELRISKITENDALIGNTHIS LAST EXPRESSION IS MINIMIZED BY MINIMIZING THE INTEGRAND FOR EACHX THAT IS BY DEFINING PHIX TO BEBEGINDISPLAYMATHPHIX  BEGINCASESHFILL 1HFILL   TEXT IF   1PL01 FXTHETAX VARTHETA0  PL10 FXTHETAX VARTHETA110PTHFILL 0 HFILL  TEXT IF   1PL01 FXTHETAX VARTHETA0  PL10 FXTHETAX VARTHETA110PTTEXTARBITRARY  TEXT IF   1PL01 FXTHETAXVARTHETA0  P  L10 FXTHETAX VARTHETA1ENDCASESENDDISPLAYMATHFOR THIS BINARY PROBLEM THE BAYES RISK IS UNAFFECTED BY THE EQUALITYCONDITION 1PL01 FXTHETAX VARTHETA0  P L10FXTHETAX VARTHETA1 AND THEREFORE WITHOUT LOSS OFGENERALITY WE MAY PLACE ALL OF THE PROBABILITY MASS OF THE RANDOMIZEDDECISION RULE ON THE NONRANDOMIZED RULEBEGINEQUATIONPHIPX  BEGINCASES1  TEXT IF   1PL01 FXTHETAX VARTHETA0  PL10 FXTHETAX VARTHETA110PT0   TEXTOTHERWISEENDCASESLABELEQPHITENDEQUATIONWE MAY DEFINE THE SETS RC AND AC ASBEGINALIGNEDRC  LEFT XMC 1PL01 FXTHETAX VARTHETA0  P L10 FXTHETAX VARTHETA1RIGHT10PTAC  LEFT XMC 1PL01 FXTHETAX VARTHETA0 GEQ P L10 FXTHETAX VARTHETA1RIGHTENDALIGNEDTHEN REFRISKIT BECOMESBEGINALIGNRP PHIP  P  L10 LEFT 1INTXC  FXTHETAXVARTHETA1  IRCX DX RIGHT 1PL01INTXC FXTHETAXVARTHETA0 IRCX DXNONUMBER 10PT   P L10 INTXC FXTHETAXVARTHETA1 IACX DX  1PL01 INTXC FXTHETAXVARTHETA0 IRCX DX  LABELPROBERRORENDALIGNSINCE WE DECIDE THETA  VARTHETA1 IF XINRC AND THETAVARTHETA0 IF XINAC WE OBSERVE THAT BY SETTING L01 L10 1 THE BAYES RISK REFPROBERROR BECOMES THE TOTALPROBABILITY OF ERRORBEGINEQUATIONLABELTOTALPROBRP PHIP  UNDERBRACEPRC  THETA VARTHETA0PFAPTHETA THETA0 UNDERBRACEPAC  THETA  VARTHETA1PMD PTHETA THETA1ENDEQUATIONOBSERVE THAT PHIPX OF REFEQPHIT MAY BE WRITTEN AS ALIKELIHOOD RATIO TESTBEGINEQUATIONLABELEQBAYESDECISIONRULEPHIPX   BEGINCASES1  TEXTIF  FRACDISPLAYSTYLE FXTHETAXVARTHETA1DISPLAYSTYLE F XTHETAX VARTHETA0  FRACDISPLAYSTYLE L01PH0DISPLAYSTYLE L10PH1 0  TEXTOTHERWISEENDCASESENDEQUATIONIT IS IMPORTANT TO NOTE THAT FOR BINARY DECISION PROBLEMS UNDERSIMPLE HYPOTHESES THIS TEST IS IDENTICAL IN FORM TO THE SOLUTION TOTHE NEYMANPEARSON TEST ONLY THE THRESHOLD IS CHANGED  WHEREAS FORTHE NEYMANPEARSON TEST THE THRESHOLD WAS DETERMINED BY THE SIZE OFTHE TEST THE BAYESIAN FORMULATION PROVIDES THE THRESHOLD AS AFUNCTION OF THE PRIOR DISTRIBUTION ON THETA AND THE COSTSASSOCIATED WITH THE DECISIONSBEGINEXAMPLE BINARY CHANNEL  CONSIDER AGAIN THE BINARY CHANNEL OF  EXAMPLE REFEXMBINCHANNEL2  WE WANT TO DEVISE A BAYES TEST FOR THIS  CHANNELBEGINALIGNAT2FX11   PX1THETA1  1LAMBDA1  QQUAD FX01 PX0THETA1  LAMBDA1  FX10  PX1THETA0  LAMBDA0  QQUAD  FX00 PX0THETA0  1LAMBDA0 ENDALIGNATWE CAN WRITE A LIKELIHOOD RATIO ELLX  FRACFXX1FXX0  BEGINCASES  FRAC1LAMBDA1LAMBDA0  X1 FRACLAMBDA11LAMBDA0  X  0ENDCASESIF ARE COSTS ARE APPROPRIATE FOR COMMUNICATIONS L01  L101THEN THE DECISION RULE ISBEGINEQUATION  LABELEQBINCOM1  PHIPY    BEGINCASES    1  ELLY  FRAC1PP     0  TEXTOTHERWISE  ENDCASESENDEQUATIONFOR EXAMPLE WHEN P12 THE DECISION RULE IS BEGINALIGNEDPHI1  BEGINCASES1  1LAMBDA1 GEQ LAMBDA0 0  TEXTOTHERWISEENDCASES PHI0  BEGINCASES1  LAMBDA1 GEQ 1LAMBDA0 0  TEXTOTHERWISEENDCASESENDALIGNEDENDEXAMPLEBEGINEXAMPLE  LET THETA  THETA0THETA1  MBF0MBF1  LET US  ASSUME THAT UNDER HYPOTHESIS H1 THAT XBF SIM NCMBF1R  AND UNDER HYPOTHESIS H0 THAT XBF SIM NCMBF0R WHERE  XBF IS AN NDIMENSIONAL RANDOM VECTOR  DENOTING THE MEAN OF  THE DISTRIBUTION BY MBF WE ASSUME THAT WE HAVE THE FOLLOWING  PRIOR INFORMATION BEGINALIGNEDPMBF  MBF1   PPMBF  MBF0   1 PENDALIGNEDTHE LIKELIHOOD RATIO IS BEGINALIGNEDELLXBF    FRACFXBFXBFTHETA1FXBFXBFTHETA010PT   FRACEXPFRAC12XBFMBF1T R1 XBFMBF1EXPFRAC12XBFMBF0T R1 XBFMBF0ENDALIGNEDAFTER CANCELING COMMON TERMS AND TAKING THE LOGARITHM WE HAVE THELOGLIKELIHOOD RATIOBEGINEQUATIONLABELLLRT5LAMBDAXBF  LOG ELLXBF  MBF1 MBF0T R1 XBF  XBF0ENDEQUATIONWHERE XBF0  FRAC12MBF1  MBF0SUPPOSE THE PROBLEM NOW IS TO DETECT THE MEAN OF XBF AND THE ONLYCRITERION IS CORRECTNESS OF THE DECISION  THEN FOR A COST FUNCTION WECAN TAKE L01 L10  1  BASED ON THE DECISION RULE FROMREFEQBAYESDECISIONRULE WE HAVEBEGINDISPLAYMATHPHIPXBF   BEGINCASES1  TEXTIF  LAMBDAXBF  LOG FRACDISPLAYSTYLE  1PDISPLAYSTYLE P  ETA 10PT0   TEXTOTHERWISEENDCASESENDDISPLAYMATHAS BEFORE WE COULD COMPUTE THE PROBABILITIES OF ERROR PFA PPHIP1H0 AND THE PROBABILITY OF MISSED DETECTIONPMD  PPHIT  0H1  IN THIS CASEBEGINALIGNEDPFA  PTHETA0LAMBDAXBF  ETA  QETA  S22S  QZENDALIGNEDWHERE S2  WBFT R WBFQQUADTEXTANDQQUAD WBF  R1MBF1 MBF0AND Z  ETAS  S2ALSO BEGINALIGNEDPD  PTHETA1LAMBDAXBF  ETA  QETA  S22F  QZSENDALIGNEDGIVEN OUR MODEL OF THE PRIOR WE CAN ALSO COMPUTE THE TOTALPROBABILITY OF ERRORBEGINALIGN  PEC  PPHIT1H0PH0  PPHIT  0H1 PH1 NONUMBER   P QS2LOG1PPS  1P QLOG1PP  S2S  NONUMBER  P1QZS  1PQZ  LABELEQPROBERBAYESENDALIGNENDEXAMPLESUBSECTIONPOSTERIOR DISTRIBUTIONSINDEXPOSTERIOR DISTRIBUTION FOR BAYESIF THE DISTRIBUTION OF THE PARAMETER THETA BEFORE OBSERVATIONS AREMADE IS CALLED THE PRIOR DISTRIBUTION THEN IT IS NATURAL TO CONSIDERDEFINING A POSTERIOR DISTRIBUTION AS THE DISTRIBUTION OF THE PARAMETERAFTER OBSERVATIONS ARE TAKEN AND PROCESSED  WE FIRST CONSIDER THE CASE FOR X AND THETA BOTH CONTINUOUSASSUMING WE CAN REVERSE THE ORDER OF INTEGRATION INREFRISKCONTINUOUS WE OBTAINBEGINALIGNRFTHETA PHI    INTTHETAINTXC            LVARTHETA PHIXFXTHETAXVARTHETA              FTHETAVARTHETA DX DVARTHETA NONUMBER 10PT    INTXC  INTTHETA LVARTHETA PHIXUNDERBRACEFXTHETAXVARTHETA FTHETAVARTHETAFXTHETAXVARTHETADVARTHETA DXNONUMBER 10PT    INTXC  LEFT INTTHETA LVARTHETA PHIXFTHETAXVARTHETA X DVARTHETA RIGHT FXX DX LABELREVERSEENDALIGNWHERE WE HAVE USED THE FACT THATBEGINDISPLAYMATHFXTHETAXVARTHETAFTHETAVARTHETA FXTHETAXVARTHETA  FTHETAXVARTHETA XFXXENDDISPLAYMATHIN OTHER WORDS A CHOICE OF THETA BY THE MARGINAL DISTRIBUTIONFTHETAVARTHETA FOLLOWED BY A CHOICE OF X FROM THECONDITIONAL DISTRIBUTION FXTHETAXVARTHETA DETERMINES AJOINT DISTRIBUTION OF THETA AND X WHICH IN TURN CAN BEDETERMINED BY FIRST CHOOSING X ACCORDING TO ITS MARGINALDISTRIBUTION FXX AND THEN CHOOSING THETA ACCORDING TO THECONDITIONAL DISTRIBUTION FTHETAXVARTHETAX OF THETAGIVEN XXWITH THIS CHANGE IN ORDER OF INTEGRATION SOME VERY USEFUL INSIGHT MAYBE OBTAINED  WE SEE THAT WE MAY MINIMIZE THE BAYES RISK GIVEN BYREFREVERSE BY FINDING A DECISION FUNCTION PHIX THAT EM  MINIMIZES THE INSIDE INTEGRAL SEPARATELY FOR EACH X THAT IS WEMAY FIND FOR EACH X A RULE THAT MINIMIZESBEGINEQUATIONLABELPOSTERIORCONDINTTHETA LVARTHETA PHIXFTHETAXVARTHETA X DVARTHETA ENDEQUATIONBEGINDEFINITION  THE CONDITIONAL DISTRIBUTION OFTHETA GIVEN X DENOTED FTHETAXVARTHETA  X ISCALLED THE BF POSTERIOR OR BF A POSTERIORI DISTRIBUTION OFTHETA  IT IS FREQUENTLY WRITTEN USING BAYES THEOREM AS FTHETAXVARTHETA1X   FRACDISPLAYSTYLEFXTHETAXVARTHETA1FTHETAVARTHETA1INT  FXTHETAXVARTHETAFTHETAVARTHETA DVARTHETAENDDEFINITIONTHE EXPRESSION GIVEN IN REFPOSTERIORCOND IS THE EXPECTED LOSSGIVEN THAT XX AND WE MAY THEREFORE INTERPRET A BAYES DECISIONRULE AS ONE THAT EM MINIMIZES THE POSTERIOR CONDITIONAL EXPECTEDLOSS GIVEN THE OBSERVATION  THE ABOVE RESULTS NEED BE MODIFIED ONLY IN NOTATION FOR THE CASE WHEREX AND THETA ARE DISCRETE  FOR EXAMPLE IF THETA IS DISCRETESAY THETA VARTHETA1 LDOTS VARTHETAK WE REVERSE THE ORDER OF SUMMATION AND INTEGRATION INREFRISKDISCRETE TO OBTAINBEGINALIGNRP PHI     SUMI1K INTXC             LVARTHETAI PHIXFXTHETAXVARTHETAI              FTHETAVARTHETAI DXNONUMBER 10PT     INTXC SUMI1K            LVARTHETAI PHIXFXTHETAXVARTHETAI              FTHETAVARTHETAIDXNONUMBER 10PT     INTXC LEFTSUMI1K            LVARTHETAI PHIXFTHETAXVARTHETAIX   RIGHTFXXDX  LABELREVERSE2 ENDALIGNSUPPOSE THAT THERE ARE ONLY TWOHYPOTHESES THETA  VARTHETA0VARTHETA1 AND DECISIONSCORRESPONDING TO EACH OF THESE  THEN  SUMI12  LVARTHETAIPHIXFTHETAXVARTHETAIX  BEGINCASES  L00 FTHETAXVARTHETA0X  L10  FTHETAXVARTHETA1X  PHIX  0   L01 FTHETAXVARTHETA0X  L11  FTHETAXVARTHETA1X  PHIX  1ENDCASESDETERMINATION OF PHIX ON THE BASIS OF MINIMUM RISK CAN BE STATEDAS SET PHIX1 IF L01 FTHETAXVARTHETA0X  L11  FTHETAXVARTHETA1X   L00  FTHETAXVARTHETA0X  L10  FTHETAXVARTHETA1XWHICH LEADS TO THE LIKELIHOOD RATIO TESTBEGINEQUATION PHIX  BEGINCASES  1  FRACDISPLAYSTYLE  FTHETAXVARTHETA1 XDISPLAYSTYLE F  THETAXVARTHETA0X  FRACDISPLAYSTYLE L01   L00DISPLAYSTYLE L10  L11  0  TEXTOTHERWISEENDCASESLABELEQBAYESDEC2ENDEQUATIONIT IS INTERESTING TO CONTRAST THIS RULE WITH THAT DERIVED INREFEQBAYESDECISIONRULE WHICH IS REPRODUCED HERE PHIPX   BEGINCASES1  TEXTIF  FRACDISPLAYSTYLE FXTHETAXVARTHETA1DISPLAYSTYLE F XTHETAX VARTHETA0  FRACDISPLAYSTYLE L011PDISPLAYSTYLE L10P  ETA0  TEXTOTHERWISEENDCASESIN REFEQBAYESDEC2 THE THRESHOLD IS DETERMINED ONLY BY THE BAYESCOSTS AND THE RATIO IS A RATIO OF EM POSTERIOR DENSITIES STRICTLYSPEAKING NOT A LIKELIHOOD RATIOBEGINEXAMPLE LABELEXMPOSTERIORBAYESLET US CONSIDER THE SIMPLE HYPOTHESIS VERSUS SIMPLE ALTERNATIVEPROBLEM FORMULATION AND LET THETA  VARTHETA0 VARTHETA1AND DELTA  0  1  ASSUME WE OBSERVE A RANDOMVARIABLE X TAKING VALUES IN  X0  X1 WITH THE FOLLOWING CONDITIONAL DISTRIBUTIONSBEGINDISPLAYMATHBEGINARRAYCFXTHETAX1VARTHETA0  PX  X1 THETA VARTHETA0 FRAC34 QQUAD FXTHETAX0VARTHETA0  PX  X0THETA VARTHETA0  FRAC1410PTFXTHETAX1VARTHETA1  PX  X1 THETA VARTHETA1 FRAC13 QQUAD FXTHETAX0VARTHETA1  PX  X0THETA VARTHETA1  FRAC23ENDARRAYENDDISPLAYMATH WE WILL TAKE THE LOSS FUNCTION FOR THIS PROBLEM AS GIVEN BY THE MATRIXIN FIGURE REFGAME1  THIS EXAMPLE COULD BE THOUGHT OF AS AGENERALIZATION OF THE BINARY CHANNEL WITH CROSSOVER PROBABILITIES34 AND 23 AND WITH DIFFERENT COSTS ASSOCIATED WITH THEDIFFERENT KINDS OF ERRORBEGINFIGUREHTBBEGINCENTERINPUTPICTUREDIRLOSSFUNCTLATEXENDCENTERCAPTIONLOSS FUNCTIONLABELGAME1ENDFIGURELET PTHETA  VARTHETA1  P AND PTHETA  VARTHETA0 1P BE THE PRIOR DISTRIBUTION FOR THETA FOR 0 LEQ P LEQ1  WE WILL ADDRESS THIS PROBLEM BYSOLVING FOR THE EM A POSTERIORI PMF THE POSTERIOR PMF IS GIVEN VIA BAYES THEOREM ASBEGINALIGNFTHETAXVARTHETA1X    FRACDISPLAYSTYLEFXTHETAXVARTHETA1FTHETAVARTHETA1 FXTHETAXVARTHETA0FTHETAVARTHETA0  FXTHETAXVARTHETA1FTHETAVARTHETA110PT     BEGINCASESFRAC FRAC13PFRAC341P  FRAC13P  TEXTIF   XX110PTFRAC FRAC23PFRAC141P  FRAC23P  TEXTIF  XX0ENDCASESENDALIGNNOTE THAT BEGINDISPLAYMATHFTHETAXVARTHETA0X  1FTHETAXVARTHETA1XENDDISPLAYMATHAFTER THE VALUE XX HAS BEEN OBSERVED A CHOICE MUST BE MADE BETWEENTHE TWO ACTIONS DELTA 0 AND DELTA 1  THE BAYES DECISION RULE ISBEGINALIGNPHIPX    ARGMINPHILEFTLVARTHETA1PHIFTHETAXVARTHETA1X LVARTHETA0PHIFTHETAXVARTHETA0X RIGHT NONUMBER 10PT   BEGINCASESARGMINPHILEFT LVARTHETA1PHI FRAC FRAC13PFRAC341P  FRAC13P LVARTHETA0PHIFRAC FRAC341PFRAC341P FRAC13PRIGHT  TEXTIF   XX110PTARGMINPHILEFT LVARTHETA1PHI FRAC FRAC23PFRAC141P  FRAC23P LVARTHETA0PHIFRAC FRAC141PFRAC141P FRAC23PRIGHT  TEXTIF  XX0ENDCASESLABELPHITAUENDALIGNFOR PHIIN0 1  CONSIDER THE CASE WHEN X  X1  THEN THE RISK FUNCTION IS EITHERBEGINEQUATION 10FRAC FRAC13PFRAC341P  FRAC13P QQUADTEXTOR QQUAD 5 FRAC FRAC341PFRAC341P FRAC13PLABELEQLOSS1ENDEQUATIONDEPENDING UPON WHETHER PHI  0 OR PHI  1  A PLOTOF THESE TWO RISK FUNCTIONS IS SHOWN ON THE LEFT OF FIGUREREFFIGLOSS1  EQUATING THE TWO RISK FUNCTIONS IN REFEQLOSS1TO FIND THE POINT OF INTERSECTION WE FIND THATBEGINDISPLAYMATHPHIPX1  BEGINCASES0  TEXTIF  P LEQ FRAC91710PT1  TEXTIF  P  FRAC917ENDCASESENDDISPLAYMATHTHE RIGHT OF FIGURE REFFIGLOSS1 SIMILARLY SHOWS THE RISK FUNCTIONWHEN XX0  AGAIN THE THRESHOLD CAN BE FOUND AND THE DECISION RULEIN THIS CASE ISBEGINDISPLAYMATHPHIPX0  BEGINCASES0  TEXTIF   P LEQ FRAC31910PT1  TEXTIF   P  FRAC319ENDCASESENDDISPLAYMATHBEGINFIGUREHTBPBEGINCENTER BAYES1MEPSFIGFILEPICTUREDIRBAYES1EPSWIDTH09TEXTWIDTHENDCENTER  CAPTIONBAYES RISK FOR A DECISION  LABELFIGLOSS1ENDFIGUREWE MAY COMPUTE THE BAYES RISK FUNCTION AS FOLLOWS  IF 0 LEQ P FRAC319 THEN IT FOLLOWS THAT PHIPX  EQUIV 0 WILL BE THE BAYESRULE WHATEVER THE VALUE OF X  THE CORRESPONDING BAYES RISK IS0CDOT 1P  10 P  10 P  IF FRAC319 LEQ P LEQ FRAC917 THENPHIPX0  1 AND PHIPX1  0 IS THE BAYESDECISION FUNCTION AND THE CORRESPONDING RISK ISBEGINALIGNEDRP PHIP    P RVARTHETA1 PHIP  1P RVARTHETA0 PHIP 10PT    P  10 CDOT FRAC13  0 CDOTFRAC23  1P0 CDOT FRAC34  5CDOTFRAC1410PT    FRAC103P  FRAC541P  FRAC2512 P  FRAC54ENDALIGNEDIF FRAC917  P LEQ 1 THEN PHIPX EQUIV 1 IS THEBAYES RULE AND THE BAYES RISK IS 51P  BEGINFIGUREHTBUNITLENGTH 2INBEGINCENTERBEGINPICTURE1020PUT55PSFIGFILEDETESTPICTDIRWIERDPS HSCALE50VSCALE50HOFFSET0PUT55SPECIALPSFILETUSERSWYNNTEXCLASSEE513WIERDPS   HSCALE 50 VSCALE  50 HOFFSET 0PUT00MAKEBOX00PUT55MAKEBOX00PUT359MAKEBOX00RP PHIPPUT50MAKEBOX00PENDPICTUREENDCENTERCAPTIONBAYES ENVELOPE FUNCTIONLABELBAYESENVELOPE1ENDFIGUREENDEXAMPLESUBSECTIONDETECTION AND SUFFICIENCYLABELSECDETSUFFICIENTWE HAVE SEEN THAT FOR BINARY TESTS IN BOTH THE NEYMANPEARSON ANDBAYES DETECTORS THE DECISION FUNCTION CAN BE EXPRESSED IN TERMS OFTHE LIKELIHOOD RATIO ELLXBF  FRACFXBFXBF THETA1FXBFXBF THETA0IF TBFXBF IS SUFFICIENT FOR THETA SO THATFXBFXBF THETAI  BTBFXBFTHETAIAXBF THE LIKELIHOOD RATIO BECOMESBEGINEQUATION ELLXBF  FRACBTBFXBFTHETA1BTBFXBFTHETA0LABELEQXTENDEQUATIONTHE RATIO IN REFEQXT IS EQUIVALENT TO THE RATIO OF DENSITYFUNCTIONS ELLXBF FRACFTBFTBFTHETA1FTBFTBFTHETA0WHICH IS NATURALLY DENOTED AS ELLTBF  ON THIS BASIS THEDECISION FUNCTION FOR A BINARY TEST BECOMES A FUNCTION ONLY OF THESUFFICIENT STATISTIC NOT OF THE ENTIRE SET OF OBSERVED DATA PHITBF  BEGINCASES  1  ELLTBF  NU   GAMMA  ELLTBF  NU   0  ELLTBF  NUENDCASESFOR SOME SUITABLE CHOSEN THRESHOLD NU  IN THE NEYMANPEARSON TESTTHE THRESHOLD IS SELECTED TO PRODUCE THE DESIRED SIZE ALPHA FOR THETEST  IN THE BAYES TEST IT IS SELECTED FOR MINIMUM RISKBEGINEXAMPLE  SUPPOSE XI SIM PCLAMBDA THAT IS XI IS POISSON  DISTRIBUTED FOR I12LDOTSN  WE DESIRE TO TEST H0MC  LAMBDA  LAMBDA0 VERSUS H1MC LAMBDA  LAMBDA1 WHERE  LAMBDA1  LAMBDA0  THE RANDOM VARIABLE T  SUMI1N XIIS SUFFICIENT FOR LAMBDA  T IS POISSON DISTRIBUTED T SIM PCNLAMBDAFOR A GIVEN THRESHOLD NU THE PROBABILITY OF FALSE ALARM IS ALPHA  SUMK  NUINFTY FRACENLAMBDA0  NLAMBDA0KK  GAMMA  FRACENLAMBDA0NLAMBDA0NUNUENDEXAMPLESUBSECTIONSUMMARY OF BINARY DECISION PROBLEMSTHE FOLLOWING OBSERVATIONS SUMMARIZE THE RESULTS WE HAVE OBTAINED FORTHE BINARY DECISION PROBLEMBEGINENUMERATEITEM USING EITHER NEYMANPEARSON OR A BAYES CRITERION WE SEE THAT  THE OPTIMUM TEST IS A LIKELIHOOD RATIO TEST  IF THE DISTRIBUTION  IS NOT CONTINUOUS A RANDOMIZED TEST MAY BE NECESSARY FOR THE  NEYMANPEARSON DECISION  THUS REGARDLESS OF THE DIMENSIONALITY OF  THE OBSERVATION SPACE THE TEST CONSISTS OF COMPARING A SCALAR  VARIABLE ELLXBF WITH A THRESHOLDITEM IN MANY CASES CONSTRUCTION OF THE LIKELIHOOD RATIO TEST CAN BESIMPLIFIED BY USING A SUFFICIENT STATISTIC ITEM A COMPLETE DESCRIPTION OF THE LIKELIHOOD RATIO TEST PERFORMANCECAN BE OBTAINED BY PLOTTING THE CONDITIONAL PROBABILITIES PD VERSUSPFA AS THE THRESHOLD IS VARIED  THE RESULTING ROC CURVE CAN BEUSED TO CALCULATE EITHER THE POWER FOR A GIVEN SIZE AND VICE VERSAOR THE BAYES RISK THE PROBABILITY OF ERRORITEM THE MINIMAX CRITERION IS A SPECIAL CASE OF A BAYES RULE WITH ALEAST FAVORABLE PRIORITEM A BAYES RULE MINIMIZES THE EXPECTED LOSS UNDER THE POSTERIOR  DISTRIBUTIONENDENUMERATESECTIONSOME MARY PROBLEMSLABELSECMARYINDEXMARY DETECTIONMARY DETECTIONUP TO THIS POINT IN THE CHAPTER ALL OF THE TESTS HAVE BEEN BINARYWE NOW GENERALIZE TO MARY TESTS  SUPPOSE THERE ARE MGEQ 2POSSIBLE OUTCOMES EACH OF WHICH CORRESPONDS TO ONE OF THE MHYPOTHESES  WE OBSERVE THE OUTPUT AND ARE REQUIRED TO DECIDE WHICHSOURCE WAS USED TO GENERATE IT  PUT IN THE LIGHT OF THE RADARDETECTION PROBLEM WE DISCUSSED EARLIER SUPPOSE THERE ARE MDIFFERENT TARGET POSSIBILITIES AND WE NOT ONLY HAVE TO DETECT THEPRESENCE OF A TARGET BUT TO CLASSIFY IT AS WELL  FOR EXAMPLE WE MAYBE REQUIRED TO CHOOSE BETWEEN THREE ALTERNATIVES H0MC MBOXNO  TARGET PRESENT H1MC MBOXTARGET IS PRESENT AND HOSTILEH2MC MBOXTARGET IS PRESENT AND FRIENDLY  ANOTHER COMMONEXAMPLE IS DIGITAL COMMUNICATION IN WHICH THERE ARE MORE THAN 2POINTS IN THE SIGNAL CONSTELLATIONFORMALLY THE PARAMETER SPACE THETA IS OF THE FORM THETA  VARTHETA0 VARTHETA1 LDOTS VARTHETAM1  LET H0MC THETA  VARTHETA0 H1MC THETA  VARTHETA1 LDOTSHM1MC THETA VARTHETAM1 DENOTE THE M HYPOTHESES TOTEST  WE WILL EMPLOY THE BAYES CRITERION TO ADDRESS THIS PROBLEM ANDASSUME THAT PBF  P0 LDOTS PM1T IS THE CORRESPONDINGEM A PRIORI PROBABILITY VECTOR WHERE PJ  FTHETAVARTHETAJTHAT IS PBF REPRESENTSFTHETA  WE WILL DENOTE THE COST OF EACH COURSE OF ACTION ASLJI WHERE THE SUBSCRIPT I SIGNIFIES THAT THE ITH HYPOTHESISIS CHOSEN AND THE SUBSCRIPT J SIGNIFIES THAT THE JTH HYPOTHESISIS TRUE  LJI IS THE COST OF CHOOSING HI WHEN HJ IS TRUEWE OBSERVE A RANDOM VARIABLE XBF TAKING VALUES IN XC SUBSETRBBKWE WISH TO GENERALIZE THE NOTION OF A THRESHOLD TEST THAT WAS SOUSEFUL FOR THE BINARY CASE  OUR APPROACH WILL BE TO COMPUTE THEPOSTERIOR CONDITIONAL EXPECTED LOSS FOR XBFXBFTHE NATURAL GENERALIZATION OF THE BINARY CASE IS TOPARTITION THE OBSERVATION SPACE INTO M DISJOINT REGIONS S0LDOTS SM1 THAT IS XC  S0 CUP CDOTS CUP SM1 ANDTO INVOKE A DECISION RULE OF THE FORMBEGINEQUATIONLABELBAYESRULEPHIXBF  N QUAD MBOXIF  XBF IN SN  N0 LDOTS M1ENDEQUATIONTHE LOSS FUNCTION THEN ASSUMES THE FORMBEGINDISPLAYMATHLVARTHETAJ PHIXBF  SUMI0M1 LJIISIXBFENDDISPLAYMATHWHERE ISIXBF IS THE INDICATOR FUNCTION EQUAL TO 1 IF XBFIN SI  FROM REFREVERSE2 THE BAYES RISK ISBEGINALIGNEDRFTHETAPHI  RPBF PHI   INTXC LEFTSUMJ0M1            LVARTHETAJ PHIXBFFTHETAXBFVARTHETAJXBF   RIGHTFXBFXBFDXBF10PT    INTXC LEFT SUMJ0M1 SUMI0M1 LJIISIXBF FTHETAXBFVARTHETAIXBF   RIGHTFXBFXBFDXBF ENDALIGNEDAND WE MAY MINIMIZE THIS QUANTITY BY MINIMIZING THE QUANTITY IN BRACESFOR EACH XBF  IT SUFFICES TO MINIMIZE THE POSTERIOR CONDITIONALEXPECTED LOSS BEGINEQUATIONLABELNEWMINRPRIMEPBF PHI  SUMJ0M1 SUMI0M1LJIISIXBF FTHETAXBFVARTHETAIXBF ENDEQUATIONTHE PROBLEM REDUCES TO DETERMINING THE SETS SI I0 LDOTSM1 THAT RESULT IN THE MINIMIZATION OF RPRIMEFROM BAYES RULE WE HAVEBEGINEQUATION  LABELEQBAYESCONDFTHETAXBFVARTHETAJXBF FRACFXBFTHETAXBFVARTHETAJFTHETAVARTHETAJFXBFXBF ENDEQUATIONWE WILL DENOTE THE PRIOR PROBABILITIES FTHETAVARTHETAJ AS PJ  FTHETAVARTHETAJTHEN REFEQBAYESCOND BECOMES FTHETAXBFVARTHETAJXBF FRACFXBFTHETAXBFVARTHETAJPJFXBFXBFWHICH WHEN SUBSTITUTED INTO REFNEWMIN YIELDSBEGINDISPLAYMATHRPRIMEPBF PHI  SUMJ0M1 SUMI0M1 LJIISIXBF FRACFXBFTHETAXBFVARTHETAJPJFXBFXBFENDDISPLAYMATHWE NOW MAKE AN IMPORTANT OBSERVATION GIVEN XBFXBF WE CANMINIMIZE THE POSTERIOR CONDITIONAL EXPECTED LOSS BY MINIMIZINGBEGINDISPLAYMATHSUMJ0M1 SUMI0M1 LJIISIX FXTHETAXVARTHETAJ PJENDDISPLAYMATHTHAT IS FXBFXBF IS SIMPLY A SCALE FACTOR FOR THIS MINIMIZATIONPROBLEM SINCE XBF IS ASSUMED TO BE FIXED  SINCEBEGINDISPLAYMATHSUMJ0M1 SUMI0M1 LJIISIXBF FXBFTHETAXBFVARTHETAJPJ  SUMI0M1 ISIXBFSUMI0M1 LJIFXBFTHETAXBFVARTHETAJPJENDDISPLAYMATHWE MAY NOW ASCERTAIN THE STRUCTURE OF THE SETS SI THAT RESULT INTHE BAYES DECISION RULE PHIXBF GIVEN BY REFBAYESRULEBEGINEQUATIONLABELEQBAYESETSK   XBFIN XCMC SUMJ0M1LJKFXBFTHETAXBFVARTHETAJPJ LEQ SUMJ0M1LJIFXBFTHETAXVARTHETAJPJ QUAD FORALL I NOT  KENDEQUATIONTHE DECISION DETERMINED BY THE SETS IN REFEQBAYESET CAN BEWRITTEN ANOTHER WAY  WE SET OUR ESTIMATE VARTHETAHAT EQUAL TO THATVALUE VARTHETAK WHICH MINIMIZES SUMJ0M1LJKFXBFTHETAXBFVARTHETAJPJTHAT IS VARTHETAHAT  VARTHETAK IFBEGINEQUATIONK  ARGMINK SUMJ0M1LJKFXBFTHETAXBFVARTHETAJ PJLABELEQVARTHETAHATENDEQUATIONTHE GENERAL STRUCTURE OF THESE DECISION REGIONS IS RATHER MESSY TOVISUALIZE AND LENGTHY TO COMPUTE BUT WE CAN LEARN ALMOST ALL THERE ISTO KNOW ABOUT THIS PROBLEM BY SIMPLIFYING IT A BIT  IN THE IMPORTANTCASE OF DIGITAL COMMUNICATION IT IS APPROPRIATE TO CONSIDER ADECISION COST WHICH DEPENDS ONLY UPON INCORRECT DECISIONS  THUSWE SETBEGINALIGNEDLII    0LJI    1 INOT  JENDALIGNEDTHEN  SK   XBF IN XCMC SUMJ J NEQ KFXBFTHETAXBFVARTHETAJ PJ LEQ SUMJ J NEQ I FXBFTHETAXBFVARTHETAJ PJ QUAD FORALL I NEQ KTHIS IS EQUIVALENT TO SK   XBF IN XCMC FXBFTHETAXBFVARTHETAK PKGEQ MAXI NEQ K FXBFTHETAXBFVARTHETAI PISTATED IN TERMS OF DECISIONS THE BEST DECISION IS VARTHETAHAT VARTHETAK WHEREBEGINEQUATIONK  ARGMAXK FXBFTHETAXBFVARTHETAK PKLABELEQMAPENDEQUATIONSTATED IN WORDS THE BEST BAYES DECISION IS THAT WHICH EM  MAXIMIZES THE POSTERIOR PROBABILITYFXBFTHETAXBFVARTHETAK PK  SUCH A TEST IS SOMETIMES  CALLED THE EM MAXIMUM EM A POSTERIORI TEST OR THE MAP TESTINDEXMAXIMUM EM A POSTERIORI MAP DETECTIONBEGINEXAMPLE DETECTION IN GAUSSIAN NOISEGIVEN BEGINALIGNEDH0MC XBF SIM NCMBF0SIGMA2I H1MC XBF SIM NCMBF1SIGMA2I H2MC XBF SIM NCMBF2SIGMA2IENDALIGNEDWITH PRIOR PROBABILITIES P0P1P2  WE WILL CONSIDER THEBOUNDARIES BETWEEN DECISION REGIONS  LET IS CONSIDER THE BOUNDARYBETWEEN THE DECISION REGION FOR H0 AND H1  FORMING THELIKELIHOOD RATIO WE HAVE FRACFXTHETAXBF VARTHETA1FXTHETAXBF VARTHETA0UNDERSETH0OVERSETH1TWOCOMP FRACP0P1AFTER SOME SIMPLIFICATION WE FIND THE TEST MBF1  MBF0T XBFXBF0UNDERSETH0OVERSETH1TWOCOMPLOGFRACP0P1WHERE XBF0  FRAC12MBF1MBF0  THE BOUNDARY BETWEEN THEDECISION REGIONS OCCURS WHERE BEGINEQUATIONMBF1  MBF0T XBFXBF0  LOGFRACP0P1LABELEQSEPPLANE1ENDEQUATIONEQUATION REFEQSEPPLANE1 IS THE EQUATION OF A PLANE ORTHOGONAL TOMBF1  MBF0  IN THE COMPARISON BETWEEN H0 AND H1 IFXBF FALLS ON SIDE OF THE PLANE NEAREST MBF0 THEN H0 ISSELECTED AND IF XBF FALLS ON THE SIDE OF THE PLANE NEARESTMBF1 THEN H1 IS SELECTED  WE CAN GET A BETTER UNDERSTANDINGOF THE SEPARATING PLANE BY LETTING DBF  LEFTLOGFRACP0P1RIGHTFRACMBF1MBF0MBF1MBF02 SO THAT MBF1MBF0T DBF  LOG P0P1  THEN THE EQUATIONFOR THE SEPARATING PLANE OF REFEQSEPPLANE1 CAN BE WRITTENBEGINEQUATION  LABELEQSEPPLANE2  MBF1MBF0TXBF XBFTILDE  0ENDEQUATIONWHERE XBFTILDE  XBF0  DBF  EQUATION REFEQSEPPLANE2REPRESENTS A PLANE ORTHOGONAL TO THE VECTOR MBF1MBF0 BETWEENTHE MEANS THAT PASSES THROUGH THE POINT XBFTILDE   SEE FIGUREREFFIGDETEST4BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRDETEST4    CAPTIONGEOMETRY OF THE DECISION SPACE FOR MULTIVARIATE GAUSSIAN    DETECTION    LABELFIGDETEST4  ENDCENTERENDFIGURETHE SITUATION IS EVEN MORE CLEAR WHEN P0P1  THEN XBFTILDE XBF0 THE POINT MIDWAY BETWEEN THE MEANS SO THE SEPARATING PLANELIES MIDWAY BETWEEN MBF1 AND MBF0SIMILAR SEPARATING PLANES CAN BE FOUND BETWEEN EACH PAIR OF MEANSWHICH DIVIDE SPACE UP INTO DECISION REGIONS  ENDEXAMPLEBEGINEXAMPLE  NOW CONSIDER THE 4ARY DETECTION PROBLEM WITH THE MEANS MBF0  BEGINBMATRIX4  4 ENDBMATRIX QQUADMBF1  BEGINBMATRIX2  2 ENDBMATRIX QQUADMBF2  BEGINBMATRIX2  5 ENDBMATRIX QQUADMBF3  BEGINBMATRIX1 1 ENDBMATRIXFIGURE REFFIGBOUNDREG ILLUSTRATES THE DECISION REGIONS FOR THISPROBLEM  THE DASHED LINES ARE THE LINES BETWEEN THE MEANS  THE HEAVYSOLID LINES INDICATE THE BOUNDARIES OF THE DECISION REGIONS AND THELIGHT SOLID LINES ARE PORTIONS OF THE DECISION LINES WHICH DO NOTCONTRIBUTE TO THE DECISION BOUNDARIES  IN FIGUREREFFIGBOUNDREGA EACH SELECTION IS EQUALLY PROBABLE AND INFIGURE REFFIGBOUNDREGB MBF0 OCCURS WITH PROBABILITY 099WITH THE REMAINING PROBABILITY SPLIT EQUALLY BETWEEN THE OTHERS  THEEFFECT OF THIS CHANGE IN PROBABILITY IS TO MAKE THE DECISION REGIONFOR H0 LARGERBEGINFIGUREHTBP  CENTERING BAYES4MMBOXSUBFIGUREEQUAL  PROBABILITIESEPSFIGFILEPICTUREDIRBAYESBOUND1EPS      WIDTH045TEXTWIDTHQUADSUBFIGUREUNEQUAL PROBABILITIESEPSFIGFILEPICTUREDIRBAYESBOUND2EPS       WIDTH045TEXTWIDTH  CAPTIONDECISION BOUNDARIES FOR A 4WAY DECISION PROBLEM  LABELFIGBOUNDREGENDFIGUREENDEXAMPLETHE BAYES RISK OR PROBABILITY OF ERROR FOR THE MARY CLASSIFIERCAN BE IN TERMS OF THE PROBABILITY OF MAKING A EM CORRECT DECISION PEC  1PCCWHERE PCC  SUMJ1M PCCVARTHETAJ PJWHERE PCCVARTHETAJ INTSJFXBFTHETAXBFVARTHETAJDXBFIN THE GENERAL CASE IT CAN BE DIFFICULT TO COMPUTE THESE PROBABILITIESEXACTLY  IN SOME SPECIAL CASES  SUCH AS AS DECISION REGIONS WITHRECTANGULAR BOUNDARIES WITH GAUSSIAN OBSERVATIONS  THE COMPUTATIONIS STRAIGHTFORWARD  A RECENT RESULT CITESIMON1998 PROVIDESEXTENSION OF PROBABILITY COMPUTATIONS TO MORE COMPLICATED POLYGONALREGIONSBEGINEXERCISESITEM FOR SOME DISTRIBUTIONS OF MEANS THE PROBABILITY OF  CLASSIFICATION ERROR IS STRAIGHTFORWARD TO COMPUTE  FOR THE SET OF  POINTS REPRESENTING MEANS SHOWN IN FIGURE REFFIGDETERRPROB  COMPUTE THE PROBABILITY OF ERROR ASSUMING THAT EACH HYPOTHESIS  OCCURS WITH EQUAL PROBABILITY AND THAT THE NOISE IS  NC0SIGMA2I  THESE SETS OF MEANS COULD REPRESENT SIGNAL  CONSTELLATIONS IN A DIGITAL COMMUNICATIONS SETTING  IN EACH  CONSTELLATION THE DISTANCE BETWEEN NEAREST SIGNAL POINTS IS D    ALSO COMPUTE TOTAL ENERGY E OF THE SIGNAL CONSTELLATION AS A  FUNCTION OF D  IF THE MEANS ARE AT MBFI THEN THE AVERAGE  ENERGY IS E  FRAC1M SUMI1M MBFI2FOR EXAMPLE FOR THE 4PSK CONSTELLATION E  FRAC144LEFTD22  D22RIGHT  D22FOR EACH CONSTELLATION EXPRESS THE PROBABILITY OF ERROR AS A FUNCTIONOF EITEM LET M  2K WHERE K IS AN EVEN NUMBER  DETERMINE THE  PROBABILITY OF ERROR FOR A SIGNAL CONSTELLATION WITH M POINTS  ARRANGED IN A SQUARE CENTERED AT THE ORIGIN WITH MINIMUM DISTANCE  BETWEEN POINTS EQUAL TO D AND NOISE VARIANCE SIGMA2  EXPRESS  THIS AS A FUNCTION OF E THE AVERAGE ENERGY FOR THE CONSTELLATIONINDEXQUADRATUREAMPLITUDE MODULATION QAMINDEXPHASESHIFT KEYING PSK  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODESUBFIGURE4PSKINPUTPICTUREDIRDETPROBA QQUADSUBFIGURE8QAMINPUTPICTUREDIRDETPROBB SUBFIGURE16QAMINPUTPICTUREDIRDETPROBC      CAPTIONSOME SIGNAL CONSTELLATIONS      LABELFIGDETPROB    ENDCENTER  ENDFIGUREENDEXERCISESSECTIONMAXIMUMLIKELIHOOD DETECTIONLABELSECMLDETTHE DECISION CRITERION SPECIFIED IN REFEQMAP REQUIRES KNOWLEDGEOF THE FUNCTIONS FXBFTHETAXBFVARTHETAK AND THE PRIORPROBABILITIES  IN MANY CIRCUMSTANCES THE PRIOR PROBABILITIES AREALL EQUAL P0P1CDOTS  PM1  FRAC1MOR LACKING INFORMATION TO THE CONTRARY THEY ARE ASSUMED TO BEEQUAL  A DECISION MADE ON THIS BASIS CAN BE STATED AS SETVARTHETAHAT  VARTHETAK IF  K  ARGMAXK FXBFTHETAXBFVARTHETAKA DECISION MADE ON THE BASIS OF THIS CRITERION IS SAID TO BE A EM  MAXIMUMLIKELIHOOD ESTIMATE AND THE CONDITIONAL PROBABILITYFXBFTHETAXBFVARTHETA IS SAID TO BE THE EM  LIKELIHOOD FUNCTION BEING VIEWED USUALLY AS A FUNCTION OF  VARTHETAINDEXMAXIMUM LIKELIHOOD ESTIMATIONINDEXLIKELIHOOD FUNCTIONSECTIONAPPROXIMATIONS TO THE PERFORMANCE THE UNION BOUNDLABELSECUBAS HAS BEEN OBSERVED OBTAINING EXACT EXPRESSIONS FOR THE PROBABILITYOF ERROR FOR MARY DETECTION IN GAUSSIAN NOISE CAN BE DIFFICULTHOWEVER IT IS STRAIGHTFORWARD TO OBTAIN AN EM UPPER BOUND ON THEPROBABILITY OF ERROR USING WHAT IS KNOWN AS THE UNION BOUNDCONSIDER THE PROBLEM OF COMPUTING THE PROBABILITY OF ERROR FOR THEUNION OF TWO EVENTS A AND B  THIS PROBABILITY CAN BE EXPRESSED AS PA CUP B  PA  PB  PA CAP BWHERE THE TERM SUBTRACTED OFF PREVENTS THE EVENT IN THE INTERSECTIONFROM BEING COUNTED TWICE SEE FIGURE REFFIGUNION1BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRUNION1    CAPTIONVENN DIAGRAM FOR THE UNION OF TWO SETS    LABELFIGUNION1  ENDCENTERENDFIGURESINCE EVERY PROBABILITY GEQ 0 WE MUST HAVE PA CUP B LEQ PA  PBBY THIS BOUND THE PROBABILITY OF A COMPLICATED EVENT A CUP B ISBOUNDED BY THE PROBABILITIES OF MORE ELEMENTARY EVENTS A AND BNOW CONSIDER THE PROBLEM OF FINDING THE PROBABILITY OF ERROR FOR A PSKSIGNAL CONSTELLATION SUCH AS THAT SHOWN IN FIGURE REFFIGPSK2ASSUME ALL SIGNALS ARE SENT WITH EQUAL PROBABILITY  SUPPOSE THATMBF0 IS SENT  THEN THE RECEIVED SIGNAL WILL BE CORRECTLY DETECTEDONLY IF THE RECEIVED SIGNAL FALLS IN THE WHITE WEDGE  LOOKED AT FROMANOTHER POINT OF VIEW THE SIGNAL WILL BE DETECTED IF EITHER EVENT AOCCURS WHICH IS THE EVENT THAT THE RECEIVED SIGNAL LIES ABOVE THELINE L1 OR EVENT B OCCURS WHICH IS THE EVENT THAT THE RECEIVEDSIGNAL LIES BELOW THE LINE L2  IT IS ALSO POSSIBLE FOR BOTH EVENTSTO OCCUR THE DARKLY SHADED WEDGE  USING THE UNION BOUND WE HAVE PECMBF0  PA CUP B LEQ PA  PBBUT PA IS THE EM BINARY PROBABILITY OF ERROR BETWEEN THESIGNALS MBF0 AND MBF1 AND PB IS THE BINARY PROBABILITY OFERROR BETWEEN THE SIGNALS MBF0 AND MBF7 SO THAT PA  PB  QD2SIGMAWHERE D IS THE DISTANCE BETWEEN ADJACENT SIGNALS IN THE PSKCONSTELLATION  THUSBEGINEQUATION PEC LEQ 2 QD2SIGMALABELEQPSDKPROBENDEQUATIONAS THE SNR INCREASES THE PROBABILITY OF FALLING IN THE DARKLY SHADEDWEDGE BECOMES SMALLER AND THE BOUND REFEQPSDKPROB BECOMESINCREASINGLY TIGHTBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIRPSK2    CAPTIONBOUND ON THE PROBABILITY OF ERROR FOR PSK SIGNALING    LABELFIGPSK2  ENDCENTERENDFIGUREBEGINEXERCISES  ITEM IN AN NDIMENSIONAL ORTHOGONAL DETECTION PROBLEM THERE ARE    N HYPOTHESES HIMC XBF NCMBFISIGMA2I WHERE MBFI PERP MBFJ QQUAD I NEQ JDETERMINE A BOUND ON THE PROBABILITY OF CORRECT DETECTION PCCUSING THE UNION BOUNDITEM COMPUTING EXERCISE SIGNAL SPACE SIMULATION  IN THIS EXERCISE  YOU SIMULATE SEVERAL DIFFERENT DIGITAL COMMUNICATIONS SIGNAL  CONSTELLATIONS AND THEIR DETECTION  SUPPOSE THAT AN MARY  TRANSMISSION SCHEME IS TO BE SIMULATED WHERE M2K  THE  FOLLOWING IS THE GENERAL ALGORITHM TO ESTIMATE THE PROBABILITY OF  ERROR SMALLBEGINPROGTABSQUAD  QUAD  QUAD  QUAD QUAD  QUAD  QUAD KILLGENERATE K RANDOM BITS MAP THE BITS INTO THE MARY CONSTELLATION TO PRODUCE THE TRANSMITTEDSIGNAL SBF  THIS IS ONE SYMBOL GENERATE A GAUSSIAN RANDOM NUMBER NOISE WITH VARIANCE SIGMA2 N02 IN EACH SIGNAL COMPONENT DIRECTION ADD THE NOISE TO THE SIGNAL CONSTELLATION POINT RBF  SBF  NBF PERFORM A DETECTION ON THE RECEIVED SIGNAL RBF MAP THE DETECTED POINT XBFHAT BACK TO BITS COMPARE THE DETECTED BITS WITH THE TRANSMITTED BITS AND COUNT BITS INERRORENDPROGTABSREPEAT THIS UNTIL MANY PREFERABLY AT LEAST 100 BITS IN ERROR HAVEBEEN COUNTED  THE ESTIMATED EM BIT ERROR PROBABILITY IS PB APPROX FRACTEXTNUMBER OF BITS IN ERRORTEXTTOTAL NUMBER OF BITS GENERATEDTHE ESTIMATED EM SYMBOL ERROR PROBABILITY IS PE APPROX FRACTEXTNUMBER OF SYMBOLS IN ERRORTEXTTOTAL NUMBER OF SYMBOLS GENERATEDIN GENERAL PB NEQ PE SINCE A SYMBOL IN ERROR MAY ACTUALLY HAVESEVERAL BITS IN ERRORTHE PROCESS ABOVE SHOULD BE REPEATED FOR VALUES OF SNR EBN0 INTHE RANGE FROM 0 TO 10 DBTHE ASSIGNMENTBEGINENUMERATEITEM PLOT THE THEORETICAL PROBABILITY OF ERROR FOR BPSK DETECTION WITH EQUAL  PROBABILITIES AS A FUNCTION OF SNR IN DB EM VS PB ON A LOG  SCALE  YOUR PLOT SHOULD LOOK LIKE FIGURE REFFIGBPSKITEM BY SIMULATION ESTIMATE THE PROBABILITY OF ERROR FOR BPSK  TRANSMISSION USING THE METHOD OUTLINED ABOVE  PLOT THE RESULTS ON  THE SAME AXES AS THE THEORETICAL PLOT  THEY SHOULD BE VERY  SIMILARITEM PLOT THE THEORETICAL PROBABILITY OF EM SYMBOL ERROR FOR QPSK  SIMULATE USING QPSK AND PLOT THE ESTIMATED SYMBOL ERROR PROBABILITYITEM PLOT THE UPPER BOUND FOR THE PROBABILITY OF 8PSK    SIMULATE USING 8PSK AND PLOT THE ESTIMATED ERROR PROBABILITYITEM REPEAT PARTS A AND B USING UNEQUAL PRIOR PROBABILITIES PMBF0  08 QQUAD PMBF1  02ITEM COMPARE THE THEORETICAL AND EXPERIMENTAL PLOTS AND COMMENTENDENUMERATEENDEXERCISESINPUTDETESTDIRINVARIANTINPUTDETESTDIRCONTDECSECTIONMINIMAX BAYES DECISIONSLABELSECMINIMAXBAYESUP TO THIS POINT WE HAVE ASSUMED EITHER NOTHING ABOUT THE PRIORPROBABILITIES FOCUSING ON THE CONDITIONAL PROBABILITIES OF ERROR ASIN THE NEYMANPEARSON TEST OR ON THE MINIMAL RISK BASED ON SOMEASSUMED PRIOR PROBABILITIES AS IN THE BAYES DECISION  IN THISSECTION WE RETURN AGAIN TO THE BAYES DECISION THEORY BUT ADDRESS THEPROBLEM OF FINDING DECISION FUNCTIONS WHEN THE PRIOR PROBABILITIES ARENOT KNOWN  THIS WILL LEAD US TO THE MINIMAX BAYES DECISION PROCEDUREALONG THE WAY WE WILL NEED TO EMPLOY THE THEORY RANDOMIZED RULESSOME UNDERSTANDING OF THE MINIMAX PROBLEM CAN BE OBTAINED BY MEANS OFTHE BAYES ENVELOPE FUNCTION  WE WILL THEN INTRODUCE THE MINIMAXPRINCIPLE IN THE CONTEXT OF MULTIPLE HYPOTHESIS TESTINGSUBSECTIONBAYES ENVELOPE FUNCTIONSUBSECTIONRANDOMIZED DECISION RULESSUPPOSE RATHER THAN INVOKING A RULE THAT ASSIGNS A SPECIFIC ACTIONDELTA FOR A GIVEN OBSERVATION X WE INSTEAD INVOKE A RULE THATATTACHES A SPECIFIC PROBABILITY DISTRIBUTION TO THE ACTIONS AND THEDECISIONMAKER THEN CHOOSES ITS ACTION BY SAMPLING THE ACTION SPACEACCORDING TO THAT DISTRIBUTION  FOR EXAMPLE LET DELTA0 ANDDELTA1 BE TWO CANDIDATE ACTIONS AND LET PHI BE A RULE THATYIELDS FOR EACH X A PROBABILITY PI SUCH THAT THEDECISIONMAKER CHOOSES ACTION DELTA1 WITH PROBABILITY PI ANDCHOOSES ACTION DELTA0 WITH PROBABILITY 1PI  INDEED IT ISEASY TO SEE THAT ANY FINITE CONVEX COMBINATION OF ACTIONS CORRESPONDSTO A RANDOMIZED RULE  IN FACT EVEN THE DETERMINISTIC RULES WE HAVESEEN IN BAYESIAN DETECTION EXAMPLES UP TO THIS POINT CAN BE VIEWED ASDEGENERATE RANDOMIZED RULE WHERE WE HAVE SET PI  1 FOR SOMEACTION DELTA  LET D DENOTE THE SET OF ALL RANDOMIZED DECISION RULES  LET PHIIN D AND PHIPRIME IN D BE TWO RULES AND LET PHIPI BETHE RANDOMIZED DECISION RULE CORRESPONDING TO USING RULE PHI WITHPROBABILITY PI WHERE PIIN 0 1 AND USING RULEPHIPRIME WITH PROBABILITY 1PI  THEN PHIPI IN DANDFOR BINARY HYPOTHESIS TESTING WE CAN INTRODUCE THE BAYES ENVELOPEFUNCTION  SUPPOSE RATHER THAN INVOKING A RULE THAT ASSIGNS ASPECIFIC ACTION DELTA FOR A GIVEN OBSERVATION X WE INSTEADINVOKE A RANDOMIZED RULE  LET PHI IN D AND PHIPRIME IN DBE TWO NONRANDOMIZED RULES AND LET VARPHIPIIN D BE THERANDOMIZED DECISION RULE CORRESPONDING TO USING RULE PHI WITHPROBABILITY PI WHERE PIIN 0 1 AND USING RULEPHIPRIME WITH PROBABILITY 1PI  TO COMPUTE THE RISKFUNCTION CORRESPONDING TO THIS RANDOMIZED RULE WE MUST TAKE THEEXPECTATION WITH RESPECT TO THE RULE ITSELF IN ADDITION TO TAKING THEEXPECTATION WITH RESPECT TO X  THIS YIELDS SEE REFEQNRANDOMRISKDBEGINDISPLAYMATHRVARTHETA VARPHIPI   PIRVARTHETA PHI  1PIRVARTHETAPHIPRIMEENDDISPLAYMATHBEGINDEFINITIONINDEXBAYES ENVELOPE FUNCTION  THE FUNCTION RHOCDOT DEFINED BYBEGINEQUATIONLABELBAYESENVELOPERHOP  RP VARPHIP  MINVARPHIIN DRP PHIENDEQUATIONIS CALLED THE BF BAYES ENVELOPE FUNCTION  IT REPRESENTS THE MINIMALGLOBAL EXPECTED LOSS ATTAINABLE BY ANY DECISION FUNCTION WHEN THETAIS A RANDOM VARIABLE WITH EM A PRIORI DISTRIBUTION PTHETA VARTHETA1  P AND PTHETA  VARTHETA0  1P  ENDDEFINITIONWE OBSERVE THAT FOR P 0 RHOP  0 AND ALSO FOR P  1RHOP  0  FURTHERMORE IT EASY TO SEE THAT RHOP MUST BECONCAVE DOWNWARD FOR IF IT WERE NOT WE COULD CONSTRUCT A RANDOMIZEDRULE THAT WOULD IMPROVE PERFORMANCE IN A MANNER ANALOGOUS TO THE WAYWE ANALYZED THE CONSTRUCTION OF A RANDOMIZED RULE IN THE ROC CURVECONTEXT  FIGURE REFBAYESENVELOPEPLOT IS AN EXAMPLE OF A BAYESENVELOPE FUNCTION THE PARABOLICALLY SHAPED CURVE IN THE FIGUREBEGINTHEOREM  EM CONCAVITY OF BAYES ENVELOPE FUNCTION  FOR ANY DISTRIBUTIONS  P1 AND P2 OF THETA AND FOR ANY NUMBER Q SUCH THAT 0 LEQ  Q LEQ 1BEGINDISPLAYMATHRHO QP1  1QP2 GEQ QRHO P1   1QRHO P2  ENDDISPLAYMATHENDTHEOREMBEGINPROOF  SINCE THE BAYES RISK DEFINED IN REFBAYESRISK IS LINEAR IN P  IT FOLLOWS THAT FOR ANY DECISION RULE PHIBEGINDISPLAYMATHR QP1  1QP2 PHI QR P1 PHI  1QR P2 PHI ENDDISPLAYMATHTO OBTAIN THE BAYES ENVELOPE WE MUST MINIMIZE THIS EXPRESSION OVERALL DECISION RULES PHI  BUT THE MINIMUM OF THE SUM OF TWOQUANTITIES CAN NEVER BE SMALLER THAN THE SUM OF THEIR INDIVIDUALMINIMA HENCEBEGINALIGNEDMINPHIR QP1  1QP2 PHI    MINPHIQR P1 PHI  1QR P2 PHI10PT   GEQ MINPHI QR P1 PHI  MINPHI 1QR P2 PHIENDALIGNEDENDPROOFNOW CONSIDER THE FUNCTION DEFINED BYBEGINALIGNEDYPIP  PRVARTHETA1VARPHIPI  1PRVARTHETA0VARPHIPI  RPVARPHIPIENDALIGNEDAS A FUNCTION OF P YPIP IS A STRAIGHT LINE FROM Y0 RVARTHETA0VARPHIPI TO Y1  RVARTHETA1VARPHIPI  WE SEETHAT FOR EACH FIXED PI THE CURVE RHOP LIES ENTIRELY BELOWTHE STRAIGHT LINE YPIP  RP VARPHIPI  THE QUANTITYYPIP MAY BE REGARDED AS THE EXPECTED LOSS INCURRED BY ASSUMINGTHAT PTHETA  VARTHETA1  PI AND HENCE USES THE DECISION RULEVARPHIPI WHEN IN FACT PTHETA  VARTHETA1  P THE EXCESSOF YPIP OVER RHOP IS THE COST OF THE ERROR IN INCORRECTLYESTIMATING THE TRUE VALUE OF THE EM A PRIORI PROBABILITY P PTHETA VARTHETA1  SEE FIGURE REFBAYESENVELOPEPLOTTHE MINIMAX ESTIMATOR ADDRESSES THE QUESTION WHAT IF THE PRIORPROBABILITY P IS UNKNOWN  WHAT IS BEST DETECTOR RULE PHIPITHAT WE CAN USE TO MINIMIZE THE MAXIMUM COST OF THE DECISIONBEGINFIGUREHTBBEGINCENTERINPUTPICTUREDIRBAYESENVENDCENTERCAPTIONBAYES ENVELOPE FUNCTIONLABELBAYESENVELOPEPLOTENDFIGUREBEGINEXAMPLECONSIDER AGAIN THE PROBLEM OF DETECTING BEGINALIGNEDH0MC XBF SIM NCMBF0SIGMA2IH1MC XBF SIM NCMBF1SIGMA2IENDALIGNEDAS WE HAVE SEEN THE PROBABILITY OF ERROR IS PEC  1P QSIGMAD LOG1PP  D2SIGMA  PQD2SIGMA  SIGMAD LOG1PPSETTING L01  L101 THE BAYES RISK IS THE TOTAL PROBABILITYOF ERROR  FIGURE REFFIGERRORPROB ILLUSTRATES THECORRESPONDING BAYES ENVELOPE FUNCTIONS FOR VARIOUS VALUES OF D  MBF0MBF1BEGINFIGUREHTBP  BEGINCENTER BAYES2MEPSFIGFILEPICTUREDIRBAYES2EPSENDCENTERCAPTIONBAYES ENVELOPE FUNCTION NORMAL VARIABLES WITH UNEQUAL MEANS AND EQUAL VARIANCES  LABELFIGERRORPROBENDFIGUREBEGINFIGUREHTBUNITLENGTH 2INBEGINCENTERBEGINPICTURE1020PUT55PSFIGFILEDETESTPICTDIRTRYNPS HSCALE50VSCALE50HOFFSET0PUT55SPECIALPSFILETUSERSWYNNTEXCLASSEE513TRYNPS   HSCALE 50 VSCALE  50 HOFFSET 0PUT60MAKEBOX00PPUT369MAKEBOX00RPPHIPPUT64MAKEBOX00D3PUT68MAKEBOX00D2PUT615MAKEBOX00D1PUT00MAKEBOX00PUT55MAKEBOX00ENDPICTUREENDCENTERCAPTIONBAYES ENVELOPE FUNCTION NORMAL VARIABLES WITHUNEQUAL MEANS AND EQUAL VARIANCESLABELERRORPROBENDFIGUREENDEXAMPLEBEGINEXAMPLECONSIDER AGAIN THE DETECTION PROBLEM OF EXAMPLEREFEXMPOSTERIORBAYES WHERE THE RISK FUNCTION CORRESPONDING TOTHE OPTIMAL DECISION RULE WAS FOUND TO BE RPPHIP  BEGINCASES  10P  0 LEQ P  FRAC319   FRAC2512P  FRAC54  FRAC319 LEQ P  FRAC917   51P  FRAC917 LEQ P LEQ 1ENDCASESA PLOT OF THE BAYES ENVELOPE FUNCTION IS PROVIDED IN FIGUREREFBAYESENVELOPE1BEGINFIGUREHTBP BAYES1M  BEGINCENTER    EPSFIGFILEPICTUREDIRBAYES3EPS  ENDCENTER  CAPTIONBAYES ENVELOPE FUNCTION  LABELBAYESENVELOPE1ENDFIGUREENDEXAMPLEBEGINEXAMPLE LABELEXMBINCHAN5  CONSIDER THE BINARY CHANNEL OF EXAMPLE REFEXMBINCHAN5 AND  ASSUME THAT LAMBDA0  14 AND LAMBDA1  13  THE BAYES RISK  FUNCTIONS FOR EACH DECISION FUNCTION ARE BEGINALIGNEDRPPHI1  P RPPHI2  1PLAMBDA0  PLAMBDA1 RPPHI3  1P1LAMBDA0  P1LAMBDA1 RPPHI4  1PENDALIGNEDFIGURE REFFIGBINENV SHOWS THE BAYES RISK FUNCTION RPPHIIFOR EACH OF THE POSSIBLE DECISION FUNCTIONS  ALSO SHOWN IN THEDARKER LINE IS THE MINIMUM BAYES RISK FUNCTION  THE BAYESENVELOPE  BEGINFIGUREHTBP    BEGINCENTER      LEAVEVMODE      EPSFIGFILEPICTUREDIRBINCHAN1EPS      CAPTIONBAYES ENVELOPE FOR BINARY CHANNEL BINCHAN1M      LABELFIGBINENV    ENDCENTER  ENDFIGUREENDEXAMPLESUBSECTIONMINIMAX RULESAN INTERESTING APPROACH TO DECISIONMAKING IS TO CONSIDER ORDERINGDECISION RULES ACCORDING TO THE WORST THAT COULD HAPPEN  CONSIDER THE VALUE P  PIM ON THE BAYES ENVELOPE PLOT GIVENIN FIGURE REFBAYESENVELOPEPLOT  AT THIS VALUE WE HAVE THAT BEGINDISPLAYMATHR0 VARPHIPIM  R1 VARPHIPIM MAXPRHOP ENDDISPLAYMATHTHUS FOR P  PIM THE MAXIMUM POSSIBLE EXPECTED LOSS DUE TO IGNORANCE OF THE TRUESTATE OF NATURE IS MINIMIZED BY USING VARPHIPIM  THISOBSERVATION MOTIVATES THE INTRODUCTION OF THE SOCALLED MINIMAXDECISION RULESBEGINDEFINITION  WE SAY THAT A DECISION RULE VARPHI1 IS BF PREFERRED  RULE  VARPHI2 IFBEGINDISPLAYMATHMAXVARTHETAINTHETARVARTHETA VARPHI1  MAXVARTHETAINTHETARVARTHETA VARPHI2ENDDISPLAYMATHENDDEFINITIONTHIS NOTION OF PREFERENCE LEADS TO A LINEAR ORDERING OF THE RULES IND  A RULE THAT IS MOST PREFERRED IN THIS ORDERING ISCALLED A EM MINIMAX DECISION RULE  THAT IS A RULE VARPHI0 IS SAIDTO BE EM MINIMAX IF BEGINEQUATIONLABELMINIMAXMAXVARTHETAINTHETARVARTHETA VARPHI0 MINVARPHI IN DMAXVARTHETAINTHETARVARTHETA VARPHI ENDEQUATIONTHE VALUE ON THE RIGHT SIDE OF REFMINIMAX IS CALLED THE EMMINIMAX VALUE OR EM UPPER VALUE OF THE GAME INDEXMINIMAX DECISIONIN WORDS REFMINIMAX MEANS ESSENTIALLY THAT IF WE FIRST FINDFOR EACH RULE VARPHIIN DTHE VALUE OF VARTHETA THAT MAXIMIZES THE RISKAND THEN FIND THE RULE VARPHI0IN D THAT MINIMIZES THERESULTING SET OF RISKS WE HAVE THE MINIMAX DECISION RULE  THIS RULECORRESPONDS TO AN ATTITUDE OF CUTTING OUR LOSSES  WE FIRSTDETERMINE WHAT STATE NATURE WOULD TAKE IF WE WERE TO USE RULE VARPHIAND IT WERE PERVERSE THEN WE TAKE THE ACTION THAT MINIMIZES THEAMOUNT OF DAMAGE THAT NATURE CAN DO TO USIF AN AGENT IS PARANOID HE WOULD BE INCLINED TOWARD A MINIMAX RULEBUT AS THEY SAY JUST BECAUSE IM PARANOID DOESNT MEAN THEYREEM NOT OUT TO GET ME AND INDEED NATURE MAY HAVE IT IN FOR ADECISIONMAKING AGENT  IN SUCH A SITUATION NATURE WOULD SEARCHTHROUGH THE FAMILY OF POSSIBLE PRIOR DISTRIBUTIONS AND WOULD CHOOSEONE THAT DOES THE AGENT THE MOST DAMAGE EVEN IF HE ADOPTS ADOPT AMINIMAX STANCEBEGINDEFINITIONA DISTRIBUTION P0INTHETA IS SAID TO BE ABF LEAST FAVORABLE PRIOR IF INDEXLEAST FAVORABLE PRIORBEGINEQUATIONLABELMAXIMINMINVARPHI IN D RP0 VARPHI MAXPINTHETAMINVARPHI IN DRP VARPHIENDEQUATIONTHE VALUE ON THE RIGHT SIDE OF REFMAXIMIN IS CALLED THE EMMAXIMIN VALUE OR EM LOWER VALUE OF THE BAYES RISKENDDEFINITIONTHE TERMINOLOGY LEAST FAVORABLE DERIVES FROM THE FACT THAT IFI WERE TOLD WHICH PRIOR NATURE WAS USING I WOULD LIKE LEAST TO BETOLD A DISTRIBUTION P0 SATISFYING REFMAXIMIN BECAUSE THATWOULD MEAN THAT NATURE HAD TAKEN A STANCE THAT WOULD ALLOW ME TO CUTMY LOSSES BY THE LEAST AMOUNTSUBSECTIONMINIMAX BAYES IN MULTIPLE DECISION PROBLEMSIN DEVELOPING THE SOLUTION TO THE MINIMAX DECISION PROBLEM WE WILLGENERALIZE BEYOND THE BINARY HYPOTHESIS TEST TO THE MARY DECISIONPROBLEM  SUPPOSE THAT THETA CONSISTS OF MGEQ 2 POINTS THETA VARTHETA1 LDOTS VARTHETAM  THE GENERAL DECISION PROBLEMIS TO DETERMINE A TEST TO SELECT AMONG THESE M OPTIONSSUPPOSE THAT THE PRIOR DISTRIBUTION ON THETA IS  PTHETA  VARTHETA1  P1 PTHETA  VARTHETA2  P2LDOTS PTHETA  VARTHETAM  PMWE CAN REPRESENT THE VECTOR OF PRIORS USING THE VECTOR PBF  P1P2LDOTSPMTAS IN THE BINARY CASE WE CAN TALK ABOUT THE RISK AND THE BAYES RISKWHERE RISK IS DENOTED AS RVARTHETAI VARPHI AND THE BAYES RISKIS RPBFVARPHI  SUMI1M PI RVARTHETAIVARPHIUSING THE NOTATION YBFVARPHI RVARTHETA1VARPHILDOTSRVARTHETAMVARPHIT WE HAVE RPBFVARPHI  PBFT YBFBEGINDEFINITION INDEXRISK SET  THE BF RISK SET S SUBSET RBBM IS IS THE SET OF THE FORM S   RVARTHETA1 VARPHI LDOTS RVARTHETAMVARPHIWHERE VARPHI RANGES THROUGH D THE SET OF ALL RANDOMIZED DECISIONRULES  IN OTHER WORDS S IS THE SET OF ALL MTUPLES Y1LDOTS YM SUCH THAT YI  RVARTHETAI VARPHI I1 LDOTSM FOR SOME VARPHI IN DENDDEFINITIONTHE RISK SET WILL BE FUNDAMENTAL TO OUR UNDERSTANDING OF MINIMAX TESTSBEGINTHEOREMTHE RISK SET S IS A CONVEX SUBSET OF RBBMENDTHEOREMBEGINPROOFLET YBF  Y1 LDOTS YMT AND YBFPRIME  Y1PRIME LDOTS YMPRIMET BE ARBITRARYPOINTS IN S  ACCORDING TO THE DEFINITION OF S THERE EXIST DECISION RULES VARPHI  AND VARPHIPRIME IN D FOR WHICH YI RVARTHETAIVARPHI AND YIPRIMERVARTHETAIVARPHIPRIME FOR I1 LDOTS M  LET PI BEARBITRARY SUCH THAT 0 LEQ PI LEQ 1 AND CONSIDER THE DECISIONRULE VARPHIPI WHICH CHOOSES RULE VARPHI WITH PROBABILITYPI AND RULE VARPHIPRIME WITH PROBABILITY 1PICLEARLY VARPHIPIIN D ANDBEGINDISPLAYMATHRVARTHETAI VARPHIPI  PIRVARTHETAI VARPHI  1PIRVARTHETAIVARPHIPRIMEENDDISPLAYMATHFOR I1 LDOTS M  IF ZBF DENOTES THE POINT WHOSE ITHCOORDINATE IS RVARTHETAIVARPHIPI THEN ZBF  PIYBF  1PIYBFPRIME THUS ZBFIN SENDPROOFA PRIOR DISTRIBUTION FOR NATURE IS A MTUPLE OF NONNEGATIVE NUMBERSP1 LDOTS PM SUCH THAT SUMI1M PI1 WITH THEUNDERSTANDING THAT PI REPRESENTS THE PROBABILITY THAT NATURECHOOSES VARTHETAI  LET PBF P1 LDOTS PMT  FOR ANYPOINT YBFIN S DETERMINED BY SOME RULE VARPHI THE BAYES RISK ISTHEN THE INNER PRODUCTBEGINDISPLAYMATHRPBFVARPHI  PBFTYBF  SUMI1M PI YI  SUMI1M PIRVARTHETAIVARPHI  ENDDISPLAYMATHWE MAKE THE FOLLOWING OBSERVATIONSBEGINENUMERATEITEM THERE MAY BE MULTIPLE POINTS WITH THE SAME BAYES RISK FOREXAMPLE SUPPOSE ONE OR MORE ENTRIES IN PBF IS ZEROCONSIDER THE SET OF ALL VECTORS YBF THAT SATISFY FOR A GIVENPBF THE RELATIONSHIP BEGINEQUATIONLABELEQUIVPBFTYBF  BENDEQUATION FOR ANY REAL NUMBERB  THEN ALL OF THESE POINTS AND THE CORRESPONDING DECISION RULESARE EQUIVALENTITEM THE SET OF POINTS YBF THAT SATISFY REFEQUIV LIE IN A  HYPERPLANE THIS PLANE IS PERPENDICULAR TO THE VECTOR FROM THE  ORIGIN TO THE POINT P1 LDOTS PM  TO SEE THIS CONSIDER  FIGURE REFRISKSET WHERE FOR M2 THE RISK SET AND SETS OF  EQUIVALENT POINTS ARE DISPLAYED THE CONCEPTS CARRY OVER TO THE  GENERAL CASE FOR M2  IF YBF IS SUCH THAT PBFTYBF  B  THEN FOR A VECTOR XBF PERP PBF PBFTYBF  XBF  BITEM THE QUANTITY B CAN BE VISUALIZED BY NOTING THAT THE POINT OFINTERSECTION OF THE DIAGONAL LINE Y1  CDOTS  YM WITH THE PLANEPBFTYBF  SUMI PI YI  B MUST OCCUR AT BLDOTS  BTITEM TO FIND THE BAYES RULES WE FIND THE MINIMUM OF THOSE VALUES OF  B CALL IT B0 FOR WHICH THE PLANE PBFTYBF  B0 INTERSECTS  THE SET S  DECISION RULES CORRESPONDING TO POINTS IN THIS  INTERSECTION ARE BAYES WITH RESPECT TO THE PRIOR PBFENDENUMERATETHE MINIMAX PROBLEM CAN THUS BE VISUALIZED USING THE RISK SET  ASPBF VARIES HOW DOES THE POINT B0 OF MINIMUM RISK VARY  THEPOINT OF MINIMUM RISK IS THE MINIMAX RISK AS WE SHALL NOW EXPLOREBEGINFIGUREHTBBEGINCENTERINPUTPICTUREDIRRISKSET1TEXENDCENTERCAPTIONGEOMETRICAL INTERPRETATION OF THE RISK SETLABELRISKSETENDFIGURETHE MAXIMUM RISK FOR A FIXED RULE VARPHI IS GIVEN BYBEGINDISPLAYMATHMAXIRVARTHETAIVARPHIENDDISPLAYMATHALL POINTS YBFIN S THAT YIELD THIS SAME VALUE OF MAXI YIARE EQUIVALENT WITH RESPECT TO THE MINIMAX PRINCIPLE  THUS ALLPOINTS ON THE BOUNDARY OF THE SET BEGINDISPLAYMATHQC   Y1 LDOTS YM MC YI LEQ C QUAD MBOXFOR  I1LDOTS M ENDDISPLAYMATHFOR ANY REAL NUMBER C ARE EQUIVALENT  TO FIND THE MINIMAX RULES WEFIND THE MINIMUM OF THOSE VALUES OF C CALL IT C0  SUCH THAT THESET QC0 INTERSECTS S  THEN WE OBSERVE THE FOLLOWINGBEGINQUOTEANY DECISION RULE VARPHI WHOSEASSOCIATED RISK POINT RVARTHETA1VARPHI LDOTSRVARTHETAMVARPHIT IS AN ELEMENT OF QC0 CAP S IS AMINIMAX DECISION RULE  ENDQUOTEFIGURE REFMINIMAX1 DEPICTS A MINIMAX RULEFOR M2  THUS FOR A MINIMAX RULE WE MUST HAVE RISK EQUALIZATION  FOR THEMINIMAX RULE VARPHIBEGINEQUATION RVARTHETA1VARPHI  RVARTHETA2VARPHI  CDOTS RVARTHETAMVARPHILABELEQEQUALRISKENDEQUATIONDUE TO THE EQUAL RISK AT THE POINT OF MINIMAX RISK RPBFVARPHI  RVARTHETA1VARPHIP0  P1  CDOTS  PM RVARTHETA1VARPHI SO THAT THE EM BAYES RISK IS INDEPENDENT OF THE PRIOR  ANYATTEMPTS BY NATURE TO FIND A LESS FAVORABLE PRIOR ARE NEUTRALIZEDFIGURE REFMINIMAX1 ALSO DEPICTS THE LEAST FAVORABLE PRIOR WHICH ISVISUALIZED AS FOLLOWS  AS WE HAVE SEEN A STRATEGY FOR NATURE IS APRIOR DISTRIBUTION PBF   P1 LDOTS PMT WHICH REPRESENTS THEFAMILY OF PLANES PERPENDICULAR TO PBF  IN USING A BAYES RULE TOMINIMIZE THE RISK WE MUST FIND THE PLANE OUT OF THIS FAMILY THAT ISTANGENT TO AND BELOW S  BECAUSE THE MINIMUM BAYES RISK IS B0WHERE B0 LDOTS B0T IS THE INTERSECTION OF THE LINE Y1 LDOTS  YM AND THE PLANE TANGENT TO AND BELOW S ANDPERPENDICULAR TO PBF A LEAST FAVORABLE PRIOR DISTRIBUTION IS THECHOICE OF PBF THAT MAKES THE INTERSECTION AS FAR UP THE LINE ASPOSSIBLE  THUS THE LEAST FAVORABLE PRIOR LFP IS A BAYES RULE WHOSERISK IS B0  C0BEGINFIGUREHTBBEGINCENTERINPUTPICTUREDIRMINIMAX1ENDCENTERCAPTIONGEOMETRICAL INTERPRETATION OF THE MINIMAX RULELABELMINIMAX1ENDFIGUREBEGINEXAMPLE  WE CAN BE MORE EXPLICIT ABOUT THE RISK SET S IN M2 DIMENSIONSFROM REFPROBERRORBEGINALIGNRP PHIP     1P RVARTHETA0PHI P RVARTHETA1PHI    1PL01 ALPHA  P L101BETAENDALIGNLET PHI BE A NEYMANPEARSON TEST ASSOCIATED WITH THE BINARYHYPOTHESIS PROBLEM  THE ROC ASSOCIATED WITH THE NEYMANPEARSON TESTIS A PLOT OF BETA VERSUS ALPHA FOR THE TEST PHI  LETPHIHAT  1PHI DENOTE THE TEST WHICH IS EM CONJUGATE TOPHI   LET PFA AND PHATFA DENOTE THE PROBABILITY OFCHOOSING DECISION 1 GIVEN THAT THETA  THETA0 FOR PHI ANDPHIHAT RESPECTIVELY AND LET PD AND PHATD BE DEFINEDSIMILARLY  THEN FROM TABLE REFTABCONJNP WE NOTE THAT PHIHATHAS PHATFA  1ALPHA AND PHATD  1BETA  A PLOT OF THEROC FOR PHI AND PHIHAT IS SHOWN IN FIGUREREFFIGROCBAYES1A  THERE ARE NO TESTS OUTSIDE THE SHAPE SHOWNSINCE SUCH POINTS WOULD VIOLATE THE NEYMANPEARSON LEMMABEGINTABLEHTBP  BEGINCENTER    LEAVEVMODE    BEGINTABULARLLLL HLINE                        THETATHETA0  THETA  THETA1  HLINEPHI0  PHIHAT  1  1ALPHA  PHATFA   1BETA  PHATD PHI1  PHIHAT  0  ALPHA  PFA BETA  PD HLINE    ENDTABULAR    CAPTIONPROBABILITIES FOR NEYMANPEARSON AND CONJUGATE      NEYMANPEARSON TESTS    LABELTABCONJNP  ENDCENTERENDTABLEFIGURE REFFIGROCBAYES1B SHOWS THE BOUNDARIES OF THE RISK SETFOUND BY PLOTTING L01 PFA AND L101PD FOR EACH OF THETWO SETS L0104 AND L10  15 IN THIS FIGURE  BEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODEMBOXSUBFIGUREROC FOR PHI AND 1PHIEPSFIGFILEPICTUREDIRROCBAYES1EPS      WIDTH045TEXTWIDTH QUADSUBFIGURERISK SETEPSFIGFILEPICTUREDIRROCBAYES2EPSWIDTH045TEXTWIDTHCAPTIONTHE RISK SET AND ITS RELATION TO THE NEYMANPEARSON TEST  ENDCENTER    LABELFIGROCBAYES1ENDFIGUREENDEXAMPLESUBSECTIONDETERMINING THE LEASTFAVORABLE PRIORLABELSECLFPROC3MGIVEN THAT A MINIMAX SOLUTION IS FOUND IT MAY BE OF INTEREST TODETERMINE THE LEASTFAVORABLE PRIOR  AS DISCUSSED ABOVE THEPROBABILITY VECTOR PBF  P0P1LDOTSPMT IS ORTHOGONAL TOTHE BOUNDARY OF THE RISK SET AT THE POINT WHERE RVARTHETA0PHI RVARTHETA1PHI  CDOTS  RVARTHETAMPHI  DETERMINING THELEASTFAVORABLE PRIOR REQUIRES FINDING A VECTOR TANGENT TO THEBOUNDARY OF THE RISK SET THEN FINDING A VECTOR NORMAL TO THATSURFACE NORMALIZED TO BE A PROBABILITY VECTORLET THE BOUNDARY B OF S THAT INTERSECTS QC BE A SURFACEPARAMETERIZED BY SOME PARAMETER QBF IN RBBM1 AND ASSUME THATB IS A DIFFERENTIABLE FUNCTION OF THE COMPONENTS OF QBF FOR SOMEQBF IN AN OPEN NEIGHBORHOOD OF QBF0  THAT IS THE POINTRQBFVARTHETA0PHIALLOWBREAKRVARTHETA1PHIALLOWBREAK LDOTSALLOWBREAK RQBFVARTHETAM1PHI IS A POINT ON B AND WE TAKE THE POINT QBF  QBF0 ASTHAT VALUE OF PARAMETER WHICH IS THE MINIMAX RISK  THEN THE VECTORS BEGINBMATRIX PARTIALDRQBFVARTHETA0PHIQ1 EXMATSPPARTIALDRQBFVARTHETA1PHIQ1 EXMATSP VDOTS PARTIALDRQBFVARTHETAM1PHIQ1 ENDBMATRIXQUADBEGINBMATRIX PARTIALDRQBFVARTHETA0PHIQ2 EXMATSPPARTIALDRQBFVARTHETA1PHIQ2 EXMATSP VDOTS PARTIALDRQBFVARTHETAM1PHIQ2 ENDBMATRIXQUAD CDOTS QUADBEGINBMATRIX PARTIALDRQBFVARTHETA0PHIQM1 EXMATSPPARTIALDRQBFVARTHETA1PHIQM1 EXMATSP VDOTS PARTIALDRQBFVARTHETAM1PHIQM1 ENDBMATRIXEVALUATED AT QBF  QBF0 ARE TANGENT TO B AT THE MINIMAX RISKPOINT   SEE FIGURE REFFIGLFPSURFACEA VECTOR WHICH IS ORTHOGONAL TO ALL OF THESE VECTORSNORMALIZED TO BE A PROBABILITY VECTOR IS THUS A LEASTFAVORABLE PRIORPROBABILITYIN TWO DIMENSIONS THE LEASTFAVORABLE PRIOR P0P1 CAN BEDETERMINED WITH LESS SOPHISTICATION  AT THE POINT OF EQUAL RISK THEMINIMAX TEST IS A BAYES TEST WITH LIKELIHOOD RATIO TEST THRESHOLDBEGINEQUATION NU  FRACP0 L01P1 L10LABELEQNUBAYESENDEQUATIONIF THE THRESHOLD NU CAN BE DETERMINED THEN REFEQNUBAYES CANBE SOLVED FOR P0SUBSECTIONA MINIMAX EXAMPLE AND THE MINIMAX  THEOREMLABELSECMINIMAXTHEOREMBEGINEXAMPLE  WE NOW CAN DEVELOP SOLUTIONS TO THE ODD OR EVEN GAME WE  INTRODUCED EARLIER IN THE CHAPTER  AS YOU RECALL NATURE AND  YOURSELF SIMULTANEOUSLY PUT UP EITHER ONE OR TWO FINGERS  NATURE  WINS IF THE SUM OF THE DIGITS SHOWING IS ODD AND YOU WIN IF THE SUM  OF THE DIGITS SHOWING IS EVEN  THE WINNER IN ALL CASES RECEIVES IN  DOLLARS THE SUM OF THE DIGITS SHOWING THIS BEING PAID TO HIM BY THE  LOSER  BEFORE THE GAME IS PLAYED YOU ARE ALLOWED TO ASK NATURE HOW  MANY FINGERS IT INTENDS TO PUT UP AND NATURE MUST ANSWER TRUTHFULLY  WITH PROBABILITY 34 HENCE UNTRUTHFULLY WITH PROBABILITY 14  YOU THEREFORE OBSERVE A RANDOM VARIABLE X THE ANSWER NATURE  GIVES TAKING THE VALUES OF 1 OR 2  IF THETA  1 IS THE TRUE  STATE OF NATURE THE PROBABILITY THAT X1 IS 34 THAT IS  PTHETA11  34  SIMILARLY PTHETA12  14THE FOUR NONRANDOMIZED DECISION RULES AREBEGINALIGNEDPHI11  1  QQUAD  PHI12  1PHI21  1  QQUAD  PHI22  2PHI31  2  QQUAD  PHI32  1PHI41  2  QQUAD  PHI42  2ENDALIGNEDTHE RISK MATRIXGIVEN IN FIGURE REFGAME2 CHARACTERIZES THIS STATISTICAL GAMEBEGINFIGUREHTBBEGINCENTERINPUTPICTUREDIRRISKFUNLATEXENDCENTERCAPTIONRISK FUNCTION FOR STATISTICAL ODD OR EVEN GAMELABELGAME2ENDFIGURETHE RISK SET FOR THIS EXAMPLE IS GIVEN IN FIGURE REFFIGODDEVENWHICH MUST CONTAIN ALL OF THE LINES BETWEEN ANY TWO OF THE POINTS23 34 94 74 54 34  ACCORDING TO OUREARLIER ANALYSIS THE MINIMAX POINT CORRESPONDS THE POINT INDICATED INTHE FIGURE WHICH IS ON THE LINE L CONNECTING THE R1 PHI1 R2 PHI1 WITH R1 PHI2 R2 PHI2  THE PARAMETRICEQUATION FOR THIS LINE IS Y1Y2  Q23  1Q3494AS Q RANGES OVER THE INTERVAL 0  1WHICH CAN BE WRITTEN AS BEGINALIGNEDY1  FRAC54 Q FRAC34 EXMATSPY2  FRAC214 Q  FRAC94ENDALIGNEDTHIS LINE INTERSECTS THE LINE Y1  Y2 AT FRAC54Q  2FRAC214Q  3 THAT IS WHEN Q FRAC313 THE MINIMAXRISK IS FRAC54 FRAC313  FRAC34  FRAC2726THE RANDOMIZED DECISION RULE IS THIS USE RULE D1 WITH PROBABILITY Q FRAC313 AND USE D2 WITH PROBABILITY 1Q FRAC1013BEGINFIGUREHTBUNITLENGTH 1INBEGINCENTERBEGINPICTURE1027PUT1010PSFIGFILEDETESTPICTDIRODDEVENPSHEIGHT25IN HSCALE45VSCALE45HOFFSET0PUT50SPECIALPSFILETUSERSWYNNTEXCLASSEE513ODDEVENPS   HSCALE 45 VSCALE  45 HOFFSET 0PUT00MAKEBOX00PUT2523MAKEBOX0023PUT216MAKEBOX00LPUT917MAKEBOX00FRACSCRIPTSTYLE 7SCRIPTSTYLE 4 FRACSCRIPTSTYLE 5SCRIPTSTYLE 4PUT1575MAKEBOX00FRACSCRIPTSTYLE 3SCRIPTSTYLE 4FRACSCRIPTSTYLE 9SCRIPTSTYLE 4PUT1152MAKEBOX0034PUT311MAKEBOX00SMALL MINIMAX POINTPUT8 10MAKEBOX00SPUT515MAKEBOX00SMALL LFPENDPICTUREENDCENTERCAPTIONRISK SET FOR ODD OR EVEN GAMELABELODDEVENENDFIGUREBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE ODDEVENMEPSFIGFILEPICTUREDIRODDEVENEPS  WIDTH08TEXTWIDTH        CAPTIONRISK SET FOR ODD OR EVEN GAME  ENDCENTER    LABELFIGODDEVENENDFIGUREWE MAY COMPUTE THE LEAST FAVORABLE PRIOR AS FOLLOWS  LET NATURE TAKEACTION VARTHETA1 WITH PROBABILITY P AND VARTHETA2 WITHPROBABILITY 1P  THE VECTOR PBF  P 1PT IS PERPENDICULARTHE SURFACE OF S WHICH IN THIS CASE IS THE LINE L PARAMETERIZEDABOVE  THE TANGENT TO THIS LINE HAS SLOPE FRACDISPLAYSTYLE FRACDY2DQDISPLAYSTYLE FRACDY1DQ  215BY THE ORTHOGONALITY OF THE LEASTFAVORABLE PRIOR VECTOR WE REQUIRE FRAC1PP  FRAC521OR P FRAC2126  THUS IF NATURE CHOOSES TO HOLD UP ONE FINGER WITH PROBABILITY2126 IT WILL MAINTAIN YOUR EXPECTED LOSS TO AT LEASTFRAC2726 AND IF THE AGENT SELECTS DECISION RULE D1 WITHPROBABILITY FRAC313 HE WILL RESTRICT HIS AVERAGE LOSS TO NOMORE THAN FRAC2726  IT SEEMS REASONABLE TO CALLFRAC2726 THE EM VALUE OF THE GAME  IF A REFEREE WERE TOARBITRATE THIS GAME IT WOULD SEEM FAIR TO REQUIRE NATURE TO PAY YOUFRAC2726 DOLLARS IN LIEU OF PLAYING THE GAME  IT SHOULD BE POINTED OUT WHAT IS ACHIEVED IN THE LEAST FAVORABLEPRIOR  THE SELECTION IS ONLY THE PROBABILITY P OF CHOOSING SOMEPARTICULAR OUTCOME  WHAT IS EM NOT CHANGED IS THE CONDITIONALPROBABILITY UPON WHICH MEASUREMENTS ARE MADE FXTHETAXVARTHETAENDEXAMPLETHE ABOVE EXAMPLE DEMONSTRATES A SITUATION IN WHICH THE BEST YOU CANDO IN RESPONSE TO THE WORST NATURE CAN DO YIELDS THE SAME EXPECTEDLOSS AS WOULD BE OBTAINED IF NATURE DID ITS WORST IN RESPONSE TO THEBEST YOU CAN DO  THIS RESULT IS SUMMARIZED IN THE FOLLOWING THEOREMWHICH WE WILL NOT PROVE HEREBEGINTHEOREMEM THE MINIMAX THEOREM  IF FOR A GIVEN DECISION PROBLEMTHETA D R WITH FINITE THETA  VARTHETA1 LDOTSVARTHETAK THE RISK SET S IS BOUNDED BELOW THENBEGINDISPLAYMATHMINVARPHI IN DMAXPIN THETARP VARPHI MAXPIN THETAMINVARPHIIN DRP VARPHI ENDDISPLAYMATHAND THERE EXISTS A LEAST FAVORABLE DISTRIBUTION P0  ENDTHEOREMINDEXMINMAXTHIS CONDITION IS CALLED THE EM SADDLEPOINT CONDITION  MORE ONSADDLEPOINT OPTIMALITY IS PRESENTED IN SECTION REFSECDUALITYTHIS EXAMPLE DEMONSTRATES STILL ANOTHER PROPERTY OF BAYES DECISIONTHEORY WHICH IS ESSENTIALLY THAT IF WE USE A BAYES DECISION RULETHAT IS A RULE THAT MINIMIZES THE BAYES RISK WE MAY RESTRICTOURSELVES TO NONRANDOMIZED RULES  FROM OUR RULES DESCRIBING THECONSTRUCTION OF THE BAYES POINT FOR THIS PROBLEM WE SEE THAT EVERYPOINT ON THE LINE L IS A BAYES POINT CONSEQUENTLY THE VERTICES 23 AND FRAC34 FRAC94 ARE BAYES POINTSCORRESPONDING TO NONRANDOMIZED DECISION RULES  CAN YOU CONSTRUCT THESET OF BAYES POINTS CORRESPONDING TO EVERY POSSIBLE PRIORBEGINEXAMPLE  CONSIDER AGAIN THE BINARY CHANNEL OF EXAMPLE  REFEXMBINCHAN5 AND TAKE LAMBDA0  14 AND LAMBDA1   13  FIGURE REFFIGBINCHAN2 ILLUSTRATES THE RISK SET FOR THIS  CASE  THE LINE OF THE MINIMAX SOLUTION LIES ON THE RISK FOR  PHI2 AND PHI4 IT IS PARAMETERIZED BY Y1Y2  Q1413  1Q10SO THAT THE MINIMAX SOLUTION WHEN Y1Y2 OCCURS WHEN Q1213THAT IS PHI2 SHOULD BE EMPLOYED WITH PROBABILITY 1213  THECORRESPONDING MINIMUM BAYES RISK IS 413  THIS IS THE MINIMAXPROBABILITY OF ERROR  THE LEAST FAVORABLE PRIOR IS FOUND BY FINDINGTHE SLOPE OF THE RISK FUNCTION FRACDISPLAYSTYLE FRACDY2DQDISPLAYSTYLE FRACDY1DQ FRAC49THE LFP HAS PERPENDICULAR SLOPE FRAC94  FRACP1PSO THAT P913 IS LEAST FAVORABLEIT IS INTERESTING TO COMPARE THESE RESULTS WITH THE BAYES ENVELOPE OFFIGURE REFFIGBINENV  THE LFP AND MINIMAX BAYES RISK ARE BOTHAPPARENT IN THIS FIGUREBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE BINCHANM    EPSFIGFILEPICTUREDIRBINCHAN2EPS    CAPTIONRISK SET FOR THE BINARY CHANNEL    LABELFIGBINCHAN2  ENDCENTERENDFIGUREENDEXAMPLEBEGINEXAMPLEGIVEN GIVEN THETA   VARTHETA0 VARTHETA1 CORRESPONDING RESPECTIVELY TO THE  HYPOTHESES H0 AND H1 DEFINED BY BEGINALIGNEDH0MC XBF SIM NCMBF0SIGMA2 I H1MC XBF SIM NCMBF1SIGMA2 IENDALIGNED WE FORM THE LOGLIKELIHOOD LAMBDAXBF  FRAC1SIGMA2MBF1  MBF0T XBF FRAC12 MBF1  MBF0THE LOGLIKELIHOOD FUNCTION IS GAUSSIAN DISTRIBUTED UNDER H0LAMBDAXBF SIM NCF22F2 AND UNDER H1 LAMBDAXBFSIM NCF22F2 WHERE F2  FRAC1SIGMA2MBF1  MBF0THE DECISION IS PHIXBF  BEGINCASES  1  LAMBDAXBF  LOG NU  ETA   0  LAMBDAXBF  ETAENDCASESTHE SIZE AND POWER AREBEGINEQUATIONBEGINSPLITALPHA  QMU BETA  QMU  FENDSPLITLABELEQALPHABETAQENDEQUATIONWHERE BEGINEQUATION  LABELEQMUETAMU  ETAF  F2    ENDEQUATIONNOW SUPPOSE THAT WE IMPOSE THE COSTS L00  L11  0 AND L01 CL10  THAT IS THE COST OF A FALSE ALARM IS K TIMES MORE THANTHE COST OF A MISSED DETECTION  WE DESIRE TO DETERMINE THE THRESHOLDETA WHICH MINIMIZES THE RISK AGAINST ALL POSSIBLE PRIORS  THERISKS ARE BEGINALIGNEDRVARTHETA0PHI  L01 ALPHA RVARTHETA1PHI  L10 1BETAENDALIGNEDBY REFEQEQUALRISK THE MINIMAX RULE MUST SATISFY L01 ALPHA  L10 1BETAOR C L10 ALPHA  L101BETASO BETA  1CALPHA  FROM REFEQALPHABETAQ WE MUST HAVEBEGINEQUATION  LABELEQQMUQMUF  1C QMU  ENDEQUATIONDETERMINATION OF MU AND FROM REFEQMUETA THE LOGLIKELIHOODTHRESHOLD ETA CAN BE ACCOMPLISHED BY NUMERICAL SOLUTION OFREFEQQMU WHICH CAN BE DONE BY ITERATING MUK1  Q11QMUKFCSTARTING FROM SOME INITIAL MU0  ONCE MU IS FOUND THE LEAST FAVORABLE PRIOR IS FOUND  WE CANDESCRIBE THE BOUNDARY OF S USING THE ROC CURVE AS A FUNCTION OF THETHRESHOLDBEGINALIGNED RMUVARTHETA0PHI  L01 ALPHA  L01 QMU  RMUVARTHETA1PHI  L10 1BETA  L10 1QMUFENDALIGNEDTHEN THE TANGENT VECTOR HAS SLOPE DRMUVARTHETA1PHIDRMUVARTHETA0PHI  FRACL10  GMUF L01 GMU  FRACGMUFCGMUWHERE GX  FRAC1SQRT2PIEX22 AND THE ORTHOGONALVECTOR P1P MUST SATISFY FRAC1PP  C FRACGMUGMUFSC MATLAB CODE WHICH COMPUTES THE MU EPSILON THE MINIMAXVALUE AND THE LEAST FAVORABLE PRIOR IS SHOWN IN ALGORITHMREFALGBAYESEXBEGINNEWPROGENVEXAMPLE BAYES MINIMAX CALCULATIONSBAYES3MBAYESEXEXAMPLE BAYES MINIMAX CALCULATIONSENDNEWPROGENVENDEXAMPLEBEGINEXERCISESITEM FOR THE BINARY CHANNEL REPRESENTED BY    BEGINCENTER      INPUTPICTUREDIRBSCLAMBDA    ENDCENTER  ENDFIGURE  BEGINENUMERATE  ITEM DETERMINE THE LIKELIHOOD RATIO TEST  ITEM DETERMINE THE THRESHOLD NU TO OBTAIN A TEST OF SIZE    ALPHA WHEN LAMBDA0  LAMBDA1  LAMBDA AS A FUNCTION OF    LAMBDA  ITEM IF LAMBDA0LAMBDA1 LAMBDA DETERMINE AND PLOT THE ROC    FOR A NEYMANPEARSON TEST ON THE CHANNEL FOR LAMBDA18    LAMBDA14 LAMBDA38 AND LAMBDA12  ITEM DETERMINE THE BAYES DECISION RULE WHEN THE PRIOR PROBABILITIES    P0  PTHETA0 AND P1  PTHETA1 ARE EQUAL AND THE    COSTS ARE UNIFORM  ITEM PLOT THE BAYES ENVELOPE FUNCTION WHEN LAMBDA0 01 AND    LAMBDA1  02  ENDENUMERATEITEM CONSIDER TWO BOXES A AND B EACH OF WHICH CONTAINS BOTH RED BALLSAND GREEN BALLS  IT IS KNOWN THAT IN ONE OF THE BOXES FRAC12OF THE BALLS ARE RED AND FRAC12 ARE GREEN AND THAT IN THEOTHER BOX FRAC14 OF THE BALLS ARE RED AND FRAC34 AREGREEN  LET THE BOX IN WHICH FRAC12 ARE RED BE DENOTED BOX WAND SUPPOSE PW  A  XI AND PW  B  1XI  SUPPOSE YOU MAYSELECT ONE BALL AT RANDOM FROM EITHER BOX A OR BOX B AND THATAFTER OBSERVING ITS COLOR MUST DECIDE WHETHER WA OR WB  PROVETHAT IF FRAC12  XI  FRAC23 THEN IN ORDER TO MAXIMIZETHE PROBABILITY OF MAKING A CORRECT DECISION HE SHOULD SELECT THEBALL FROM BOX B  PROVE ALSO THAT IF FRAC23LEQ XI LEQ 1 THEN ITDOES NOT MATTER FROM WHICH BOX THE BALL IS SELECTEDITEM A WILDCAT OILMAN MUST DECIDE HOW TO FINANCE THE DRILLING OF A WELLIT COSTS 100000 TO DRILL THE WELL  THE OILMAN HAS AVAILABLE THREEOPTIONS BEGINDESCRIPTIONITEMH0 FINANCE THE DRILLING HIMSELF AND RETAIN ALL THE PROFITSITEMH1 ACCEPT 70000 FROM INVESTORS IN RETURN FOR PAYING THEM  50 OF THE OIL PROFITSITEMH2 ACCEPT 120000 FROM INVESTORS IN RETURN FOR PAYING THEM  90 OF THE OIL PROFITSENDDESCRIPTIONTHE OIL PROFITS WILL BE 3THETA WHERE THETA IS THE NUMBER OFBARRELS OF OIL IN THE WELL  FROM PAST DATA IT IS BELIEVED THAT THETA  0 WITH PROBABILITY09 AND THE DENSITY FOR THETA  0 ISBEGINDISPLAYMATHGVARTHETA  FRAC01300000EVARTHETA300000I0    INFTYVARTHETAENDDISPLAYMATHA SEISMIC TEST IS PERFORMED TO DETERMINE THE LIKELIHOOD OF OIL IN THEGIVEN AREA  THE TEST TELLS WHICH TYPE OF GEOLOGICAL STRUCTURE X1X2 OR X3 IS PRESENT  IT IS KNOWN THAT THE PROBABILITIES OFTHE XI GIVEN THETA AREBEGINALIGNEDFXTHETAX1VARTHETA    08EVARTHETA100000FXTHETAX2VARTHETA    02FXTHETAX3VARTHETA    081 EVARTHETA100000ENDALIGNEDBEGINITEMIZEITEM FOR MONETARY LOSS WHAT IS THE BAYES ACTION IF X  X1 IS  OBSERVEDITEM FOR MONETARY LOSS WHAT IS THE BAYES ACTION IF X  X2 IS  OBSERVEDITEM FOR MONETARY LOSS WHAT IS THE BAYES ACTION IF X  X3 IS  OBSERVEDENDITEMIZEITEM A DEVICE HAS BEEN CREATED WHICH CAN SUPPOSEDLY CLASSIFY BLOOD AS TYPEA B AB OR O  THE DEVICE MEASURES A QUANTITY X WHICH HASDENSITYBEGINDISPLAYMATHFXTHETAXVARTHETA  EX  VARTHETAIVARTHETA INFTYXENDDISPLAYMATHIF 0  THETA  1 THE BLOOD IS OF TYPE AB IF 1  THETA  2 THEBLOOD IS OF TYPE A IF 2  THETA  3 THE BLOOD IS OF TYPE B ANDIF THETA  3 THE BLOOD IS OF TYPE O  IN THE POPULATION AS A WHOLETHETA IS DISTRIBUTED ACCORDING TO THE DENSITYBEGINDISPLAYMATHFTHETAVARTHETA  EVARTHETAI0 INFTYVARTHETAENDDISPLAYMATHTHE LOSS IN MISCLASSIFYING THE BLOOD IS GIVEN BY THE FOLLOWING TABLEVSPACE2INDEFLIMMSHATADEFSPANNHATBDEFRANKHATCDEFKERHATDDEFCOVHATEDEFVARHATFDEFTRHATGDEFDIAGHATHBEGINCENTERBEGINTABULARCCCCCCMULTICOLUMN6CCLASSIFICATION   AB  A  B  OCLINE26  AB  0  1  1  2 CLINE26TRUE  A  1  0  2  2CLINE26TYPE  B  1  2  0  2CLINE26  O  3  3  3  0CLINE26ENDTABULARENDCENTERIF X  4 IS OBSERVED WHAT IS THE BAYES ACTIONITEM FOR THE BINARY CHANNEL TAKE LAMBDA0  13 AND LAMBDA1   14  DETERMINE  BEGINENUMERATE  ITEM THE RISK SET  ITEM THE MINIMAX BAYES RISK  ITEM THE OPTIMUM DECISION RULE  ITEM THE LEAST FAVORABLE PRIOR  ENDENUMERATE  ITEM LABELEXDETECHANGE1 IN THESE LAST EXERCISES WE INTRODUCE  BRIEFLY SOME OTHER TOPICS IN DETECTION THEORY  THIS PROBLEM DEALS  WITH BF DETECTION OF CHANGE  SUPPOSE THAT A SIGNAL CHANGES ITS  MEAN AT SOME UNKNOWN TIME N0 AND THE PROBLEM IS TO DETECT THE  CHANGE  WE SET UP THE FOLLOWING HYPOTHESIS TEST BEGINALIGNEDH0MC  XI SIM NCM0SIGMA2QQUAD I12LDOTSN H1MC   XI SIM NCM0SIGMA2 QQUAD I12LDOTSN01           XI SIM NCM1SIGMA2 QQUAD IN0N01LDOTSNENDALIGNEDWHERE WE ASSUME M1  M0 AND ARE ASSUMED TO BE KNOWN AS ISSIGMA2  ASSUME THAT N0 IS KNOWN  BEGINENUMERATEITEM BASED UPON A LIKELIHOODRATIO TEST SHOW THAT A TEST FOR THE CHANGE IS DECIDE H1 IF TXBF  FRAC1NN0 SUMIN0N XI  M0  ETAITEM DETERMINE THE DISTRIBUTION OF TXBF UNDER THE TWO  HYPOTHESES AND DETERMINE AN EXPRESSION FOR PFA AS A FUNCTION  OF THE THRESHOLD ETAENDENUMERATEITEM LABELEXDETECHANGE2 IN THE DETECTION OF CHANGE PROBLEM  PRESENT PREVIOUSLY ASSUME NOW THAT WE DONT KNOW N0  FORMING  THE LIKELIHOOD RATIO ELLN0XBF  FRACFXBF1XBF1FXBF0XBF0WE CHOOSE THE MAXIMUM LIKELIHOOD ESTIMATE OF N0 TO BE THAT VALUEWHICH MAXIMIZES ELLN0XBF  SHOW THAT THIS REDUCES TO  MAXN0 SUMIN0N1 XI  M0  FRACM1M02ENDEXERCISES LOCAL VARIABLES TEXMASTER TEST ENDSECTIONBAYES ESTIMATION THEORYLABELSECBE1INDEXBAYES ESTIMATION THEORYSUPPOSE YOU OBSERVE A RANDOM VARIABLE X WHOSE DISTRIBUTION DEPENDSON A PARAMETER THETA  THE MAXIMUM LIKELIHOOD APPROACH TOESTIMATION SAYS THAT YOU SHOULD TAKE AS YOUR ESTIMATE OF AN UNKNOWNPARAMETER THAT VALUE THAT IS THE MOST LIKELY OUT OF ALL POSSIBLEVALUES OF THE PARAMETER TO HAVE GIVEN RISE TO THE OBSERVED DATABEFORE OBSERVATIONS ARE TAKEN THEREFORE THE MAXIMUM LIKELIHOODMETHOD IS SILENT AS TO ANY PREDICTIONS IT WOULD MAKE ABOUT EITHER THEVALUE OF THE PARAMETER OR THE VALUES FUTURE OBSERVATIONS WOULD TAKERATHER THE ATTITUDE OF A RABID MAXLIKE ENTHUSIAST WOULD BEWAIT UNTIL ALL OF THE DATA ARE COLLECTED GIVE THEM TO ME BEPATIENT AND SOON I WILL GIVE YOU AN ESTIMATE OF WHAT THE VALUES OFTHE PARAMETERS WERE THAT GENERATED THE DATA  IF YOU WERE TO ASK HIMFOR HIS BEST GUESS BEFORE YOU COLLECTED THE DATA AS TO WHAT VALUESWOULD BE ASSUMED BY EITHER THE DATA OR THE PARAMETERS HIS RESPONSEWOULD SIMPLY BE DONT BE RIDICULOUSON THE OTHER HAND A BAYESIAN WOULD BE ALL TOO HAPPY TO GIVE YOU ESTIMATES BOTH BEFORE AND AFTER THE DATA HAVE BEEN OBTAINEDBEFORE THE OBSERVATION SHE WOULD GIVE YOU PERHAPS THE MEAN VALUE OFTHE EM A PRIORI DISTRIBUTION OF THE PARAMETER AND AFTER THE DATAWERE COLLECTED SHE WOULD GIVE YOU THE MEAN VALUE OF THE EM APOSTERIORI DISTRIBUTION OF THE PARAMETER  SHE WOULD OFFER ASPREDICTED VALUES OF THE OBSERVATIONS THE MEAN VALUE OF THECONDITIONAL DISTRIBUTION OF X GIVEN THE EXPECTED VALUE OF THETABASED ON THE EM A PRIORI DISTRIBUTION  SOME INSIGHT MAY BE GAINED INTO HOW THE PRIOR DISTRIBUTION ENTERS INTOTHE PROBLEM OF ESTIMATION THROUGH THE FOLLOWING EXAMPLEBEGINEXAMPLELET X1 LDOTS XM DENOTE A RANDOM SAMPLE OF SIZE M FROM THENORMAL DISTRIBUTION NCTHETA SIGMA2 SUPPOSE SIGMA IS KNOWN ANDWE WISH TO ESTIMATE THETA  WE ARE GIVEN THE PRIOR DENSITY THETA SIM NCVARTHETA0 SIGMATHETA2 THAT ISBEGINDISPLAYMATHFTHETAVARTHETA  FRAC1SQRT2PI SIGMATHETAEXPLEFT FRACVARTHETAVARTHETA022SIGMATHETA2RIGHTENDDISPLAYMATHBEFORE GETTING INVOLVED IN DEEP BAYESIAN PRINCIPLES LETS JUST THINKABOUT WAYS WE COULD USE THIS PRIOR INFORMATION  BEGINENUMERATEITEM WE COULD CONSIDER COMPUTING THE MAXIMUM LIKELIHOOD  ESTIMATE OF THETAWHICH WE SAW EARLIER IS JUST THE SAMPLE AVERAGEAND THEN SIMPLY AVERAGING THIS RESULT WITH THE MEAN VALUE OF THE PRIORDISTRIBUTION YIELDING BEGINDISPLAYMATHTHETAHATA  FRACVARTHETA0  THETAHATML2ENDDISPLAYMATHTHIS NAIVE APPROACH WHILE IT FACTORS IN THE PRIOR INFORMATIONGIVES EQUAL WEIGHT TO THE PRIOR INFORMATION AS COMPARED TOEM ALL OF THE DIRECT OBSERVATIONS  SUCH A RESULT MIGHT BEHARD TO JUSTIFY ESPECIALLY IF THE DATA QUALITY IS HIGHITEM WE COULD TREAT VARTHETA0 AS ONE EXTRA DATA POINT ANDAVERAGE IT IN WITH ALL OF THE OTHER XIS YIELDINGBEGINDISPLAYMATHTHETAHATB  FRACVARTHETA0  SUMI1MXIM1ENDDISPLAYMATHTHIS APPROACH HAS A VERY NICE INTUITIVE APPEAL WE SIMPLY TREAT THEEM A PRIORI INFORMATION IN EXACTLY THE SAME WAY AS WE DO THE REALDATA  THETAHATB IS THEREFORE PERHAPS MORE REASONABLE THANTHETAHATA BUT IT STILL SUFFERS A DRAWBACK IT IS TREATED AS BEINGEXACTLY EQUAL IN INFORMATIONAL CONTENT TO EACH OF THE XISWHETHER OR NOT SIGMA THETA2 EQUALS SIGMA2ITEM WE COULD TAKE A WEIGHTED AVERAGE OF THE EM A PRIORI MEAN ANDTHE MAXIMUM LIKELIHOOD ESTIMATE EACH WEIGHTED INVERSEPROPORTIONALLY TO THE VARIANCE YIELDINGBEGINDISPLAYMATHTHETAHATC  FRACDISPLAYSTYLE VARTHETA0DISPLAYSTYLE SIGMATHETA2 FRAC DISPLAYSTYLE THETAHATMLDISPLAYSTYLE SIGMAML2 OVERFRACDISPLAYSTYLE 1DISPLAYSTYLE SIGMATHETA2 FRACDISPLAYSTYLE 1DISPLAYSTYLE  SIGMAML2ENDDISPLAYMATHWHERE SIGMAML2 IS THE VARIANCE OF THETAHATML AND ISGIVEN BYBEGINDISPLAYMATHSIGMAML2  ELEFT FRAC1MSUMI1M XI  THETARIGHT2ENDDISPLAYMATHTO CALCULATE THE ABOVE EXPECTATION WE TEMPORARILY TAKE OFF OUR BAYESIANHAT AND PUT ON OUR MAXLIKE HAT VIEW THETA AS SIMPLY AN UNKNOWNPARAMETER AND TAKE THE EXPECTATION WITH RESPECT TO THE RANDOMVARIABLES XI ONLY  IN SO DOING IT FOLLOWS AFTER SOME MANIPULATIONS THATSIGMAML2  SIGMA2M  CONSEQUENTLYBEGINEQUATIONLABELWEIGHTEDTHETAHATC  FRACSIGMA2MSIGMATHETA2 SIGMA2MVARTHETA0 FRACSIGMATHETA2SIGMATHETA2 SIGMA2MTHETAHATMLENDEQUATIONTHE ESTIMATE THETAHATC SEEMS TO INCORPORATE ALL OF THEINFORMATION BOTH EM A PRIORI AND EM A POSTERIORI THAT WE HAVEABOUT THETA  WE SEE THAT AS M BECOMES LARGE THE EM A PRIORIINFORMATION IS FORGOTTEN AND THE MAXIMUM LIKELIHOOD PORTION OF THEESTIMATOR DOMINATES  WE ALSO SEE THAT IF SIGMATHETA2 SIGMA2 THEN THE EM A PRIORI INFORMATION TENDS TO DOMINATE  THE ESTIMATE PROVIDED BY THETAHATC APPEARS TO BE OF THE THREE WEHAVE PRESENTED THE ONE MOST WORTHY OF OUR ATTENTION  WE SHALLEVENTUALLY SEE THAT IT IS INDEED A BAYESIAN ESTIMATE  ENDENUMERATEENDEXAMPLESECTIONBAYES RISKLABELSECBR1INDEXBAYES RISK THE STARTING POINT FOR BAYESIAN ESTIMATION AS ITWAS FOR BAYESIAN DETECTION IS THE SPECIFICATION OF A LOSS FUNCTIONAND THE CALCULATION OF THE BAYES RISK  RECALL THAT THE COST FUNCTIONIS A FUNCTION OF THE STATE OF NATURE AND THE DECISION FUNCTION THATIS IT IS OF THE GENERAL FORM LTHETA PHIX  FOR OURDEVELOPMENT IN BAYES ESTIMATION THEORY WE WILL RESTRICT THE STRUCTUREOF THE LOSS FUNCTION TO BE FUNCTION OF THE EM DIFFERENCE THAT ISTO BE OF THE FORM LTHETA PHIX  ALTHOUGH THIS RESTRICTS US TOONLY A SMALL SUBSET OF ALL POSSIBLE LOSS FUNCTIONS WE WILL SEE THATIT STILL LEADS US TO SOME VERY INTERESTING AND USEFUL RESULTS  WEWILL EXAMINE THREE DIFFERENT COST FUNCTIONALS BEGINENUMERATEITEM SQUARED ERRORITEM ABSOLUTE ERROR AND ITEM UNIFORM COST  ENDENUMERATEOF THESE THE SQUARED ERROR CRITERION WILL EMERGE AS BEING THE MOSTIMPORTANT AND DESERVING OF STUDYRECALL FROM SECTION REFSECBAYESDEC THAT THE RISK FUNCTION RMC THETA TIMES DELTA RIGHTARROW DELTA IS THE AVERAGE LOSS WHERETHE AVERAGE IS WITH RESPECT TO X RTHETAPHI  E LTHETATHETAX  INTINFTYINFTYLTHETAPHIXFXTHETAXTHETADXTHE BAYES RISK IS THE EXPECTATION OF THE RISK WITH RESPECT TO ANASSUMED PRIOR DISTRIBUTION ON THETA RFTHETAPHI  ERTHETAPHI  INTTHETARVARTHETAPHIFTHETAVARTHETADVARTHETAWE SAW EARLIER THAT UNDER APPROPRIATE REGULARITY CONDITIONS WE MAYREVERSE THE ORDER OF INTEGRATION IN THE CALCULATION OF THE BAYES RISKFUNCTION TO OBTAINBEGINDISPLAYMATHRTAU PHI   INTXC  LEFT INTTHETA LVARTHETA PHIXFTHETAXVARTHETA X DVARTHETA RIGHT FXX DX ENDDISPLAYMATHAND NOTED THAT WE COULD MINIMIZE THE BAYES RISK BY MINIMIZING THEINNER INTEGRAL EM FOR EACH X SEPARATELY THAT IS WE MAY FINDFOR EACH X THE ACTION CALL IT PHIX THAT MINIMIZESBEGINDISPLAYMATHINT LVARTHETA PHIXFTHETAXVARTHETAXDVARTHETAENDDISPLAYMATHIN OTHER WORDS BFTHE BAYES DECISION RULE MINIMIZES THE POSTERIOR CONDITIONAL EXPECTEDLOSS GIVEN THE OBSERVATIONS  LET US NOW EXAMINE THE STRUCTURE OF THE BAYES RULE UNDER THE THREECOST FUNCTIONALS WE HAVE DEFINEDSUBSUBSECTIONSQUARED ERROR LOSSINDEXSQUARED ERROR LOSS BAYESIANLET US FIRST CONSIDER SQUARED ERROR LOSS AND INTRODUCE THE CONCEPTVIA THE FOLLOWING EXAMPLEBEGINEXAMPLECONSIDER THE ESTIMATION PROBLEM IN WHICH THETA  DELTA  0INFINITY AND  LTHETA DELTA  THETA DELTA2OUR PROBLEM IS TO ESTIMATE THE VALUE OF THETA  THAT IS THEDECISION DELTA IN DELTA IS OUR ESTIMATE OF THETA SO WECAN WRITE THETAHAT  DELTASUPPOSE WE OBSERVE THE VALUE OF A RANDOM VARIABLE X HAVING A UNIFORMDISTRIBUTION ON THE INTERVAL 0 THETA WITH DENSITYFXTHETAXVARTHETA  BEGINCASES1VARTHETA  MBOXIF  0  X  VARTHETA5PT0  TEXTOTHERWISEENDCASESNOTE THAT WE MAY WRITE BEGINDISPLAYMATHFXTHETAXVARTHETA  FRAC1VARTHETAI0VARTHETAX  FRAC1VARTHETAIXINFINITYVARTHETAENDDISPLAYMATHWE ARE TO FIND A BAYES RULE WITH RESPECT TO THE PRIOR DISTRIBUTIONASSUME FOR SOME REASON THAT WE KNOW OR SUSPECT THAT THE PARAMETERTHETA IS DISTRIBUTED ACCORDING TO AN EXPONENTIAL DENSITYBEGINDISPLAYMATHFTHETAVARTHETA  BEGINCASESVARTHETA EVARTHETA  TEXTIF  VARTHETA  05PT0  TEXTOTHERWISEENDCASESENDDISPLAYMATHTHIS IS A SIGNIFICANT POINT OF DEPARTURE FROM MAXIMUM LIKELIHOODESTIMATION AT THIS POINT WE HAVE NO PHYSICAL OR MATHEMATICALJUSTIFICATION FOR THIS ASSUMPTION  FOR NOW THIS DENSITY SIMPLYAPPEARS IN THE DEVELOPMENT  THE JOINT DENSITY OF X AND THETAIS THEREFOREBEGINDISPLAYMATHFXTHETAXVARTHETA FXTHETAXVARTHETAFTHETAVARTHETAENDDISPLAYMATHAND THE MARGINAL DISTRIBUTION OF X HAS THE DENSITYBEGINDISPLAYMATHFXX INTINFINITYINFINITYFXTHETAXVARTHETADVARTHETA BEGINCASES EX  MBOXIF  X  05PT0  TEXTOTHERWISEENDCASESENDDISPLAYMATHHENCE THE POSTERIOR DISTRIBUTION OF THETA GIVEN XX HAS THEDENSITY BEGINDISPLAYMATHFTHETAXVARTHETAX FRACFXTHETAXVARTHETAFXX BEGINCASES EXVARTHETA  MBOXIF  VARTHETA  X5PT0  TEXTOTHERWISEENDCASESENDDISPLAYMATHWHERE X  0  AGAIN WE SEE A SIGNIFICANT DIFFERENCE BETWEENBAYESIAN ESTIMATION AND ML ESTIMATION IN ML ESTIMATION THERE WAS NOCONCEPT OF A POSTERIOR BECAUSE THERE WAS NO CONCEPT OF A PRIOR  THEPOSTERIOR EXPECTED LOSS GIVEN XX ISBEGINDISPLAYMATHELTHETA DELTAXX  EXINTXINFINITYVARTHETA DELTA2EVARTHETADVARTHETAENDDISPLAYMATHTO FIND THE DELTA THAT MINIMIZES THIS EXPECTED LOSS WE MAY SET THEDERIVATIVE WITH RESPECT TO DELTA TO ZEROBEGINDISPLAYMATHFRACDDDELTAELTHETA DELTAXX  2EXINTXINFINITYVARTHETA DELTAEVARTHETADVARTHETA  0ENDDISPLAYMATHTHIS IMPLIES BEGINDISPLAYMATHPHIX  DELTA  FRACINTXINFINITYVARTHETA EVARTHETADVARTHETAINTXINFINITY EVARTHETADVARTHETA FRACX1 EXEX  X1ENDDISPLAYMATHTHIS THEREFORE IS A BAYES DECISION RULE WITH RESPECT TO FTHETA IFXX IS OBSERVED THEN THE ESTIMATE OF THETA IS X1ENDEXAMPLETHE PROBLEM OF POINT ESTIMATION OF A REAL PARAMETER USING QUADRATICLOSS OCCURS SO FREQUENTLY IN ENGINEERING APPLICATIONS THAT IT ISWORTHWHILE TO MAKE THE FOLLOWING OBSERVATION THE POSTERIOR EXPECTED LOSS GIVEN XX FOR A QUADRATIC LOSS FUNCTION AT DELTA IS THE SECOND MOMENT ABOUT DELTA OF THE POSTERIORDISTRIBUTION OF THETA GIVEN X  THAT ISBEGINDISPLAYMATHELTHETA DELTAXX  INTINFINITYINFINITYVARTHETADELTA2FTHETAXVARTHETAXDVARTHETAENDDISPLAYMATHBEGINTHEOREM  LABELTHMCONDMEANTHE POSTERIOR EXPECTED LOSS GIVEN XX   BEGINEQUATION    LABELEQPOSTLOSSELTHETA DELTAXX  INTINFINITYINFINITYVARTHETADELTA2FTHETAXVARTHETAXDVARTHETA  ENDEQUATIONIS MINIMIZED BY TAKING DELTA AS THE MEAN OF THE DISTRIBUTION THATISBEGINDISPLAYMATHPHIX  DELTA  ETHETAXXENDDISPLAYMATHENDTHEOREMBEGINPROOF TAKING THE DERIVATIVE OF REFEQPOSTLOSS WITH  RESPECT TO DELTA AND SIMPLIFYING WE OBTAIN INT VARTHETA FTHETAXVARTHETAXDVARTHETA  DELTA INTFTHETAXVARTHETAXD VARTHETAON THE LHS WE RECOGNIZE ETHETAXX AND ON THE RIGHT HAND WE HAVESIMPLY DELTAENDPROOFNOTE STRICTLY SPEAKING DELTA IS A EM FUNCTION AND SO A FIRSTVARIATION NOT A DERIVATIVE SHOULD BE EMPLOYED HERE  HOWEVER THEDERIVATION WORKS BECAUSE FOR EVERY XX DELTA IS A CONSTANTINDEPENDENT OF THE VARIABLE OF INTEGRATION  THE ESTIMATE OF THETAGIVEN BY THIS THEOREM IS TERMED MINIMUM EM MEANSQUARE ESTIMATE OFTHETA AND IS DENOTED THETAHATMS INDEXMINIMUM  MEANSQUAREBAYESIAN ESTIMATESUBSUBSECTIONABSOLUTE ERROR LOSSINDEXABSOLUTE ERROR LOSS BAYESIANANOTHER IMPORTANT LOSS FUNCTION IS ABSOLUTE VALUE OF THE DIFFERENCE  LTHETA DELTA  THETA DELTATHE BAYES RISK IS MINIMIZED BY MINIMIZINGBEGINEQUATIONELTHETA DELTAXX  INTINFINITYINFINITYVARTHETADELTAFTHETAXVARTHETAXDVARTHETALABELEQABSERRLOSENDEQUATION THE MINIMIZATION HERE IS MORE AWKWARD THAN FOR THE SQUARED ERROR LOSSSINCE THE ABSOLUTE VALUE FUNCTION IS NOT DIFFERENTIABLE EVERYWHEREOUR APPROACH IS TO  CONSIDER TWO CASES AND TAKE DERIVATIVES OFEACH PIECEBEGINENUMERATEITEM WHEN VARTHETADELTA THEN  PARTIALDDELTA INTTHETAVARTHETA FTHETAXVARTHETAX DVARTHETA  INT 1FTHETAXVARTHETAXDVARTHETAITEM WHEN VARTHETADELTA THEN  PARTIALDDELTA INTTHETAVARTHETA FTHETAXVARTHETAX DVARTHETA  INT 1FTHETAXVARTHETAXDVARTHETAENDENUMERATECOMBINING THESE TWO TOGETHER BY MEANS OF THE LIMITS OF INTEGRATIONAND SETTING THE DERIVATIVE WITH RESPECT TO DELTA EQUAL TO ZERO WEOBTAIN INTINFTYDELTA FTHETAXVARTHETAXDVARTHETA INTDELTAINFTY FTHETAXVARTHETAXDVARTHETA0OR INTINFTYDELTA FTHETAXVARTHETAXDVARTHETA INTDELTAINFTY FTHETAXVARTHETAXDVARTHETATHAT IS THE INTEGRAL UNDER THE DENSITY TO THE LEFT OF DELTA IS THESAME AS THAT TO THE RIGHT OF DELTA  WE HAVE THUS PROVEN THEFOLLOWING THEOREMBEGINTHEOREMELTHETA DELTAXX  INTINFINITYINFINITYVARTHETADELTAFTHETAXVARTHETAXDVARTHETAIS MINIMIZED BY TAKING BEGINDISPLAYMATHPHIX  DELTA  TEXTMEDIAN FTHETAXVARTHETAXENDDISPLAYMATHTHAT IS BAYES RULE CORRESPONDING TO THE ABSOLUTE ERROR CRITERION ISTO TAKE DELTA AS THE MEDIAN OF THE POSTERIOR DISTRIBUTION OFTHETA GIVEN XXENDTHEOREMINDEXMEDIAN AS LEASTABSOLUTE ERROR ESTIMATESUBSUBSECTION EM UNIFORM COST  THE LOSS FUNCTION ASSOCIATED WITH UNIFORM COST IS DEFINED ASBEGINDISPLAYMATHLVARTHETA DELTA  BEGINCASES0  TEXTIF  VARTHETADELTA LEQ EPSILON210PT1  TEXTIF  VARTHETADELTA  EPSILON2ENDCASESENDDISPLAYMATHIN OTHER WORDS AN ERROR LESS THAN EPSILON2 IS AS GOOD AS NO ERRORAND IF THE ERROR IS GREATER THAN EPSILON2 WE ASSIGN A UNIFORMCOST  THE BAYES RISK IS MINIMIZED BY MINIMIZINGBEGINALIGNEDINTINFINITYINFINITY LVARTHETA DELTAFTHETAXVARTHETAXDVARTHETA    INTINFINITYDELTAEPSILON2FTHETAXVARTHETAXDVARTHETA INTDELTAEPSILON2INFINITYFTHETAXVARTHETAXDVARTHETA 10PT    1  INTDELTAEPSILON2DELTAEPSILON2FTHETAXVARTHETAXDVARTHETAENDALIGNEDCONSEQUENTLY THE BAYES RISK IS MINIMIZED WHEN THE INTEGRAL BEGINDISPLAYMATHINTDELTAEPSILON2DELTAEPSILON2FTHETAXVARTHETAXDVARTHETAENDDISPLAYMATHIS MAXIMIZED  WHEN EPSILON IS SUFFICIENTLY SMALL ANDFTHETAXVARTHETAX IS CONTINUOUS IN VARTHETABEGINDISPLAYMATHINTDELTAEPSILON2DELTAEPSILON2FTHETAXVARTHETAXDVARTHETA APPROX 2EPSILONFTHETAXDELTAXENDDISPLAYMATHIN THIS CASE ITIS EVIDENT THAT THIS INTEGRAL IS MAXIMIZED WHEN VARTHETA ASSUMESTHE VALUE AT WHICH THE POSTERIOR DENSITYFTHETAXVARTHETAX IS MAXIMIZED  BEGINDEFINITIONTHE BF MODE OF A INDEXMODEOF A DISTRIBUTIONDISTRIBUTION IS THAT VALUE THAT MAXIMIZES THE PROBABILITY DENSITYFUNCTIONENDDEFINITIONWE HAVE PROVEN THE FOLLOWINGBEGINTHEOREM  THE BAYES RISK WITH UNIFORM COST IS MINIMIZED WHEN  THE INTEGRAL INTDELTAEPSILON2DELTAEPSILON2FTHETAXVARTHETAXDVARTHETAIS MINIMIZED  AS EPSILON RIGHTARROW 0 THE MINIMUM VALUE ISOBTAINED BY CHOOSING DELTA TO BE THE MODE  THE MAXIMIZING VALUE OF FTHETAXVARTHETAXENDTHEOREMSUBSECTIONMAP ESTIMATESINDEXMAXIMUM EM A POSTERIORI ESTIMATEBEGINDEFINITION  THE VALUE OF VARTHETA THATMAXIMIZES THE EM A POSTERIORI DENSITY THAT IS THE MODE OF THEPOSTERIOR DENSITY IS CALLED THE BF MAXIMUM APOSTERIORI PROBABILITY MAP ESTIMATE OF THETA  ENDDEFINITIONIF THE POSTERIOR DENSITY OF THETA GIVEN X IS UNIMODAL ANDSYMMETRIC THEN IT IS EASY TO SEE THAT THE MAP ESTIMATE AND THE MEANSQUARE ESTIMATE COINCIDE FOR THEN THE POSTERIOR DENSITY ATTAINS ITSMAXIMUM VALUE AT ITS EXPECTATION  FURTHERMORE UNDER THESECIRCUMSTANCES THE MEDIAN ALSO COINCIDES WITH THE MODE AND THEEXPECTATION  THUS IF WE ARE LUCKY ENOUGH TO BE DEALING WITH SUCHDISTRIBUTIONS THE VARIOUS ESTIMATES ALL TEND TO THE SAME THINGALTHOUGH IN THE DEVELOPMENT OF MAXIMUM LIKELIHOOD ESTIMATION THEORY WEESCHEWED THE CHARACTERIZATION OF THETA AS BEING RANDOM WE MAYGAIN SOME VALUABLE UNDERSTANDING OF THE MAXIMUM LIKELIHOOD ESTIMATE BYCONSIDERING THETA TO BE A RANDOM VARIABLE WHOSE PRIOR DISTRIBUTIONIS SO DISPERSED THAT IS HAS SUCH A LARGE VARIANCE THAT THEINFORMATION PROVIDED BY THE PRIOR IS VANISHINGLY SMALL  IF THE THEORYIS CONSISTENT WE WOULD HAVE A RIGHT TO EXPECT THAT THE MAXIMUMLIKELIHOOD ESTIMATE WOULD BE THE LIMITING CASE OF SUCH A BAYESIANESTIMATELET THETA BE CONSIDERED AS A RANDOM VARIABLE DISTRIBUTED ACCORDINGTO THE EM A PRIORI DENSITY FTHETAVARTHETA  THE EM APOSTERIORI DISTRIBUTION FOR THETA THEN IS GIVEN BYBEGINEQUATIONLABELPOSTERIORFTHETAXVARTHETAX  FRACFXTHETAXVARTHETAFTHETAVARTHETAFXXENDEQUATIONIF THE LOGARITHM OF THE EM A POSTERIORI DENSITY IS DIFFERENTIABLEWITH RESPECT TO THETA THEN THE MAP ESTIMATE IS GIVEN BY THESOLUTION TOBEGINEQUATIONLABELMAPEQNLEFT FRACPARTIAL LOG FTHETAXVARTHETAX PARTIAL VARTHETA RIGHTVARTHETA  HATTHETAMAP  0ENDEQUATIONTHIS EQUATION IS CALLED THE EM MAP EQUATIONTAKING THE LOGARITHM OF REFPOSTERIOR YIELDSBEGINDISPLAYMATHLOG FTHETAXVARTHETAX  LOG FXTHETAXVARTHETA LOG FTHETAVARTHETA LOG FXXENDDISPLAYMATHAND SINCE FXX IS NOT A FUNCTION OF THETA THE MAP EQUATIONBECOMESBEGINEQUATIONLABELGRADIENTFRACPARTIAL LOG FTHETAXVARTHETAX PARTIAL VARTHETA FRACPARTIAL LOG FXTHETAXVARTHETA PARTIAL VARTHETA FRACPARTIAL LOG FTHETAVARTHETA PARTIAL VARTHETA ENDEQUATIONCOMPARING REFGRADIENT TO THE STANDARD MAXIMUM LIKELIHOOD EQUATIONBEGINDISPLAYMATHLEFT FRACPARTIAL LTHETA XPARTIAL THETA RIGHTTHETA HATTHETAML  0ENDDISPLAYMATHWE SEE THAT THE TWOEXPRESSIONS DIFFER BY FRACPARTIAL LOG FTHETAVARTHETA PARTIAL VARTHETA  IFFTHETAVARTHETA IS SUFFICIENTLY FLAT THAT IS IF THEVARIANCE IS VERY LARGE ITS LOGARITHM WILL ALSO BE FLAT SO THEGRADIENT OF THE LOGARITHM WILL BE NEARLY ZERO AND THE EM A POSTERIORI DENSITY WILL BEMAXIMIZED IN THE LIMITING CASE AT THE MAXIMUM LIKELIHOOD ESTIMATE  BEGINEXAMPLELET X1 LDOTS XM DENOTE A RANDOM SAMPLE OF SIZE M FROM THENORMAL DISTRIBUTION NCTHETA SIGMA2 SUPPOSE SIGMA IS KNOWN ANDWE WISH TO FIND THE MAP ESTIMATE FOR THE MEAN THETA  THE JOINT DENSITYFUNCTION FOR X1 LDOTS XM ISBEGINDISPLAYMATHFX1 LDOTS XMX1 LDOTS XM THETA    PRODI1MFRAC1SQRT2PISIGMAEXPLEFTFRACXIVARTHETA22SIGMA2RIGHTENDDISPLAYMATHSUPPOSE THETA IS DISTRIBUTED NC0 SIGMATHETA2 THAT IS BEGINDISPLAYMATHFTHETAVARTHETA  FRAC1SQRT2PI SIGMATHETAEXPLEFT FRACVARTHETA22SIGMATHETA2RIGHTENDDISPLAYMATHSTRAIGHTFORWARD MANIPULATION YIELDSBEGINDISPLAYMATHFRACPARTIAL LOG FTHETAXVARTHETAX PARTIAL VARTHETA FRAC1SIGMA2SUMI1MXIVARTHETAFRACVARTHETASIGMATHETA2ENDDISPLAYMATHEQUATING THIS EXPRESSION TO ZERO AND SOLVING FOR VARTHETA YIELDSBEGINDISPLAYMATHHATTHETAMAP  FRACSIGMATHETA2SIGMATHETA2 FRACSIGMA2M FRAC1MSUMI1M XIENDDISPLAYMATHNOW IT IS CLEAR THAT AS SIGMATHETA2 RIGHTARROW INFINITYTHE LIMITING EXPRESSION IS THE MAXIMUM LIKELIHOOD ESTIMATEHATTHETAML  IT IS ALSO TRUE THAT AS MRIGHTARROWINFINITY THE MAP ESTIMATE ASYMPTOTICALLY APPROACHES THE ML ESTIMATETHUS AS THE KNOWLEDGE ABOUT THETA FROM THE PRIOR DISTRIBUTIONTENDS TO ZERO OR AS THE AMOUNT OF DATA BECOMES OVERWHELMING THE MAP ESTIMATE CONVERGES TO THE MAXIMUM LIKELIHOOD ESTIMATEENDEXAMPLESUBSECTIONSUMMARYLABELSECBAYESSUMFROM THE RESULTS ABOVE WE HAVE SEEN THAT THE BAYES ESTIMATE OFVARTHETA BASED UPON THE MEASUREMENT OF A RANDOM VARIABLE XDEPENDS UPON THE POSTERIOR DENSITY FTHETAXVARTHETAX  THECONVERSION OF THE PRIOR INFORMATION ABOUT THETA REPRESENTED BYFTHETAVARTHETA TO THE POSTERIOR DENSITY IS VIA THE EXPRESSIONBEGINEQUATION  LABELEQPOST1  FTHETAXVARTHETAX   FRACFXTHETAXVARTHETAFXX FTHETAVARTHETAENDEQUATIONTHE POSTERIOR DENSITY FTHETAXVARTHETAX REPRESENTS OUR STATEOF KNOWLEDGE AFTER THE MEASUREMENT OF X  IT IS ON THE POSTERIOR THATWE BASE OUR ESTIMATE AND FOR BAYESIAN PURPOSES CONTAINS ALL THEINFORMATION NECESSARY FOR ESTIMATION  ON THE BASIS OF THE POSTERIORESTIMATES CAN BE OBTAINED IN SEVERAL WAYSBEGINENUMERATEITEM FOR A MINIMUM VARIANCE QUADRATIC LOSS FUNCTION  VARTHETAHAT  ETHETAXITEM TO MINIMIZE VARTHETA  VARTHETAHAT SET  VARTHETAHAT TO THE MEDIAN OF FTHETAXVARTHETAXITEM TO MAXIMIZE THE PROBABILITY THAT VARTHETAHAT  VARTHETA SET  VARTHETAHAT TO THE MODE MAXIMUM VALUE OF  FTHETAXVARTHETAXENDENUMERATESUBSECTIONCONJUGATE PRIOR DISTRIBUTIONSIN GENERAL THE MARGINAL DENSITY FXX AND THE POSTERIOR DENSITYFTHETAXVARTHETAX ARE NOT EASILY CALCULATED  WE AREINTERESTED IN ESTABLISHING CONDITIONS ON THE STRUCTURE OF THEDISTRIBUTIONS INVOLVED THAT ENSURE TRACTABILITY IN THE CALCULATION OFTHE POSTERIOR DISTRIBUTION    WE SHALL INTRODUCE BELOW THE IDEA OFSEQUENTIAL ESTIMATION IN WHICH A BAYESIAN ESTIMATE IS UPDATED AFTEREACH OBSERVATION IN A SEQUENCE  IN ORDER TO HAVE TRACTABLE SEQUENTIALOBSERVATIONS WE MUST BE ABLE TO PROPAGATE ONE POSTERIOR DENSITY TOTHE NEXT BY MEANS OF AN UPDATE STEP  THIS IS MOST TRACTABLE IF THEDISTRIBUTIONS INVOLVED BELOW TO A CONJUGATE FAMILYBEGINDEFINITION  INDEXCONJUGATE FAMILY LET FC DENOTE A CLASS OF CONDITIONAL  DENSITY FUNCTIONS FXTHETA INDEXED BY VARTHETA AS  VARTHETA RANGES OVER ALL THE VALUES IN THETA  A CLASS PC  OF DISTRIBUTIONS IS SAID TO BE A BF CONJUGATE FAMILY FOR FC IF  THE POSTERIOR FTHETAXIN PC FOR ALL FXTHETAIN FC  AND ALL PRIORS FTHETAINPC  IN OTHER WORDS A FAMILY OF  DISTRIBUTIONS IS A CONJUGATE FAMILY IF IT CONTAINS BOTH THE PRIOR  FTHETA AND THE POSTERIOR DENSITY FTHETAX FOR ALL POSSIBLE  CONDITIONAL DENSITIES  A CONJUGATE FAMILY IS SAID TO BE EM CLOSED    UNDER SAMPLINGENDDEFINITIONWE GIVE SOME HERE EXAMPLES OF CONJUGATE FAMILIES  FOR MORE EXAMPLESAND ANALYSIS THE INTERESTED READER IS REFERRED TO CITEDEGROOT70BEGINEXAMPLESUPPOSE THAT X1 LDOTS XM IS A RANDOM SAMPLE FROM A BERNOULLIDISTRIBUTION WITH PARAMETER 0 LEQ THETA LEQ 1 WITH DENSITYBEGINDISPLAYMATHFXTHETAXVARTHETA  BEGINCASESVARTHETAX1VARTHETA1X  X IN 0  10  TEXTOTHERWISEENDCASESENDDISPLAYMATHSUPPOSE ALSO THAT THE PRIOR DISTRIBUTION OF THETA IS A BETADISTRIBUTION WITH PARAMETERS ALPHA0 AND BETA0 WITH DENSITYSEE BOX REFBOXBETADISTINDEXRANDOM VARIABLEBETAINDEXBETA RANDOM VARIABLEBETA RANDOM VARIABLEBEGINDISPLAYMATHFTHETAVARTHETA  BEGINCASESFRACDISPLAYSTYLE GAMMAALPHABETADISPLAYSTYLE GAMMAALPHAGAMMABETAVARTHETAALPHA11VARTHETABETA1  0  VARTHETA  110PT0  TEXTOTHERWISE ENDCASESENDDISPLAYMATHTHEN THE JOINT DISTRIBUTION OF THETA AND XBF X1X2LDOTSXMT ISBEGINALIGNEDFXBFTHETAXBFVARTHETA  FXBFTHETAXBFTHETAFTHETAVARTHETA   FRACGAMMAALPHABETAGAMMAALPHAGAMMABETATHETAALPHA  Y1 1THETABETAMY1ENDALIGNEDWHERE Y  SUMI1M XI  THE POSTERIOR DISTRIBUTION OF THETAGIVEN XBF IS FTHETAXBFVARTHETAXBF FRACFXBFTHETAXBFVARTHETA  FXBFXBF FRACFXBFTHETAXBFVARTHETA INT  FXBFTHETAXBFVARTHETADVARTHETAIT CAN BE SHOWN SEE EXERCISE REFEXCONJUGATE1 THATBEGINEQUATION FTHETAXBFVARTHETAXBF FRACGAMMAALPHABAR BETABARGAMMAALPHABARGAMMABETABARVARTHETAALPHABAR11VARTHETABETABAR1 SIMBETABFALPHABARBETABAR LABELEQCONJUGATE1ENDEQUATIONWHERE ALPHABAR  ALPHAY AND BETABAR  BETAMY  THUS BOTHFTHETA AND FTHETAX  HAVE A BETA DISTRIBUTIONENDEXAMPLEBEGINTEXTBOX09TEXTWIDTHTHE BETA DISTRIBUTION  THE BETA  PDF IS GIVEN BYLABELBOXBETADISTINDEXRANDOM VARIABLEBETAINDEXBETA RANDOM VARIABLEBETA RANDOM VARIABLE FXX  FRACGAMMAALPHABETAGAMMAALPHAGAMMABETAXALPHA1 1XBETA1FOR 0 LEQ X LEQ 1 WHERE ALPHA AND BETA ARE PARAMETERSTHIS IS DENOTED BY SAYING X SIM BETABFALPHABETA  THE MEANAND VARIANCE ARE MU  FRACALPHAALPHABETA AND SIGMA2 FRACALPHABETAALPHABETA2ALPHABETA1ENDTEXTBOXBEGINTEXTBOX09TEXTWIDTHTHE GAMMA DISTRIBUTION  THELABELBOXGAMMADISTINDEXRANDOM VARIABLEGAMMAINDEXGAMMA RANDOM VARIABLEGAMMA RANDOM VARIABLE  GAMMA PDF IS PARAMETERIZED BY TWO PARAMETERS ALPHA AND  BETA HAVING PDF FXX  FRAC1 BETAALPHAGAMMAALPHA XALPHA1 EXBETAFOR X  0  THIS IS DENOTED BY SAYING X SIMGAMMAALPHABETA  THE MEAN AND VARIANCE ARE MU ALPHABETA AND SIGMA2  ALPHABETA2  ENDTEXTBOXBEGINEXAMPLE LABELEXMCONJUGATE2SUPPOSE THAT X1 LDOTS XM IS A RANDOM SAMPLE FROM A POISSONDISTRIBUTION WITH PARAMETER THETA 0 WITH PMFBEGINDISPLAYMATHFXTHETAXVARTHETA  LEFT BEGINARRAYLLFRACEVARTHETAVARTHETAXX  X012LDOTS 10PT0  MBOXOTHERWISEENDARRAYRIGHT ENDDISPLAYMATHSUPPOSE ALSO THAT THE PRIOR DISTRIBUTION OF THETA IS A GAMMADISTRIBUTION WITH PARAMETERS ALPHA0 AND BETA0 WITH DENSITYBEGINDISPLAYMATHFTHETAVARTHETA  BEGINCASESFRAC1BETAALPHAGAMMAALPHAVARTHETAALPHA1EVARTHETABETA   VARTHETA 0 10PT0  TEXTOTHERWISEENDCASESENDDISPLAYMATHTHEN SEE EXERCISE REFEXCONJUGATE2 THE POSTERIOR DISTRIBUTION OFTHETA WHEN XI  XI I1LDOTS M IS A GAMMAALPHA Y11BETA M WHERE Y  SUMI1MXIENDEXAMPLEBEGINEXAMPLE LABELEXMCONJUGATE3SUPPOSE THAT X1 LDOTS XM IS A RANDOM SAMPLE FROM AN EXPONENTIALDISTRIBUTION WITH PARAMETER THETA 0 WITH DENSITYBEGINDISPLAYMATHFXTHETAXVARTHETA  BEGINCASESTHETA ETHETA X  X  05PT0  TEXTOTHERWISEENDCASESENDDISPLAYMATHSUPPOSE ALSO THAT THE PRIOR DISTRIBUTION OF THETA IS A GAMMADISTRIBUTION WITH PARAMETERS ALPHA0 AND BETA0 WITH DENSITYBEGINDISPLAYMATHFTHETAVARTHETA  BEGINCASESFRAC1BETAALPHAGAMMAALPHAVARTHETAALPHA1EBETA VARTHETA   VARTHETA 0 10PT0  TEXTOTHERWISEENDCASESENDDISPLAYMATHTHEN SEE EXERCISE REFEXCONJUGATE3 THE POSTERIOR DISTRIBUTION OFTHETA WHEN XI  XI I1LDOTS M IS A GAMMAALPHA M11BETA Y WHERE Y  SUMI1MXIENDEXAMPLEBEGINEXAMPLE  IMPORTANT SUPPOSE THAT X1 LDOTS XM IS A RANDOM SAMPLE FROM  A GAUSSIAN DISTRIBUTION WITH UNKNOWN MEAN THETA AND KNOWN  VARIANCE SIGMA2  SUPPOSE ALSO THAT THE PRIOR DISTRIBUTION OF  THETA IS A GAUSSIAN DISTRIBUTION WITH MEAN VARTHETA0 AND  VARIANCE SIGMATHETA2  THEN THE POSTERIOR DISTRIBUTION OF  THETA WHEN XI  XI I1LDOTS M IS A GAUSSIAN  DISTRIBUTION WITH MEANBEGINEQUATIONLABELTHETAHATTHETAHATC  FRACFRACDISPLAYSTYLE VARTHETA0DISPLAYSTYLE SIGMATHETA2 FRAC DISPLAYSTYLE XBARDISPLAYSTYLE SIGMAM2FRACDISPLAYSTYLE 1DISPLAYSTYLE SIGMATHETA2 FRACDISPLAYSTYLE 1DISPLAYSTYLE  SIGMAM2ENDEQUATIONAND VARIANCEBEGINEQUATIONLABELTHETAVARSIGMAHATTHETA2  FRACSIGMAM2SIGMATHETA2SIGMAM2SIGMATHETA2ENDEQUATIONWHERE BEGINDISPLAYMATHXBAR  FRAC1MSUMI1M XI QQUAD MBOXAND QQUAD SIGMAM2  SIGMA2MENDDISPLAYMATHDUE TO ITS IMPORTANCE WE PROVIDE A DEMONSTRATION OF THE ABOVE CLAIMFOR INFINITY  VARTHETA  INFINITY THE CONDITIONAL DENSITY OFX1 LDOTS XM SATISFIESBEGINALIGNFXBFTHETA XBFVARTHETA   PRODI1MFRAC1SQRT2PISIGMAEXPLEFTFRACXIVARTHETA22SIGMA2RIGHTNONUMBER 10PT    2PIFRACM2SIGMAMEXPLEFTFRAC12SIGMA2SUMI1MXIXBAR2RIGHTEXPLEFTFRACM2SIGMA2VARTHETAXBAR2RIGHT LABELLIKEENDALIGNTHE PRIOR DENSITY OF THETA SATISFIESBEGINEQUATIONLABELPRIORDENFTHETAVARTHETA  FRAC1SQRT2PISIGMATHETAEXPLEFTFRACVARTHETAVARTHETA022SIGMATHETA2RIGHTENDEQUATIONAND THE POSTERIOR DENSITY FUNCTION OF THETA WILL BE PROPORTIONAL TOTHE PRODUCT OF REFLIKE AND REFPRIORDEN  LETTING THE SYMBOLPROPTO DENOTE PROPORTIONALITY WE HAVE BEGINALIGNEDFTHETAXBFVARTHETAXBF  PROPTO EXPLEFTFRACM2SIGMA2VARTHETAXBAR2RIGHTEXPLEFTFRACVARTHETAVARTHETA022SIGMATHETA2RIGHT10PT   EXPLEFTFRACVARTHETAXBAR22SIGMAM2   FRACVARTHETAVARTHETA022SIGMATHETA2RIGHTENDALIGNEDSIMPLIFYING THE EXPONENT WE OBTAINBEGINDISPLAYMATHFRACVARTHETAXBAR2SIGMAM2   FRACVARTHETAVARTHETA02SIGMATHETA2 FRACSIGMAM2 SIGMATHETA2SIGMAM2SIGMATHETA2VARTHETATHETAHATC2 FRAC1SIGMAM2 SIGMATHETA2XBARVARTHETA02ENDDISPLAYMATHWHERE THETAHATC IS GIVEN BY REFTHETAHATTHUSBEGINDISPLAYMATHFTHETAXBFVARTHETAXBF  PROPTO EXPLEFT 12  FRACSIGMAM2SIGMATHETA2SIGMAM2SIGMATHETA2 VARTHETATHETAHATC2RIGHTENDDISPLAYMATHCONSEQUENTLY SUITABLY NORMALIZED WE SEE THAT THE POSTERIOR DENSITYOF THETA GIVEN X1 LDOTS XM IS NORMAL WITH MEAN GIVEN BYREFTHETAHAT AND VARIANCE GIVEN BY REFTHETAVARUPON REARRANGING REFTHETAHAT WE SEE THATBEGINDISPLAYMATHTHETAHATC  FRACSIGMAM2SIGMATHETA2 SIGMAM2VARTHETA0 FRACSIGMATHETA2SIGMATHETA2 SIGMAM2XBARENDDISPLAYMATHWHICH IS EXACTLY THE SAME AS THE ESTIMATE GIVEN BY REFWEIGHTEDTHUS THE WEIGHTED AVERAGE AS PROPOSED AS A REASONABLE WAY TOINCORPORATE PRIOR INFORMATION INTO THE ESTIMATE TURNS OUT TO BEEXACTLY A BAYES ESTIMATE FOR THE PARAMETER GIVEN THAT THE PRIOR IS AMEMBER OF THE NORMAL CONJUGATE FAMILYENDEXAMPLEAS THIS EXAMPLE SHOWS THE CONJUGATE PRIOR FOR A GAUSSIAN DISTRIBUTIONIS A GAUSSIAN DISTRIBUTION  YET ANOTHER REASON FOR ENGINEERINGINTEREST IN THESE DISTRIBUTIONSWE WILL SEE BELOW THAT CONJUGATE CLASSES OF DISTRIBUTIONS ARE USEFULIN SEQUENTIAL ESTIMATION IN WHICH A POSTERIOR DENSITY AT ONE STAGE OFCOMPUTATION IS USED AS A PRIOR FOR THE NEXT STAGEBEGINEXERCISESITEM SUPPOSE THAT X SIM PLAMBDA1 POISSON AND N SIM  PLAMBDA2 INDEPENDENTLY  LET Y   X  N SIGNAL PLUS  NOISE  THIS MIGHT MODEL AN OPTICAL COMMUNICATIONS PROBLEM WHERE  THE RECEIVED PHOTON COUNTS Y ARE MODELED AS THE SIGNAL PHOTON  COUNTS X PLUS SOME BACKGROUND PHOTON COUNTS N  BEGINENUMERATE  ITEM FIND THE DISTRIBUTION OF Y  ITEM FIND THE CONDITIONAL PMF FOR X GIVEN Y  ITEM FIND THE MINIMUM MEANSQUARED ERROR MMSE ESTIMATOR OF X  ITEM COMPUTE THE MEAN AND MEANSQUARED ERROR FOR YOUR MMSE    ESTIMATOR  IS THE ESTIMATE UNBIASED  ENDENUMERATEITEM CITESCHARFL1991  IMPERFECT GEIGER COUNTER  A RADIOACTIVE SOURCE  EMITS N RADIOACTIVE PARTICLES  WE ASSUME THAT THE PARTICLE  GENERATION IS GOVERNED BY A POISSON DISTRIBUTION WITH PARAMETER LAMBDA FNN  PNN  FRACLAMBDANN ELAMBDA QQUAD  N GEQ  0THE N PARTICLES EMITTED ARE DETECTED BY AN IMPERFECT GEIGER COUNTERWHICH DETECTS WITH PROBABILITY P  OF THE N PARTICLES EMITTED THEIMPERFECT GEIGER COUNTER DETECTS K LEQ N OF THEM  THE PROBLEM WEEXAMINE IS ESTIMATING N FROM THE MEASUREMENT K USING BAYESIANMETHODSBEGINENUMERATEITEM SHOW THAT K THE NUMBER OF DETECTED PARTICLES IS  CONDITIONALLY DISTRIBUTED AS PKN  N CHOOSE K PK 1PNKBINOMIAL DISTRIBUTIONITEM SHOW THAT THE JOINT DISTRIBUTION IS PKN  N CHOOSE KPK 1PNK ELAMBDAFRACLAMBDANNITEM SHOW THAT K IS DISTRIBUTED AS PK  FRACLAMBDA PNKELAMBDA PPOISSON WITH PARAMETER LAMBDA PITEM COMPUTE THE POSTERIOR DISTRIBUTION PNKITEM SHOW THAT THE CONDITIONAL MEAN THE MINIMUM MEANSQUARE  ESTIMATE IS ENK  K  LAMBDA1PALSO SHOW THAT THE CONDITIONAL VARIANCE THE VARIANCE OF THE ESTIMATEIS EN ENK2K   LAMBDA1PENDENUMERATEITEM  CITESCHARFL1991 LET X1X2LDOTS XN EACH BE IID  PLAMBDA POISSON DISTRIBUTED WITH PARAMETER THETA   LAMBDA FXLAMBDAXLAMBDA  FRACLAMBDAXX ELAMBDA  QQUAD X IN ZBB QQUAD LAMBDA GEQ 0ITEM SUPPOSE THAT WE HAVE A KNOWN PRIOR ON LAMBDA THAT IS EXPONENTIAL FLAMBDALAMBDA  A EALAMBDAQQUAD LAMBDA GEQ 0 QQUADA  0BEGINENUMERATEITEM SHOW THAT T  SUMI1N XI IS SUFFICIENT FOR LAMBDAITEM SHOW THAT THE MARGINAL DENSITY FOR X IS FXX  FRACAPRODI1N XIFRACGAMMAT1NAT1ITEM SHOW THAT THE CONDITIONAL POSTERIOR DENSITY FOR LAMBDA  GIVEN X IS FLAMBDAXLAMBDAX  ENALAMBDA LAMBDAT  FRACNAT1 GAMMAT1THIS IS A GAMMA DENSITY WITH PARAMETERS T1 AND NAITEM SHOW THAT THE CONDITIONAL MEAN BAYES ESTIMATE OF LAMBDA IS LAMBDAHAT  FRACT1NAITEM SHOW THAT THE CONDITIONAL VARIANCE OF LAMBDAHAT IS T2NA2ENDENUMERATEITEM LABELEXCONJUGATE1   SHOW THAT REFEQCONJUGATE1 IS TRUEITEM LABELEXCONJUGATE2 SHOW THAT THE POSTERIOR DENSITY  FTHETAXBFVARTHETAXBF OF EXAMPLE REFEXMCONJUGATE2 IS  A GAMMAALPHA YBETA M DENSITY  ITEM LABELEXCONJUGATE3 SHOW THAT THE POSTERIOR DENSITY  FTHETAXBFVARTHETAXBF OF EXAMPLE REFEXMCONJUGATE3 IS  A GAMMAALPHA MBETA Y DENSITYITEM SHOW THAT IF X1 SIM GAMMAPLAMBDA AND X2 SIM  GAMMAQLAMBDA INDEPENDENTLY THEN  BEGINENUMERATE  ITEM Y  X1  X2 IS DISTRIBUTED AS GAMMAPQLAMBDA  SUMS    OF GAMMAS ARE GAMMAS  ITEM  Z  X1X1X2 IS DISTRIBUTED AS BETAPQ  ENDENUMERATEENDEXERCISESSUBSECTIONIMPROPER PRIOR DISTRIBUTIONSAS WE SAW WITH THE EXAMPLE DEVELOPED FOR THE MAP ESTIMATESOMETIMES THE PRIOR KNOWLEDGE AVAILABLE ABOUT A PARAMETER IS VERYSLIGHT WHEN COMPARED TO THE INFORMATION WE EXPECT TO ACQUIRE FROMOBSERVATIONS  CONSEQUENTLY IT MAY NOT BE WORTHWHILE FOR US TO SPENDA GREAT DEAL OF TIME AND EFFORT IN DETERMINING A SPECIFIC PRIORDISTRIBUTION  RATHER IT MIGHT BE USEFUL IN SOME CIRCUMSTANCES TOMAKE USE OF A STANDARD PRIOR THAT WOULD BE SUITABLE IN MANY SITUATIONSFOR WHICH IT IS DESIRABLE TO REPRESENT VAGUE OR UNCERTAIN PRIORINFORMATION NOINDENT UNDERLINEEM DEFINITION A EM PROPER DENSITY FUNCTION IS ONE WHOSE INTEGRAL OVER THEPARAMETER SPACE IS UNITY  THIS IS THE ONLY TYPE OF DENSITY FUNCTIONWITH WHICH WE HAVE HAD ANYTHING TO DO WITH THUS FAR  IN FACT WE KNOWTHAT VIRTUALLY ANY CONTINUOUS NONNEGATIVE FUNCTION WHOSE INTEGRAL OVERTHE PARAMETER SPACE IS FINITE CAN BE TURNED INTO A PROPER DENSITYFUNCTION BY DIVIDING IT BY THE INTEGRALNOINDENT UNDERLINEEM DEFINITIONAN EM IMPROPER DENSITY FUNCTION IS A NONNEGATIVE FUNCTIONWHOSE INTEGRAL OVER THE WHOLE PARAMETER SPACE THETA IS INFINITEFOR EXAMPLE IF THETA IS THE REAL LINE AND BECAUSE OF VAGUENESSTHE PRIOR DISTRIBUTION OF THETA IS SMOOTH AND VERY WIDELY SPREADOUT OVER THE LINE THEN WE MIGHT FIND IT CONVENIENT TO ASSUME AUNIFORM OR CONSTANT DENSITY OVER THE WHOLE LINE IN ORDER TO REPRESENTTHIS PRIOR INFORMATION  EVEN THOUGH THIS IS NOT A PROPER DENSITY WEMIGHT CONSIDER FORMALLY CARRYING OUT THE CALCULATIONS OF BAYES THEOREMAND ATTEMPT TO COMPUTE A POSTERIOR DISTRIBUTION  SUPPOSE THETA  INFINITY INFINITY LETFTHETAVARTHETA  1 BE AN IMPROPER PRIOR FOR THETA ANDSUPPOSE X  X IS OBSERVED  FORMALLY APPLYING BAYES THEOREM WE OBTAINBEGINDISPLAYMATHFTHETAXVARTHETAX  FXTHETAXVARTHETAFTHETAVARTHETA OVERINTTHETAFXTHETAXVARTHETAPRIMEFTHETAVARTHETAPRIME DVARTHETAPRIME  FXTHETAXVARTHETA OVERINTTHETAFXTHETAXVARTHETAPRIMEDVARTHETAPRIME ENDDISPLAYMATHWE SEE THAT IF BEGINEQUATIONLABELFINITEINTTHETAFXTHETAXVARTHETADVARTHETA  INFINITYENDEQUATIONTHEN THE POSTERIOR DENSITY FTHETAXVARTHETAX IS AT LEASTDEFINED  BEGINEXAMPLESUPPOSE X1 LDOTS XM ARE SAMPLES FROM A NORMAL POPULATION WITHMEAN THETA AND VARIANCE SIGMA2  LET THETA BE DISTRIBUTEDACCORDING TO AN IMPROPER PRIOR FTHETAVARTHETA  1  THECONDITIONAL  DENSITY OF X1 LDOTS XM GIVEN THETAVARTHETA IS  BEGINALIGNEDFX1LDOTS XMTHETA X1 LDOTS XMVARTHETA   PRODI1MFRAC1SQRT2PISIGMAEXPLEFTFRACXIVARTHETA22SIGMA2RIGHT10PT    2PIFRACM2SIGMAMEXPLEFTFRAC12SIGMA2SUMI1MXIXBAR2RIGHTEXPLEFTFRACM2SIGMA2VARTHETAXBAR2RIGHTENDALIGNEDWHERE XBAR  FRAC1MSUMI1M XI  THE FIRST EXPONENTIALTERM IN THIS EXPRESSION IS INDEPENDENT OF VARTHETA AND SINCE THEINTEGRAL OF THE ENTIRE EXPRESSION QUANTITY WITH RESPECT TO VARTHETAOVER INFINITY  INFINITY IS FINITE WE MAY NORMALIZE THISQUANTITY TO OBTAIN A POSTERIOR DENSITY FOR THETA OF THE FORMBEGINDISPLAYMATHFTHETAX1LDOTS XMVARTHETAX1 LDOTS XM FRAC1SQRT2PISIGMAMEXPLEFTFRACVARTHETAXBAR22SIGMAM2RIGHT ENDDISPLAYMATHWHERE SIGMAM  SIGMA  SQRTM  THUS THE POSTERIORDISTRIBUTION OF THETA WHEN XI  XI I1 LDOTS M IS ANORMAL DISTRIBUTION WITH MEAN XBAR AND VARIANCE SIGMA2MALTHOUGH THE PRIOR DISTRIBUTION IS IMPROPER THE POSTERIORDISTRIBUTION IS A PROPER NORMAL DISTRIBUTION AFTER JUST ONEOBSERVATION HAS BEEN MADE  UNDER SQUARED ERROR LOSS THEREFORE THEBAYES ESTIMATE FOR THETA USING AN IMPROPER PRIOR IS THE SAMPLE MEAN  COMPARING THIS WITH PREVIOUS RESULTS WE SEE THAT THIS ESTIMATEALSO COINCIDES WITH THE MAXIMUM LIKELIHOOD ESTIMATE  CONSEQUENTLY WE MAYVIEW THE MAXIMUM LIKELIHOOD AS A THE LIMIT OF A MAP ESTIMATE AS THEVARIANCE OF THE PRIOR DISTRIBUTION TENDS TO INFINITY OR B THE MEANSQUARE ESTIMATE ASSOCIATED WITH AN IMPROPER PRIOR DISTRIBUTION ENDEXAMPLESUBSECTIONSEQUENTIAL BAYES ESTIMATIONLABELSECSEQBAYESTHUS FAR IN OUR TREATMENT OF ESTIMATION WE HAVE ASSUMED THAT ALL OFTHE INFORMATION TO BE USED TO MAKE A DECISION OR ESTIMATE IS AVAILABLEAT ONE TIME  FREQUENTLY WE ARE INTERESTED IN ADDRESSING PROBLEMSWHERE THE DATA BECOMES AVAILABLE SEQUENTIALLY AND WE DESIRE TO UPDATEOUR ESTIMATE AS NEW DATA ARRIVE  TO INTRODUCE THIS TOPIC WE WILLCONSIDER FIRST THE CASE OF ESTIMATING THETA GIVEN TWO MEASUREMENTSOBTAINED AT DIFFERENT TIMESLET THETA BE THE PARAMETER TO BE ESTIMATED AND SUPPOSE X1 ANDX2 ARE TWO OBSERVED RANDOM VARIABLES  SUPPOSE THAT X1 AND X2HAVE A JOINT CONDITIONAL PROBABILITY DENSITY FUNCTIONFX1X2THETAX1X2VARTHETA FOR EACH VARTHETAINTHETATHE POSTERIOR DENSITY FUNCTION OF THETA CONDITIONED ON X1 X1AND X2X2 ISBEGINEQUATIONLABELCOND0FTHETAX1X2VARTHETAX1X2  FX1X2THETAX1X2VARTHETAFTHETAVARTHETA OVERINTTHETAFX1X2THETAX1X2VARTHETAPRIMEFTHETAVARTHETAPRIME DVARTHETAPRIME ENDEQUATIONIF WE HAD BOTH X1 AND X2 AT OUR DISPOSAL THEN WE WOULD SIMPLYUSE THIS POSTERIOR DENSITY TO FORM OUR ESTIMATE ACCORDING TO THE LOSSFUNCTION WE CHOOSE SAY FOR EXAMPLE SQUARED ERROR LOSS  BUT SUPPOSEWE FIRST OBSERVE X1 AND AT SOME FUTURE TIME HAVE THE PROSPECT OFOBSERVING X2  THERE ARE TWO WAYS WE MIGHT PROCEED A WECOULD PUT X1 ON THE SHELF AND WAIT UNTIL X2 IS OBTAINED TOCALCULATE OUR ESTIMATE B WE COULD USE X1 AS SOON AS IT ISOBTAINED TO ESTIMATE THETA USING THAT INFORMATION ONLY THEN UPDATETHAT ESTIMATE ONCE X2 BECOMES AVAILABLE  OUR GOAL IS TO SHOW THATTHESE TWO APPROACHES YIELD THE SAME RESULTWE FIRST COMPUTE THE POSTERIOR DISTRIBUTION OF THETA GIVENX1 ONLYBEGINEQUATIONLABELCOND1FTHETAX1VARTHETAX1  FX1THETAX1VARTHETAFTHETAVARTHETA OVERINTTHETAFX1THETAX1VARTHETAPRIMEFTHETAVARTHETAPRIME DVARTHETAPRIME ENDEQUATIONWE NEXT COMPUTE THE CONDITIONAL DISTRIBUTION OF X2 GIVENTHETAVARTHETA EM AND X1  X1 YIELDING  BEGINEQUATIONLABELCOND2FX2THETA X1X2VARTHETA X1 FX1X2THETAX1X2VARTHETA OVERFX1THETAX1VARTHETAENDEQUATIONAND COMPUTE THE CORRESPONDING POSTERIOR DENSITY OF THETABEGINEQUATIONLABELCOND3FTHETAX1X2PRIMEVARTHETAX1  FX2THETA X1X2VARTHETAX1FTHETAX1VARTHETAX1OVERINTTHETA FX2THETA X1X2VARTHETAPRIMEX1FTHETAX1VARTHETAPRIMEX1DVARTHETAPRIMEENDEQUATIONSUBSTITUTING REFCOND1 AND REFCOND2 INTO REFCOND3YIELDS AFTER SOME SIMPLIFICATION THE CONDITIONAL DENSITY GIVEN INREFCOND0 THUS WE SEE THAT IF THE OBSERVATIONS ARE RECEIVEDSEQUENTIALLY THE POSTERIOR DISTRIBUTION CAN ALSO BE COMPUTEDSEQUENTIALLY THAT ISBEGINDISPLAYMATHFTHETAX1X2PRIMEVARTHETAX1X2FTHETAX1X2VARTHETAX1X2ENDDISPLAYMATH  IT ALSO FOLLOWS FROM THIS DERIVATION THAT IF THE POSTERIORDISTRIBUTION OF THETA WHEN X1 X1 AND X2  X2 IS COMPUTEDIN TWO STAGES THE FINAL RESULT IS THE SAME REGARDLESS OF WHETHERX1 OR X2 IS OBSERVED FIRSTIT IS STRAIGHTFORWARD TO GENERALIZE THIS RESULT TO THE CASE OFOBSERVING X1 X2 X3 LDOTS  AND SEQUENTIALLY UPDATING THE ESTIMATEOF THETA AS TIME PROGRESSES  THERE IS A GENERAL THEORY OFSEQUENTIAL SAMPLING WHICH WE WILL NOT DEVELOP IN THIS CLASS THATTREATS THIS PROBLEM IN DETAIL  FOR DETAILS SEECITEFERGUSON67DEGROOT70  ALTHOUGH WE WILL NOT PURSUE SEQUENTIALDETECTION THEORY FURTHER IN THIS COURSE WE WILL DEVELOP THE CONCEPTOF A CLOSELY RELATED SUBJECT THAT OF SEQUENTIAL ESTIMATION THEORYBEGINEXERCISESITEM SHOW THAT SUBSTITUTING REFCOND1 AND REFCOND2 INTOREFCOND3 YIELDS REFCOND0ENDEXERCISESSUBSECTIONCONNECTIONS WITH MINIMUM MEANSQUARED ESTIMATIONLABELSECBAYESMMSEIN CHAPTER REFCHAPVECTAP CONSIDERABLE EFFORT WAS DEVOTED TOEXPLAINING AND EXPLORING MINIMUM MEANSQUARED MMS ESTIMATION  INTHAT CONTEXT AN ESTIMATE XHAT OF A SIGNAL X WHERE XHAT IS ALINEAR COMBINATION OF SOME SET OF DATA XHAT  C1 P1  C2 P2  CDOTS CM PMWAS DETERMINED SO THAT THE AVERAGE OF THE SQUARED ERROR WHERE E  X  XHATIS MINIMIZED  THAT IS EE2  EXXHAT2 IS MINIMIZED  NOW RECALL FROM THEOREM REFTHMCONDMEAN THAT FOR A BAYESESTIMATOR USING A QUADRATIC LOSS FUNCTION THE BEST ESTIMATE OF ARANDOM PARAMETER THETAHAT GIVEN A MEASUREMENT X IS THECONDITIONAL EXPECTATION THETAHAT  ETHETAXAND THAT THE BAYES COST ELTHETADELTAX WAS TERMED THEMEANSQUARED ERROR OF THETAHAT    THUS THE CONDITIONAL MEAN IS THEESTIMATOR WHICH MINIMIZES THE MEANSQUARED ERROR  OBVIOUSLY THERE MUST BE SOME CONNECTION BETWEEN THE TWO TECHNIQUESSINCE BOTH OF THEM RELY ON A MINIMUM MEANSQUARED ERROR CRITERIONWE MAKE SOME OBSERVATIONS IN THIS REGARD  OUR COMPARISON WILL BEAIDED BY USING A NOTATION WHICH IS MORE SIMILAR IN EACH CASE  FOR THEFIRST CASE WE WILL WRITE OUR ESTIMATE ASBEGINEQUATION THETAHAT  C1 X1  C2 X2  CDOTS  CM XMLABELEQMMSEEX1ENDEQUATIONTHAT IS WE ARE ESTIMATING THE PARAMETER THETA AS A LINEARCOMBINATION OF THE M RANDOM VARIABLES X1X2LDOTSXM  WE WILLREFER TO THIS AS A LINEAR ESTIMATOR  IN THE SECOND CASE WE MIGHTACTUALLY HAVE SEVERAL OBSERVATIONS SO OUR ESTIMATOR WILL BE OF THEFORMBEGINEQUATION THETAHAT  ETHETAX1X2LDOTSXMLABELEQMMSEEX2ENDEQUATIONWE WILL REFER TO THIS AS A CONDITIONAL MEAN ESTIMATORBY THE FORMULATION OF THE LINEAR ESTIMATOR REFEQMMSEEX1 WE HAVERESTRICTED ATTENTION TO ONLY THAT CLASS OF ESTIMATORS WHICH AREEM LINEAR FUNCTIONS OF THE OBSERVATIONS  THE CONDITIONAL MEANESTIMATOR REFEQMMSEEX2 HAS NO SUCH RESTRICTIONS THE CONDITIONALMEAN MAY NOT BE A LINEAR FUNCTION OF THE OBSERVATIONS  THECONDITIONAL MEAN ESTIMATOR MAY IN FACT BE A NONLINEAR FUNCTION OFTHE OBSERVATIONS  THE CONDITIONAL MEAN ESTIMATE THUS GUARANTEESMINIMUM MEAN SQUARED ERROR ACROSS ALL POSSIBLE ESTIMATES  HOWEVERFOR SOME DISTRIBUTIONS THE RESULTING NONLINEARITY MAY MAKE THECOMPUTATION INTRACTABLEHOWEVER IN THE CASE OF ESTIMATING THE MEAN OF A GAUSSIANDISTRIBUTION THE CONDITIONAL MEAN ESTIMATE EM IS LINEAR AS WESHALL SEE IN THE NEXT SECTION SO THAT THE LINEAR ESTIMATOR AND THECONDITIONAL MEAN ESTIMATOR COINCIDE  THIS IS YET ANOTHER REASON WHYTHE GAUSSIAN DISTRIBUTION IS OF PRACTICAL INTERESTSUBSECTIONBAYES ESTIMATION WITH THE GAUSSIAN DISTRIBUTIONLABELSECGAUSSBAYESWE HAVE ENCOUNTERED THROUGHOUT THE BOOK THE GAUSSIAN DISTRIBUTION IN AVARIETY OF SETTINGS  WE WILL CONSIDER AGAIN THE PROBLEM OFJOINTLYDISTRIBUTED GAUSSIAN RANDOM VARIABLES SUCH AS XY ORRANDOM VECTORS SUCH AS XBFYBF  SINCE THE DISTRIBUTION OFGAUSSIAN RANDOM VARIABLES IS UNIMODAL AND SYMMETRIC  AND SINCE THECONDITIONAL DISTRIBUTION  FXY IS ALSO GAUSSIAN THIS CONDITIONALDISTRIBUTION PROVIDES WHAT IS NEEDED FOR ESTIMATING THE RANDOMVARIABLE X FOR A VARIETY OF COST FUNCTIONSBEGINENUMERATEITEM FOR A SQUAREDERROR LOSS FUNCTION THE BEST ESTIMATE IS THE  CONDITIONAL MEANITEM FOR AN ABSOLUTEERROR LOSS FUNCTION THE BEST ESTIMATE IS THE  MEDIAN WHICH FOR A GAUSSIAN IS THE SAME AS THE MEANITEM FOR A UNIFORM COST FUNCTION THE BEST ESTIMATE IS THE MODE  WHICH FOR GAUSSIAN IS THE SAME AS THE MEANENDENUMERATETHUS DETERMINING THE CONDITIONAL DISTRIBUTION AND IDENTIFYING THEMEAN PROVIDES THE NECESSARY ESTIMATES FOR THE MOST COMMON BAYES LOSSFUNCTIONS  IT SHOULD BE NOTED THAT IN THIS SECTION WE WILL DENOTETHE OBJECT OF OUR INTEREST IN ESTIMATION AS THE RANDOM VARIABLEXBF RATHER THAN THE RANDOM VARIABLE THETABF  THIS PROVIDES ANOTATIONAL TRANSITION TOWARD CONSIDERING XBF AS A STATE VARIABLE TOBE ESTIMATED AS IS DONE IN FOLLOWING SECTIONS  RECALL THAT IN SECTION REFSECINVPART WE COMPUTED THE DISTRIBUTION OF THE CONDITIONALRANDOM VARIABLE XBFYBF USING THE FORMULAS FOR INVERSE OF APARTITIONED MATRIX  THESE RESULTS WILL NOW BE PUT TO WORKIN EXAMPLE REFEXMCONDGAUSS THE DISTRIBUTION OF THE RANDOMVARIABLE ZBF  XBFYBF WHERE XBF IN RBBM AND YBF INRBBN XBF SIM NCMUBFXRXX AND YBF SIM NCMUBFYRYY IS FOUND TO BE FZBFZBF  FZBFXBFYBF  FRAC12PIP2DETRZZEXPFRAC12 ZBFMUBFZTRZZ1 ZBF MUBFZWHERE PMN AND  RZZ  COVZBF  BEGINBMATRIX RXX  RXY  RYX   RYY ENDBMATRIXWE NOW CONSIDER THE ESTIMATION PROBLEM  GIVEN A MEASUREMENT OFYBF WE WANT TO ESTIMATE XBF  THIS REQUIRES FINDINGFXBFYBFXBFYBF  HOWEVER WE HAVE ALREADY DEALT WITH THISPROBLEM IN EXAMPLE REFEXMCONDGAUSS FXBFYBFXBFYBF WASSHOWN TO BE GAUSSIAN WITH MEANBEGINEQUATION MUBFXY  EXBFYBF  YBF  MUBFX  RXYRY1YBF  MUBFYLABELEQCONDGAUSS1ENDEQUATIONAND COVARIANCEBEGINEQUATION COVXBFYBF  YBF  RXX  RXYRYY1RYX  PLABELEQCONDGAUSS2ENDEQUATIONTHE QUANTITY XBFHAT  MUBFXY IS THE BAYES ESTIMATE OF XBFGIVEN THE MEASUREMENT YBF IN THE SENSE OF BEING THE MEAN MODEAND MEDIAN OF THE DISTRIBUTION  IT CAN BE INTERPRETED AS FOLLOWSPRIOR TO ANY MEASUREMENTS THE BEST ESTIMATE OF XBF IS OBTAINED VIATHE PRIOR DENSITY FXBFXBF TO BE MUBFX THE MEAN OFXBF  BY MEANS OF THE MEASUREMENT THE PRIOR DISTRIBUTIONFXBFXBF EVOLVES INTO THE POSTERIOR DISTRIBUTION BYBEGINEQUATION FXBFYBFXBFYBF FRACFYBFXBFYBFXBFFYBFYBF FXBFXBFLABELEQKALMANDER0ENDEQUATIONON THE BASIS OF THE POSTERIOR DENSITY THE PRIOR ESTIMATE IS MODIFIEDBY AN AMOUNT PROPORTIONAL TO HOW FAR THE MEASUREMENT YBF IS FROMITS EXPECTED VALUE  THE PROPORTIONALITY DEPENDS UPON THE HOW STRONGLYX AND Y ARE CORRELATED BY MEANS OF RXY AND INVERSELY ON THEVARIANCE OF RY1 MEASUREMENTS WITH HIGH VARIANCE ARE NOTACCORDED AS MUCH WEIGHT AS MEASUREMENTS WITH LOW VARIANCE WOULD BELET US  EXAMINE THE ESTIMATOR XBFHAT  MUBFXY FURTHER  BEGINENUMERATEITEM THE ESTIMATOR IS UNBIASED E XBFHAT  E MUBFX  RXYRY1 EYBF  MUBFY  MUBFXITEM THE ESTIMATOR ERROR EBF  XBF  XBFHAT IS UNCORRELATED WITH  XBFHATMUBFXBEGINEQUATIONE EBFXBFHAT  MUBFXT   0LABELEQUNCORR1ENDEQUATIONITEM THE ERROR IS UNCORRELATED WITH THE YBF  MUBFYBEGINEQUATIONLABELEQUNCORR2 EEBFYBF  MUBFYT  0ENDEQUATIONTHE ERROR IS ORTHOGONAL TO THE DATAITEM THE COVARIANCE OF XBFHAT IS COVXBFHAT  EEBF EBFT  RXX  RXYRYY1RYXTHUS THIS HAS SMALLER COVARIANCE THAN THE EM A PRIORICOVARIANCE RXXENDENUMERATEIN THE CASE OF THE LINEAR MODELBEGINEQUATION  LABELEQLINGAUSSMOD  YBF  H XBF  NUBFENDEQUATIONWHERE NUBF IS A ZEROMEAN RANDOM VARIABLE WITH COVNUBF  RTHEN RXY  RXXHT QQUAD TEXTANDQQUAD RYY  R QQUADTEXTANDQQUAD  MUBFY  HMUBFXTHEN  REFEQCONDGAUSS1 CAN BE WRITTEN ASBEGINEQUATION MUBFXY  MUBFX  RXXHTR1YBF  MUBFYLABELEQCONDGAUSS3ENDEQUATIONIT WILL BE CONVENIENT TO WRITE K  RXXHT R1 WHERE K ISCALLED THE EM KALMAN GAIN  THEN MUBFXY  MUBFX  KYBF  HMUBFXBEGINEXERCISESITEM CITESCHARFL1991 LABELEXGAUSSMODEL THERE ARE OTHER WAYS TO  CONSIDER THE JOINT DISTRIBUTION MODEL THAT ARE USEFUL IN DEVELOPING  INTUITION ABOUT THE PROBLEM  IN THIS EXERCISE WE EXPLORE SOME OF  THESE  IN EACH CASE XBF AND YBF ARE JOINTLY DISTRIBUTED  GAUSSIAN RANDOM VARIABLES WITH MEAN AND COVARIANCE MUBFX RX  AND MUBFY RY RESPECTIVELY  THEY CAN BE REGARDED AS BEING  GENERATED BY THE DIAGRAM SHOWN IN FIGURE REFFIGGAUSSGENA    BEGINENUMERATE    ITEM SHOW THAT CONDITIONED UPON MEASURING XBF THE RANDOM      VARIABLE YBF  YBFXBF SIM NCMUBFY  RYXRX1XBF  MUBFX QWHERE Q  RY  RYX RX1 RXYTHIS INTERPRETATION IS AS SHOWN IN FIGURE REFFIGGAUSSGENBITEM SHOW THAT AN EQUIVALENT WAY OF GENERATING XBF AND YBF  HAVING EQUIVALENT JOINT DISTRIBUTION IS TO MODEL THIS AS A  SIGNALPLUSNOISE MODEL YBF  HXBF  NBFWHERE H  RYXRX1 AND XBF SIMMUBFXRX AND NBFSIM NC0Q  THIS MODEL IS ILLUSTRATED IN FIGUREREFFIGGAUSSGENCITEM CONDITIONING NOW ON A MEASUREMENT OF YBF  SHOW THAT AN  EQUIVALENT REPRESENTATION FOR THE JOINT DISTRIBUTION IS AS SHOWN IN  FIGURE REFFIGGAUSSGENC SCHARF P 297 CWHERE G  RXYRY1THAT IS  XTILDE  MUBFX  G YBFMUBFY  NBFHAS THE SAME DISTRIBUTION AS XBF    ENDENUMERATE  ITEM SHOW THAT REFEQUNCORR1 AND REFEQUNCORR2 ARE    CORRECT    BEGINFIGUREHTBP      BEGINCENTER        LEAVEVMODE SUBFIGUREJOINTLY DISTRIBUTED XBF AND YBFINPUTPICTUREDIRGAUSSGEN1 SUBFIGUREMARGINALLY DISTRIBUTED XBF AND CONDITIONALLY DISTRIBUTED YBFINPUTPICTUREDIRGAUSSGEN2 SUBFIGURECHANNEL MODEL LINEARLY TRANSFORMED XBF PLUS NOISE NBFINPUTPICTUREDIRGAUSSGEN3 SUBFIGURESIGNAL PLUS NOISE MODELINPUTPICTUREDIRGAUSSGEN4        CAPTIONEQUIVALENT REPRESENTATIONS FOR THE GAUSSIAN ESTIMATION PROBLEM        LABELFIGGAUSSGEN      ENDCENTER    ENDFIGUREENDEXERCISESSECTIONRECURSIVE ESTIMATIONLABELSECSEQESTINDEXRECURSIVE ESTIMATIONWE NOW EXAMINE THE PROBLEM OF ESTIMATING THE STATE  OF ASYSTEM USING OBSERVATIONS OF THE SYSTEM WHERE THE STATE EVOLVES INTHE PRESENCE OF NOISE AND THE OBSERVATIONS ARE MADE SEQUENTIALLY INTHE PRESENCE OF NOISE  WE WILL USE THE NOTATION XBFT TO INDICATETHE PARAMETER TO BE ESTIMATED INSTEAD OF THETA AND USE YBFTTO INDICATE THE OBSERVATION DATAOUR PROBLEM IS TO ESTIMATE THE STATE XBFTT01LDOTS BASED ON A SEQUENCE OF OBSERVATIONS YBFTT01LDOTS  IN THIS DEVELOPMENT WE WILL ASSUME THAT THE STATE SEQUENCE XBFT IS A MARKOV RANDOM PROCESS  INDEXMARKOV RANDOM PROCESS THAT IS FOR ANY RANDOM VARIABLE Z   THAT IS A FUNCTION OF XBFS SGEQ T BEGINEQUATION FZXBFTXBFT1LDOTSXBF0  FZXBFTLABELEQXMARKOVENDEQUATIONIN PARTICULAR WE HAVEBEGINEQUATION FXBFT1XBFTXBFT1LDOTSXBF0  FXBFT1XBFTENDEQUATIONALSO WE WILL ASSUME THATTHE OBSERVATION YBFT1 DEPENDS UPON XBFT1 AND  POSSIBLY ON SOME RANDOM NOISE WHICH IS INDEPENDENT FROM SAMPLE TO  SAMPLE BUT IS CONDITIONALLY INDEPENDENT OF PRIOR OBSERVATIONS  GIVEN XBFT1 THAT ISBEGINEQUATION FYBFT1XBFT1 YBFTLDOTSYBF0  FYBFT1XBFT1LABELEQYINDEPENDEQUATIONNOTATION THE VECTOR XBFT IS A EM RANDOM VECTOR AS ISYBFT  IN MAKING THE CHANGE TO LOWER CASE RATHER THANUPPER CASE AS PREVIOUSLY IN THIS PART WE ARE FOLLOWING A NOTATIONALCONVENTION NOW DECADES OLD  IN STATISTICS THE STANDARD NOTATION FORA RANDOM VARIABLE IS TO USE A CAPITAL SYMBOL AND WE HAVE RETAINEDTHAT USAGE UP TO THIS POINT MAINLY TO REINFORCE THE CONCEPT THAT WEARE DEALING WITH RANDOM VARIABLES AND NOT THEIR ACTUAL VALUES  BUT WEWILL NOW DEPART FROM THE TRADITIONAL NOTATION OF STATISTICSTHE DIRECTION WE ARE HEADED IN THIS DEVELOPMENT IS THE KALMAN FILTERAN IMPORTANT RECURSIVE ESTIMATOR  THIS WILL BE PRESENTED IN DETAIL INTHE FOLLOWING CHAPTER BUILDING UPON THE CONCEPTS PRESENTED HEREWE WILL EMPLOY THE FOLLOWING NOTATION  THE SET OF MEASUREMENTSY0Y1LDOTSYT IS DENOTED AS YCT  THE NOTATIONXBFHATTTAU IS USED TO DENOTE THE BAYES ESTIMATE OF XBFTGIVEN THE DATA YBF0YBF1LDOTSYBFTAU  YCTAU  FOREXAMPLE THE ESTIMATE XBFHATTT1 INDICATES THE ESTIMATE OFXBFT USING THE DATA YCT1  WE WILL DENOTE THE COVARIANCEOF THE ESTIMATE OF XBFHATTTAU AS PTTAUBEGINEQUATION PTTAU  E XBFHATTTAU EXBFHATTTAUXBFHATTTAUEXBFHATTTAUTLABELEQCOVDEFENDEQUATIONFOR NOTATIONAL CONVENIENCE WE WILL ALSO ELIMINATE THE SUBSCRIPTNOTATION ON THE DENSITY FUNCTIONS FOR NOW USING THE ARGUMENTS TOINDICATE THE RANDOM VARIABLES AS FXBFT1YCT1 FXBFT1YCT1XBFT1YCT1STARTING FROM A PRIOR DENSITY FXBF0 THE FIRST OBSERVATIONYBF0 IS USED TO COMPUTE A POSTERIOR DISTRIBUTION USING BAYESTHEOREM REFEQPOST1 AS FXBF0YBF0  FRACFYBF0XBF0FYBF0 FXBF0BASED ON FXBF0YBF0 AN ESTIMATE XBFHAT00 IS OBTAINEDTHIS IS THE UPDATE STEP  THIS DENSITY IS NOW PROPAGATED AHEADIN TIME BY SOME MEANS USING THE STATE UPDATE EQUATION FOR XBFTTO OBTAIN FXBF1YBF0 FROM WHICH THE ESTIMATE XBFHAT10IS OBTAINEDWE NOW WANT TO GENERALIZE THIS FIRST STEP TO UPDATING AN ESTIMATECONDITIONED ON THE YCT TO ONE CONDITIONED ON YCT1  FROMTHE POINT OF VIEW OF BAYESIAN ESTIMATION THE PROBLEM NOW IS TODETERMINE THE POSTERIOR DENSITY FXBFT1YCT1 RECURSIVELYFROM THE POSTERIOR DENSITY FXBFTYCT  THAT IS WE WISH TOFIND A FUNCTION FC SUCH THAT FXBFT1YCT1  FCFXBFTYCTYBFT1IDENTIFICATION OF THE FUNCTION FC WILL PROVIDE THE DESIREDPOSTERIOR DENSITY FROM WHICH THE ESTIMATE MAY BE OBTAINED  LET USBEGIN BY WRITING DOWN THE DESIRED RESULT USING BAYES THEOREMBEGINALIGN  FXBFT1YCT1  FXBFT1YCTYBFT1 NONUMBER  FRACFYBFT1XBFT1YCTFYBFT1YCTFXBFT1YCTLABELEQKALMANDER1ENDALIGNWE NOW OBSERVE THATBEGINEQUATION FYBFT1XBFT1YCT  FYBFT1XBFT1LABELEQASSUMPRESENDEQUATIONTO BE EXPLICIT ABOUT WHY THIS IS TRUE NOTE THAT WE CAN WRITE BEGINALIGNEDFYBFT1XBFT1YCT FYBFT1XBFT1LDOTSXBF0YBFTLDOTS YBF0  FYBFT1XBFT1LDOTSXBF0 QQUAD TEXTBY REFEQXMARKOV  FYBFT1XBFT1 QQUAD TEXTBY  REFEQYINDEPENDALIGNEDSUBSTITUTING REFEQASSUMPRES INTO REFEQKALMANDER1 YIELDSBEGINEQUATION  LABELEQKALMANDER3  UNDERBRACEFXBFT1YCT1TEXTRMPOSTERIOR   FRACFYBFT1XBFT1FYBFT1 YCT  UNDERBRACEFXBFT1YCTTEXTRMPRIORENDEQUATIONEQUATION REFEQKALMANDER3 IS DIRECTLY ANALOGOUS TO REFEQPOST1 WITH THE FOLLOWING IDENTIFICATION  THE PRIORPROBABILITY FTHETAVARTHETA IS IDENTIFIED ASFXBFT1YCT AND FVARTHETAX IS IDENTIFIED AS THEPOSTERIOR FXBFT1YCT1  WE MAY CALLREFEQKALMANDER3 THE UPDATE STEPCOMPUTATION OF THE UPDATE STEP REQUIRES FINDING FXBFT1YCTTHE DENSITY FXBFT1YCT IS THE PROPAGATION STEP  THISSTEP CAN BE WRITTEN ASBEGINALIGNFXBFT1YCT  INT FXBFT1XBFT YCTFXBFTYCT  DXBFT NONUMBER  INT FXBFT1XBFTFXBFTYCT  DXBFT LABELEQKALMANDER4ENDALIGNWHERE THE EQUALITY IN REFEQKALMANDER4 FOLLOWS BY THE MARKOVPROPERTY OF XBFT AND SINCE YBFT DEPENDS UPON XBFT  THE TWO STEPS REPRESENTED BY REFEQKALMANDER3 UPDATE ANDREFEQKALMANDER4 PROPAGATE ARE ILLUSTRATED IN FIGUREREFFIGBAYESUPDATE  THE PRIOR DISTRIBUTION FXBF0 IS UPDATEDBY MEANS OF REFEQKALMANDER3 TO PRODUCE THE POSTERIORFXBF0YBF0 FROM WHICH THE ESTIMATE XBFHAT00 ISOBTAINED  THE DENSITY IS PROPAGATED BY REFEQKALMANDER4 TOFXBF1YBF0 WHICH IS THEN USED AS THE PRIOR FOR THE NEXT STAGEAND FROM WHICH XBFHAT10 IS OBTAINED  ITERATING THESE TWOEQUATIONS PROVIDE FOR AN UPDATE OF THE BAYES ESTIMATE AS NEW DATAARRIVEBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODEINPUTPICTUREDIRSEQEST 15PTINPUTPICTUREDIRSEQEST2    CAPTIONILLUSTRATION OF THE UPDATE AND PROPAGATE STEPS IN      SEQUENTIAL ESTIMATION     LABELFIGBAYESUPDATE  ENDCENTERENDFIGUREIN GOING FROM ONE STAGE TO THE NEXT THE CONDITIONALFXBFT1YCT BECOMES THE PRIOR FOR THE NEXT STAGE  IN ORDERTO PRESERVE THE COMPUTATIONAL STRUCTURE FROM ONE STAGE TO THE NEXT ITIS EXPEDIENT TO HAVE THE CONDITIONAL DENSITY BE OF THE SAME TYPE ASTHE PRIOR DENSITY THIS MEANS THAT THEY SHOULD BE MEMBERS OF ACONJUGATE CLASS  IN PRACTICE HOWEVER IT IS ONLY GAUSSIAN RANDOMVARIABLES WHICH ADMIT FINITEDIMENSIONAL FILTERING IMPLEMENTATIONSSUBSECTIONAN EXAMPLE OF NONGAUSSIAN RECURSIVE BAYESLABELSECNONLINBAYESWE WILL DEMONSTRATE THE CONCEPT OF RECURSIVE ESTIMATION WITH A SIMPLEPROBABILITY STRUCTURE  THE KEY EQUATIONS ARE THE BAYES UPDATEEQUATION REFEQKALMANDER3 AND THE UPDATE EQUATIONREFEQKALMANDER4 WHICH ARE USED TO PROVIDE THE EVOLUTION OF THEDISTRIBUTIONS ARE NEW INFORMATION IS OBTAINED  FOR CONVENIENCE ANEXAMPLE WITH A DISCRETE DISTRIBUTION HAS BEEN SELECTED SO THAT ALLINTEGRALS ARE REPLACED BY SUMMATIONSIN THIS EXAMPLE CITEPAGE 385BRYSONHO THE STATE OF THE SCALARSIGNAL XT IN 01 IS GOVERNED BY A BERNOULLI DISTRIBUTION WITH PMFBEGINEQUATION PXT  XT  1Q0DELTAXT  Q0 DELTA1XTLABELEQXPRIORENDEQUATIONWHERE DELTAX  BEGINCASES  1  X  0  0  X NEQ 0ENDCASES  THIS DISTRIBUTION HOLDS FOR ALL TLET NT BE A SCALAR MARKOV BERNOULLI SEQUENCE WITHBEGINEQUATIONPN0  N0  1A0DELTAN0  A0 DELTA1N0 QQUAD A0  12LABELEQNPRIORENDEQUATIONSUPPOSE ALSO THAT PNT EVOLVES ACCORDING TO PNT1  NT1NT  NT 1ATFRACNT2DELTANT1  AT  FRACNT2 DELTA1NT1THIS CONDITIONAL UPDATE TENDS TO FAVOR THE REOCCURRENCE OF A 1 IFNT1 THEN NT1 IS MORE LIKELY TO BE SO  THE MEASUREMENT EQUATION IS YT  XT VEE NTINDEXVEE MAXWHERE VEE INDICATES THE MAXIMUM VALUE OF ITS ARGUMENTS  BASED ON ASEQUENCE OF OBSERVATIONS Y0Y1LDOTS WE DESIRE TO ESTIMATEX0X1LDOTS AND N0N1LDOTS  THESE EQUATIONS REPRESENT ASIMPLE BUT IMPERFECT MODEL OF A DETECTION SYSTEM IN WHICH THE STATEXT INDICATES THE PRESENCE OF A TARGET  OCCURRING IN ISOLATEDSAMPLES  AND THE NOISE NT REPRESENTS BLOCKING OF THE SIGNAL BYSOME LARGE BODY WHICH GIVES A FALSE INDICATION OF THE TARGET IF THEBLOCKING WAS PRESENT AT THE LAST MEASUREMENT IT WILL BE MORE LIKELYTO APPEAR IN THE NEXT MEASUREMENT  FOR EXAMPLE THE SYSTEM MIGHTAPPLY TO AN INFRARED DETECTION SYSTEM IN WHICH CLOUDS MIGHT BLOCK THEVIEW AND GIVE A FALSE SIGNALFROM THE PRIOR PROBABILITIES IN REFEQXPRIOR ANDREFEQNPRIOR UPDATED PMFS BASED UPON THE OBSERVATION Y0 CANBE COMPUTED FROMBEGINALIGNPN0N0Y0Y0   FRACPY0Y0N0N0PY0Y0  PN0N0 LABELEQBUPDATE1 PX0X0Y0Y0   FRACPY0Y0X0X0PY0Y0  PX0X0 LABELEQBUPDATE2 ENDALIGNWHERE PY0Y0 IS OBTAINED FROM EXPLICIT ENUMERATIONBEGINALIGNED PY00  PX00 N00  1A01Q0  PY01  PX00 N01  PX01N00  PX01N01 NONUMBER   A01Q0  Q01A0   A0Q0ENDALIGNEDTHEN FROM REFEQBUPDATE1 WE HAVEBEGINEQUATION PN01Y0Y0  A00  BEGINCASES  FRACPY01N01PY01PN01  TEXT IF  Y01 EXMATSP  FRACPY00Y01PY00PN01  TEXT IF  Y00ENDCASES FRACA0 DELTAY01A0Q0A0Q0LABELEQA00ENDEQUATIONAND SIMILARLY PX01Y0Y0  Q00  FRACQ0 DELTAY01A0Q0A0Q0THE UPDATED DENSITIES CAN BE THEN WRITTEN ASBEGINALIGNEDPN0N0Y0Y0  1A00DELTAN0  A00 DELTA1N0 PX0X0Y0Y0  1Q00DELTAX0  Q00 DELTA1X0ENDALIGNEDWHICH ARE OF THE SAME FORM AS THE ORIGINAL PMFS  IN REFEQNPRIORAND REFEQXPRIOR EXCEPT THAT THE PROBABILITIES HAVE CHANGEDTHE UPDATE STEP IS STRAIGHTFORWARD USING REFEQKALMANDER4BEGINALIGNEDPN1N1Y0Y0  SUMN0 PN1N1N0N0 PN0N0Y0Y0  1A10DELTAN1  A10 DELTA1N1ENDALIGNEDWHEREBEGINEQUATION  LABELEQAUPDATE  A10  A  FRACA002ENDEQUATIONALSO PX1X1Y0Y0  PX1X1LETTING YCT  Y0LDOTSYT  WE HAVEBEGINEQUATION PNTNTYCT  1ATTDELTANT  ATTDELTA1NTLABELEQSEQBEX1ENDEQUATIONWHERE ATT  FRACATT1 DELTA1YT 1ATT11Q DELTAYT  ATT1  Q   ATT1QDELTA1YTAND PNT1NT1YCT  1AT1TDELTANT1 AT1TDELTA1NT1WHERE  AT1T  A  FRACATT2SIMILARLY WE HAVEBEGINEQUATION PXTXTYCT  1QTTDELTAXT  QTT DELTA1XTLABELEQSEQBEX2ENDEQUATIONWHERE XTT  FRACQ DELTA1YT1ATT11QDELTAYT  ATT1  Q   ATT1QDELTA1YTAND PXT1XT1YCT  PXT1XT1SUPPOSE A14 AND Q16  IF THE SEQUENCE YC3  0001 ISOBSERVED THEN PX31YC3  4444QQUAD PN31YC3  6667IF THE SEQUENCE YC3  0111 IS OBSERVED THEN PX31YC3  2230QQUAD PN31YC3  9324AS AN INTERPRETATION CONSIDER PX31YC3 AS THE PROBABILITYTHAT A TARGET IS PRESENT AS OPPOSED TO THE BLOCKING  COMPARING THEFIRST CASE P4444 WITH THE SECOND CASE P2230 THERE IS MOREPROBABILITY THAT THE TARGET IS PRESENT IN THE FIRST CASE  THESEQUENCE OF 1S IS SUGGESTIVE OF THE BLOCKINGBECAUSE THE DISTRIBUTIONS WERE CHOSEN IN THIS EXAMPLE TO BE DISCRETETHIS ESTIMATION PROBLEM CAN REALLY BE INTERPRETED AS A DETECTION PROBLEMBEGINEXERCISES  ITEM SHOW THAT REFEQA00 IS CORRECT  ITEM WRITE SC MATLAB CODE TO COMPUTE THE PROBABILITIES    FXTYCT AND FNTYCT OF REFEQSEQBEX1 AND    REFEQSEQBEX2 GIVEN A SEQUENCE OF    OBSERVATIONS AND THE INITIAL PROBABILITIES A AND Q  BAYESEST1M  ITEM SUPPOSE THAT THE OBSERVATION EQUATION IS YT  XT OPLUS NTWHERE OPLUS REPRESENTS ADDITION MODULO 2 WITH EVERYTHING ELSEBEING THE SAME AS BEFORE  DERIVE THE UPDATE AND PROPAGATION EQUATIONSIN THIS CASEENDEXERCISES LOCAL VARIABLES TEXMASTER TEST ENDSETCOUNTERPAGE1SETCOUNTERFIGURE0SETCOUNTEREQUATION0SETCOUNTERLEMMA0SETCOUNTERTHEOREM0SETCOUNTERDEFINITION0SETCOUNTEREXAMPLE0SETCOUNTEREXERCISE0CHAPTERESTIMATION THEORYLABELCHAPESTBEGINQUOTESOURCE  GEORGE BERKELEYEM THE FIRST DIALOGUE BETWEEN HYLAS AND PHILONOUSSC HYLAS YOU STILL TAKE THINGS IN A STRICT LITERAL SENSE THAT ISNOT FAIR PHILONOUSSC PHILONOUS I AM NOT FOR IMPOSING ANY SENSE ON YOUR WORDS YOUARE AT LIBERTY TO EXPLAIN THEM AS YOU PLEASE  ONLY I BESEECH YOUMAKE ME UNDERSTAND SOMETHING BY THEMENDQUOTESOURCEESTIMATION IS THE PROCESS OF MAKING DECISIONS OVER A CONTINUUM OFPARAMETERS  WE HAVE SEEN THAT THERE ARE TWO MAJOR PHILOSOPHIES TODETECTION THE NEYMANPEARSON APPROACH IN WHICH NO PRIORPROBABILITIES ARE ASSUMED ON THE PARAMETERS AND THE BAYES APPROACHIN WHICH A PRIOR PROBABILITY IS ASSUMED  THE SAME DICHOTOMY EXISTSHERE AS WITH THE DETECTION PROBLEM SINCE WE MAY VIEW THEUNKNOWN PARAMETER AS EITHER AN UNKNOWN BUT DETERMINISTIC QUANTITY ORAS A RANDOM VARIABLE  CONSEQUENTLY THERE ARE MULTIPLE SCHOOLS OFTHOUGHT REGARDING ESTIMATION  ON THE ONE HAND WHEN NO PRIORDISTRIBUTION IS ASSUMED THE ESTIMATION IS COMMONLY DONE BASED UPONTHE EM MAXIMUM LIKELIHOOD PRINCIPLE  WHEN A PRIOR DISTRIBUTION FORTHE PARAMETER IS ASSUME A EM BAYES ESTIMATE IS FORMEDSECTIONTHE MAXIMUM LIKELIHOOD PRINCIPLELABELSECML1INDEXMAXIMUM LIKELIHOOD ESTIMATIONTHE ESSENTIAL FEATURE OF THE PRINCIPLE OF MAXIMUM LIKELIHOOD AS ITAPPLIES TO ESTIMATION THEORY IS THAT IS REQUIRES ONE TO CHOOSE AS ANESTIMATE OF A PARAMETER THAT VALUE FOR WHICH THE PROBABILITY OFOBTAINING A GIVEN SAMPLE ACTUALLY OBSERVED IS AS LARGE AS POSSIBLETHAT IS HAVING OBTAINED OBSERVATIONS ONE LOOKS BACK AND COMPUTESTHE PROBABILITY FROM THE POINT OF VIEW OF ONE ABOUT TO PERFORM THEEXPERIMENT THAT THE GIVEN SAMPLE VALUES WILL BE OBSERVED  THISPROBABILITY WILL IN GENERAL DEPEND ON THE PARAMETER WHICH IS THENGIVEN THAT VALUE FOR WHICH THIS PROBABILITY IS MAXIMIZEDTHIS IS REMINISCENT OF STORY ABOUT THE CRAFTY POLITICIAN WHO ONCE HEOBSERVES WHICH WAY THE CROWD IS GOING HURRIES TO THE FRONT OF THEGROUP AS IF TO LEAD THE PARADESUPPOSE THAT THE RANDOM VARIABLE X HAS A PROBABILITY DISTRIBUTIONWHICH DEPENDS ON A PARAMETER THETA  THE PARAMETER THETA MUSTLIE IN A SPACE OF POSSIBLE PARAMETERS THETA  LETFXXTHETA DENOTE EITHER A PMF OR PDF OF X WE SUPPOSE THATTHE FORM OF FX IS KNOWN BUT NOT THE VALUE OF THE PARAMETERTHETA  THE JOINT PMF OF M SAMPLE RANDOM VARIABLES EVALUATED ATTHE SAMPLE POINTS X1 LDOTS XM ISBEGINEQUATIONLABELLIKELIHOODFUNCTIONELLTHETA X1 LDOTS XM  ELLTHETAXBF  FXBFXBFTHETA  PRODI1M FXXI THETAENDEQUATIONTHIS FUNCTION IS ALSO KNOWN AS THE EM LIKELIHOOD FUNCTIONINDEXLIKELIHOOD FUNCTION OF THESAMPLE WE ARE PARTICULARLY INTERESTED IN IT AS A FUNCTION OF THETAWHEN THE SAMPLE VALUES X1LDOTS XM ARE FIXED  THE PRINCIPLE OFMAXIMUM LIKELIHOOD REQUIRES US TO CHOOSE AS AN ESTIMATE OF THE UNKNOWNPARAMETER THAT VALUE OF THETA FOR WHICH THE LIKELIHOOD FUNCTIONASSUMES ITS LARGEST VALUE IF THE PARAMETER THETA IS A VECTOR SAY THETABF  THETA1LDOTS THETAKT THEN THE LIKELIHOOD FUNCTION WILL BE A FUNCTIONOF ALL OF THE COMPONENTS OF THETABF  THUS WE ARE FREE TO REGARDTHETABF AS A VECTOR IN REFLIKELIHOODFUNCTION AND THE MAXIMUMLIKELIHOOD ESTIMATE OF THETA IS THEN THE VECTOR OF NUMBERS WHICHRENDER THE LIKELIHOOD FUNCTION A MAXIMUM  BEGINEXAMPLEEM A MAXIMUM LIKELIHOOD DETECTOR  SUPPOSE YOU ARE GIVEN A COINAND TOLD THAT IT IS BIASED WITH ONE SIDE FOUR TIMES AS LIKELY TO TURNUP AS THE OTHER YOU ARE ALLOWED THREE TOSSES AND MUST THEN GUESSWHETHER IT IS BIASED IN FAVOR OF HEAD OR IN FAVOR OF TAILS  LET THETA BE THE PROBABILITY OF HEADS H WITH T CORRESPONDING TOTAILS ON A SINGLE TOSS  DEFINE THE RANDOM VARIABLE XMC HTMAPSTO 0 1 XH  1 AND XT  0  THE PMF FOR X ISGIVEN BYBEGINDISPLAYMATHBEGINARRAYCCCFX045  15  QQUAD  FX145  4510PTFX015  45  QQUAD  FX115  15ENDARRAYENDDISPLAYMATHSUPPOSE YOU THROW THE COIN THREE TIMES RESULTING IN THE SAMPLESHTH THE SAMPLE VALUES ARE X1  1 X2  0 X3  1  THELIKELIHOOD FUNCTION IS BEGINALIGNEDELLTHETA X1 X2 X3    FX1X2X3X1 X2X3THETA    FX1X2X31  0 1 THETA    FX11THETAFX20THETAFX3 1 THETAENDALIGNEDOR BEGINALIGNEDELL45 101    451545  1612510PTELL15 101    154515  4125ENDALIGNEDCLEARLY THETA  45 YIELDS THE LARGER VALUE OF THE LIKELIHOODFUNCTION SO BY THE LIKELIHOOD PRINCIPLE WE ARE COMPELLED TO DECIDETHAT THE COIN IS BIASED IN FAVOR OF HEADSENDEXAMPLEALTHOUGH AS THIS EXAMPLE DEMONSTRATES THE PRINCIPLE OF MAXIMUMLIKELIHOOD MAY BE APPLIED TO DISCRETE DECISION PROBLEMS IT HAS FOUNDGREATER UTILITY FOR PROBLEMS WHERE THE DISTRIBUTION IS CONTINUOUS ANDDIFFERENTIABLE IN THETA  THE REASON FOR THIS IS THAT WE WILLUSUALLY BE TAKING DERIVATIVES IN ORDER TO FIND MAXIMA  BUT IT ISIMPORTANT TO REMEMBER THAT GENERAL DECISION PROBLEMS CAN INPRINCIPLE BE ADDRESSED VIA THE PRINCIPLE OF MAXIMUM LIKELIHOODNOTICE FOR THIS EXAMPLE THAT NEITHER COST FUNCTIONS NOR EM A  PRIORI KNOWLEDGE OF THE DISTRIBUTION OF THE PARAMETERS IS NEEDED TOFASHION A MAXIMUM LIKELIHOOD ESTIMATEBEGINEXAMPLEINDEXEMPIRIC DISTRIBUTION ESTIMATIONEM EMPIRIC DISTRIBUTIONS  LET X BE A RANDOM VARIABLE OFUNKNOWN DISTRIBUTION AND THAT X1 LDOTS XM ARE SAMPLE RANDOM VARIABLESFROM THE POPULATION OF X  SUPPOSE WE ARE REQUIRED TO ESTIMATE THEDISTRIBUTION FUNCTION OF X  THERE ARE MANY WAYS TO APPROACH THISPROBLEM  ONE WAY WOULD BE TO ASSUME SOME GENERAL STRUCTURE SUCH AS ANEXPONENTIAL FAMILY AND TRY TO ESTIMATE THE PARAMETERS OF THIS FAMILYBUT THEN ONE HAS THE SIMULTANEOUS PROBLEMS OF A ESTIMATING THEPARAMETERS AND B JUSTIFYING THE STRUCTURE  ALTHOUGH THERE ARE MANYWAYS OF DOING BOTH OF THESE PROBLEMS IT IS NOT EASY  THE MAXIMUMLIKELIHOOD METHOD GIVES US A FAIRLY SIMPLE APPROACH THAT IF FOR NOOTHER REASON WOULD BE VALUABLE AS A BASELINE FOR EVALUATING OTHERMORE SOPHISTICATED APPROACHESTO APPLY THE PRINCIPLE OF MAXIMUM LIKELIHOOD TO THIS PROBLEM WE MUSTFIRST DEFINE THE PARAMETERS  WE DO THIS BY SETTING BEGINDISPLAYMATHTHETAI  PXI  XI QUAD I1 LDOTS  MENDDISPLAYMATHTHE EVENT BEGINDISPLAYMATHX1  X1 CDOTS XM  XMENDDISPLAYMATHIS OBSERVED AND ACCORDING TO THE MAXIMUM LIKELIHOOD PRINCIPLE WE WISH TO CHOOSE THE VALUES OF THETAI THAT MAXIMIZE THEPROBABILITY THAT THIS EVENT WILL OCCUR  SINCE THE EVENTS XI XI I1 LDOTS M ARE INDEPENDENT WE HAVEBEGINDISPLAYMATHPX1  X1 CDOTS XM  XM  PRODI1MPXI  XI PRODI1M THETAIENDDISPLAYMATHWHICH WE WISH TO MAXIMIZE SUBJECT TO THE CONSTRAINT SUMI1MTHETAI  1 WHICH WE SHALL DO VIA LAGRANGE MULTIPLIERSINDEXCONSTRAINED OPTIMIZATION INDEXLAGRANGE MULTIPLIERLETBEGINDISPLAYMATHJ  PRODI1MTHETAI  LAMBDA SUMI1M THETAI1ENDDISPLAYMATHAND SET THE GRADIENT OF J WITH RESPECT TO THETAI I1 LDOTSM AND WITH RESPECT TO LAMBDA TO ZERO BEGINALIGNEDFRACPARTIAL JPARTIAL THETAJ    PRODINOT JTHETAI LAMBDA 0 QUAD J1 LDOTS M10PTFRACPARTIAL JPARTIAL LAMBDA   SUMI1M THETAI 1  0ENDALIGNEDBUT THE ONLY WAY ALL OF THE PRODUCTS PRODINOT  JTHETAI CAN BEEQUAL IS IF THETA1  CDOTS  THETAM AND THE CONSTRAINTTHEREFORE REQUIRES THAT THETAI  1M I1 LDOTS M  WE DEFINE THE MAXIMUM LIKELIHOOD ESTIMATE FOR THE DISTRIBUTION ASFOLLOWS  LET TILDEX BE A RANDOM VARIABLE CALLED THE EM EMPIRICRANDOM VARIABLE WHOSE DISTRIBUTION FUNCTION IS BEGINDISPLAYMATHFTILDEXX  PTILDEX LEQ X  FRAC1MSUMI1MIXI INFINITYXENDDISPLAYMATHFIGURE REFFIGEMPIRIC ILLUSTRATES THE STRUCTURE OF THE EMPIRICDISTRIBUTION FUNCTIONBEGINFIGUREHTBP  BEGINCENTER    LEAVEVMODE    INPUTPICTUREDIREMPIRIC    CAPTIONEMPIRIC DISTRIBUTION FUNCTION    LABELFIGEMPIRIC  ENDCENTERENDFIGUREFOR LARGE SAMPLES IT IS CONVENIENT TO QUANTIZE THE OBSERVATIONS ANDCONSTRUCT THE EMPIRIC DENSITY FUNCTION BY BUILDING A HISTOGRAMTHUS THE EMPIRIC DISTRIBUTION IS PRECISELY THAT DISTRIBUTION FOR WHICHTHE INFLUENCE OF THE SAMPLE VALUES ACTUALLY OBSERVED IS MAXIMIZED ATTHE EXPENSE OF OTHER POSSIBLE VALUES OF X  OF COURSE THE ACTUALUTILITY OF THIS DISTRIBUTION IS LIMITED SINCE THE NUMBER OF PARAMETERSMAY BE VERY LARGE  BUT IT IS A MAXIMUM LIKELIHOOD ESTIMATE OF THEDISTRIBUTION FUNCTIONENDEXAMPLESUBSECTIONMAXIMUM LIKELIHOOD FOR CONTINUOUS DISTRIBUTIONSSUPPOSE NOW THAT THE RANDOM VARIABLE X IS CONTINUOUS AND HAS APROBABILITY DENSITY FUNCTION FXXTHETA WHICH DEPENDS ON THEPARAMETER THETA THETA MAY BE A VECTOR  THE JOINT PROBABILITYDENSITY FUNCTION OF THE SAMPLE RANDOM VARIABLES EVALUATED AT THESAMPLE POINTS X1 LDOTS XM IS GIVEN BYBEGINDISPLAYMATHELLLTHETA X1 LDOTS XM  FX1CDOTS XMX1 LDOTS XMTHETA  PRODI1M FXXITHETAENDDISPLAYMATHFOR SMALL DX1 LDOTS DXM THE M1DIMENSIONAL VOLUMEFX1CDOTS XMX1 LDOTS XMTHETADX1CDOTS DXM REPRESENTS APPROXIMATELY THE PROBABILITYTHAT A SAMPLE WILL BE CHOSEN FOR WHICH THE SAMPLE POINTS LIE WITHIN ANNDIMENSIONAL RECTANGLE AT X1 LDOTS XM WITH SIDES DX1LDOTS DXM  CONCEPTUALLY WE CAN CONSIDER CALCULATING THISVOLUME FOR FIXED XI AND DXI AS THETA IS VARIED OVER ITSRANGE OF PERMISSIBLE VALUES ACCORDING TO THE MAXIMUM LIKELIHOODPRINCIPLE WE TAKE AS THE MAXIMUM LIKELIHOOD ESTIMATE OF THETATHAT VALUE THAT MAXIMIZES THE VOLUME THE IDEA BEING THAT IF THATWERE THE ACTUAL VALUE OF THETA THAT NATURE USED IT WOULDCORRESPOND TO THE DISTRIBUTION THAT YIELDS THE LARGEST PROBABILITY OFPRODUCING SAMPLES NEAR THE OBSERVED VALUES X1 LDOTS XMSINCE THE RECTANGLE IS FIXED THE VOLUME AND HENCE THE PROBABILITYIS MAXIMIZED BY MAXIMIZING THE LIKELIHOOD FUNCTION ELLLTHETA X1 LDOTS XMIT MUST BE STRESSED THAT THE LIKELIHOOD FUNCTION ELLTHETA X ISTO BE VIEWED AS A FUNCTION OF THETA WITH X BEING A FIXEDQUANTITY RATHER THAN A VARIABLE  THIS IS IN CONTRADISTINCTION TO THEWAY WE VIEW THE DENSITY FUNCTION FXXTHETA WERE THETA ISA FIXED QUANTITY AND X IS VIEWED AS A VARIABLE  SO REMEMBER EVENTHOUGH WE MAY WRITE ELLTHETAX  FXXTHETA WE VIEW THEROLES OF X AND THETA IN THE TWO EXPRESSIONS ENTIRELY DIFFERENTLYIT IS ACTUALLY MORE CONVENIENT FOR MANY APPLICATIONS TO CONSIDER THELOGARITHM OF THE LIKELIHOOD FUNCTION WHICH WE DENOTEBEGINDISPLAYMATHLAMBDATHETA XBF  LOG FXBFXBF THETAENDDISPLAYMATHAND CALL THE EM LOGLIKELIHOOD FUNCTION  INDEXLOGLIKELIHOOD  FUNCTION SINCE THE LOGARITHM IS A MONOTONIC FUNCTION THEMAXIMIZATION OF THE LIKELIHOOD AND LOGLIKELIHOOD FUNCTIONS ISEQUIVALENT THAT IS THETAML MAXIMIZES THE LIKELIHOOD FUNCTIONIF AND ONLY IF IT ALSO MAXIMIZES THE LOGLIKELIHOOD FUNCTION  THUSIN THIS DEVELOPMENT WE WILL DEAL MAINLY WITH THE LOGLIKELIHOODFUNCTIONIF THE LOGLIKELIHOOD FUNCTION IS DIFFERENTIABLE IN THETA A NECESSARYBUT NOT SUFFICIENT CONDITION FOR THETA TO BE A MAXIMUM OF THELOGLIKELIHOOD FUNCTION IS FOR THE GRADIENT OFTHE LOGLIKELIHOOD FUNCTION TO VANISH AT THAT VALUE OF THETA THATIS WE REQUIREBEGINDISPLAYMATHFRACPARTIALPARTIAL THETALAMBDATHETA XBF FRACPARTIALPARTIAL THETA LOG FXBFXBFTHETA  0ENDDISPLAYMATHTHE MAJOR ISSUE BEFORE US IS TO FIND A WAY TO MAXIMIZE THE LIKELIHOODFUNCTION  IF THE MAXIMUM IS INTERIOR TO THE RANGE OF THETA ANDAND LAMBDATHETA XBF HAS A CONTINUOUS FIRST DERIVATIVE THEN A NECESSARYCONDITION FOR HATTHETAML TO BE THE MAXIMUM LIKELIHOODESTIMATE FOR THETA IS THATBEGINEQUATIONLABELLIKELIHOODEQNLEFT FRACPARTIAL LAMBDATHETA XBFPARTIAL THETA RIGHTTHETA HATTHETAML  0ENDEQUATIONIN THE CASE OF VECTOR PARAMETERS THETABF WE WRITE THIS ASBEGINEQUATION  LABELLIKELIHOODEQN2LEFT  PARTIALDLAMBDATHETABFXBFTHETABFRIGHTTHETABF HATTHETABFML  0ENDEQUATIONEQUATION REFLIKELIHOODEQN OR REFLIKELIHOODEQN2 ISCALLED THE EM LIKELIHOOD EQUATION  INDEXLIKELIHOOD EQUATIONWE NOW GIVE SOME EXAMPLES TOILLUSTRATE THE MAXIMIZATION PROCESSBEGINEXAMPLE  THIS FIRST EXAMPLE SHOWS THAT WHILE THE LIKELIHOOD EQUATION IS  FREQUENTLY USEFUL MORE GENERAL PRINCIPLES OF OPTIMIZATION CAN BE  USED TO OBTAIN MAXIMUM LIKELHOOD ESTIMATES EVEN WHEN THE MAXIMUM MAY  NOT OCCUR IN THE INTERIOR OF THE SET OF POSSIBLE VALUES  LET X1 LDOTS XM DENOTE A RANDOM SAMPLE OF SIZE M FROM AUNIFORM DISTRIBUTION OVER 0 THETA  WE WISH TO FIND THE MAXIMUMLIKELIHOOD ESTIMATE OF THETA  LET IAX  BEGINCASES  1  X IN A  0  X NOTIN AENDCASESBE THE INDICATOR FUNCTION FOR THE SET A  USING THIS NOTATION THELIKELIHOOD FUNCTION CAN BE WRITTEN AS BEGINALIGNEDELLTHETA X1 LDOTS XM    THETAMPRODI1M I0 THETAXI10PT    THETAMPRODI1M I0 MAXIXIMINI XIIMINI XITHETAMAXIXI10PT    THETAMPRODI1M IMINI XITHETAMAXIXI10PT   THETAMPRODI1M IMAXIXIINFINITYTHETAENDALIGNEDSINCE THE MAXIMUM OF THIS QUANTITY DOES NOT OCCUR ON THEINTERIOR OF THE RANGE OF THETA WE CANT TAKE DERIVATIVES AND SETTO ZERO  BUT WE DONT NEED TO DO THAT FOR THIS EXAMPLE SINCETHETAM IS MONOTONICALLY DECREASING IN THETA  CONSEQUENTLYTHE LIKELIHOOD FUNCTION IS MAXIMIZED AT BEGINDISPLAYMATHHATTHETAML  MAXI XIENDDISPLAYMATHINTUITIVELY WE SHOULD EXPECT THE RANGE OF A UNIFORMLY DISTRIBUTED TOBE DETERMINED BY THE LARGEST VALUE THAT IS OBSERVEDENDEXAMPLEBEGINEXAMPLELET X1 LDOTS XM DENOTE A RANDOM SAMPLE OF SIZE M FROM THENORMAL DISTRIBUTION NCMU SIGMA2 WE WISH TO FIND THE MAXIMUMLIKELIHOOD ESTIMATES FOR MU AND SIGMA2  THE DENSITY FUNCTION ISBEGINDISPLAYMATHFXBFXBF MU SIGMA    PRODI1MFRAC1SQRT2PISIGMAEXPLEFTFRACXIMU22SIGMA2RIGHTENDDISPLAYMATHAND THE LOGLIKELIHOOD FUNCTION IS THENBEGINDISPLAYMATHLAMBDAMUSIGMAXBF  MLOG SQRT2PI MLOGSIGMAFRAC12SIGMA2 SUMI1MXIMU2ENDDISPLAYMATHTAKING THE GRADIENT AND EQUATING TO ZERO YIELDS BEGINALIGNEDFRACPARTIAL LAMBDAPARTIAL MU   FRAC1SIGMA2SUMI1MXIMU  0 10PT  RIGHTARROW HATMUML  FRAC1MSUMI1M XIENDALIGNEDAND BEGINALIGNEDFRACPARTIAL LAMBDAPARTIAL SIGMA   FRACMSIGMA  SIGMA3SUMI1MXIMU2  0 10PT  RIGHTARROW   HATSIGMAML2  FRAC1MSUMI1MXIMU2 ENDALIGNEDWHEN THE MEAN IS NOT KNOWN WE CAN WRITE SIGMAHATML  FRAC1M SUMI1M XI  MUHATML2IT IS SATISFYING IS THAT THESE ESTIMATES COINCIDE WITH WHAT OURINTUITION WOULD SUGGESTIF WE VIEW THE ESTIMATORS AS RANDOM VARIABLES MUHATML  FRAC1MSUMI1M XI QQUADQQUADSIGMAHATML2  FRAC1MSUMI1M XIMUHATML2WE CAN EXAMINE THEIR MEANSBEGINALIGNE MUHATML  MU LABELEQMUMEAN E SIGMAHATML2  SIGMA2FRACM1M LABELEQSIGMAMEANENDALIGNWE NOTE THAT MUHATML IS AN UNBIASED ESTIMATOR ANDSIGMAHATML2 IS A BIASED ESTIMATOR  SO A MAXIMUM LIKELIHOODESTIMATE IS NOT NECESSARILY AN UNBIASED ESTIMATE  HOWEVER AS MRIGHTARROW INFTY THE ESTIMATE BECOMES UNBIASEDWE CAN ALSO EXAMINE THE VARIANCE OF THE ESTIMATORS  FOR EXAMPLE ITCAN BE SHOWN THATBEGINEQUATION  LABELEQMUVAR  VAR MUHATML  FRACSIGMA2MENDEQUATIONSO THAT THE VARIANCE DECREASES THE MORE SAMPLES ARE USED TO DETERMINETHE ESTIMATE  ENDEXAMPLEBEFORE WE GET TOO EUPHORIC OVER THE SIMPLICITY AND SEEMINGLY MAGICALPOWERS OF THE MAXIMUM LIKELIHOOD APPROACH CONSIDER THE FOLLOWINGEXAMPLEBEGINEXAMPLELET X1SIM NCTHETA 1 AND X2SIMNCTHETA 1 AND DEFINEBEGINDISPLAYMATHY  BEGINCASESX1  MBOXRM WITH PROBABILITY  1210PTX2  MBOXRM WITH PROBABILITY  12ENDCASESENDDISPLAYMATHTHEN BEGINDISPLAYMATHFYYTHETA  FRAC12FRAC1SQRT2PIEFRAC12Y THETA2 FRAC12FRAC1SQRT2PIEFRAC12Y THETA2ENDDISPLAYMATHNOW LET YYPRIME BE A GIVEN SAMPLE VALUE  ACCORDING TO OUR PROCEDURE WEWOULD EVALUATE THE LIKELIHOOD FUNCTION AT YPRIME YIELDINGBEGINDISPLAYMATHLTHETA YPRIME  FRAC12FRAC1SQRT2PIEFRAC12YPRIME THETA2 FRAC12FRAC1SQRT2PIEFRAC12YPRIME THETA2ENDDISPLAYMATHAND CHOOSE AS THE MAXIMUM LIKELIHOOD ESTIMATE OF THETA THAT VALUETHAT MAXIMIZES LTHETA YPRIME  BUT THIS FUNCTION DOES NOTHAVE A UNIQUE MAXIMUM SO THERE IS NOT A UNIQUE ESTIMATE  BOTHHATTHETAML  YPRIME AND HATTHETAML  YPRIMEQUALIFY AS MAXIMUM LIKELIHOOD ESTIMATES FOR THETAENDEXAMPLESECTIONML ESTIMATES AND SUFFICIENCYIF TXBF IS A SUFFICIENT STATISTIC FOR THETA INFXBFXBFTHETA THEN FXBFXBF THETA  BTBFXBF THETAAXBFAND FTBFTBFTHETA  BTBFTHETALEFTINT AWBF1TBFUBF LEFTFRACPARTIAL    WBF1YBFPARTIAL YBFRIGHTRIGHT DUBFSO THAT FXBFXBF THETA IS PROPORTIONAL TOFTBFTBFTHETA AND THE CONSTANT OF PROPORTIONALITYDEPENDS UPON XBF BUT NOT ON THETA  THUSTHE LOGLIKELIHOOD CAN BE WRITTEN AS LAMBDATHETAXBF  CXBF  LOG FTTBFTHETAWHERE CXBF DOES NOT DEPEND UPON THETA  FOR PURPOSES OFMAXIMIZING THE LOG LIKELIHOOD FUNCTION WE CAN IGNORE THE CONSTANTCXBF AND WRITE LAMBDATHETATBF  LOG FTTBFTHETABEGINEXAMPLE  LET X1X2LDOTSXN BE INDEPENDENT BERNOULLI BCP RANDOM  VARIABLES WHERE P IS THE UNKNOWN PARAMETER  THEN FXBFXBFP  PRODI1N PXI1P1XIA SUFFICIENT STATISTIC FOR THIS DISTRIBUTION IS K  SUMI1N XIWHICH IS BINOMIAL BCNP DISTRIBUTEDBEGINEQUATION FKKP  N CHOOSE K PK1PNKLABELEQKDISTENDEQUATIONIN FINDING A MAXIMUM LIKELIHOOD ESTIMATE FOR P WE MAY MAXIMIZE THEDISTRIBUTION OF THE SUFFICIENT STATISTIC IN REFEQKDIST  THE LOGLIKELIHOOD BASED ON THIS DISTRIBUTION ISBEGINEQUATION LAMBDAPK  LOG FKKP  LOGNCHOOSE K  K LOG P NKLOG1PLABELEQKIDST2ENDEQUATIONTHE CONSTANT LOG N CHOOSE K DOES NOT DEPEND UPON P AND CAN BEIGNORED  TAKING THE DERIVATIVE OF REFEQKIDST2 WITH RESPECT TOP AND EQUATING TO ZERO WE OBTAIN PHAT  FRACKNENDEXAMPLESECTIONESTIMATION QUALITYLABELSECEQBEGINQUOTESOURCEA JAZWINSKICITEPAGE 150JAZWINSKI70AN ESTIMATE IS MEANINGLESSUNLESS ONE KNOWS HOW GOOD IT ISENDQUOTESOURCEESTIMATION THEORISTS ARE SOMETIMES CONSUMED NOT ONLY WITH DEVISINGAND UNDERSTANDING VARIOUS ALGORITHMS FOR ESTIMATION BUT WITHEVALUATIONS OF HOW RELIABLE THEY ARE  WE USUALLY ASK THE QUESTION INTHE SUPERLATIVE WHAT IS THE BEST ESTIMATEWE MIGHT BE TEMPTED TO ANSWER THAT THE BEST ESTIMATE IS THE ONECLOSEST TO THE TRUE VALUE OF THE PARAMETER TO BE ESTIMATED  BUT EVERYESTIMATE IS A FUNCTION OF THE SAMPLE VALUES AND THUS IS THE OBSERVEDVALUE OF SOME RANDOM VARIABLE  THERE IS NO MEANS OF PREDICTING JUSTWHAT THE INDIVIDUAL VALUES ARE TO BE FOR ANY GIVEN EXPERIMENT SO THEGOODNESS OF AN ESTIMATE CANNOT BE JUDGED RELIABLY FROM INDIVIDUALVALUES  AS WE REPEATEDLY SAMPLE THE POPULATION HOWEVER WE MAY FORMSTATISTICS SUCH AS THE SAMPLE MEAN AND VARIANCE WHOSE DISTRIBUTIONSWE MAY CALCULATE  IF WE ARE ABLE TO FORM ESTIMATORS FROM THESESTATISTICS THEN THE BEST WE CAN HOPE FOR IS THAT THE BULK OF THE MASSIN THE DISTRIBUTION IS CONCENTRATED IN SOME SMALL NEIGHBORHOOD OF THETRUE VALUE  IN SUCH CIRCUMSTANCES THERE IS A HIGH PROBABILITY THATTHE ESTIMATE WILL ONLY DIFFER FROM THE TRUE VALUE BY A SMALL AMOUNTFROM THIS POINT OF VIEW WE MAY ORDER THE QUALITY OF ESTIMATORS AS AFUNCTION OF HOW THE SAMPLE DISTRIBUTION IS CONCENTRATED ABOUT THE TRUEVALUETHIS INTUITIVE NOTION IS EMBODIED BY THE CHEBYSHEV INEQUALITYINDEXCHEBYSHEV INEQUALITY WHICHSTATES THAT FOR A RANDOM VARIABLE Y WITH MEAN MUY AND VARIANCESIGMA2Y THE PROBABILITY THAT AN OBSERVATION YY DIFFERS FROMTHE MEAN BY EPSILON IS PYMUY EPSILON LEQ FRACSIGMA2EPSILON2THE SMALLER SIGMA2 IS THE LESS THE PROBABILITY THAT ANOBSERVATION IS FAR FROM THE MEAN  IN THE CONTEXT OF ESTIMATION WEWANT THE ESTIMATE OF A PARAMETER TO BE CLOSE TO THE MEAN  THE TRUEVALUE  SO WE WANT THE VARIANCE OF THE ESTIMATE TO BE AS SMALL ASPOSSIBLEEXTENDING THIS SIMPLE CONCEPT TO VECTOR PARAMETERS FOR VECTORPARAMETERS THETABF WITH A VECTOR ESTIMATE THETABFHAT A GOODESTIMATOR IS ONE FOR WHICH THE COVARIANCE C  ETHETABFHAT  ETHETABFTHETABFHAT  ETHETABFTIS AS SMALL AS POSSIBLE  SMALL HERE MEANS THE FOLLOWING  LET AAND B BE HERMITIAN MATRICES  THEN WE SAY THAT A  B IF BA ISPOSITIVE DEFINITE XBFT BA XBF  0 FOR ALL NONZERO VECTORSXBF  INDEXPOSITIVE DEFINITECOMPARING MATRICESWHILE THERE ARE OTHER MEASURES OF THE QUALITY OF AN ESTIMATE MOSTESTIMATION TECHNIQUES USE THE VARIANCE OR COVARIANCE FORMULTIDIMENSIONAL PARAMETERS EXCLUSIVELY AS A MEANS OF EVALUATING THEQUALITY OF THE ESTIMATE  THIS CHOICE IS MOTIVATED STRONGLY BY THEIMPORTANT CASE WHEN THE SAMPLING DISTRIBUTIONS OF THE ESTIMATES ARE ATLEAST APPROXIMATELY NORMAL SINCE THEN THE SECONDORDER MOMENT IS THENTHE UNIQUE MEASURE OF DISPERSIONBASED UPON THE ABOVE ARGUMENTS WE SHOULD FEEL JUSTIFIED IN FOCUSINGPRIMARILY ON  THE VARIANCE OF THE ESTIMATION ERROR AS THE MEASURE OF DISPERSION ANDHENCE OF GOODNESS  BUT I WANT TO SENSITIZE YOU TO THE FACT THAT THISIS A SOMEWHAT ARBITRARY ALBEIT VERY REASONABLE MEASURE OF GOODNESSAND LATER IN THIS COURSE I HOPE TO REVISIT THESE ISSUES IN A LITTLEMORE DEPTH AND BUILD A CASE FOR MEASURES OTHER THAN DISPERSION ASBEING VALID MEASURES OF QUALITY  BUT FOR NOW WE WILL FOLLOW THECONVENTIONAL DEVELOPMENT AND FOCUS ON THE MEASURE OF QUALITY BEINGEQUIVALENT TO MEASURES OF DISPERSION THAT IS TO THE VARIANCE OF THEESTIMATION ERRORSUBSECTIONTHE SCORE FUNCTIONINDEXSCORE FUNCTIONSUBSECTIONTHE CRAMERRAO BOUNDTHE MAXIMUM LIKELIHOOD METHOD OF ESTIMATION DOES NOT PROVIDE AS ABYPRODUCT OF CALCULATING THE ESTIMATE ANY MEASURE OF THECONCENTRATION THAT IS THE VARIANCE OF THE ESTIMATION ERRORALTHOUGH THE VARIANCE CAN BE CALCULATED FOR MANY IMPORTANT EXAMPLESIT IS DIFFICULT FOR OTHERS  RATHER THAN APPROACH THE PROBLEM OFCALCULATING THE VARIANCE FOR AN ESTIMATE DIRECTLY THEREFORE WE WILLFIRST CALCULATE A LOWER BOUND FOR THE VARIANCE OF THE ESTIMATION ERRORFOR EM ANY UNBIASED ESTIMATOR THEN WE WILL SEE HOW THE VARIANCE OFTHE MAXIMUM LIKELIHOOD ESTIMATION ERROR COMPARES WITH THIS LOWERBOUND  BEFORE STATING THE MAIN RESULT OF THIS SECTION WE NEED TOESTABLISH SOME NEW NOTATION AND TERMINOLOGY AND PROVE SOME PRELIMINARYRESULTSBEGINDEFINITIONLET XBF  X1 LDOTS XNT DENOTE AN NDIMENSIONAL RANDOMVECTOR AND THETABF  THETA1 LDOTS THETAPTDENOTE A PDIMENSIONAL PARAMETER VECTOR  THE BF SCORE FUNCTIONSBFTHETABF XBF IS THE GRADIENT OF THE LOGLIKELIHOOD FUNCTIONBEGINEQUATIONSBFTHETABFXBF  PARTIALDLAMBDATHETABFXBFTHETABF FRAC1ELLTHETABFXBF PARTIALDELLTHETABFXBFTHETABFLABELEQDEFSCOREENDEQUATIONENDDEFINITIONWE SEE THAT AT A ML ESTIMATE THETABFML ON THE INTERIOR OF THERANGE OF THETABF STHETABFMLXBF  0THE ML ESTIMATE IS A ZERO OF THE SCORE FUNCTION  WE ALSO SAY THATGOOD SCORES ARE THOSE WITH VALUES NEAR ZERO  IT IS IMPORTANT TONOTICE THAT SINCE THE SCORE IS RELATED TO THE GRADIENT THE RESULTS OFTHIS METHOD APPLY TO ESTIMATES THETABFHAT LYING ON THE INTERIOR OFTHETABFBEFORE CONTINUING WE PROVE SOME USEFUL FACTS ABOUT THE SCOREFUNCTION  WE BEGIN WITH THE FOLLOWING THEOREM  BEGINTHEOREM LABELSCORETHEOREMIF SBFTHETABFXBF IS THE SCORE OF A LIKELIHOODFUNCTION ELLTHETABFXBF AND IF TBF IS ANY VECTORVALUEDFUNCTION OF XBF AND THETABF THEN UNDER CERTAIN REGULARITYCONDITIONSFOOTNOTETHIS IS NICE WAY OF SAYING THAT WE WILL ASSUME  WHATEVER ADDITIONAL ASSUMPTIONS MAY BE REQUIRED TO ACCOMPLISH ALL OF  THE STEPS OUTLINED IN THE PROOF  THIS ISNT TOO BAD OF A COPOUT  SINCE THE REGULARITY CONDITIONS TURN OUT TO BE QUITE MILDBEGINEQUATIONLABELFUNNYTHEOREMESBFTHETABF XBFTBFTTHETABFXBF  PARTIALDTHETABFETBFTTHETABFXBF  E PARTIALDTHETABF TBFTTHETABFXBFENDEQUATIONENDTHEOREMBEFORE EMBARKING ON THE PROOF WE NOTE THAT IF TBFTHETABF T1THETABFT2THETABFLDOTSTKTHETABFT THEN PARTIALDTHETABFTBFT  BEGINBMATRIX PARTIALDT1THETA1 PARTIALDT2THETA1  CDOTS  PARTIALDTKTHETA1EXMATSP VDOTS PARTIALDT1THETAP PARTIALDT2THETAP  CDOTS  PARTIALDTKTHETAP ENDBMATRIXBEGINPROOF WE HAVE BEGINALIGNEDETBFTTHETABFXBF   INTTBFTTHETABFXBFFXBFXBFTHETABFDXBF10PT    INTTBFTTHETABFXBFELLTHETABFXBFDXBFENDALIGNEDUPON DIFFERENTIATING BOTH SIDES WITH RESPECT TO THETABF AND TAKINGTHE DIFFERENTIATION UNDER THE INTEGRAL SIGN IN THE RIGHTHAND SIDEASSUMING DIFFERENTIABILITY CONDITIONS AS APPROPRIATE WE OBTAINBEGINALIGNPARTIALDTHETABF ETBFTTHETABFXBF INT ELLTHETABFXBF PARTIALDTHETABF LOG ELLTHETABFXBFTBFTTHETABFXBFDXBF  INT ELLTHETABFXBF  PARTIALDTHETABF TBFTHETABFXBF DXBFNONUMBER  E SBFTHETABFXBF TBFTTHETABFXBF  E PARTIALDTHETABFTBFTTHETABFXBF LABELBIGMESSENDALIGNTHE RESULT FOLLOWS ON SIMPLIFYING AND REARRANGING THIS EXPRESSIONENDPROOFWE MAY QUICKLY OBTAIN TWO USEFUL COROLLARIES OF THIS THEOREMBEGINCOROLLARY  IF SBFTHETABF XBF IS THE SCORE CORRESPONDING TO A  DIFFERENTIABLE LIKELIHOOD FUNCTION ELLTHETABFXBF THENBEGINEQUATIONLABELCOROLLARY1ESBFTHETABFXBF  ZEROBFENDEQUATIONENDCOROLLARYBEGINPROOF  CHOOSE TBF AS ANY CONSTANTVECTOR  THEN SINCE TBF IS NOT A FUNCTION OFTHETABF ITS DERIVATIVE VANISHES SO BY REFFUNNYTHEOREMBEGINDISPLAYMATHESBFTHETABFXBFTBFT      ESBFTHETABFXBFTBFT ZEROBF ENDDISPLAYMATHWHICH CAN HAPPEN FOR ARBITRARY TBF ONLY IF ESBFTHETABFXBFZEROBFENDPROOFWE NOTE THAT REFCOROLLARY1 CAN BE WRITTEN ASBEGINALIGNESBFTHETABFXBF  E PARTIALDTHETABF LOG ELLTHETABFXBFNONUMBER  PARTIALDTHETABF INT ELLTHETABFXBFDX  0 LABELEQESZENDALIGNTHIS IS ALSO A MANIFESTATION OF THE TRVIAL OBSERVATION THATPARTIALDTHETABFINT FXBFXBFTHETABFDXBF PARTIALDTHETABF1  0 SINCE FXBF CDOT  THETABF ISA DENSITY FUNCTIONBEGINCOROLLARY  IF SBFTHETABF XBF IS THE SCORE CORRESPONDING TO A  DIFFERENTIABLE LIKELIHOOD FUNCTION ELLTHETABFXBF AND IF  TBFXBF IS ANY UNBIASED ESTIMATOR OF THETABF THENBEGINEQUATIONLABELCOROLLARY2ESBFTHETABF XBF TBFTXBF  IBFENDEQUATIONENDCOROLLARYBEGINPROOF  SINCE THE ESTIMATE IS UNBIASED WEHAVE ETBFXBF  THETABF AND SINCE TBF IS NOT A FUNCTION OFTHETABF WE HAVE PARTIALDTHETABF TBFT  0THUS BY REFFUNNYTHEOREM BEGINDISPLAYMATHESBFTHETABF XBF TBFTXBF  PARTIALDTHETABF THETABFT  IBFENDDISPLAYMATHENDPROOFSUBSECTIONTHE CRAMERRAO LOWER BOUNDLABELSECCRLBINDEXCRAMERRAO LOWER BOUND CRLBBEGINDEFINITIONTHE COVARIANCE MATRIX OF THESCORE FUNCTION IS THE BF FISHER INFORMATION MATRIX INDEXFISHER  INFORMATION MATRIX DENOTEDJTHETABF  SINCE BY REFCOROLLARY1 THE SCORE FUNCTION ISZEROMEAN WE HAVEBEGINEQUATIONLABELFISHERINFOJTHETABF  ESBFTHETABF XBFSBFTTHETABF XBF  EPARTIALDTHETABF LOG ELLTHETABFXBFPARTIALDTHETABF LOGELLTHETABFXBFTENDEQUATIONENDDEFINITIONBEGINLEMMA LABELLEMJOTHER  THE FISHER INFORMATION JTHETABF OF REFFISHERINFO CAN  BE WRITTEN ASBEGINEQUATION JTHETABF  E PARTIALDTHETABFLEFTPARTIALDTHETABF LOGELLTHETABFXBFRIGHTTLABELEQFISHERINFO2ENDEQUATIONENDLEMMABEGINPROOFLETTING TBF  SBF IN THEOREM REFSCORETHEOREM WE OBTAINBEGINALIGNESBFTHETABF XBFSBFTTHETABFXBF   PARTIALDTHETABFESBFTTHETABFXBF  E PARTIALDTHETABF SBFTTHETABFXBF    E PARTIALDTHETABF SBFTTHETABFXBFENDALIGNSINCE ESBFTTHETABFXBF 0 BY COROLLARY REFCOROLLARY1  THE INDICATED GRADIENTS CAN BE WRITTEN ASBEGINALIGNEDPARTIALDTHETABFLEFTPARTIALDTHETABF LOG  ELLTHETABFXBFRIGHTT PARTIALDTHETABF FRAC1ELLTHETABFXBF LEFTPARTIALDTHETABFELLTHETABFXBFRIGHTT    FRAC1ELLTHETABFXBF PARTIALDTHETABFLEFTPARTIALDTHETABFELLTHETABFXBFRIGHTT  FRAC1L2THETABFXBF PARTIALDTHETABFELLTHETABFXBF LEFTPARTIALDTHETABF ELLTHETABFXBFRIGHTT  FRAC1ELLTHETABFXBF PARTIALDTHETABFLEFTPARTIALDTHETABFELLTHETABFXBFRIGHTT   PARTIALDTHETABFLOG ELLTHETABFXBF LEFTPARTIALDTHETABF LOG ELLTHETABFXBFRIGHTTENDALIGNEDTAKING EXPECTATIONS OF BOTH SIDE AND USING REFEQESZ ON THEFIRST TERM ON THE RIGHTHAND SIDE WE FIND EFRAC1ELLTHETABFXBF PARTIALDTHETABFPARTIALDTHETABFELLTHETABFXBFT  0SO THAT WE OBTAIN E PARTIALDTHETABFPARTIALDTHETABF ELLTHETABFXBFT   EPARTIALDTHETABF LOG ELLTHETABFXBFPARTIALDTHETABF LOGELLTHETABFXBFTENDPROOFBEGINTHEOREMEM CRAMERRAO  IF TBFXBF IS ANY UNBIASED ESTIMATOR OFTHETABF BASED ON A DIFFERENTIABLE LIKELIHOOD FUNCTION THENBEGINEQUATIONLABELCRAMERRAOETBFXBFTHETABFTBFXBFTHETABFT GEQ J1THETABFENDEQUATIONWHERE JTHETABF IS THE FISHER INFORMATION MATRIXENDTHEOREMTHAT IS JTHETABF PROVIDES A LOWER BOUND ON THE COVARIANCE OF OFEM ANY UNBIASED ESTIMATOR OF THETABFTHE PROOF OF THIS THEOREM WILL BE YET ANOTHER APPLICATION OF THECAUCHYSCHWARZ INEQUALITY  INDEXCAUCHYSCHWARZ INEQUALITY BEFOREPROVING THE CRAMERRAO LOWER BOUND WE WILL INTRODUCE AN AUXILIARYRESULT THAT WILL BE NEEDED IN THE PROOFBEGINLEMMA LABELLEMAUXLEMLET J BE A POSITIVE DEFINITE MATRIX AND LET ABF BE A FIXED VECTORTHE MAXIMUM OF ABFTCBF SUBJECT TO THE CONSTRAINT BEGINEQUATIONLABELCONSTRAINTCBFTJCBF1ENDEQUATIONIS ATTAINED ATBEGINDISPLAYMATHCBF  FRACJ1ABFABFTJ1ABFFRAC12ENDDISPLAYMATHENDLEMMATHE PROOF OF THIS LEMMA IS EXPLORED IN THE EXERCISESBEGINPROOF  OF CRAMERRAO THEOREMLET ABF AND CBF BE TWO PDIMENSIONAL VECTORS AND LET SBFTHETABFXBF BETHE SCORE FUNCTION  FORM THE TWO RANDOM VARIABLES  ALPHA ABFTTBFXBFAND BETA CBFTSBFTHETABFXBF  SINCE THE CORRELATION COEFFICIENTBEGINDISPLAYMATHRHOALPHABETA FRACEALPHABETASQRTVARALPHAVARBETAENDDISPLAYMATH IS BOUNDED IN MAGNITUDE BY ONE WE HAVE THATBEGINEQUATIONLABELBOUNDFRACE2ALPHABETAVARALPHAVARBETA LEQ 1ENDEQUATIONBUT SINCE THE SCORE FUNCTION IS ZERO MEAN IT IS IMMEDIATE THAT BEGINALIGNEDVARBETA    ECBFTSBFTHETABFXBFSBFTTHETABFXBFCBF     CBFTVARSBFTHETABFXBFCBF    CBFTJTHETABFCBFENDALIGNEDALSO BEGINDISPLAYMATHVARALPHA   ABFTCOVTBFABFENDDISPLAYMATHFURTHERMORE BY REFCOROLLARY2 WE HAVE THAT  BEGINALIGNEDEALPHABETA     ABFT ETBFXBFSBFTTHETABFXBFCBF    ABFT IBF CBF    ABFTCBFENDALIGNEDSUBSTITUTING THESE EXPRESSIONS INTO REFBOUND AND SQUARINGBEGINEQUATIONLABELTRICKFRACEALPHABETA2VARALPHAVARBETA  FRACABFTCBF2ABFTCOVTBFABFCBFTJTHETABFCBFLEQ1 ENDEQUATIONIN THE INTEREST OF FINDING THE LARGEST VALUE OF THIS EXPRESSION ANDWITH THE ASSISTANCE OF CONSIDERABLE HINDSIGHT LET US SUBSTITUTE THERESULT OF LEMMA REFLEMAUXLEM INTO REFTRICK BEGINALIGNEDFRACABFTCBF2ABFTCOVTBFABFCBFTJTHETABFCBF LEQ  FRACLEFTFRACABFTJ1THETABFABF  SQRTABFTJ1THETABFABFRIGHT2ABFTVARTBFABF   FRACABFTJ1THETABFABFABFTCOVTBFABF LEQ 1ENDALIGNEDBEGINEQUATIONLABELGOTITENDEQUATIONWE OBSERVE THAT THIS INEQUALITY MUST HOLD FOR ALL ABF SO ABFT J1THETABF ABF LEQ ABFT COVTBF ABFORBEGINDISPLAYMATHABFTLEFTCOVTBFJ1THETABFRIGHTABF GEQ 0ENDDISPLAYMATH FOR ALL ABF WHICH IS EQUIVALENT TO REFCRAMERRAOENDPROOFTHE INVERSE OF THE FISHER INFORMATION MATRIX IS A LOWER BOUND ON THEVARIANCE THAT MAY BE ATTAINED BY ANY UNBIASED ESTIMATOR OF THEPARAMETER THETABF GIVEN THE OBSERVATIONS XBF  IT IS INTERESTINGTO DETERMINE CONDITIONS UNDER WHICH CRAMERRAO LOWER BOUND MAY BEACHIEVED  FROM REFBOUND WE SEE THAT EQUALITY IS POSSIBLE IFBEGINDISPLAYMATHE2ALPHABETAVARALPHAVARBETAENDDISPLAYMATHORBEGINDISPLAYMATHEALPHABETASQRTVARALPHASQRTVARBETAENDDISPLAYMATHBUT FROM THE CAUCHYSCHWARZ INEQUALITY EQUALITY IS POSSIBLE IF ANDONLY IF ALPHA AND BETA ARE LINEARLY RELATED THAT IS IFBEGINDISPLAYMATHTBFXBF  KTHETABFSBFTHETABF XBFENDDISPLAYMATHFOR SOME FUNCTION KTHETABFSUBSECTIONEFFICIENCYBEGINDEFINITIONINDEXEFFICIENCY OF AN ESTIMATOR  AN ESTIMATOR IS SAID TO BE BF EFFICIENT IF IT IS UNBIASED AND THE  COVARIANCE OF THE ESTIMATION ERROR ACHIEVES THE CRAMERRAO LOWER  BOUND  THAT IS IF HATTHETABF  TBFXBF IS AN ESTIMATOR  FOR THETABF THEN HATTHETABF IS EFFICIENT IF BEGINALIGNEDEHATTHETABF   THETABFEHATTHETABFTHETABFHATTHETABF THETABFT  J1THETABF ENDALIGNEDENDDEFINITIONBEGINTHEOREM LABELTHMEFFTHMEM EFFICIENCY  AN UNBIASED ESTIMATOR HATTHETABF IS EFFICIENTIF AND ONLY IFBEGINEQUATIONLABELEFFICIENTJTHETABFHATTHETABFTHETABF  SBFTHETABFXBFENDEQUATIONFURTHERMORE ANY UNBIASED EFFICIENT ESTIMATOR IS A MAXIMUM LIKELIHOOD ESTIMATORENDTHEOREM  BEGINPROOF SUPPOSE JTHETABFHATTHETABFTHETABF  SBFTHETABFXBF  THENFROM THE DEFINITION BEGINALIGNEDJTHETABF   ESBFTHETABFXBFSBFTTHETABFXBF    JTHETABFEHATTHETABFTHETABFHATTHETABFTHETABFTJ  THETABFENDALIGNEDBUT THIS RESULTIMPLIES EHATTHETABFTHETABFHATTHETABFTHETABFTJTHETABF IBF WHICH YIELDS EFFICIENCYCONVERSELY SUPPOSE HATTHETABF IS EFFICIENT  FROMREFCOROLLARY1 AND REFCOROLLARY2 IT FOLLOWS THAT BEGINDISPLAYMATHESBFTHETABFXBFHATTHETABFTHETABFT  IBFENDDISPLAYMATHSO BY THE CAUCHYSCHWARZ INEQUALITY INDEXCAUCHYSCHWARZ INEQUALITY BEGINALIGNEDIBF   LEFTESBFTHETABFXBFHATTHETABFTHETABFTRIGHT2  LEQ ESBFTHETABFXBFSBFTTHETABFXBFEHATTHETABFTHETABF HATTHETABFTHETABFT    JTHETABFEHATTHETABFTHETABFHATTHETABFTHETABFT    IBF MBOX BY EFFICIENCY ASSUMPTIONENDALIGNEDEQUALITY CAN HOLD WITH THE CAUCHYSCHWARZ INEQUALITY IF AND ONLY IFBEGINDISPLAYMATHSTHETABF XBF  KTHETABFHATTHETABF THETABFENDDISPLAYMATH FOR SOME CONSTANT  KTHETABF  MULTIPLYING BOTH SIDES OF THIS EXPRESSION BY  HATTHETABFTHETABFT AND TAKING EXPECTATIONS YIELDS KTHETABF JTHETABF    TO SHOW THAT ANY UNBIASED EFFICIENT ESTIMATOR IS A MAXIMUM  LIKELIHOOD ESTIMATOR LET HATTHETABF BE EFFICIENT AND  UNBIASED AND LET TILDETHETABF BE A MAXIMUM LIKELIHOOD  ESTIMATE OF THETABF  EVALUATING REFEFFICIENT AT THETABF   TILDETHETABF YIELDSBEGINDISPLAYMATHJTILDETHETABFHATTHETABFTILDETHETABF SBFTILDETHETABFXBFENDDISPLAYMATHBUT THE SCORE FUNCTION IS ZERO WHEN EVALUATED AT THE MAXIMUMLIKELIHOOD ESTIMATE CONSEQUENTLYBEGINDISPLAYMATHHATTHETABFTILDETHETABFENDDISPLAYMATHENDPROOFSUBSECTIONASYMPTOTIC PROPERTIES OF MAXIMUM LIKELIHOOD ESTIMATORSUNFORTUNATELY IT IS THE EXCEPTION RATHER THAN THAN THE RULE THAT ANUNBIASED EFFICIENT ESTIMATOR CAN BE FOUND FOR PROBLEMS OF PRACTICALIMPORTANCE  THIS FACT MOTIVATES US TO ANALYZE JUST HOW CLOSE WE CANGET TO THE IDEAL OF AN EFFICIENT ESTIMATE  OUR APPROACH WILL BE TOEXAMINE THE LARGESAMPLE PROPERTIES OF MAXIMUM LIKELIHOOD ESTIMATES  IN OUR PRECEDING DEVELOPMENT WE HAVE CONSIDERED THE SIZE OF THE SAMPLEAS A FIXED INTEGER M GEQ 1  LET US NOW SUPPOSE THAT AN UNBIASEDESTIMATE CAN BE DEFINED FOR ALL M AND CONSIDER THE ASYMPTOTICBEHAVIOR OF HATTHETAML AS M TENDS TO INFINITY  IN THISSECTION WE PRESENT WITHOUT PROOFS THREE KEY RESULTS SUBJECT TOSUFFICIENT REGULARITY OF THE DISTRIBUTIONSBEGINENUMERATEITEM MAXIMUM LIKELIHOOD ESTIMATES ARE CONSISTENTITEM MAXIMUM LIKELIHOOD ESTIMATES ARE ASYMPTOTICALLY NORMALLY  DISTRIBUTEDITEM MAXIMUM LIKELIHOOD ESTIMATES ARE ASYMPTOTICALLY  EFFICIENT ENDENUMERATEIN THE INTEREST OF CLARITY WE WILL TREAT ONLY THE CASE FOR SCALARTHETA  WE ASSUME IN THE STATEMENT OF THE FOLLOWING THREETHEOREMS THAT ALL OF THE APPROPRIATE REGULARITY CONDITIONS ARESATISFI