University of Surrey

Test tubes in the lab Research in the ATI Dance Research

Information Theory and Neural Networks

Taylor, John G. and Plumbley, Mark D. (1993) Information Theory and Neural Networks In: Mathematical Approaches to Neural Networks. North-Holland Mathematical Library, 51 . Elsevier Science Publishers, Netherlands, pp. 307-340. ISBN 978-0-444-81692-4

Information Theory and Neural Networks.pdf - Accepted version Manuscript
Available under License Creative Commons Attribution Non-commercial No Derivatives.

Download (824kB) | Preview


This chapter discusses the role of information theory for analysis of neural networks using differential geometric ideas. Information theory is useful for understanding preprocessing, in terms of predictive coding in the retina and principal component analysis and decorrelation processing in early visual cortex. The chapter introduces some concepts from information theory. In particular, the entropy of a random variable and the mutual information between two random variables are focused. One of the major uses for information theory has been in interpretation and guidance for unsupervised neural networks: networks that are not provided with a teacher or target output that they are to emulate. The chapter describes how information relates to the more familiar supervised learning schemes, and discusses the use of error back propagation (BackProp) to minimize mean squared error (MSE) in a multi-layer perceptron (MLP). Other distortion measures are possible in place of MSE. In particular, the information theoretic cross-entropy distortion has been focused in the chapter.

Item Type: Book Section
Divisions : Faculty of Engineering and Physical Sciences > Electronic Engineering
Authors :
Taylor, John G.
Plumbley, Mark
Date : 1 December 1993
DOI : 10.1016/S0924-6509(08)70042-4
Copyright Disclaimer : © 1993. This manuscript version is made available under the CC-BY-NC-ND 4.0 license
Related URLs :
Additional Information : ISSN: 0924-6509
Depositing User : Symplectic Elements
Date Deposited : 17 May 2017 13:35
Last Modified : 17 Jan 2019 11:41

Actions (login required)

View Item View Item


Downloads per month over past year

Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800