|
NATS 1700 6.0 COMPUTERS, INFORMATION AND SOCIETY
Lecture 6: Information : Shannon's Definition
| Previous | Next | Syllabus | Selected References | Home |
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
8 | 9 | 10 | 11 | 12 | 13 |
14 | 15 | 16 | 17 | 18 | 19 |
20 | 21 | 22 |
Introduction
-
For a nice introduction to the concept of information, read
 Information
Theory and Practice: Once Upon a Time Information Didn't Even Exist, a review of two recent books on the subject, on Salon 21st.
Other classical texts are briefly reviewed by Laurence Heglar in Information Theory.
- A rather unique project, How Much Information?, has been undertaken by a group of
"faculty and students at the School of Information Management and Systems at the University of California at
Berkeley....This study is an attempt to measure how much information is produced in the world each year...The world's
total yearly production of print, film, optical, and magnetic content would require roughly 1.5 billion gigabytes of
storage. This is the equivalent of 250 megabytes per person for each man, woman, and child on earth." Luckily, much
of this 'information' has probably little 'meaning' for most people.
-
A rather famous introduction to communication is provided by Colin Cherry in On Human Communication (3rd edition. 1978 MIT Press).
-
For a good summary of Shannon's ideas read Brief Excerpts
from Warren Weaver’s Introduction to Claude Shannon’s The Mathematical Theory of Communication.
- It is always wise to approach anything with some measure of humour. Rich Tennant, "the father of the computer cartoon,"
can certainly help us in this respect. Look for is work at The 5th Wave.
Topics
- The word information is used everywhere nowadays. 'Information technology' (and its acronym 'IT'), 'the
information age,' 'the information revolution,' and other variations on the theme are waved about like flags. We also know
that information and computers somehow belong together. It is therefore necessary to ask ourselves if the concept behind
this word is indeed still the familiar one. Is information just another term for facts, experience, knowledge?
- Although the two concepts are by no means identical, we will assume as a given that information and communication are
related. For example, in The Evolution of Communication (The MIT Press, 1997, p. 6), Marc Hauser writes:
"The concepts of information and signal form integral components of most definitions
of communication...Thus information is a feature of an interaction between sender and receiver. Signals carry certain
kinds of information content, which can be manipulated by the sender and differentially acted upon by the receiver.."
Statements such as this seem pretty clear, but they are qualitative, and may not be suitable in situations where
it is necessary to quantify information, signals, etc. This is obviously the case when we deal with computers and other
devices that process information. Various attempts to sharpen the concept of information were thus made, mostly in the first
part of the 20th century.
- For instance, in 1917 and later in 1928, Nyquist found the theoretical limits on how much information can be transmitted with
a telegraph, and the frequency band (or bandwidth) needed to transmit a given amount of information. In this case,
Nyquist defined information in terms of number of dots and dashes a telegraph system can exchange in a given time.
In 1928, Hartley defined the information content of a message as a function (the logarithm) of the number of messages
which might have occurred. For example, if we compare two situations, one in which six messages are possible, and
another in which fifty-two messages are possible (think of a dice, and of a deck of cards, respectively), Hartley would say
that a message about which face of the dice came up would carry less information than a message about which card came up on
top of the deck of cards.
- In 1948, Claude E Shannon, and American engineer, took a fresh look at information from the point of view of
communication. For Shannon, not unlike Hartley, information is that which reduces a receiver's uncertainty.
In a given situation, any message can be thought of as belonging to a set of possible messages. For example, if I ask
a weather forecaster how's the sky over a certain region, the possible answers I could receive would be something
like 'clear,' 'partly cloudy,' 'overcast.' Before I get the answer, I am uncertain as to which of the three possibilities
is actually true. The forecaster's answer reduces my uncertainty. The larger the set of possible messages, the greater
is the reduction of uncertainty. If the set contains only one possibility, then I already know the answer, I am certain
about it, and Shannon would say that no information is conveyed to me by the actual answer. This last observation is
important, because in our daily lives we may not be prepared to accept it. A message may not contain any new factual
information, but the expression of my interlocutor's face or the tone of his voice do carry information that is meaningful
to me. We must distinguish therefore between the vernacular sense of information, and the technical one used in
telecommunications. As it always happens in science, to quantity the description of a phenomenon makes that
description usable in a precise, predictable, controllable way. But this happens at the expenses of most of their familiar
denotations and connotations. That's what we mean when we complain that science is drier, poorer than life. To make
concepts unambiguous, so that they can be logically manipulated, we have no choice but to discard the ambiguities that
language allows.
Claude Shannon
- For Shannon therefore information is a form of processed data about things, events, people, which is meaningful to the
receiver of such information because it reduces the receiver's uncertainty. Here is
another way to express Shannon's definition: "The more uncertainty there is about the contents of a message that is about
to be received, the more information the message contains. He was not talking about meaning. Instead, he was talking about
symbols in which the meaning is encoded. For example, if an AP story is coming over the wire in Morse code, the first
letter contains more information than the following letters because it could be any one of 26. If the first letter
is a 'q,' the second letter contains little information because most of the time 'q' is followed by 'u.' In other words,
we already know what it will be."
- Shannon formalized this definition in a formula which is now so famous that it deserves at least one appearance
in these lectures: H = log2n . H is the quantity
of information carried by one of n possible (and equally probable, for simplicity) messages; log2n is the
logarithm in base 2 of n. For example, log28 = 3, log232 = 5, etc. Notice in particular that
when n = 1, then H = 0. In words, when a given message is the only possible one, then no information is conveyed by
such message. Finally, notice that if there are only two possible messages, say 'yes' and 'no,' then the amount of
information of each message will be H = log22 = 1. In this case we say that the message carries
1 bit of information. The word "bit' is a contraction of the expression 'binary digit.'
- Shannon was able to prove that his formula H = log2n represents the
maximum amount of information that a message can carry. Noise and other factors will usually reduce
the actual figure.
- Let's now return to communication. What happens when two people communicate? They do exchanges messages, of course.
But there is more. For example, they may be speaking to each other on a subway car during rush hour, and the surrounding
buzz may make communications difficult. Or lightning may introduce spurious signals in the telegraph wires. Or the cat
may have spilled some milk on your newspaper smudging some of the words. In fact noise is always present, and in a
fundamental sense it can never be completely eliminated. Shannon proceeded therefore to formalize the concept of
noise, but I will spare you the technicalities.
- Here, in conclusion, is the model of communication that Shannon adopted.
|
