The Ultimate Machine by Claude Elwood Shannon, "the father of information age"

If we speak about people who shaped informatics / software engineering, we cannot forget "the father of information age", Shannon, who died nearly exactly 10 years ago, on february 24th, 2001. One of his writing was claimed the "Magna Charta of the Information Age" - and probably this essay shaped our present much more than the now 10 years old other "Magna Charta".

{jb_dropcap}W{jb_dropcap} ithout him, we may not be able to listen to CDs or MP3s, or make phone calls on our mobile, let alone connect to a WiFi network. But who was this guy? Shannon was born in Michigan, in 1916; he did his bachelor's degree on two courses parallel: in mathematics and in electrical engineering.

His master's thesis from MIT is the most influental master's theses ever written, as of today: it's about the application of boolean algebra to digital circuits, so, in short: he invented digital logic.

But it wasn't his only invention. He joined Bell Labs, where he worked on digital transmission of phone talks. During the war he worked on crytography.

Following the war, two of his essays appeared to the public: "A Mathematical Theory of Communication" in 1948 and "Communication Theory of Secrecy Systems" in 1949.

The boundaries of the effects of these are still to be seen:

• - taking the bit as not only a measurement for transmission, but of the information itself makes us in itself able to digitize the things surrounding us;
• - setting the theoretical boundaries of transmission speed made us able to reach today's high-speed networks, both wired and unwired; two of his pupils founded Qualcomm, one of the leading UMTS chipset manufacturers, but the effects are general to the industry itself
• - showing the behaviour of information vs noise we were able to invent the CD, and even the MP3, besides the above mentioned communication netwokrs
• - he set the basic theory of modern cryptography systems

His hindsights paved the way for future generations to be there where we are now in terms of communication, and understanding the nature of the information.

Besides these not so small things, he constructed the formula of joggling - his favourite hobbi - but also implemented a well-known geek toy: The Ultimate Machine.
Dim lights Embed Embed this video on your site
While most of the inventions of the world were a result of social change, and were invented paralelly by multiple people at the same time, it's still of question, wether we would be here without him.

Biography of Claude Elwood Shannon

This biography, written by N.J.A. Sloane and A.D. Wyner, is one of two biographies of Claude Shannon that appears in the book we edited, Shannon's Collected Papers.

Claude Elwood Shannon was born in Petoskey, Michigan, on Sunday, April 30, 1916. His father, Claude Sr. (1862-1934), a descendant of early New Jersey settlers, was a businessman and, for a period, Judge of Probate. His mother, Mabel Wolf Shannon (1880-1945), daughter of German immigrants, was a language teacher and for a number of years Principal of Gaylord High School, in Gaylord, Michigan.

The first sixteen years of Shannon's life were spent in Gaylord, where he attended the Public School, graduating from Gaylord High School in 1932. As a boy, Shannon showed an inclination toward things mechanical. His best subjects in school were science and mathematics, and at home he constructed such devices as model planes, a radio-controlled model boat and a telegraph system to a friend's house half a mile away. The telegraph made opportunistic use of two barbed wires around a nearby pasture. He earned spending money from a paper route and delivering telegrams, as well as repairing radios for a local department store. His childhood hero was Edison, who he later learned was a distant cousin. Both were descendants of John Ogden, an important colonial leader and the ancestor of many distinguished people. Shannon's recent hero list, without deleting Edison, includes more academic types such as Newton, Darwin, Einstein and Von Neumann.

In 1932 he entered the University of Michigan, following his sister Catherine, who had just received a master's degree in mathematics there. While a senior, he was elected a member of Phi Kappa Phi and an associate member of Sigma Xi. In 1936 he obtained the degrees of Bachelor of Science in Electrical Engineering and Bachelor of Science in Mathematics. This dual interest in mathematics and engineering continued throughout his career.
Dim lights Embed Embed this video on your site
In 1936 he accepted the position of research assistant in the Department of Electrical Engineering at the Massachusetts Institute of Technology. The position allowed him to continue studying toward advanced degrees while working part-time for the department. The work in question was ideally suited to his interests and talents. It involved the operation of the Bush differential analyzer, the most advanced calculating machine of that era, which solved by analog means differential equations of up to the sixth degree. The work required translating differential equations into mechanical terms, setting up the machine and running through the needed solutions for various initial values. In some cases as many as four assistants would be needed to crank in functions by following curves during the process of solution.

Also of interest was a complex relay circuit associated with the differential analyzer that controlled its operation and involved over one hundred relays. In studying and servicing this circuit, Shannon became interested in the theory and design of relay and switching circuits. He had studied symbolic logic and Boolean algebra at Michigan in mathematics courses, and realized that this was the appropriate mathematics for studying such two-valued systems. He developed these ideas during the summer of 1937, which he spent at Bell Telephone Laboratories in New York City, and, back at M.I.T., in his master's thesis, where he showed how Boolean algebra could be used in the analysis and synthesis of switching and computer circuits. The thesis, his first published paper, aroused considerable interest when it appeared in 1938 in the A.I.E.E. Transactions [1].* In 1940 it was awarded the Alfred Noble Prize of the combined engineering societies of the United States, an award given each year to a person not over thirty for a paper published in one of the journals of the participating societies. A quarter of a century later H. H. Goldstine, in his book The Computer from Pascal to Von Neumann, called this work ``one of the most important master's theses ever written...a landmark in that it helped to change digital circuit design from an art to a science.''
Dim lights Embed Embed this video on your site
During the summer of 1938 he did research work at M.I.T. on the design of the Bush Rapid Selector, and was mainly involved with the vacuum tube circuits employed in this device. In September of 1938, at the suggestion of Vannevar Bush, Shannon changed from the Electrical Engineering Department to the Mathematics Department. He was awarded the Bolles Fellowship and was also a teaching assistant while working toward a doctorate in mathematics. Bush had just been made President of the Carnegie Institution in Washington, one of whose branches, in Cold Spring Harbor, N.Y., dealt with the science of genetics. He suggested to Shannon that algebra might be as useful in organizing genetic knowledge as it was in switching, and Shannon decided to look into this matter with a view toward using it for a doctoral thesis in mathematics. He spent the summer of 1939 at Cold Spring Harbor working under geneticist Dr. Barbara Burks exploring the possibility, and found it a suitable subject for a dissertation under the title ``An Algebra for Theoretical Genetics'' [3]. His Ph.D. supervisor at M.I.T. was Professor Frank L. Hitchcock, an algebraist. In the Spring of 1939 he was elected to full membership in Sigma Xi.
Dim lights Embed Embed this video on your site
At about this time Shannon was also developing ideas both in computers and communications systems. In a letter of February 16, 1939 now in the Library of Congress archives ([2], included in Part A), he writes to Bush about trading relations between time, bandwidth, noise and distortion in communication systems, and also about a computer design for symbolic mathematical operations.

As the Spring of 1940 approached, Shannon had passed all requirements for both a master's in electrical engineering and a doctorate in mathematics -- except for satisfying the language requirements, always his weakest subjects. Facing reality, he buckled down in the last few months, hired a French and German tutor and repeatedly worked his way through stacks of flash cards. He finally passed the language exams (it took two tries with German) and in the Spring of 1940 received the S.M. degree in Electrical Engineering and the degree of Doctor of Philosophy in Mathematics at the same commencement. His Ph.D. dissertation, [3], is published here for the first time (in Part D).

The Summer of 1940 was spent at Bell Telephone Laboratories doing further research on switching circuits. A new method of design was developed which greatly reduced the number of contacts needed to synthesize complex switching functions from earlier realizations. This was later published in a paper, ``The Synthesis of Two-Terminal Switching Circuits'' [50].

The academic year 1940-1941 was spent on a National Research Fellowship at the Institute for Advanced Study in Princeton working under Hermann Weyl. It was during this period that Shannon began to work seriously on his ideas relating to information theory and efficient communication systems.

Thornton C. Fry, head of the mathematics department at Bell Labs, was in charge of a committee on fire control systems for anti-aircraft use -- the country was arming up at the time because of the spreading European war threats -- and asked Shannon to join in this effort. Returning to Bell Labs, Shannon joined a team working on anti-aircraft directors -- devices to observe enemy planes or missiles and calculate the aiming of counter missiles. This problem became crucial with the development of the German V1 and V2 rockets. Without the American anti-aircraft directors, the ravaging of England, bad as it was, would have been vastly worse.

Shannon spent fifteen years at Bell Laboratories in a very fruitful association. Many first-rate mathematicians and scientists were at the Labs -- men such as John Pierce, known for satellite communication; Harry Nyquist, with numerous contributions to signal theory; Hendrik Bode of feedback fame; transistor inventors Brattain, Bardeen and Shockley; George Stibitz, who built an early (1938) relay computer; Barney Oliver, engineer extraordinaire; and many others.

During this period Shannon worked in many areas, most notably in information theory, a development which was published in 1948 as ``A Mathematical Theory of Communication'' [37]. In this paper it was shown that all information sources -- telegraph keys, people speaking, television cameras and so on -- have a ``source rate'' associated with them which can be measured in bits per second. Communication channels have a ``capacity'' measured in the same units. The information can be transmitted over the channel if and only if the source rate does not exceed the channel capacity (see the Preface to Part A).

This work on communication is generally considered to be Shannon's most important scientific contribution. In 1981 Professor Irving Reed, speaking at the International Symposium on Information Theory in Brighton, England, said, ``It was thirty-four years ago, in 1948, that Professor Claude E. Shannon first published his uniquely original paper, `A Mathematical Theory of Communication,' in the Bell System Technical Journal. Few other works of this century have had greater impact on science and engineering. By this landmark paper and his several subsequent papers on information theory he has altered most profoundly all aspects of communication theory and practice.''
Dim lights Embed Embed this video on your site
Shannon has rung many changes on the problems of information and noise. In a paper ``Communication Theory of Secrecy Systems'' [25] cryptography is related to communication in a noisy channel, the ``noise'' being in this case the scrambling by the key of the cryptographic system. This work later led to his appointment as a consultant on cryptographic matters to the United States Government.

Another problem, which he investigated jointly with E. F. Moore [88]--[90], was that of increasing the reliability of relay circuits by redundant use of contacts, each of which may be unreliable. Again this is a problem related to transmission in noisy channels.

Shannon has also applied these concepts to the problem of optimal investment strategies. The ``noisy signal'' is the stock market and related time series, and the problem is to maximize a utility function by proper choice and adjustment of a portfolio.

In a lighter vein and in the computer and artificial intelligence area, Shannon wrote a paper ``Programming a Computer for Playing Chess'' in 1950 [54]. At that time computers were slow, inept and very difficult to program. Since then, many chess-playing programs have been written, most of them following quite closely the system described in that early paper.

In 1965 he was invited to Russia to give lectures at an engineering conference. While there, he took the opportunity to meet Mikhail Botvinnik, for many years the World Chess Champion. Botvinnik, also an electrical engineer, had become interested in the chess programming problem. Shannon remembers the discussion as interesting but carried on through a somewhat noisy channel since the interpreters knew little of either chess or computers.

After the discussion, he asked Botvinnik for the pleasure of a chess game. Translators, guides and members of the American party watched with rapt attention as the epic battle unfolded. At one point Shannon had a slight edge (a rook for a knight and pawn), but alas the result was foregone -- after forty-two moves Shannon tipped over his king -- a message that needed no translation.

Further advances in chess programming continued through the next decades and in 1980 Shannon was invited, as an honored guest, to an International Computer Chess Championship held in Linz, Austria. Eighteen computers from Sweden, Germany, Russia, France, England, Canada and several from the United States were entered. Most of the computers remained at home but were linked electronically to the tournament hall in Linz. The winner, ``Belle,'' developed by Ken Thompson and Joe Condon of Bell Laboratories, was not far from master playing strength.
Dim lights Embed Embed this video on your site
Dr. Shannon enjoys constructing amusing if not utilitarian devices, and his house is filled with such brainchildren. Among these might be mentioned THROBAC (Thrifty ROman numerical BAckward looking Computer) [76], a calculator which performs all the arithmetic operations in the Roman numerical system; ``turtles'' which wander around the floor, backing up and turning from obstacles; game-playing machines of various types and sizes; and a three-ball juggling machine with two hands that bounce-juggles three balls on a drumhead.

The ``Ultimate Machine,'' based on an idea of Mervin Minsky, was built in the early fifties. The operation and spirit were well described by Arthur C. Clarke in Voice Across the Sea: ``Nothing could be simpler. It is merely a small wooden casket, the size and shape of a cigar box, with a single switch on one face. When you throw the switch, there is an angry, purposeful buzzing. The lid slowly rises, and from beneath it emerges a hand. The hand reaches down, turns the switch off and retreats into the box. With the finality of a closing coffin, the lid snaps shut, the buzzing ceases and peace reigns once more. The psychological effect, if you do not know what to expect, is devastating. There is something unspeakably sinister about a machine that does nothing -- absolutely nothing -- except switch itself off.''

The maze-solving mouse Theseus, built in 1950, took a more positive approach to its universe. Controlled by a relay circuit, a lifesize magnetic mouse moved around a maze of twenty-five squares. The maze could be altered at will and the mouse would then search through the passageways until it found the arbitrarily placed goal. Having been through the maze, the mouse could be placed anywhere it had been and would go directly to the goal -- placed in unfamiliar ground, it would search until it reached a known position and then proceed to the goal, adding the new knowledge to its memory. It appears to have been the first learning device of this level.

In the case of Theseus, both the ``brain'' and the ``muscles'' were separate from the mouse itself and were in fact under the maze. The brain was a circuit of about 100 relays, and the muscles a pair of motors driving an electromagnet which by magnetic action moved the mouse through the maze. With the development of solid state circuitry, self-contained mice became feasible. Compared to Theseus, the brains were smaller but the mice were bigger. By 1978 enough engineers had built maze-solving mice for the IEEE Spectrum to hold an ``Amazing Micro Mouse Maze Contest,'' at which Theseus made a guest appearance.

A happy consequence of Shannon's sojourn at Bell Labs was his marriage to Mary Elizabeth (Betty) Moore. Betty, a graduate in mathematics of Douglass College, Rutgers University, worked as a numerical analyst (what was then called a ``computer'') in John Pierce's group. Her interests in handweaving and computing are currently combined in work with a computer-controlled loom, an area in which she pioneered in the sixties. Claude and Betty were married in 1949 and have three children, Robert, Andrew and Margarita. They live on Mystic Lake, in Winchester, Massachusetts.

In 1956 Dr. Shannon was invited to be a visiting professor at M.I.T. and, in 1957-58, a fellow at the Center for the Study of the Behavioral Sciences in Palo Alto, California. The following year he became a permanent member of the M.I.T. faculty as Donner Professor of Science, where he continued research in various areas of communication theory. Among these were communications systems with feedback and a study of the rate at which it is possible to approach ideal coding as a function of delay. He continued his affiliation with Bell Telephone Laboratories until July 1, 1972.
Dim lights Embed Embed this video on your site
Many of Shannon's papers have been translated into various foreign languages. Perhaps the most thorough job was that of Russian scientists, who have long been interested in information theory and computers and have contributed greatly to these fields. In 1963 he received three copies of an 830-page collection, in Russian, of his scientific papers [121]. Years later, on a visit to Russia, he was informed that his book had been collecting royalties to the amount of several thousand rubles, which translated roughly into the same number of dollars. Unfortunately, there was a catch -- this could not be taken out of the country as money, but could only be spent in Russia. Curiously, nothing they might buy seemed suitable. The books were in Russian, Betty already had a fur coat, furniture was difficult to transport. They finally ended up with an array of eight musical instruments ranging from a bassoon to a balalaika. On the trip home the party was often taken for a traveling orchestra.

In his leisure time Shannon, in addition to the gadgeteering mentioned above, has a number of recreations. He tries to jog a mile or two each day, and enjoys sports like juggling which require good coordination. One Christmas, Betty, knowing his proclivities, gave him a unicycle. Within a few days he was riding around the block; in a few weeks he could juggle three balls while riding. In a few months he was building unusual cycles such as one with an eccentric wheel (the rider moved up and down as he pedalled forward). He is an easy mark for any new intellectual challenge -- he designed a machine to solve the Rubik cube, and was observed trying to equal his son's record at Pac-Man.

Shannon plays the clarinet and enjoys music, especially the Dixieland popular in his youth. He likes poetry with a nod to T. S. Eliot, the Rubaiyat and Ogden Nash, and has been known to dash off a bit of light verse from time to time [127].

He holds honorary degrees from Yale (Master of Science, 1954), Michigan (1961), Princeton (1962), Edinburgh (1964), Pittsburgh (1964), Northwestern (1970), Oxford (1978), East Anglia (1982), Carnegie-Mellon (1984), Tufts (1987) and the University of Pennsylvania (1991).

His awards include the Alfred Noble Prize (1940), Morris Liebmann Memorial Award of the Institute of Radio Engineers (1949), Stuart Ballantine Medal of the Franklin Institute (1955), Research Corporation Award (1956), Rice University Medal of Honor (1962), Marvin J. Kelly Award (1962), I.E.E.E. Medal of Honor (1966), National Medal of Science (1966) presented by President Johnson, Golden Plate Award (1967), Harvey Prize, Technion, Haifa (1972) presented by the President of Israel, Jacquard Award (1978), Harold Pender Award (1978), Audio Engineering Society Gold Medal (1985), the Kyoto Prize (1985) and the Eduard Rhein Prize (1991).

He delivered the Vanuxem Lectures, Princeton (1958); the Steinmetz Lecture, Schenectady (1962); the Gibbs Lecture, American Mathematical Society (1965); the first Shannon Lecture, I.E.E.E. (1973); and the Chichele Lecture, Oxford (1978).

He has been Bolles Fellow at M.I.T. (1938-40); National Research Fellow at the Institute for Advanced Study in Princeton (1940-41); Fellow of the Center for Advanced Study in the Behavioral Sciences, Stanford (1957-58), Visiting Fellow at All Souls College, Oxford (1978); and is a Fellow of Muir College of the University of California, the I.E.E.E., and the Royal Society. He is (or has been) a member of the National Academy of Sciences, the National Academy of Engineering, the American Mathematical Society, the American Philosophical Society, the Royal Irish Academy, the American Academy of Arts and Sciences, the Royal Netherlands Academy, the Leopoldina Academy of Leipzig, and Tau Beta Pi, Sigma Xi, Phi Kappa Phi and Eta Kappa Nu. For many years he was a member of the board of directors of Teledyne, Inc.

In 1983, Dr. Shannon wrote concerning information technologies: ``The growth of both communication and computing devices has been explosive in the last century. It was about a hundred years ago that the telephone and phonograph were invented, and these were followed by radio, motion pictures and television. We now have vacuum tubes, transistors, integrated circuits, satellite communication and microwave cable. We have even talked to astronauts on the moon. Our life style has been totally changed by advances in communication.
Dim lights Embed Embed this video on your site
``On the computing side we started the twentieth century with slide rules and adding machines. These were followed in quantum jumps by the Bush analog computers, Stibitz and Aiken relay computers, Eckert and Mauchly vacuum tube machines (ENIAC), transistor computers and, finally, the incredibly compact integrated circuit and chip computers. At each step the computers became faster, cheaper and more powerful. These hardware revolutions were matched by equally impressive developments in programming.

``What can we expect in the future? Three advances in the artificial intelligence area would be most welcome. (1) An optical sensor-computer combination capable of learning to recognize objects, people, etc., as our eyes and occipital cortex do. (2) A manipulator-computer combination capable of the purposeful operations of the human hand. (3) A computer program capable of at least some of the concept formation and generalizing abilities of the human brain.

``In the communication area our government might consider diverting a small fraction of its `defense' budget to the construction of giant radio telescopes as proposed by the SETI (Search for Extraterrestrial Intelligence) program, to listen for evidence of intelligent life on other star systems -- possibly as a joint venture with the Soviets. Who knows, perhaps E.T. would have words of wisdom for all of us!''

Shannon was recently interviewed by the Scientific American and the interviewer, John Horgan, reports that: ``Claude E. Shannon can't sit still. We're at his home, a stuccoed Victorian edifice overlooking a lake north of Boston, and I'm trying to get him to recall how he came up with the theory of information. But Shannon, who is a boyish 73, with an elfish grin and a shock of snowy hair, is tired of expounding on his past. Wouldn't I rather see his toys?

``Without waiting for an answer, and over the mild protests of his wife, Betty, he leaps from his chair and disappears into the other room. When I catch up with him, he proudly shows me his seven chess-playing machines, gasoline-powered pogostick, hundred-bladed jackknife, two-seated unicycle and countless other marvels. Some of his personal creations -- such as a juggling W. C. Fields mannequin and a computer called THROBAC that calculates in Roman numerals -- are a bit dusty and in disrepair, but Shannon seems as delighted with everything as a 10-year-old on Christmas morning.
Dim lights Embed Embed this video on your site
Is this the man who, as a young engineer at Bell Laboratories in 1948, wrote the Magna Carta of the information age: The Mathematical Theory of Communication? Whose work Robert W. Lucky, executive director of research at AT&T Bell Laboratories, calls the greatest `in the annals of technological thought?' Whose `pioneering insight' IBM Fellow Rolf W. Landauer equates with Einstein's? Yes. This is also the man who invented a rocket-powered Frisbee and who juggled while riding a unicycle through the halls of Bell Labs. `I've always pursued my interests without much regard to financial value or value to the world,' Shannon says. `I've spent lots of time on totally useless things.'

... ... ...

``Shannon's ideas were almost too prescient to have an immediate practical impact. Vacuum-tube circuits simply could not calculate the complex codes needed to approach the Shannon limit. In fact, not until the early 1970's -- with the advent of high-speed integrated circuits -- did engineers begin fully to exploit information theory. Today Shannon's insights help shape virtually all systems that store, process or transmit information in digital form, from compact disks to computers, from facsimile machines to deep-space probes such as Voyager.

``Information theory has also infiltrated fields outside of communications, including linguistics, psychology, economics, biology, even the arts. In the early 1950's the IEEE Transactions on Information Theory published an editorial, titled ``Information Theory, Photosynthesis and Religion,'' decrying this trend. Yet Shannon himself suggests that applying information theory to biological systems may not be so farfetched, because in his view common principles underlie mechanical and living things. `You bet,' he replies, when asked whether he thinks machines can think. `I'm a machine and you're a machine, and we both think, don't we?'

``He built a `mind-reading' machine [73] that played the game of penny-matching, in which one person tries to guess whether the other has chosen heads or tails. A colleague at Bell Labs, David W. Hagelbarger, built the prototype; the machine recorded and analyzed its opponent's past choices, looking for patterns that would foretell the next choice. Because it is almost impossible for a human to avoid falling into such patterns, the machine won more than 50 percent of the time. Shannon then built his own version and challenged Hagelbarger to a legendary dual. Shannon's machine won.''

This biographical sketch was based on the booklet Claude E. Shannon, Medalist for 1983 that was issued when he was awarded the John Fritz medal. It has been supplemented by material from other sources, including a profile by John Horgan that appeared in the Scientific American of January 1990.

Books

A Mathematical Theory of Communication

Theory of Data Compression