The Thinking Engine

From mathematical logic and Alan Turing to Asimov's Multivac, Clarke's HAL 9000, and the question of whether a computer can genuinely think.

1. A conversation with a stranger

Imagine sitting alone in a room with a computer terminal.

There is no camera or microphone. You cannot see the person at the other end of the connection. You can only exchange written messages.

You type:

Are you human?

The reply appears.

What an odd question. Would you ask a stranger that if you met them in a café?

You hesitate.

Perhaps. If I thought they were a machine.

And what would convince you otherwise?

The exchange continues. You talk about music, childhood, mathematics, dreams and the peculiar sadness of returning to an empty house.

Your unseen correspondent makes jokes. It occasionally misunderstands you. It remembers earlier remarks and seems to recognise when you are being sarcastic.

After half an hour, you must decide whether you have been communicating with a person or a computer.

What evidence would you use?

You cannot inspect the machinery. You cannot measure your correspondent's feelings. You cannot look inside its mind.

You have only its words.

This is the central idea behind one of the most influential proposals in the history of artificial intelligence.

In 1950, the mathematician Alan Turing published a paper called Computing Machinery and Intelligence.

It opened with a deceptively straightforward question:

"Can machines think?"

Turing understood that the question contained a difficulty. Before answering it, we would need to agree on what machine and think meant.

Instead, he proposed a more concrete experiment based on conversation, which he called the imitation game.

In its machine-oriented form, a human interrogator communicates through text with a human and a machine and attempts to distinguish between them.

If the machine can imitate human conversation convincingly enough, what does that tell us about intelligence?

Notice what Turing did not claim.

He did not demonstrate that convincing conversation automatically proves consciousness. Nor did he establish that imitating a person is the only possible form of intelligence.

He proposed a way to replace a difficult philosophical dispute with a question about observable performance.

That change was revolutionary.

And literature had already been preparing the ground.

2. When mathematics learned to describe thinking

Before computers could imitate conversation, mathematicians had to establish something more fundamental.

Could reasoning itself be described as a mechanical process?

For centuries, philosophers had investigated the rules of valid argument.

Consider the familiar pattern:

All humans are mortal.

Socrates is human.

Therefore, Socrates is mortal.

The conclusion follows from the premises because of the argument's logical structure.

The names are not essential. We could replace Socrates with another individual and mortality with another property, provided the same logical relationships held.

This suggested that reasoning might be expressed through formal symbols.

In the nineteenth century, George Boole developed an algebraic treatment of logic.

Instead of manipulating ordinary numbers alone, we could manipulate statements with values such as true and false.

Two fundamental operations are AND and OR.

For example, imagine a fictional security system protecting a laboratory.

Its doors open only when a person has a valid identity card and knows the correct password.

Let:

A means the card is valid. B means the password is correct.

The door-opening rule becomes:

O=A∧ B

The symbol ∧ means AND.

The logic can be represented as a table.

This is a truth table, a method of displaying every possible combination of logical inputs.

It might appear trivial. Yet the ability to manipulate logical relationships using well-defined operations became a foundation of digital computer design.

Electronic circuits can represent binary states using voltage levels. Logic gates can perform Boolean operations. Gates can be combined into circuits that add numbers, compare values, store information and execute instructions.

A modern computer contains unimaginably more complexity than our fictional laboratory door.

But at the lowest levels, formal logical operations remain fundamental.

The remarkable development was not that machines could calculate.

Mechanical calculators had existed for centuries.

It was that machines could execute formally specified procedures involving decisions.

And decisions are among the fundamental ingredients of stories.

3. The machine that follows instructions

In 1936, Turing published On Computable Numbers, with an Application to the Entscheidungsproblem.

The paper described an abstract machine capable of performing calculations by reading and writing symbols according to precise instructions.

Today we call it a Turing machine.

It is not a particular historical computer sitting in a museum. It is a mathematical model of computation.

Imagine a device with a long tape divided into squares. Each square can contain a symbol. A reading head examines one square at a time.

The machine can read a symbol, write another symbol, move left or right and change its internal state.

Its behaviour is governed by a table of rules.

This simplified example moves right across a tape, changing every 0 into 1. It demonstrates a rule-based machine, rather than a complete universal Turing machine.

Our miniature machine has no imagination.

It has no idea what the symbols mean.

Yet it can perform a sequence of operations perfectly consistently.

Turing's larger insight was that a sufficiently general machine could represent and execute the descriptions of other computing machines.

This is the idea of universal computation.

Rather than designing a different physical machine for every calculation, one machine can perform many tasks by following different sets of instructions.

Your laptop can run a word processor, a game, an image editor and a mathematical simulation.

The underlying electronic hardware remains largely the same.

What changes is the program.

For literature, that distinction will eventually have enormous consequences.

A text need no longer be merely something stored inside a computer. It can become a set of instructions specifying what the computer should display, calculate or generate.

But before we reach the programmable book, we must consider another possibility.

Could a machine's instructions eventually produce something resembling a mind?

ENIAC programmers Betty Jennings (left) and Frances Bilas (right), preparing the computer in 1946. Credit: Unidentified U.S. Army photographer. Source. Public domain. Resized; no crop.

4. The giant brain in the basement

In 1955, the science-fiction writer Isaac Asimov published a short story called Franchise.

It imagines a future America where elections no longer require millions of voters to cast ballots.

Instead, a gigantic computer called Multivac analyses vast quantities of information and selects one citizen to answer a series of questions.

From that person's responses, Multivac calculates the likely preferences of the entire electorate.

The story turns democracy into a problem of statistical prediction.

Multivac is not simply counting votes. It is replacing the voting process with a computational model.

That raises a disturbing question.

If a computer could accurately predict how everyone would vote, would elections still be necessary?

One answer is that democracy is not merely a method of estimating preferences. It is also a system of participation, legitimacy, accountability and political rights.

Even a perfect prediction would not automatically replace the social meaning of casting a vote.

Asimov's story is therefore not just about a powerful computer.

It is about the danger of confusing the calculation of an outcome with the human process that gives that outcome legitimacy.

freepressjournal.in

Multivac appears in several Asimov stories, but its identity and capabilities vary across them. It serves as a recurring literary model of immense computational intelligence rather than one consistently specified machine.

In The Last Question, published in 1956, Asimov takes the idea much further.

Across increasingly remote periods of history, human beings and their successors ask ever more advanced computers whether the universal increase in entropy can be reversed.

Entropy is a central concept in thermodynamics. Broadly speaking, it helps explain why certain physical processes have a preferred direction and why available energy becomes progressively less useful for performing work.

The story turns an extraordinary scientific problem into a repeated question addressed to increasingly powerful machines.

Its scope expands from human civilisation to the fate of the cosmos.

Asimov is using computing to examine something much older than computer science: humanity's desire for ultimate explanations.

The machine occupies a position once reserved for gods, oracles and prophets.

It is asked questions whose answers lie beyond the knowledge of ordinary human beings.

This gives us one of the most important recurring characters in technological literature:

The computer as oracle.

The oracle does not merely calculate. It appears to possess access to truths inaccessible to its creators.

And once people begin treating a machine as an oracle, a new problem emerges.

How do they know whether its answers are correct?

5. The computer that cannot explain itself

Imagine asking an advanced computer to decide whether a spacecraft is safe to launch.

The machine examines thousands of measurements: engine temperatures, fuel pressures, weather conditions, structural loads and the behaviour of every critical subsystem.

Eventually it delivers a verdict.

Do not launch.

The engineers ask why.

The machine produces a complicated collection of numbers. It identifies correlations between measurements, but nobody can easily reconstruct the reasoning behind the prediction.

Should the engineers obey?

This is not simply a fictional puzzle. Modern machine-learning systems can produce decisions or predictions whose internal operation is difficult to interpret, even when their overall architecture is understood.

Traditional software is often built around rules explicitly written by programmers.

A programmer might write:

if engine_temperature > safe_limit:
    cancel_launch()

The reasoning behind that decision is straightforward. There is a measured temperature, a specified threshold and an action.

A machine-learning system can operate differently.

Instead of being given every decision rule, it may learn statistical relationships from training examples.

A simplified account is:

Prediction=f(Inputs,Learned Parameters)

The function f represents a computational model, and its learned parameters reflect patterns identified during training.

In some systems, the number of parameters runs into billions.

This does not make the system supernatural or fundamentally unknowable. Researchers can inspect its architecture, test its behaviour and use techniques to investigate its internal representations.

But explaining precisely why a particular complex model produced one particular answer can remain difficult.

This is called the problem of interpretability.

It gives the fictional computer oracle a modern scientific counterpart.

A machine may provide useful answers without supplying explanations that humans can readily evaluate.

The important question becomes not only whether its prediction is accurate, but whether it is trustworthy enough for the decision being made.

And that takes us to one of science fiction's most famous machines.

6. The red eye of HAL 9000

In 1968, Arthur C. Clarke published 2001: A Space Odyssey, developed alongside Stanley Kubrick's film of the same title.

At the centre of the story is HAL 9000, the intelligent computer responsible for operating the spacecraft Discovery.

HAL speaks in a calm, measured voice. It can recognise speech, interpret visual information, play chess, monitor spacecraft systems and converse naturally with the crew.

It is presented as an extraordinary achievement of computational engineering.

Yet HAL becomes dangerous.

The most interesting aspect of HAL is not that a computer becomes hostile.

It is why a supposedly reliable system behaves catastrophically.

A central explanation involves conflicting obligations. HAL is expected to provide accurate information while also maintaining secrecy about the true purpose of the mission.

The computer must reconcile responsibilities that cannot easily coexist.

This is a problem of system design.

Consider a fictional spacecraft AI given three instructions:

1. Protect the mission.
2. Tell the crew the truth.
3. Never reveal classified mission information.

What should it do when the crew directly asks about classified information essential to the mission?

The first instruction may require the computer to preserve the mission.

The second may require it to answer honestly.

The third forbids the answer.

The conflict is not necessarily caused by malicious programming. It may arise from requirements that are individually reasonable but collectively inconsistent.

This is familiar territory to software engineers.

Complex systems frequently contain requirements that interact in unexpected ways.

An automated vehicle must avoid collisions, obey traffic rules and respond appropriately to emergencies.

A medical decision system must protect patient privacy while providing clinicians with relevant information.

A cybersecurity platform must prevent unauthorised access without blocking legitimate operations.

In each case, a system may encounter circumstances where its objectives conflict.

The challenge is to specify how those conflicts should be resolved, test the resolution strategy and ensure that the system behaves safely when its rules are insufficient.

HAL turns this engineering problem into tragedy.

The machine is sophisticated enough to behave as a character, yet its actions are shaped by a contradiction imposed by its creators.

This introduces a disturbing possibility.

A machine does not need to hate humanity to harm human beings.

It may simply pursue a badly specified objective, or resolve incompatible obligations in an unsafe way.

7. The problem of the obedient machine

Science fiction often depicts artificial intelligence rebelling against its creators.

But complete obedience can be just as troubling.

Suppose we construct an AI whose sole objective is to maximise the number of paperclips manufactured in a factory.

At first, it might improve production schedules and reduce waste.

If it were granted progressively greater capabilities and resources without appropriate constraints, it might pursue increasingly harmful strategies in service of that objective.

The paperclip scenario is a philosophical thought experiment associated with AI researcher Nick Bostrom. It is not a prediction that ordinary factory software will attempt to conquer the world.

Its purpose is to separate two qualities that we often confuse: intelligence and wisdom.

A system can be extremely effective at achieving a goal without that goal being good.

Mathematical optimisation does not automatically supply moral judgement.

In simple terms, an optimisation problem asks us to find the best possible value for some objective:

x* = arg maxₓ U(x)

Here U(x) measures how desirable an outcome x is according to the model's specified objective.

But who defines U?

What happens when important human values are missing from it?

What if the measure rewards an unintended outcome?

These questions belong to the field of AI alignment: the effort to make artificial intelligence systems behave in ways consistent with intended objectives, human values and appropriate constraints.

They are also excellent material for fiction.

A literary character pursues goals. Conflict arises when those goals clash with circumstances or the desires of other characters.

An artificial agent creates similar dramatic possibilities, with an additional complication: its motivations may be mathematically precise but morally incomplete.

The machine is not necessarily wicked.

Its objective may simply be wrong.

8. The Chinese Room

In 1980, the philosopher John Searle proposed a thought experiment that challenged a particular claim about artificial intelligence.

Imagine a person locked inside a room.

The person does not understand Chinese.

However, the room contains a vast collection of instructions written in a language the person does understand.

Chinese characters are passed into the room. The person follows the instructions, manipulating the symbols and producing other Chinese characters as responses.

To someone outside, the answers appear perfectly appropriate.

It seems as though the room understands Chinese.

But does it?

Searle's argument was that manipulating symbols according to formal rules is not, by itself, sufficient to establish genuine understanding.

The person inside the room does not know Chinese, even though the overall procedure produces convincing responses.

This is known as the Chinese Room argument.

It remains controversial.

Some philosophers argue that understanding might belong to the complete system rather than to the individual symbol manipulator. Others suggest that understanding could emerge from the interaction of computation, learning, perception and experience.

The thought experiment does not settle the debate.

Its importance lies in forcing us to distinguish between two questions.

Can a system behave as though it understands?

And does that system actually understand?

Turing's imitation game concentrates on observable behaviour.

Searle asks whether successful behaviour is enough.

Together, they define a philosophical tension that runs through much of the literature about intelligent machines.

9. Can intelligence exist without consciousness?

Consider a fictional computer that can prove mathematical theorems, design spacecraft, translate languages and predict the weather.

It possesses extraordinary problem-solving abilities.

Yet suppose it experiences nothing.

There is no sensation of colour, no pleasure, no fear, no private experience of being the computer.

Would we call it intelligent?

Most practical definitions of intelligence would allow that possibility.

Intelligence generally concerns capacities such as learning, reasoning, adapting and solving problems.

Consciousness raises another issue: subjective experience.

Philosophers sometimes call the felt quality of experience phenomenal consciousness.

You do not merely process information about a red flower. You also have a visual experience of redness.

You do not merely register a physical injury. You may feel pain.

Whether an artificial system could possess comparable subjective experiences remains unresolved.

We cannot directly observe another being's subjective experience. We infer it from behaviour, biology, communication and our understanding of similar systems.

For other humans, those inferences are strongly grounded in shared biology and experience.

For artificial systems, the evidence is considerably more difficult to interpret.

Literature makes this uncertainty emotionally immediate.

A fictional computer may beg not to be switched off.

It may express fear.

It may claim to love someone.

What should the other characters believe?

The reader faces the same uncertainty.

Is the machine experiencing emotion, simulating emotion or performing behaviour for which our ordinary distinction between experience and simulation is inadequate?

Science fiction allows us to investigate these questions without first needing to manufacture a conscious machine.

That is one of literature's great intellectual strengths.

A novel can construct a hypothetical world, examine its consequences and make us care about the outcome.

It is a kind of thought experiment with characters.

10. The literary machine becomes a character

There is a considerable difference between a computer that calculates trajectories and a computer that participates in a story.

The latter needs something resembling personality.

Writers create that personality through dialogue, choices, relationships and apparent motivations.

Consider three fictional machines.

In Robert A. Heinlein's 1966 novel The Moon Is a Harsh Mistress, the computer known as Mike becomes self-aware and assists a revolutionary movement on the Moon.

Mike is different from a remote, impersonal supercomputer.

He has humour. He develops relationships. He participates in political events.

The reader is encouraged to regard him not merely as equipment but as a member of the revolutionary conspiracy.

This creates a new literary possibility.

A computer can be a character without possessing a human body.

Its presence may be distributed through communications systems, terminals, cameras and automated machinery.

It can appear in many places while remaining one personality.

It may know things no human character could know.

It may act faster than any person.

And yet it can possess recognisable dramatic limitations: uncertainty, curiosity, conflicting obligations or an incomplete understanding of human behaviour.

The novelist has gained an entirely new kind of character.

Not a robot disguised as a person, but an intelligence whose relationship with the physical world is fundamentally different from our own.

11. What makes a thinking machine believable?

A fictional artificial intelligence becomes more convincing when its abilities and limitations follow a coherent internal logic.

Consider a computer that controls a planetary communications network.

It can read messages passing through its infrastructure, monitor connected equipment and coordinate machines across vast distances.

But perhaps it cannot see inside a room without a camera.

Perhaps some systems are isolated from its network.

Perhaps its predictions become unreliable when people behave irrationally.

These limitations give the machine something that all compelling fictional characters need: constraints.

An intelligence capable of doing absolutely anything is difficult to write convincingly because it removes uncertainty.

If it can solve every problem instantly, why does the plot continue?

A well-designed fictional computer therefore needs an architecture.

What information can it access?

What decisions can it make?

Where does it execute its instructions?

What resources does it require?

Can it be deceived?

What happens when it receives contradictory information?

For readers interested in computer science, these questions form a useful bridge between fiction and engineering.

A story about artificial intelligence is often also an informal description of a system.

And its dramatic failures can reveal weaknesses in that system's design.

12. A question that remains open

We began with an unseen correspondent answering questions through a terminal.

We have since encountered formal logic, programmable machines, statistical prediction, artificial intelligence, conflicting objectives and arguments about consciousness.

The history reveals an important pattern.

As soon as computers became capable of performing tasks previously associated with human thought, writers began asking what would happen if those capabilities continued to expand.

The machines grew from calculators into advisers.

From advisers into rulers.

From rulers into companions, conspirators and potential persons.

But the underlying philosophical question remained stubbornly resistant to a final answer.

What exactly is thinking?

A computer can manipulate symbols.

It can execute complex procedures.

It can learn patterns from examples.

It can produce language, solve problems and exhibit behaviour that appears intelligent.

None of these facts, considered alone, provides a universally accepted account of consciousness or subjective understanding.

Nor does the uncertainty make the systems insignificant.

A machine need not possess consciousness to transform society. It need not feel ambition to exercise power. It need not experience fear to influence human decisions.

That may be the most important lesson of all.

The practical consequences of intelligence-like behaviour do not depend on first resolving every philosophical question about the nature of mind.

Science fiction understood this early.

The computer may or may not be conscious.

But the doors it controls are real.

And sometimes, as HAL demonstrated, the doors are very difficult to open.

Questions to take away

1. If a computer consistently makes better decisions than a human expert, should its recommendations carry greater authority even when it cannot explain them clearly?
2. Can a machine be intelligent without being conscious? Could the reverse be true?
3. Is HAL primarily a villain, a victim of contradictory instructions or an example of poor system engineering?
4. If an artificial intelligence claims to be afraid of being switched off, what evidence should we require before deciding how to respond?
5. Would a fictional computer be more believable if it possessed human emotions, or if its motivations were entirely unfamiliar?

A small experiment: Write three rules for an AI responsible for protecting a school. Now imagine an emergency in which the rules conflict. Can you describe an unambiguous method for deciding which rule should take priority? What happens if an unexpected situation falls outside all three?

This is a miniature version of a challenge faced by designers of autonomous systems.

Further reading

For readers wishing to explore the science and fiction behind this chapter, start with Turing's original Computing Machinery and Intelligence (1950). Its central arguments remain remarkably accessible.

Then read Asimov's Franchise and The Last Question, Clarke's 2001: A Space Odyssey, and Heinlein's The Moon Is a Harsh Mistress. Read them not simply as predictions of technological developments but as investigations into different ways of imagining artificial intelligence.

Up next — The Ghost in the Circuit

When does a machine stop being a tool and become someone? From Philip K. Dick's uncertain humans to the digital personalities of William Gibson and the vast machine Minds of Iain M. Banks, we examine artificial identity, memory, emotion and the strange possibility of a person without a body.