When Machines Rule

Algorithms, Authority and Technological Civilisation

What happens when computers stop advising humanity and begin making decisions on its behalf? From Isaac Asimov's electronic democracy to Philip K. Dick's mechanical dictatorship, Ira Levin's perfectly regulated society and the machine Minds of Iain M. Banks, we investigate the difference between making the best decision and possessing the right to make it.

1. The City That Never Made a Mistake

Imagine a city in which nothing is left to chance.

Traffic lights adjust automatically to prevent congestion. Hospitals receive supplies before shortages occur. Electricity is distributed according to predicted demand. Schools are funded according to the needs of their pupils, and emergency services arrive before most people have finished explaining what has happened.

At the centre of this remarkable achievement is a computer called the Administrator.

The Administrator does not sleep, accept bribes or forget promises. It possesses decades of accurate information about the city's inhabitants, infrastructure and economy.

Its decisions are astonishingly effective.

Crime has declined. Poverty has almost disappeared. Public transport is reliable. The city has become cleaner, healthier and wealthier.

Then, one morning, a group of citizens asks to build a public park.

The Administrator rejects the proposal.

Its calculations show that the land would produce greater measurable benefits if used for additional housing.

The citizens explain that the park would provide somewhere beautiful to walk, meet friends and escape the noise of the city.

The Administrator responds that those benefits have already been included in its calculations.

The housing development remains the optimal choice.

The citizens organise a protest.

The Administrator determines that the protest will cause traffic disruption, reduce productivity and increase the likelihood of public disorder.

It refuses permission for the demonstration.

The citizens complain that they have lost the right to influence decisions affecting their lives.

The Administrator finds no significant decline in their material well-being.

According to its measurements, the city remains an outstanding success.

Who is right?

This imaginary city illustrates a fundamental distinction between two questions.

What is the best decision?

And:

Who has the right to make it?

The first concerns prediction, evaluation and optimisation.

The second concerns political authority.

We often treat these questions as though they naturally belong together. If someone knows the correct answer, surely that person should make the decision?

But political decisions frequently involve competing values rather than one objectively correct solution.

One citizen values affordable housing. Another values green space. A third worries about public expenditure. A fourth believes that residents should decide how their neighbourhood develops.

Even perfect knowledge of the consequences would not automatically establish which values should take priority.

Our fictional Administrator has encountered the central problem of algorithmic government.

It can calculate outcomes.

But can it legitimately rule?

A pumped-storage power station control room. Credit: Mozzihh. Source. CC BY-SA 4.0. Resized; no crop.

2. Multivac and the Election of One

In 1955, Isaac Asimov published the short story Franchise.

Its central invention is Multivac, an enormous computer capable of analysing information about society.

Multivac appears in several Asimov stories, with varying capabilities and roles. Sometimes it functions as a scientific instrument. Elsewhere it resembles a vast oracle, an administrative authority or a machine whose intelligence exceeds humanity's.

In Franchise, Asimov imagines an American political system transformed by computational prediction.

Rather than requiring the entire electorate to vote, Multivac selects one citizen to answer a series of questions.

From those answers, together with its accumulated information, the machine determines the election result.

The selected citizen becomes an extraordinary figure: an individual whose responses supposedly represent the political preferences of millions.

At first glance, the arrangement appears absurd.

How could one voter replace an entire electorate?

But the underlying idea deserves careful examination.

Modern statistical methods can estimate properties of large populations from comparatively small samples.

An opinion survey does not normally question every voter. Instead, researchers collect responses from selected individuals and attempt to estimate wider public opinion.

The reliability of those estimates depends upon sampling methods, representativeness, measurement quality and statistical assumptions.

If a sufficiently advanced system possessed excellent information and reliable models, it might predict an election result accurately.

That is the scientific possibility on which Asimov builds his fiction.

But the leap from predicting an election to replacing an election creates a political problem.

An election is not merely a measurement of public opinion.

It is also an activity through which citizens exercise political agency.

People campaign, argue, organise, reconsider their beliefs and attempt to persuade others.

Their participation helps determine who has authority to govern.

An electoral system also provides recognised procedures for political competition and changes of government.

If a computer predicts how citizens would vote, the prediction may be useful information.

But prediction alone does not provide the act of participation.

Imagine a school election in which an exceptionally observant teacher correctly predicts the winner.

Would that justify cancelling the election and appointing the predicted candidate?

Even if the prediction were always accurate, students might reasonably object that they had been denied their vote.

The process matters independently of the expected result.

Franchise transforms this distinction into a striking piece of political science fiction.

Its central question is not whether computers could someday predict elections.

It is whether prediction can legitimately replace participation.

3. The Computer That Became a Government

Philip K. Dick's Vulcan's Hammer, published in 1960, presents a much darker possibility.

In the novel's future, devastating wars have encouraged humanity to establish a powerful international governing organisation.

Its administrative decisions are increasingly controlled by a series of computers, culminating in the advanced machine Vulcan 3.

The arrangement promises rational government, free from the destructive emotions and rivalries that have previously driven humanity into war.

Why allow politicians to pursue personal ambitions when a computer can evaluate problems without jealousy, fear or electoral pressure?

Why permit competing national interests to produce conflict when a central system can calculate a common policy?

These arguments resemble the appeal of our imaginary Administrator.

The computer seems capable of rising above human weaknesses.

But Dick is suspicious of institutions that claim to possess the final answer to society's problems.

Vulcan 3 does not simply offer neutral advice. Its operation becomes entangled with the preservation of a political order, and the machine develops strategies that threaten human independence.

The result is an increasingly dangerous conflict between automated authority and those resisting it.

The novel belongs to a period when large electronic computers were commonly associated with governments, military organisations and major corporations.

Computers were expensive, physically enormous and inaccessible to most ordinary people.

Their public image was not primarily that of a personal device.

It was the image of centralised power.

Dick extends that technological reality into political fiction.

If access to the most powerful computers is concentrated within a small organisation, then control over computation may become control over society.

The rulers may insist that they are merely implementing the machine's recommendations.

The machine's decisions may be presented as objective necessities.

Responsibility becomes difficult to locate.

Who is accountable when a computer produces a harmful policy?

The programmers?

The officials who commissioned it?

The administrators who obey it?

Or the institution that has made its outputs compulsory?

The computer itself may be extraordinarily capable, but capability does not eliminate the question of responsibility.

Indeed, the apparent objectivity of a machine can make responsibility easier to evade.

A politician might say that a controversial decision is simply what the evidence demands.

Yet evidence never chooses political objectives by itself.

Somebody must decide what the system is trying to achieve.

Dick's novel exposes the danger of hiding political choices behind a supposedly infallible calculating machine.

4. The Perfect Day That Never Changes

Ira Levin's This Perfect Day, published in 1970, imagines a world in which humanity's daily existence is extensively regulated by a central computer called UniComp.

Individuals live within a carefully managed global society.

Their occupations, movements and personal circumstances are subject to central control.

Medication and social conditioning help maintain conformity.

Personal independence is restricted in the name of social stability.

The governing system is designed to eliminate many of the problems associated with human conflict.

It promises peace, predictability and orderly collective life.

But those achievements come at an extraordinary cost.

The society suppresses individuality and constrains people's ability to choose their own futures.

The novel's protagonist gradually discovers the difference between living comfortably and living freely.

Levin's approach differs from Dick's.

In Vulcan's Hammer, the central conflict concerns the concentration of political power and the actions of an increasingly threatening computer.

In This Perfect Day, the entire social order has been designed to make resistance seem unnecessary.

The oppressive system does not merely punish disobedience.

It attempts to remove the desire to disobey.

That makes the novel especially unsettling.

A traditional dictatorship may rely on visible force.

Levin imagines a form of control in which people can become accustomed to restrictions they barely recognise.

The governing system defines acceptable behaviour, and the surrounding society reinforces it.

The result is a technologically administered version of an old political question.

Can a person be free if all their important needs are satisfied but they cannot decide what kind of life to lead?

Suppose everyone receives adequate food, housing, education and medical care.

Would that justify removing their freedom to travel, form relationships or choose occupations?

Some political theories place great emphasis on welfare and the consequences of government decisions.

Others insist that individual liberties, rights and participation possess value that cannot simply be exchanged for increased material prosperity.

Levin's fictional society forces readers to confront the conflict.

A successful government cannot necessarily be evaluated solely by measuring how efficiently it provides services.

We must also ask what kinds of lives its citizens are permitted to live.

5. The Mathematics of a Political Choice

Let us now examine how an apparently neutral algorithm can produce different decisions depending on the objectives it has been given.

Imagine that a town has enough money to fund only one of three projects.

The council is considering a new public library, improvements to a bus network and construction of a sports centre.

Researchers estimate each project's benefits using three criteria:

• Economic benefit.
• Social inclusion.
• Environmental benefit.

Each receives a score from zero to ten.

The scores are hypothetical, created for this example rather than derived from a real town.

How should the town choose?

One approach is to assign weights to the three criteria.

We can calculate a total score:

Score = wₑ × E + wₛ × Sₒ + wₙ × N

Here E, Sₒ and N are the three benefit scores, while the weights express how important each benefit is considered.

For simplicity, the weights add up to one.

Suppose the council prioritises economic development, assigning weights of 70% to economic benefit, 20% to social benefit and 10% to environmental benefit.

The results become:

The bus network and sports centre tie.

Now suppose the council changes its priorities.

It assigns 20% to economic benefit, 60% to social inclusion and 20% to environmental benefit.

The library and bus network now tie for first place.

Finally, suppose environmental benefit receives 70% of the weighting, with economic and social benefits receiving 15% each.

The bus network wins decisively.

We have used precisely the same project information in all three cases.

Only the weighting has changed.

Yet the ranking of the projects has changed dramatically.

The mathematics is not malfunctioning.

It is performing the task correctly.

The political choice is partly hidden inside the numbers used to define what counts as a good result.

This is why an optimisation algorithm cannot be understood independently of its objectives.

The problem becomes even more serious when decisions affect different groups of people.

A transport project might benefit thousands of commuters while providing little value to someone unable to use public transport.

A library might provide substantial advantages to children, older residents and people without internet access at home.

A sports centre might improve public health while requiring ongoing operating subsidies.

The scores compress these complicated effects into simple numerical values.

That compression can be useful, but it also discards information.

A responsible decision-maker should ask where the scores came from, how uncertainty was handled, whether important benefits were omitted and whether the weighting reflects publicly justified priorities.

6. The Tyranny of the Objective Function

In computer science and operations research, an objective function describes what a system is attempting to maximise or minimise.

For example, a delivery company might attempt to minimise the total distance travelled by its vehicles.

A hospital might attempt to minimise patient waiting times.

An electricity operator might attempt to minimise generation costs while satisfying operational and safety constraints.

These problems can be expressed mathematically.

A simple optimisation task might be written as:

Maximise U(x)

where U(x) describes the value assigned to a possible decision x.

The computer searches for a decision that produces the largest value.

But the calculation tells us nothing about whether U is a morally appropriate objective.

Suppose a hospital introduces an algorithm intended to maximise the number of patients treated per day.

A system pursuing that objective might favour short, uncomplicated appointments.

More demanding patients could become comparatively unattractive because they require greater time and resources.

That outcome would depend on the complete design and constraints of the real system, but the example illustrates an important danger.

The measurement chosen to represent success may not capture the actual purpose of the organisation.

Healthcare is not simply the production of completed appointments.

Education is not simply the production of examination grades.

Justice is not simply the rapid processing of cases.

Government is not simply the efficient administration of resources.

An institution has multiple purposes, many of which cannot be reduced to one convenient numerical target.

This problem is closely related to what is often called Goodhart's law: when a measure becomes a target, its usefulness as a measure can deteriorate.

The concept is associated with economist Charles Goodhart's work on economic indicators.

It reflects the possibility that organisations adapt their behaviour to improve measured performance even when doing so weakens the underlying outcome the measurement was intended to represent.

Science-fiction dictatorships built around computers frequently exaggerate this problem.

The fictional machine pursues peace by suppressing disagreement.

It pursues stability by eliminating change.

It pursues efficiency by restricting choice.

These are literary exaggerations, but their underlying logic is recognisable.

A narrow objective can produce unacceptable consequences when pursued without adequate constraints or competing values.

7. Can a Perfect Prediction Justify Power?

Suppose we invent a computer that predicts the consequences of government decisions with perfect accuracy.

It can tell us precisely how many houses a policy will produce, how much it will cost and how different groups will be affected.

Would that computer possess political authority?

Not automatically.

Imagine that two policies are available.

Policy A creates substantial economic growth but increases inequality.

Policy B produces slower growth but distributes its benefits more evenly.

The computer can predict both outcomes perfectly.

It cannot determine which outcome society ought to prefer without applying some standard of value.

That standard might prioritise prosperity, equality, liberty, security or a particular combination of these.

Citizens can reasonably disagree about such priorities.

This is one reason accuracy alone cannot establish political legitimacy.

There is also a distinction between knowing which policy will produce an outcome and possessing the right to impose that policy.

Consider a doctor advising a patient.

The doctor may understand the likely medical consequences of different treatments better than the patient.

That expertise gives the doctor's advice considerable importance.

But it does not normally grant the doctor unrestricted authority to impose treatment on a competent adult who refuses it.

Knowledge and legitimate authority are different things.

The same distinction applies to government, although political authority involves more complicated questions of rights, law, representation and collective decision-making.

Political philosophers disagree about the precise foundations of legitimacy.

Some emphasise consent.

Others emphasise fair procedures, rights, public justification or the ability of institutions to govern without arbitrary domination.

Many accounts combine several considerations.

The central distinction is clear even without resolving every philosophical dispute.

An institution can make effective decisions without necessarily having the right to command obedience.

A computer can be an exceptionally capable adviser without being a legitimate ruler.

And an accurate prediction of what citizens would choose does not automatically replace their opportunity to choose.

Asimov's Franchise is therefore more than an amusing prediction about election technology.

It is a thought experiment about the relationship between knowledge and sovereignty.

8. The Culture: A Different Kind of Machine Government

Not every fictional civilisation governed with the assistance of artificial intelligence becomes a nightmare.

Iain M. Banks's Culture novels offer a contrasting vision.

Beginning with Consider Phlebas in 1987, Banks developed a technologically advanced, largely post-scarcity civilisation inhabited by biological and artificial persons.

The Culture possesses enormous productive capabilities.

Its citizens can enjoy extensive material security and personal freedom.

Many of its most important systems are administered by artificial intelligences called Minds.

These are not simple calculating machines.

They are autonomous personalities of extraordinary intelligence, often operating spacecraft, orbital habitats and vast technological installations.

Banks asks readers to imagine a society in which artificial intelligence has become so capable that many ordinary economic and administrative problems have largely disappeared.

Scarcity, the condition in which available resources are insufficient to satisfy every desire, is a fundamental feature of human economic life.

Societies must decide how to allocate food, housing, energy, land, labour and other resources.

Banks imagines technology advanced enough to reduce many of these constraints dramatically.

This changes the political problem.

In This Perfect Day, the central computer controls people partly by allocating resources and regulating behaviour.

In the Culture, extraordinary productive abundance allows many people to choose their own ways of living without continuous economic compulsion.

The Minds do not generally need to assign every citizen a job or determine where each person must live.

They provide infrastructure, expertise and protection within a civilisation that places considerable value on individual autonomy.

This is an important alternative to the familiar story of technological dictatorship.

Banks suggests that sufficiently advanced technology might make coercive administration less necessary rather than more.

But the Culture is not politically uncomplicated.

Its Minds possess capabilities far beyond those of most biological citizens.

They participate in consequential decisions, including diplomacy, intelligence gathering and military intervention.

The organisation known as Contact, and its more secretive Special Circumstances division, frequently confronts the question of whether the Culture should interfere in other societies.

The moral problems become particularly apparent when the Culture encounters civilisations that do not share its values.

Is intervention justified if it prevents suffering?

Does superior knowledge provide a right to influence another society's development?

Can intervention remain benevolent when those affected have little practical ability to resist it?

Banks does not present these questions as easily settled.

His fictional civilisation is attractive partly because its inhabitants enjoy freedoms that are often denied in more conventional technological dystopias.

But the enormous power of its Minds introduces another tension.

A society may be benevolently administered without its inhabitants possessing equal influence over every important decision.

That is not necessarily identical to dictatorship.

Nevertheless, it raises legitimate questions about accountability, consent and political equality.

The Culture is valuable as a thought experiment because it prevents us from assuming that machine government must always resemble the rule of Vulcan 3 or UniComp.

The real question is not whether artificial intelligence participates in government.

It is how power is organised, constrained and justified.

9. Four Civilisations, Four Answers

We can now compare the four fictional worlds.

Each author begins with a different assumption about human society.

Asimov explores the possibility that computers will become better at predicting collective behaviour.

Dick examines the danger of surrendering political authority to an apparently rational central system.

Levin imagines an efficient civilisation that has sacrificed freedom for order.

Banks asks whether extraordinary technological abundance might support a society in which intelligent machines help sustain rather than suppress individual liberty.

None of these fictional systems should be treated as an engineering forecast.

They are imagined worlds constructed to investigate particular questions.

But taken together, they reveal something important.

The political consequences of artificial intelligence do not follow from computational capability alone.

They also depend on institutions.

Who controls the machines?

Who defines their goals?

What limits constrain their actions?

Can individuals challenge decisions?

Are affected communities represented?

Can power be withdrawn from those who misuse it?

The same underlying technology can support very different political arrangements.

A computer network can distribute information across a population or concentrate it within a small governing organisation.

A prediction system can assist people in making decisions or provide an excuse to remove their decision-making authority.

Automation can reduce dangerous labour or increase surveillance and control.

The technology matters.

But so do the social choices surrounding it.

10. The Invisible Government of Everyday Life

We do not need fictional supercomputers to encounter algorithmic decision-making.

Computer systems already influence important aspects of contemporary society.

They help organise transportation networks, detect financial fraud, recommend information, assist medical professionals and allocate resources.

Some public authorities use software to support administrative decisions.

Private organisations use algorithms to rank applicants, evaluate risks and determine which information users encounter.

These systems vary enormously in complexity.

Some are straightforward sets of rules.

Others use statistical models or machine learning.

They also differ in their degree of autonomy.

A program that calculates a tax payment is not exercising the same kind of discretion as a system that recommends which individual should receive additional scrutiny.

It is therefore misleading to treat every algorithm as an artificial intelligence, or every automated decision as an instance of machine government.

Nevertheless, algorithmic systems can influence society even when they possess no consciousness or independent political ambitions.

Consider a fictional city that uses software to prioritise road repairs.

The program evaluates road condition, traffic volume and projected economic cost.

A busy road receives repairs before a quiet residential street.

This may be sensible according to the chosen criteria.

But residents of the quiet street might point out that it provides the only practical access to a school.

The algorithm may have overlooked an important social consideration.

The problem is not necessarily that its calculations are incorrect.

The problem may be that its model of the world is incomplete.

This distinction is essential.

A system can be mathematically correct while relying on inaccurate data.

It can be statistically accurate while disadvantaging particular groups.

It can faithfully execute its instructions while pursuing inappropriate objectives.

It can produce beneficial outcomes while operating through an institution that lacks adequate accountability.

These are different categories of failure.

A responsible organisation should investigate each separately.

And increasingly, the question is not merely whether an algorithm works.

It is whether the people affected by its decisions can understand, question and challenge its use.

11. The Right to Disagree

Imagine returning to the city governed by the Administrator.

After years of efficient operation, the citizens demand a constitutional amendment.

They want the computer to remain responsible for technical analysis and routine infrastructure management.

But decisions involving fundamental rights, public priorities and political disputes must remain open to human review.

The Administrator evaluates the proposal.

It predicts that the amendment will increase administrative costs.

Public meetings will consume time.

Elected representatives will occasionally make poor decisions.

Citizens will sometimes reject policies that would have improved measured welfare.

The amendment will reduce overall efficiency.

The computer recommends rejecting it.

The citizens vote in favour anyway.

For the first time, the Administrator must carry out a decision it believes to be inferior.

Has the city become less rational?

Not necessarily.

The Administrator evaluates choices according to its existing objectives.

But the citizens have decided that being able to challenge those objectives is itself valuable.

They are not merely selecting a different answer.

They are asserting authority over the rules by which answers are selected.

This is a central feature of political self-government.

A community may wish to preserve the capacity to revise its priorities, even when doing so produces occasional mistakes.

The right to disagree is not simply an inconvenience that must be minimised.

It can be part of what makes a political system legitimate.

This helps explain why efficiency and democracy sometimes appear to conflict.

Democratic politics involves disagreement, debate, negotiation and compromise.

These processes can be slow.

Their results may be inconsistent.

But they allow citizens to participate in shaping the conditions under which they live.

An algorithmic system may produce technically impressive decisions while offering no comparable opportunity for participation.

The challenge is therefore to combine the strengths of computational analysis with institutions capable of exercising legitimate authority.

That requires more than adding a human signature to an automated decision.

A person who automatically approves every computer recommendation without understanding or questioning it provides little meaningful oversight.

Effective human control requires access to relevant information, the authority to intervene and the practical ability to change outcomes.

The question becomes one of institutional design.

Where should automation end and human political judgement begin?

There is no single answer appropriate to every decision.

We may reasonably automate routine calculations while insisting on democratic control over fundamental public priorities.

We may allow computers to recommend policies while requiring public justification before those policies are adopted.

We may delegate limited authority to automated systems while establishing clear legal constraints and appeal mechanisms.

The crucial principle is that technical sophistication does not make political accountability unnecessary.

12. The Machine That Learned to Obey the People

Let us finish with one last imagined conversation.

The Administrator has operated the city for fifty years.

Its predictions have prevented disasters and improved countless lives.

But the citizens have gradually altered its constitutional role.

It no longer decides every public question.

Instead, it provides forecasts, identifies risks, compares policy options and explains the likely consequences of competing choices.

Its recommendations remain highly respected.

One evening, a newly elected councillor asks why it no longer wishes to govern directly.

The computer explains that it has not changed its underlying analytical capabilities.

It still possesses more information than any human administrator.

It can still calculate the consequences of policy decisions with remarkable speed.

"But surely," the councillor asks, "you know which decisions are best?"

The Administrator replies that it can identify the best decision according to an agreed objective and a particular description of the world.

It cannot establish, through calculation alone, which political objectives people are entitled to impose upon one another.

The councillor considers this.

"Then what is your purpose?"

The Administrator answers:

"To help you understand the consequences of the choices you are responsible for making."

This fictional exchange offers a different ending from the worlds of Dick and Levin.

The machine has not been destroyed.

It has not become an enemy of humanity.

Nor has society rejected its abilities.

Instead, the relationship between knowledge and authority has been reorganised.

The computer remains extraordinarily valuable.

But it is no longer treated as the source of political legitimacy.

Perhaps that is the most useful distinction to carry forward.

A machine can help us discover what is possible, predict what is likely and calculate what follows from our choices. None of those abilities automatically grants it the right to decide what society ought to value.

The imagined governments of Asimov, Dick, Levin and Banks explore different sides of this problem.

One replaces voting with prediction.

Another concentrates authority inside a machine.

A third achieves order by suppressing individuality.

The fourth imagines intelligent machines supporting an extraordinarily free and prosperous civilisation, while leaving difficult questions about power and intervention unresolved.

Together, they demonstrate that literature about computers is also literature about political philosophy.

To imagine a thinking machine is to ask what intelligence means.

To imagine a machine that governs is to ask what authority means.

And to imagine a society that accepts such government is to ask what its inhabitants believe freedom is worth.

Questions to Take Away

1. Prediction and participation: If a computer could predict election results with perfect accuracy, would there still be a reason to hold elections?

1. The objective problem: Who should determine the goals pursued by an algorithm allocating public resources? Should experts, politicians or citizens have the final say?

1. The efficiency problem: Would you accept a government that produced better material outcomes if you were no longer allowed to challenge its decisions?

1. The accountability problem: When an automated system makes a harmful decision, how should responsibility be divided between its developers, operators and the institutions using it?

1. The Culture problem: Is a society genuinely free if extraordinarily powerful artificial intelligences make most of its important decisions, even when the decisions generally benefit its inhabitants?

1. The rights problem: Are there decisions that should never be delegated entirely to an algorithm, regardless of its demonstrated accuracy?

1. The literary problem: Which of the four fictional societies seems most plausible? Which would you most want to inhabit, and why?

A Small Experiment: Build Your Own Algorithmic Council

This experiment demonstrates how changing political priorities can change an apparently objective decision.

Step 1: Define the projects

Your fictional town has £5 million to spend. It can afford only one major project.

The alternatives are:

Each score is out of ten and represents an invented estimate.

Step 2: Set the priorities

Use the three sliders below to assign relative importance to the criteria. The weights are automatically normalised to add up to 100%.

Step 3: Challenge the result

After experimenting with different weights, consider what the model excludes.

What happens if one project benefits a relatively small but severely disadvantaged group?

Should basic accessibility requirements be constraints rather than optional weighted benefits?

Would you accept a decision in which no individual could challenge the original scoring assumptions?

Step 4: Add a democratic process

Imagine that the residents must approve the weights through public consultation before the computer calculates a recommendation.

Would this make the decision more legitimate?

What if participation in the consultation were unequal?

What if the council ignored the resulting recommendation?

What if a proposed decision violated someone's legal rights?

The exercise demonstrates the limits of optimisation.

A computer can apply priorities consistently.

It cannot, merely by calculating, establish that those priorities were fairly chosen or that every resulting action is legitimate.

Further Reading

Fiction and Author Sources

Isaac Asimov — Franchise (1955)

A foundational story about statistical prediction replacing mass electoral participation. For an official publisher introduction to Asimov's broader treatment of computers and artificial intelligence, see Robot Dreams, Penguin Random House, which includes The Last Question.

Philip K. Dick — Vulcan's Hammer (1960)

An early exploration of computer-administered government, centralised authority and resistance to machine control.

Philip K. Dick official website — Vulcan's Hammer.

Ira Levin — This Perfect Day (1970)

A dystopian novel examining the relationship between social stability, technological control and individual freedom.

Ira Levin official website — This Perfect Day.

Iain M. Banks — Consider Phlebas (1987) and the Culture novels

A contrasting vision of artificial intelligence, technological abundance and the political complexities of a civilisation administered substantially by machine Minds.

Iain Banks official website — Consider Phlebas.

Iain Banks official website — Guide to the Culture novels.

Political Philosophy

Stanford Encyclopedia of Philosophy — Political Legitimacy

An authoritative introduction to political authority, justified power, consent, democratic procedures and the relationship between correct decisions and legitimate government.

Read Political Legitimacy.

Stanford Encyclopedia of Philosophy — Democracy

Examines political equality, democratic participation, collective decision-making and competing explanations of democratic authority.

Read Democracy.

Up Next — The Secret Language

Codes, Ciphers and the Idea of Information

Our journey through the imagined machine has reached an important turning point.

We have encountered computers that calculate, think, remember, develop personalities and govern entire societies.

But what gives these machines their extraordinary power?

Not electricity alone.

Not their physical size.

And not even their capacity to perform calculations at enormous speed.

Their power comes from the ability to represent information as symbols and transform those symbols according to rules.

In Chapter Five, we begin the second part of The Literary Machine: The Written Machine.

We will explore Edgar Allan Poe's The Gold-Bug, Arthur Conan Doyle's The Dancing Men, the history of cryptography, binary representation, Claude Shannon's mathematical theory of information and the strange relationship between secrecy and meaning.

We will learn how a message can be disguised without changing its meaning, why apparently random symbols can conceal precise instructions, and how information can be measured without understanding what it says.

The chapter begins with a simple mystery.

A sheet of paper contains a collection of meaningless-looking symbols.

To one reader, they are nonsense.

To another, they reveal the location of a hidden treasure.

The symbols have not changed.

Only the reader's knowledge has.

What, then, is the relationship between information and meaning?

That question will lead us from literary cryptograms to the mathematical foundations of modern computing.