What should AI learn to optimise?

I keep noticing the same basic idea appearing in completely different subjects. Psychology talks about reinforcement. Behavioural economics talks about decisions under gains, losses and uncertainty. Game theory talks about payoffs and strategic behaviour. Reinforcement learning talks about rewards and policies. Evolutionary algorithms talk about fitness and selection. Multi-objective optimisation gives us Pareto frontiers.

Even biology gives us systems such as slime mould, where useful network structures can emerge without a human engineer specifying every connection. These subjects are not the same, and I do not want to pretend they are. But they give us different ways of looking at a similar question:

What happens when an environment makes some behaviours or solutions more successful than others?

That question is becoming increasingly relevant to the Agape Duck drone simulator, where I am experimenting with how humans and algorithms can learn, make decisions and improve inside the same simulated environment. The difficult question is not whether we can make an AI optimise. It is what we should ask it to optimise in the first place.

A note on how I am using the research

This article deliberately connects fields that use different methods, definitions and standards of evidence. I am not arguing that operant conditioning, reinforcement learning, slime mould adaptation, evolutionary algorithms, game theory and Pareto optimisation are equivalent mechanisms.

Sometimes I describe published evidence. Sometimes I use it as an analogy. Sometimes I give my interpretation. Sometimes it produces a hypothesis I would like to test in the simulator. The process is:

Evidence → interpretation → hypothesis → simulation → measurement → revision

The objective is not to claim one grand theory connecting psychology, biology, economics and artificial intelligence. It is to ask whether ideas from these fields can help me design better experiments.

Start with a very simple system

Imagine a simulated drone has three possible strategies:

1. Fly as fast as possible.
2. Fly as safely as possible.
3. Balance speed, safety and battery use.

The drone has not become intelligent because I gave it numbers. But I have created an incentive structure. I have told the system what I currently consider good. If I reward speed, it will search for ways to become faster. If I reward collision avoidance, safer routes become more attractive.

Somebody still had to choose the objectives, decide what counted as success, and decide which trade-offs were acceptable. In this case, that somebody is me. The behaviour of an optimisation system can therefore reveal the assumptions of its designer.

Pareto frontiers: sometimes there is no single best answer

Adding every objective into one score can hide important trade-offs. A faster route might use more battery. A safer route might take longer. An efficient route might fly closer to obstacles. Imagine the simulator finds these routes:

Route A is fastest, B conserves most battery, and C has the greatest safety margin. There may be no single correct answer. A feasible solution is Pareto optimal when no other feasible solution can improve one objective without making another worse. NSGA-II is one influential method for finding diverse sets of non-dominated solutions in multi-objective problems (Deb et al., 2002).

A Pareto frontier can expose trade-offs among the objectives we chose to measure. It cannot decide our values for us.

In practice an evolutionary search finds an approximation, not necessarily the true Pareto frontier. Its quality depends on the representation, objectives, constraints, population, variation, search time and experiment design. Responsible wording is: “Here is the current approximation found by our search”, not “AI found the objectively perfect answers”.

Skinner: behaviour is shaped by consequences

B. F. Skinner's work on operant conditioning investigated how behaviour changes according to its consequences. Ferster and Skinner systematically explored how reinforcement schedules could produce different patterns of responding (1957). Later work continued examining how schedules could influence response rate, persistence and preference (Nevin et al., 2001).

Imagine a person learning the simulator. They navigate a first obstacle and receive immediate positive feedback. As ability improves, the challenge develops: avoid the wall; then avoid it while conserving battery; then avoid obstacles, conserve battery and photograph three inspection targets.

The literature does not give me a ready-made formula for an educational drone simulator. The Nevin study cited here used pigeons. I am not claiming that a pigeon responding to a schedule is equivalent to a person learning robotics. The narrower lesson is that feedback structures can produce different patterns of behaviour. How that translates to an educational interface should be tested rather than assumed.

Engagement without manipulation

I do not want to use psychology to make learning addictive. I want to make learning enjoyable enough that people voluntarily keep learning. The objective is not to make somebody unable to stop. It is to remove enough unnecessary frustration and boredom that curiosity gets a chance to take over.

Hypothesis to test: Can progressive positive feedback and appropriately increasing challenge improve voluntary persistence without producing manipulative engagement patterns?

Reinforcement learning: the computational bridge

Reinforcement learning is not Skinner's psychology converted into code, but the comparison is useful because both make consequences important. An agent observes a state, takes an action, experiences a changed environment and receives a reward signal. Through experience, it can try to learn a policy that maximises expected cumulative reward over time (Sutton and Barto, 2018).

State → action → consequence → learning → repeat

A simulated drone might receive +10 for an inspection target, +3 for conserving battery, -5 for unnecessary route length and -100 for a collision. But it does not understand my intention. If I reward photographs while barely penalising collisions, reckless policies can score well. If I reward battery preservation too strongly, doing almost nothing can look successful.

The computer has not necessarily failed. My reward design may have failed.

Slime mould and Tokyo's railway system

In 2010, Atsushi Tero and colleagues placed food sources to correspond to locations around Tokyo. Physarum polycephalum developed a connecting network. Under the experimental model, it produced characteristics comparable with Tokyo's rail system across measures that included transport efficiency, fault tolerance and connection cost (Tero et al., 2010).

This does not mean slime mould redesigned Tokyo better than human engineers. It did not model every political, geological, passenger, land-use, economic or engineering constraint. The useful finding is narrower: in a simplified environment, a decentralised biological system produced a self-organised network with strong engineering-relevant properties.

For me, this is an analogy. Useful structure does not always need to be designed connection by connection from the top down. Sometimes an environment can support adaptive processes from which useful structures emerge.

From adaptation to population-based search

Slime mould is an inspiration, not the algorithm. A drone route requires many candidate strategies that can be tested, selected and varied. An evolutionary algorithm lets me represent candidate routes as a population. Each “genome” might contain waypoints, camera actions, turns, altitude changes and a return home command.

Generate → simulate → evaluate → select → vary → repeat

I no longer need to know the perfect route. I need to specify how I will recognise useful routes. But a single weighted fitness score introduces another hidden decision. If fitness is 50% speed, 30% battery and 20% safety, I have decided how much safety can be exchanged for speed. Multi-objective optimisation lets the trade-offs remain visible.

Why NSGA-II fits this problem

NSGA-II stands for Non-dominated Sorting Genetic Algorithm II. It uses non-dominated sorting, elitist selection and a crowding mechanism intended to preserve diversity (Deb et al., 2002). Candidate routes can be evaluated separately for mission completion, collision risk, time, battery, distance, inspection quality and obstacle clearance.

If Route A is faster, safer and uses less energy than B, A dominates B. If C is faster than A but uses more battery, neither necessarily dominates the other. They represent different trade-offs. The result is not one route presented as objectively best, but an approximation containing several strong alternatives for human review.

The conceptual progression is: adaptive emergence → population-based search → multi-objective optimisation → NSGA-II → human judgement. The algorithm searches the possibility space. The human determines what is acceptable to take forward.

A positive brain may be a better learning environment

Shawn Achor's work originally made me interested in this relationship. In a widely viewed TEDx talk, Achor said a brain in what he calls a positive state can be around 31% more productive than one in a negative, neutral or stressed state (2011). I do not treat 31% as a universal scientific constant, or claim people become 31% smarter when happy. I include it because it influenced the questions I began asking.

The wider literature is more nuanced. Lyubomirsky, King and Diener's review considered 225 studies on positive affect and successful outcomes, using correlational, longitudinal and experimental designs (2005). That should not be simplified to “happiness mechanically causes success”. A narrower proposition is that psychological state and learning environment can matter for behaviour and performance.

Hypothesis to test: Could an environment designed around curiosity, visible progress and manageable challenge help people persist with difficult technical tasks for longer?

Making programming feel like a mission

Programming can become a mission. Mathematics becomes route optimisation. Physics becomes movement, acceleration and battery management. Artificial intelligence becomes autonomous navigation. Failure becomes a crashed simulated drone rather than an expensive physical one, and learning becomes visible.

Making learning enjoyable does not necessarily make it less serious. It might allow a learner to stay with a difficult problem long enough to improve. That is something I want to test, not simply assume.

Game theory, incentives and mechanism design

In strategic situations, one participant's outcome can depend partly on what others choose. The physical drone problem alone is not automatically game theoretic, but game theory gives me useful language for incentives. If speed is heavily rewarded while danger is weakly penalised, aggressive behaviour can become rational under the rules I wrote.

Game theory can help analyse behaviour given rules and expectations. Mechanism design asks how the rules themselves should be designed (Hurwicz and Reiter, 2006). I should not misuse the technical term: when adjusting one agent's reward function, “reward design” or “objective design” may be more accurate. But I take a broader lesson:

Examine the rules and incentives before blaming the participant.

Loss aversion: protect what is difficult to rebuild

Kahneman and Tversky's prospect theory describes decision-making under risk and an asymmetry in how gains and losses can be valued (1979). It does not say humans are always risk-averse, and it does not tell Agape Duck how it ought to behave. What follows is my management interpretation.

What am I extremely unwilling to lose while building Agape Duck? Integrity, trust, safety and reputation. A small performance improvement is a modest gain. Injuring somebody, misleading someone or seriously damaging trust is a much greater downside.

Safety should sometimes be a constraint, not another point

If safety is only another score, enough performance could theoretically compensate for added risk. Sometimes I do not want that trade to be available. First define a safety boundary and reject candidates outside it. Then evaluate time, energy, route efficiency and inspection quality among the feasible solutions.

Experiment aggressively where failure is cheap. Operate conservatively where failure can cause real harm.

In simulation, crash, mutate, test strange solutions and learn. When people, property or organisational trust are exposed, the rules should become more conservative. Simulation gives us somewhere to be adventurous. The physical world demands greater caution.

Repeated interactions and reputation

Axelrod and Hamilton showed how reciprocal cooperation can become viable under particular repeated-interaction structures (1981). Kreps and Wilson explored reputation in multistage games under incomplete information (1982). Neither proves that honesty guarantees business success. My decision to make integrity a long-term strategy is a value choice.

The limited lesson is that when future interactions and beliefs about past behaviour matter, today's behaviour can acquire value beyond its immediate payoff. I would rather describe an unfinished prototype accurately than gain a small short-term advantage by pretending it is complete. Protect the downside first. Optimise the upside second.

AI governance: the algorithm advises, the human remains responsible

In March 2026, I attended and spoke at the University of Winchester's Future Thinkers Forum: Voices from the AI Frontier. Professor Maria Burke was another contributor, and I know her personally. I state that transparently because neither that relationship nor her appearance should be read as an endorsement of Agape Duck or this article.

The University's public account described the continuing importance of a “human in the loop” as AI becomes more capable. I also draw on my first-hand experience of the forum. This is not a new experimental study proving one particular architecture. It is a governance principle that closely matches the philosophy I am developing in the simulator.

NIST and responsible AI risk management

The US National Institute of Standards and Technology's AI Risk Management Framework 1.0 organises its core around four connected functions: GOVERN | MAP | MEASURE | MANAGE. They are not a simple linear checklist, and GOVERN is cross-cutting. The framework is voluntary. Referencing it does not make Agape Duck NIST-certified or establish regulatory compliance. NIST also states that AI RMF 1.0 is being revised in 2026.

My experimental workflow is different: govern → simulate → measure → compare → human review → act → learn. AI explores possibilities. The simulator measures consequences. Trade-offs become visible. Risks are reviewed. A responsible human decides what enters the physical world.

Use AI to expand human judgement, not remove human responsibility.

Humans make optimisation harder

A computer does not get bored or demoralised after five crashes. A learner might. A useful educational simulator may need to support skill, enjoyment, confidence, curiosity, challenge and safety while reducing frustration, stress, unsafe behaviour and manipulation. These objectives can conflict.

I am not claiming that human learning has a mathematically established Pareto frontier. Pareto thinking is an analogy that helps me conceptualise conflicting educational objectives. It produces another hypothesis: can simulator difficulty adapt to preserve meaningful challenge while reducing unnecessary frustration?

Why this matters beyond drones

The same principle applies to business automation. Speed matters, but so do accuracy, privacy, consistency, human oversight, cost, ease of use, customer experience and auditability. Telling a system only to work as quickly as possible is already a value judgement.

A better project begins with: What does good actually mean in this environment? Only then should we ask what to automate, what should remain human, which risks are unacceptable and how we will measure whether the system helped. The technology comes second. The objective comes first.

The bigger experiment

I started building a drone simulator because drones are interesting. Increasingly, the simulator has become the experiment. It gives me somewhere to bring together psychology, behavioural economics, reinforcement learning, game theory, reward design, evolutionary computation, multi-objective optimisation, Pareto analysis, governance, robotics and education.

The process is: human idea → objectives, constraints and safety boundaries → simulation → candidate solutions → evolutionary search → multi-objective evaluation → approximate Pareto front → governance review → human judgement → physical experiment → measurement → revision.

Crash a simulated drone 4,000 times if necessary. Change the reward, incentives or representation. Run another population. Learn something. Only then expose a physical machine to the real world.

AI helps us explore possibilities that would otherwise be too expensive, dangerous or time-consuming to test, while humans remain responsible for deciding which possibilities are worth pursuing.

What should the environment reinforce?

These are different subjects and should remain different subjects. Yet they bring me back to the same questions: what happens when we carefully design the environment in which intelligent behaviour develops, and what values do we want it to reinforce?

For Agape Duck, I want the answer to remain: curiosity, learning, safety, integrity, honesty and human judgement. Technology should help us experiment boldly. Simulation should make failure cheaper. Evolutionary algorithms should expand the possibilities. Pareto optimisation should make trade-offs visible. AI should help us think. But our values should determine where we choose to fly.

References

• Achor, S. (2011) “The happy secret to better work”, TEDxBloomington.
• Achor, S. (2012) “Positive Intelligence”, Harvard Business Review.
• Axelrod, R. and Hamilton, W. D. (1981) “The evolution of cooperation”, Science, 211(4489), 1390-1396.
• Deb, K. et al. (2002) “A fast and elitist multiobjective genetic algorithm: NSGA-II”, IEEE Transactions on Evolutionary Computation, 6(2), 182-197.
• Ferster, C. B. and Skinner, B. F. (1957) Schedules of Reinforcement. New York: Appleton-Century-Crofts.
• Hurwicz, L. and Reiter, S. (2006) Designing Economic Mechanisms. Cambridge University Press.
• Kahneman, D. and Tversky, A. (1979) “Prospect theory: An analysis of decision under risk”, Econometrica, 47(2), 263-291.
• Kreps, D. M. and Wilson, R. (1982) “Reputation and imperfect information”, Journal of Economic Theory, 27(2), 253-279.
• Lyubomirsky, S., King, L. and Diener, E. (2005) “The benefits of frequent positive affect”, Psychological Bulletin, 131(6), 803-855.
• National Institute of Standards and Technology (2023) AI Risk Management Framework 1.0.
• National Institute of Standards and Technology (2026) “AI Risk Management Framework”, accessed 2 September 2026.
• Nevin, J. A. et al. (2001) “Variable-ratio versus variable-interval schedules”, Journal of the Experimental Analysis of Behavior, 76(1), 43-74.
• Sutton, R. S. and Barto, A. G. (2018) Reinforcement Learning: An Introduction, 2nd edn. MIT Press.
• Tero, A. et al. (2010) “Rules for biologically inspired adaptive network design”, Science, 327(5964), 439-442.
• University of Winchester (2026) “Experts claim humans still have vital role to play in AI revolution”, 18 March.
• Voss, C. and Raz, T. (2016) Never Split the Difference. Harper Business.