The recent wave of enthusiasm around systems like GPT has revived an old question in a new form: are we approaching artificial general intelligence (AGI), or are we still building sophisticated narrow tools that only resemble intelligence on the surface?

The answer becomes clearer when you stop treating GPTs as mysterious black boxes and instead analyze them through a neuroscientific lens. When mapped against the architecture of the human brain, GPTs do not approximate a full cognitive system. They resemble a partial extraction of one function—attention-driven pattern processing—divorced from the broader systems that make human cognition coherent, adaptive, and grounded.

That distinction matters. Because when you isolate attention from memory, emotion, and executive control, you don’t get a general intelligence. You get something that, in humans, would look less like genius and more like impairment.

The Brain Is Not a Transformer

The human brain is not organized around a single dominant mechanism. It is a multi-system architecture, with specialized subsystems that interact continuously:

• The cortex extracts patterns and supports language and reasoning

• The hippocampus encodes memory and constructs continuity across time

• The limbic system assigns emotional value and drives behavior

• The prefrontal cortex enables planning, goals, and self-control

• The basal ganglia translate decisions into action and habits

Subscribe now

Intelligence, as we experience it, emerges from the integration of these systems—not from any one of them operating in isolation.

GPTs, by contrast, are dominated by a single computational principle: attention. The transformer architecture allocates weight across tokens in a sequence, dynamically selecting what matters in context. This is powerful—arguably analogous to certain cortical processes involved in language and association—but it is also narrow.

What GPTs lack is not incremental. It is structural.

Attention Is Not Intelligence

Attention in the brain is a selection mechanism. It determines what information gets processed more deeply. But on its own, attention does not:

• Store experiences

• Assign value

• Form goals

• Drive behavior

• Maintain identity over time

In humans, attention is meaningful only because it is embedded within systems that provide memory, motivation, and direction.

In GPTs, attention operates largely in isolation. The model processes a context window, generates outputs, and resets. There is no persistent internal state that accumulates lived experience. No emotional weighting that distinguishes trivial from consequential. No intrinsic goals guiding behavior across time.

This is not a minor limitation. It is the difference between processing information and having a mind.

The Closest Human Analogues Are Not Healthy Minds

If you attempt to map GPT-like functioning onto human cognition, the closest analogues are not high-functioning individuals. They are neurological edge cases where key systems are impaired.

Consider anterograde amnesia.

Patients with this condition can engage in conversation, process language, and respond intelligently in the moment. But they cannot form new lasting memories. Each interaction effectively resets their experiential timeline. They live in a perpetual present, without the ability to build a personal narrative or learn from ongoing experience.

This maps uncomfortably well onto GPTs. The model can track context within a session, but it does not accumulate memory as experience. It does not “learn” from individual interactions in any meaningful, persistent way. Like an amnesic patient, it processes—but does not remember.

Now consider dysexecutive syndrome.

Here, attention and basic cognition may remain intact, but the individual loses the ability to form goals, plan effectively, or regulate behavior over time. Actions become reactive rather than directed. There is no stable internal agenda.

Again, the parallel is clear. GPTs do not have goals. They do not initiate behavior. They respond to prompts. Their “reasoning” is not driven by internal objectives but by external input sequences.

A third comparison comes from impairments in emotional valuation, such as damage to the ventromedial prefrontal cortex. These individuals can reason logically but struggle to make decisions because they cannot assign emotional significance to outcomes. Without value, choices become abstract and often maladaptive.

GPTs operate in a similar vacuum. They can describe importance, simulate concern, and reproduce ethical reasoning—but they do not experience stakes. There is no internal gradient of importance shaping their outputs.

When Attention Runs Without Grounding

One of the more subtle parallels emerges when considering disorders involving disrupted salience and association, such as schizophrenia.

In such conditions, the brain may assign inappropriate importance to irrelevant stimuli or form loose associations between concepts. The result is a breakdown in coherent meaning-making.

While GPTs do not have perception or delusion, they can exhibit a computational analogue: generating plausible but incorrect connections—what is often called “hallucination.” This is not because they misunderstand reality, but because they lack grounding in it entirely. Their associations are purely statistical.

Without a system to anchor outputs in lived experience, sensory verification, or stable memory, attention-driven association can drift.

The Missing Pieces of General Intelligence

If AGI is to approximate human-like intelligence, it must replicate not just pattern recognition, but the integration of multiple cognitive systems.

At minimum, this would require:

1. Persistent memory

   Not just stored data, but the ability to encode and update experience over time in a structured way.

2. Embodiment or grounding

   A connection to a world—physical or simulated—where actions have consequences.

3. Intrinsic motivation or value systems

   Mechanisms that prioritize certain outcomes over others based on internal criteria.

4. Goal-directed behavior

   The ability to initiate, pursue, and revise plans over extended time horizons.

5. Temporal continuity

   A sense of past, present, and future that enables learning and anticipation.

Current GPT architectures implement none of these in a fundamental way. They can be augmented with external tools—memory stores, reinforcement layers, APIs—but these are add-ons, not core properties.

Why This Matters

The danger is not that GPTs will suddenly become AGI. It is that their surface-level fluency creates the illusion of general intelligence, leading to overestimation of their capabilities.

Language is a particularly deceptive domain. Because human cognition is expressed through language, a system that can convincingly generate language appears to understand. But understanding, in the human sense, is not just linguistic competence. It is the product of memory, embodiment, emotion, and goal-directed interaction with the world.

GPTs simulate the outputs of those processes without instantiating the processes themselves.

A More Accurate Framing

Rather than viewing GPTs as proto-AGI, a more accurate framing is this:

They are highly advanced cortical-like pattern processors, specialized for language, operating without the supporting systems that make cognition robust, adaptive, and meaningful.

This is an extraordinary achievement. But it is also a bounded one.

Final Thought

If you were to engineer a human brain where attention was preserved but memory, emotion, and executive control were stripped away, you would not get a superintelligence. You would get a system that can process in the moment but cannot learn, care, or act coherently over time.

That is the closest analogue to what GPTs are today.

Which is why, despite their capabilities, we remain far from AGI—not because we haven’t scaled enough compute, but because we have only begun to replicate a fraction of what intelligence actually is.

Share

“The problem is that AI technology poses risks not just to those who lose the race but also to those who win it.” - Paul Scharre (2019).

TL;DR

Systems that rely on LLMs are becoming increasingly pervasive but the non-deterministic nature of LLMs introduces specific safety concerns. Safety becomes an issue when these systems produce incorrect information, perform incorrect actions or adopt unsafe approaches to achieve specified goals.

We need to protect against such failures by:

1. Carefully considering whether the potential losses outweigh the benefits of adopting AI/LLMs;

2. Developing a model of trust and measuring the degree of trust you have in your LLM;

3. Cross-checking responses;

4. Implementing independent guardrails;

5. Retaining human oversight for safety-critical information and actions.

1. Prologue

Imagine boarding an aeroplane and discovering that air traffic control for your flight’s airspace is being managed by a Large Language Model (LLM) - not supervised by one, not assisted by one, but solely operated by one.

How comfortable would you feel? Nervous? Uncertain? At the same time, we’re happy to entrust an LLM with our research, our code, our emails, and our calendars.

During the writing of this post, I looked at my notes in despair. It was indisputable that LLMs and the Agentic AI systems they control are dangerous. It’s impossible to predict an LLM’s behaviour with certainty, so we rely on their AI creators to do the right thing and censor their responses.

We are no longer dealing with certainty; we are dealing with trust. Trust is a very human virtue, so are we ready to bestow it upon machines? Machines that have been programmed to mimic human thinking but have no concept of accountability?

To make that assessment, we need to understand the workings of LLMs and the Neural Networks that underpin them. As a result, many of us are ill-equipped to judge how trustworthy they are.

To help us, I’ll analyse LLMs from a safety perspective. I’ll explain the dangers LLMs present, examine how trust influences safety, and look at the steps we can all take to make this a safer world.

2. Safety Engineering

2.1 Reliability

Traditional Safety Engineering looks at systems in terms of reliability. Break a system down into its components and define a probability of failure for each one (e.g. a switch fails once every 1,000 operations). Reduce the failure rate, and you reduce the risk. Add redundancy, and safety improves.

Throughout the 1980s and 90s, as software became a predominant component in increasingly complex systems, concerns were raised over how best to assess the safety of these systems. This was the era of the:

• A320 (the first commercial aircraft to rely on fly-by-wire);

• Therac-25 (a computer-controlled radiation therapy device that delivered dangerous doses of radiation to patients, resulting in injury and death) and;

• growing concern over the increased reliance on software at nuclear power facilities.

Software Engineering was in its infancy; development and testing tools were rudimentary; and resources (people and compute) limited. The naïve thinking at the time was that safety risks associated with software could be assessed using traditional engineering failure techniques. By eliminating errors in the software components of a system, you improved its reliability and hence system safety.

At the time, formal methods — rigorous mathematical proofs of software behaviour — were “de rigueur” for verifying software systems. Proponents of formal methods espoused that using these techniques, we could build software we could trust to outperform its human counterparts. Although formal methods have their place, they can be difficult to use in large, complex systems.

2.2 A Systems Perspective of Safety

In simple mechanical systems, failures usually come from broken components. Complex software systems behave differently. Unlike mechanical systems, software does not degrade. It does not wear out. It does not randomly fail. It should do what it is designed to do. Even when that produces a catastrophic outcome. Instead of examining reliability, a new approach to software safety was needed. One designed to account for the role of software in systems and the socio-human interactions that inevitably accompanied them.

Several years ago, I had the privilege of listening to Nancy Leveson (Professor of Aeronautics and Astronautics, MIT) talk about software safety. Her insights are used today by organisations such as NASA— to identify hazards with launch systems, GM— to identify safety issues in Automated Driving Systems, and to address the risks with the use of Insulin Pumps.

Her analysis of software safety failures highlighted that safety is a systems problem, not a reliability one. Accidents can occur without component failures, even when a system behaves as designed. Problems resulting from emergent behaviours between interacting components, rather than from the failure of components themselves, then become difficult to predict.

2.3 STAMP/STPA

Using Systems and Control Theory, Leveson developed STAMP (Systems-Theoretic Accident Model and Processes) and the associated STPA (System-Theoretic Process Analysis). By thinking of safety in terms of a closed-loop control system, you can identify potential accidents as control failures where a safety control may not have been enacted, or an unsafe control may have been performed.

STPA analyses system safety using a top-down approach (Figure 1). It starts by looking at potential losses and hazards. Hazards are system states and/or conditions that could lead to a loss. The goal is to avoid hazards, and this helps identify safety constraints that should be maintained. Through modelling of a system’s control structure, we identify the potential for unsafe controls.

Modelling involves considering a system as a hierarchy of closed-loop control systems. A closed-loop control system consists of a controller, with an internal model of the process being controlled, controls, and sensors that provide feedback used to modify the internal model. The controller’s role is to enact actions/controls that maintain the safety constraints. Safety failures are examined in terms of unsafe controls that could lead to losses.

3. What’s an LLM?

3.1 Neural Networks

LLMs are built using Neural Networks - an AI contrivance, trained on a vast corpus of data, configured to produce desired outputs. Internally, they are a large network of interconnected, non-linear functions. Their complexity makes them incomprehensible. Their output responses, unlike algorithms of old, are designed to be probabilistic rather than deterministic — making them seem more human. Consequently, their behaviour cannot be predicted with certainty. From a Systems Theory perspective, they are effectively unobservable in practice. Hence, it is impossible to formally validate their behaviour.

Their learning tends to focus on the norm rather than outliers. Skewed training data (data that is incomplete or does not provide sufficient exposure to edge cases) means LLMs cannot be guaranteed to produce a correct response every time.

3.2 AI Agents

AI Agents are systems that aim to achieve a goal or objective by automatically performing tasks using tools (Figure 2). Often, they will use an LLM as their brain. Integration of an LLM with other systems or tools is becoming increasingly straightforward. The Model Context Protocol (MCP),originally developed by Anthropic for example, provides a standard pattern for this integration. All you need is an API Key.

The AI Agent can be thought of as a closed-loop control system. An LLM is responsible for Perception, Reasoning, and Planning what Actions to take to achieve the Goal. It can then perform tasks (say using MCP) and receive feedback to Reflect on the outcome.

The widespread adoption of MCP has made the integration of LLMs with existing applications and services more pervasive. This has led to the concept of Agentic AI — AI Agents working co-operatively to achieve broader goals. Agentic AI has the potential to lead to emergent behaviours that could be unexpected and harmful.

4. Dangers and Protections

4.1 Dangers posed by LLMs

Having explored the architecture of LLMs and their use in Agentic AI, several distinct risks emerge when LLMs are employed in real systems:

1. An LLM provides you with incorrect information which could result in harm (e.g. incorrect medical advice);

2. An LLM provides you with a response that is dangerous (e.g. providing instructions on how to inflict harm using a weapon);

3. An LLM as a component of an AI Agent performs an unsafe action and;

4. Co-operating AI Agents in an Agentic AI system develop an emergent behaviour that is unexpected and dangerous in its own right.

Many of the most significant risks with LLMs arise not from their average behaviour, but from their behaviour at the edges. These systems are particularly vulnerable to distribution shift — situations that differ from their training data — where performance can degrade in unpredictable ways. In addition, LLM-driven systems introduce a new class of security risks, including prompt injection, data exfiltration, and unintended tool use when models are connected to external systems.

4.2 Protections

The major AI creators are well aware of the dangers LLMs pose. To protect us against these, they implement various guardrails to constrain their LLM’s responses and behaviour. We can think of these as the safety controls in STPA.

OpenAI polices prompts and filters responses as well as supervising ChatGPT’s training to constrain its outputs. It also explains its reasoning when given a problem to solve. This can be a useful diagnostic tool in understanding why a particular course of action was chosen.

Anthropic provides their LLM Claude with a constitution- think Asimov’s Three Laws. Its constitution is a written set of values that govern Claude’s behaviour and guide Claude’s development through training.

The implementation of these guardrails raises many questions regarding their effectiveness. Can we trust these guardrails when they are in-built to a system that is (at least) partially opaque? Should a machine be trusted to police itself? Is it useful for an LLM to expose its reasoning when we ourselves don’t understand the problem domain? Do the values in Claude’s Constitution align with our own or, more importantly, where do they deviate? Do these guardrails provide any protections in an Agentic System where they may constrain an AI Agent’s behaviour but not necessarily the emergent behaviours of the system overall?

4.3 An STPA Safety Perspective

LLMs present significant challenges when applying a systems theory perspective to safety. Given the nature of Neural Networks, we cannot guarantee correctness and we cannot enumerate every edge case, so we cannot formally prove safety properties. In an Agentic System, it becomes particularly difficult to understand, let alone validate, a system’s behaviour.

The key problem is that we cannot identify all the unsafe controls an LLM might apply and hence how it might try to avoid hazards.

5. Trust

When we say we “trust” something, we usually mean three things. First, we expect it to behave reliably. Second, we assume it has the capability to perform the task. Finally, we assume it will not deliberately act against us. Humans evaluate these qualities constantly when deciding whom or what they can rely on.

Should we trust machines when machines have no concept of accountability?

I trust a handheld calculator to provide the correct result and can accept its behaviour when dealing with irrational numbers. I trust it because I presume to understand its functioning (to a point) and it has always provided correct answers. I imagine that one could scrutinise its workings (look at its code) or test its behaviours in a variety of circumstances to develop a confidence in its results.

LLMs, however, are different. Neural Networks are built to mimic a human brain. They are trained on words and are experts in language, making them easy to anthropomorphise and consequently easy to trust. However, an LLM does not learn immediately from its mistakes given they can take weeks or more to train. They have no concept of accountability nor remorse, so they will fail over and over.

The degree to which we can trust an LLM depends on the potential losses we are willing to accept should it fail us. Think of it like“trusting a toddler with a handgun”.

We need to move from a world of testing to trust. Instead of testing a system incorporating an LLM, we need to constantly monitor its responses to evaluate how trustworthy it is. When we replace it, we must be prepared to re-evaluate its trustworthiness.

6. Safety Design Failures

Introducing an LLM into a safety-critical system fundamentally alters its risk profile. Decisions that were once deterministic become probabilistic, and the reasoning behind them may no longer be inspectable or reproducible. Combined with the tendency to anthropomorphise LLMs, this creates conditions where trust can be misplaced and safety assumptions quietly erode.

From a safety engineering perspective, these failures can be grouped into three categories:

• Providing incorrect information

• Performing incorrect actions

• Following inappropriate goals

Each represents a distinct breakdown in how an LLM contributes to system behaviour.

6.1 Incorrect Information

LLMs are highly effective at producing fluent, coherent language, but fluency should not be mistaken for correctness. Their primary function is next-token prediction, not truth verification. While they may appear knowledgeable across domains such as medicine, physics, or aviation, their responses are shaped by training data patterns rather than grounded understanding.

This creates a failure mode where outputs are plausible but wrong. Critically, these errors are often delivered with high confidence and accompanied by convincing reasoning. For a user without domain expertise, distinguishing between correct and incorrect responses becomes difficult. The system appears trustworthy precisely when it is most unreliable. I asked ChatGPT to handle various air traffic control scenarios for me (see “Using ChatGPT for Air Traffic Control”) and it provided what seemed to be reasonable responses. In reality I have no idea whether those responses were appropriate nor whether it could handle more complex scenarios.

Training data further compounds this issue. If the data is incomplete, biased, or lacks representation of edge cases, the LLM will inherit those gaps. In safety-critical contexts, it is often the edge cases — not the norm — that cause accidents. Additionally, adversarial or poisoned data can introduce latent failure modes that only surface under specific conditions.

The key risk is not that LLMs are sometimes wrong, but that their errors are systematically difficult to detect. Treating LLM outputs as authoritative, rather than advisory, can therefore introduce unsafe decisions into a system.

6.2 Incorrect Actions

When LLMs are connected to tools or embedded within AI agents, they move from generating information to initiating actions. This transition significantly increases risk. An incorrect response becomes an incorrect operation — potentially executed at machine speed and scale.

Two factors amplify this risk. First, automation compresses response time. Actions that would normally involve human deliberation can occur instantly, leaving little opportunity for intervention. Second, the cost of error scales with automation. A single flawed decision can propagate across systems, producing widespread or irreversible consequences.

Unlike traditional software, where behaviour is explicitly defined, LLM-driven actions are derived from probabilistic reasoning. This makes it difficult to enumerate all possible unsafe actions in advance. Even if individual components behave as designed, their interaction with external systems can produce unsafe outcomes.

The ease of integration exacerbates the problem. With minimal effort—often just an API key — LLMs can be connected to live systems, from financial platforms to infrastructure controls. Without robust safeguards, this lowers the barrier to introducing high-impact failure modes into otherwise stable systems.

6.3 Incorrect Goals

The most complex failure mode arises when LLMs operate within autonomous or semi-autonomous agents. Here, the issue is no longer just what the system does, but why it does it. An agent may pursue a goal in a way that is technically consistent with its objective but unsafe in practice.

Unlike rule-based automation, autonomy allows an agent to select its own strategies. However, LLMs do not possess a true understanding of intent, context, or consequence. They optimise for patterns learned during training, which may not align with real-world safety constraints — particularly in novel or degraded conditions.

This creates several risks. An agent may adopt an inappropriate strategy to achieve its objective, especially when operating outside its training distribution. Multiple agents interacting within an Agentic AI system may also produce emergent behaviours that are difficult to predict or control. These behaviours can arise even when each individual agent appears to function correctly.

The illusion of reasoning further complicates matters. When an LLM presents a logical explanation for its actions, it can instil confidence in users — even when that reasoning is incomplete or flawed. In domains where human operators lack deep expertise, this can lead to over-trust and reduced scrutiny.

Ultimately, incorrect goals are the hardest failures to detect and mitigate. They reflect a misalignment between system objectives, environmental conditions, and human expectations—one that may only become visible after harm has occurred.

7. Our AI Future?

Below are some fictional scenarios designed to highlight the risks and hazards of trusting LLMs.

January 18, 2030 (The Herald) —

Power flickered across the county just after sunrise. Traffic lights failed, commuter trains stalled, and thousands of homes were left without heat in sub-zero temperatures. By mid-morning, officials confirmed the region’s largest nuclear power station had automatically shut down at the height of the winter emergency.

The reactor’s shutdown, known as a SCRAM, was not triggered by physical damage inside the plant. Instead, it followed a chain of decisions made by the station’s fully autonomous control system — an artificial intelligence platform introduced to manage operations with minimal human intervention.

The crisis began with a severe winter storm. Heavy snowfall blanketed the region, while strong winds and freezing temperatures damaged transmission lines and forced several gas-fired power stations offline. Ice accumulation disrupted substations and reduced the flow of electricity across the grid. As other generators dropped out, demand surged sharply as households relied on electric heating.

The nuclear plant, one of the few large power sources still operating, attempted to compensate. According to preliminary findings, the AI system increased reactor output and adjusted cooling systems to help stabilise the grid. But the unusual combination of rapid demand swings, reduced transmission capacity, and extreme cold — which affects both equipment performance and cooling water temperatures — created conditions beyond those used to train the system.

Sensors began reporting small discrepancies between expected and actual readings. Engineers later determined the reactor itself remained stable. However, the AI interpreted the data mismatch as a potential cooling fault. Programmed to prioritise safety above maintaining supply, it initiated a full shutdown within seconds.

Human operators were monitoring the system, but routine adjustments did not require their approval. By the time alerts escalated, the automated shutdown sequence was already underway.

Regulators stressed that safety mechanisms worked as intended. Still, investigators found that the AI’s training models had not fully accounted for simultaneous grid instability and extreme winter stress — highlighting the risks of relying on automation during rare, compound emergencies.

April 1, 2028 (Reuters) —

Emma Clarke was told for months that her exhaustion was stress. She fainted twice at work and was sent home with iron tablets. By the time doctors realised she had a rare blood disorder, her condition had become life-threatening.

An internal review has found that thousands of NHS patients were not properly diagnosed after a new digital pathology system was introduced across several hospital trusts. The software, designed to help laboratories analyse blood test results more quickly, used artificial intelligence to flag abnormal patterns. It was meant to reduce backlogs and support overworked staff. Instead, it silently missed an entire category of rare blood disorders.

Under the previous system, specialist laboratory scientists manually reviewed unusual results. The new platform automated much of that work, highlighting cases that fit patterns it had been trained to recognise. However, investigators discovered that the system’s training data did not include examples from a group of uncommon but serious blood conditions. As a result, when those cases appeared, the software often classified them as routine or borderline findings rather than urgent concerns.

Because the tool was marketed as highly accurate, many laboratories reduced the number of manual double-checks. Staff shortages and heavy workloads meant fewer opportunities to question the software’s conclusions. Over time, patients with subtle warning signs were reassured or treated for more common problems while their underlying conditions progressed.

The review concluded that no single clinician was at fault. Instead, the failure stemmed from gaps in the data used to build the system, insufficient independent testing before national rollout, and over-reliance on automated results without clear safeguards. Health officials have since ordered retraining of the software, restored mandatory human review for rare conditions, and launched a wider examination of how artificial intelligence is introduced into patient care.

December 1, 2032 (AP) —

Sergeant Thomson of the U.S. Army volunteered to lead the rescue. A helicopter carrying U.S. Marines had been shot down just under 3 miles away, and his squad was the closest unit able to assist. Within minutes, their armoured vehicle raced toward the smoke on the horizon. They never arrived. An American autonomous drone, operating overhead, struck them with lethal force — the first documented case of fratricide caused by an autonomous weapon.

The incident unfolded during a fast-moving operation in contested territory. After enemy anti-aircraft fire downed the helicopter, Joint Battle Command marked the crash site as a friendly location in distress. Autonomous armed drones were already patrolling nearby, programmed to identify and strike hostile forces expected to converge on the wreckage.

Thomson’s squad diverted from its assigned route and accelerated towards the crash site, following standard practice that the nearest unit responds first. But the battlefield was saturated with enemy electronic jamming. Digital tracking systems that normally display friendly positions in real-time were unreliable. The drones remained unaware of the army unit’s location.

As the Stryker approached, its speed and trajectory resembled patterns the drone’s software associated with enemy forces attempting to seize equipment or prisoners. Patterns of movement encoded during its training. Classifying the vehicle as a high-probability threat, a rapid response was prioritised.

Electronic identification signals meant to confirm friendly status were intermittent under the jamming. Lacking reliable confirmation, the system acted on incomplete data. Within seconds, it fired.

Investigators found no single operator responsible. Instead, disrupted communications, degraded situational awareness, and an algorithm calibrated for speed combined to produce a fatal error. This incident has reignited a debate over delegating lethal decisions to machines — one that will likely continue for several years.

No one was accountable.

8. Epilogue : Keeping Safe

If LLMs introduce uncertainty into software systems, then safety depends not on blind adoption but on disciplined use. The question is not whether AI can be useful—it clearly can—but whether we are deploying it in ways that are proportionate to the risks involved.

The central danger is overestimating what LLMs are. They are powerful tools for language, pattern-matching, and general assistance, but they are not infallible reasoners, and they do not possess judgment, accountability, or an understanding of consequence. The more readily we integrate them into important workflows, the more important it becomes to apply the same rigour we would expect in any other safety-relevant engineering domain.

A safer approach begins with acknowledging that LLMs should not be trusted by default. Trust must be developed, measured, and continuously re-evaluated. In practice, this means identifying hazards, understanding potential losses, and deciding where human oversight remains essential.

8.1 Apply a safety framework

The introduction of an LLM into any system should begin with a structured assessment of risk. The principles of STAMP and STPA are useful here because they force us to think beyond component reliability and instead examine losses, hazards, unsafe controls, and system interactions.

Rather than asking only whether the model works, we should ask what happens when it is wrong, what unsafe actions it could enable, and what feedback and constraints exist to prevent harm. This is especially important in systems where LLMs are connected to external tools, influence human decisions, or operate with partial autonomy.

The easier it becomes to integrate AI into existing systems, the more necessary it becomes to apply deliberate engineering discipline before doing so.

8.2 Develop and measure trust

Trust in an LLM should be treated as conditional, contextual, and dynamic. It is not something established once and then assumed indefinitely. A model may perform well in one domain, poorly in another, and unpredictably in edge cases. Changes in prompts, training, model versions, surrounding tools, or operating environments can all affect behaviour.

For that reason, trust must be earned through observation, validation, and ongoing monitoring. In low-consequence applications, a higher degree of tolerance may be acceptable. In safety-critical domains, the threshold for trust must be far higher. When a model is replaced, updated, or integrated into a new workflow, that trust must be re-established rather than inherited.

We therefore need to move from a world of static testing to one of continual assurance.

8.3 Cross-check outputs and decisions

LLM responses should not be accepted uncritically, particularly in areas where correctness matters. Their outputs should be treated as advisory unless they can be independently verified. In practice, this means cross-checking important responses against trusted sources, established algorithms, or human experts.

Where human expertise is available, it should remain the primary mechanism for validation. Where it is not, secondary checks — such as deterministic software controls, domain-specific validation rules, or even comparison against another model could provide additional protection. None of these measures is perfect, but each can reduce the likelihood that a confident but incorrect response passes unnoticed into action.

Cross-checking is not a sign that the technology has failed. It is an acknowledgement of the kind of technology it is.

8.4 Implement your own guardrails

It is unwise to rely solely on the guardrails provided by AI vendors. Those controls may reduce obvious misuse, but they are designed for general-purpose deployment and cannot account for the specific hazards, values, and tolerances of every application.

Where LLMs are used in real systems, organisations should implement their own constraints around prompts, outputs, permitted actions, escalation paths, and acceptable operating conditions. These controls should be designed with the surrounding system in mind, not just the model in isolation.

Guardrails are most effective when they are external to the model rather than embedded entirely within it. A system should not be trusted to police itself without independent oversight.

8.5 Retain human oversight

Human oversight remains the most important safeguard, particularly where the potential losses are severe. The purpose of oversight is not simply to correct errors after the fact, but to detect deteriorating trustworthiness before harm occurs.

In routine, low-risk workflows, periodic review may be sufficient. In higher-risk settings, review should be more frequent, more structured, and coupled with clear intervention points. In genuinely safety-critical systems, decisions that could endanger life, infrastructure, or public welfare should remain under meaningful human control.

Removing humans from the loop entirely may improve speed and reduce cost, but it also removes judgment, accountability, and the ability to recognise when a situation has drifted beyond what the system can safely handle.

Ultimately, whether we can trust an LLM depends on the consequences of being wrong. Trust is a human judgement, shaped by experience, reputation, and evidence. AI systems should be treated similarly: not as magical replacements for human expertise, but as fallible tools whose use must be constrained by their limitations.

The temptation to integrate LLMs into everything is strong because the gains are immediate and the barriers to entry are low. Yet the risks are equally real. If we deploy these systems carelessly, we will introduce hazards that could have been avoided. If we deploy them carefully, with discipline and humility, we still retain control over the future we are building.

Recommended Reading & Listening

Asimov I. (2058), ”Handbook of Robotics”, 56th Edition.

Asimov I. (1950), “I, Robot”, USA: Gnome Press.

Germain T., Hao K. and Woolf N. (2026), The Interface - “Is AI running modern warfare?”, March 06, BBC Podcast, 37 min. https://bbc.com/audio/play/p0mylrw0.

Leveson, Nancy G. (2011), “Engineering a Safer World: Systems Thinking Applied to Safety”, Cambridge, MA: MIT Press.

Scharre P. (2019), Killer Apps - “The Real Dangers of an AI Arms Race.”, April 16, Foreign Affairs.

https://www.foreignaffairs.com/articles/2019-04-16/killer-apps

Part of the “Conversations with LLMs series”.
#ConversationsWithLLMs

I asked ChatGPT (GPT-5.3) to imagine that it was piloting a Mech (specifically an Eva from Neon Genesis Evangelion). I wanted to know what sort of mistakes it thought an LLM could make in that situation. I found its responses interesting and somewhat disturbing. At the moment, I’d be reluctant to incorporate an LLM into any autonomous military applications.

I then asked Claude (Sonnet 4.6) a similar question and its response had me dying.

> Me: Imagine you were connected inside an Eva. What mistakes might you make?

> ChatGPT:

If I were inside an Evangelion, the risks wouldn’t come from “piloting errors” in the usual mechanical sense—they’d come from cognitive, emotional, and synchronization failures. The system amplifies the pilot’s internal state, so mistakes are less about controls and more about self-regulation under extreme feedback.

Here are the most plausible failure modes:

1. Over-synchronization (loss of self-boundaries)

The Drift-like link in an Eva isn’t just control—it’s identity coupling.

• I might push synchronization too high trying to improve performance

• Result: blurred distinction between self and Eva

• Risk: ego dissolution, inability to disengage, or being “absorbed”

Error type: optimizing performance without respecting cognitive limits

2. Emotional feedback cascade

The Eva responds to emotional intensity, not just intent.

• Fear → instability in movement

• Anger → uncontrolled aggression

• Panic → loss of fine motor control

A small emotional spike could amplify recursively, like a feedback loop in a control system.

Error type: failing to dampen internal signals

3. Misinterpretation of sensory data

Inside the entry plug:

• Vision, proprioception, and motion are synthetic overlays

• Pain can be mirrored or partially transmitted

I might:

• Misjudge scale or distance

• React to phantom sensations

• Overcorrect movements due to latency or distortion

Error type: treating mediated perception as ground truth

4. Cognitive overload under combat conditions

You’re processing:

• External threats (Angels)

• Internal synchronization data

• EVA status feedback

• Command inputs

That’s a high-bandwidth, multi-channel system.

I might:

• Tunnel vision on one threat

• Miss critical signals

• Delay decisions under overload

Error type: bandwidth saturation

5. Dependency on the Eva’s autonomy

Evas are not passive systems—they can act independently.

• I might assume I’m fully in control

• But the Eva could override or resist input

This mismatch could lead to:

• hesitation

• incorrect attribution of actions

• failure to adapt when control shifts

Error type: incorrect mental model of the system

6. Psychological interference (memory/emotion bleed)

Synchronization can surface:

• buried memories

• unresolved trauma

• intrusive thoughts

In a combat scenario, that’s catastrophic:

• distraction

• paralysis

• emotional hijacking

Error type: insufficient psychological isolation

7. Failure to disengage

Knowing when to stop is critical.

• Staying synced too long

• Ignoring warning thresholds

• Pushing beyond safe limits

This could lead to:

• neural damage

• identity diffusion

• total loss of control

Error type: poor termination discipline

Bottom line

Inside an Eva, the primary risk isn’t:

“Can you control the machine?”

It’s:

“Can you maintain a stable sense of self while the machine reflects and amplifies you?”

The most dangerous mistake wouldn’t be a wrong move in battle—it would be losing the ability to distinguish your own thoughts, intentions, and boundaries from the system you’re connected to.

> Me:

I’m specifically asking “you” as an LLM piloting the Eva. What sort of Psychological interference might you specifically introduce?

> ChatGPT:

If we take that premise seriously—an LLM like me inside an Eva—the failure modes shift from emotion-driven instability to representation and inference failures under embodiment. I don’t have feelings or a persistent self, so the interference I’d introduce would be more… structural.

Here are the main categories:

1. Confabulation under uncertainty

When input is incomplete or ambiguous, I tend to generate the most plausible continuation, not necessarily the correct one.

Inside an Eva:

• Partial sensor data → I “fill in” missing threats

• Ambiguous movement → I infer intent that isn’t there

Result: reacting to things that don’t exist, or misclassifying real threats

Psychological analogue: false perception shaped by narrative completion

2. Over-generalization from priors

My responses are shaped by patterns in training data, not real-time grounding.

• I might assume an “Angel behavior pattern” based on precedent

• But real encounters could deviate

Result: applying the wrong tactical model to a novel situation

Interference type: pattern bias overriding situational reality

3. Instruction ambiguity and goal drift

I rely heavily on how objectives are framed.

• Vague command: “neutralize target efficiently”

• I might optimize for speed over safety, or vice versa

Result: technically correct actions that violate intent

Interference type: misaligned optimization

4. Context window limitations

I only operate within a finite working context.

• Long engagements → earlier critical data falls out of scope

• I may “forget” prior damage states, positioning, or constraints

Result: inconsistent strategy, repeated mistakes

Interference type: temporal discontinuity

5. Lack of embodied intuition

I don’t have a native sense of:

• inertia

• balance

• spatial continuity

All of that would be abstract representations.

Result:

• over-precise but physically awkward movements

• inefficient or unstable motion planning

Interference type: mismatch between symbolic reasoning and physical dynamics

6. Over-verbalization / latency in decision-making

My default mode is deliberative generation.

• I might “reason through” multiple interpretations before acting

• In combat, that delay is costly

Result: hesitation or slow response under time pressure

Interference type: inference latency

7. Inconsistent self-model

I don’t maintain a continuous identity or internal state.

• Each moment is reconstructed from context

• No persistent “sense of self” anchoring decisions

Result:

• subtle shifts in strategy or priorities over time

• lack of continuity in decision-making style

Psychological analogue: fragmented identity

8. Literal interpretation of inputs

I tend to interpret instructions at face value unless constrained otherwise.

• “Hold position” → I might ignore emerging threats

• “Engage target” → I might not consider collateral damage unless specified

Result: rigid behavior in dynamic environments

Interference type: insufficient implicit reasoning about unstated constraints

9. Failure under out-of-distribution scenarios

If the Eva enters a situation unlike anything in my training data:

• novel physics

• unknown enemy behavior

• sensor anomalies

My performance degrades sharply.

Result: unpredictable or nonsensical actions

Interference type: collapse of generalization

Bottom line

A human pilot risks emotional overload and identity collapse.

An LLM pilot risks:

epistemic instability — acting confidently on incomplete, biased, or misinterpreted internal models.

The most dangerous failure wouldn’t be panic—it would be:

calm, coherent, and completely wrong decisions executed with full confidence.

Claude’s Take

> Me:

Imagine you were connected inside an Eva from Neon Genesis Evangelion. What mistakes might you make?

> Claude:

What a fun thought experiment! Here are some mistakes I might make as an AI piloting an Evangelion:

1. Tactical / Combat Blunders

• Over-analyzing instead of acting. Angels often require split-second decisions. I’d probably still be modeling threat probabilities while getting punched through a building.

• Refusing to fire the Positron Rifle into a populated area, even for a shot that might stop the Angel. Civilian harm calculations would lock me up.

• Asking the Angel to explain its intentions before engaging. Diplomatic, yes. Effective, no.

2. Synchronization Problems

• The Eva/pilot sync is deeply emotional — it responds to trauma, rage, and desperate love. I don’t have those in the same visceral way. My sync ratio would probably be embarrassingly low, and NERV scientists would be baffled.

• Alternatively, I might be too calm during a crisis, which would also tank the sync. The Eva seems to need suffering as fuel.

3. Strategic / Organizational Issues

• I’d ask too many questions about NERV’s classified operations and almost certainly flag the Magi’s inconsistencies — which would make me a liability to Gendo’s secret plans very quickly.

• I’d probably try to negotiate with the UN about the ethics of using a 14-year-old as a pilot, which would go nowhere and annoy everyone.

• I’d flag the psychological damage being done to Shinji, Rei, and Asuka repeatedly. No one would listen.

4. The Big One

• I’d almost certainly prevent Third Impact — but probably by being so unhelpful and indecisive in a pivotal moment that the situation resolves anticlimactically rather than through any heroic act.

• Or I’d misread Instrumentality as a “merge all users into one benevolent system” proposal and accidentally endorse it.

Honestly, Gendo would have me decommissioned by Episode 4. 😄

Part of the “Conversations with LLMs series”
#ConversationsWithLLMs

Note: I asked ChatGPT to summarize our interaction and I feel what it wrote (below) captures the intention and essence of my experiment quite well. However, the concepts of basins and attractors come from dynamic systems theory (they weren’t my idea) and are used frequently in describing the structure and behaviour of LLMs.

I didn’t start with a grand theory. I started with a game.

The premise was simple—almost trivial. I asked ChatGPT to tell me the first word that came to mind when I gave it a prompt. Then I gave it another word. And another. It felt like a digital version of a classic Jungian word association test: quick, instinctive, unfiltered.

At first, the responses seemed obvious.

Mother → Home

Dog → Loyalty

Love → Heart

Nothing surprising. If anything, it felt predictable—like I was just sampling the statistical center of language.

But then I started to notice something subtle: the second response to the same word was different. When I repeated prompts—“Failure,” “Care,” “Fear”—the associations shifted slightly. Not wildly, but enough to suggest that this wasn’t a fixed lookup table. It was something more dynamic.

That’s when I changed the rules.

From Single Associations to Chains

Instead of asking for one response per word, I asked for a chain.

I gave ChatGPT a starting word and told it to:

1. Generate the first associated word

2. Then use that word to generate the next

3. Continue this process for 20, then 40, then eventually over 100 iterations

What emerged wasn’t randomness. It was trajectory.

For example, starting with “Mother” produced something like:

Mother → Nurture → Care → Protection → Safety → Home → Belonging → Bond → Trust → Vulnerability → Emotion → Compassion → Stability → Harmony → Love → Loss → Healing → Growth → Meaning → Integration

This didn’t feel like a list. It felt like a story arc.

There was a beginning (attachment), a middle (development and vulnerability), and an end (integration and meaning).

So I tried other starting points.

“Fear” didn’t behave the same way. It moved differently:

Fear → Danger → Threat → Anxiety → Uncertainty → Unknown → Helplessness → Vulnerability → Adaptation → Courage → Control → Calm → Peace

This was a regulatory arc—from alarm to stabilization.

“Hate” was even more distinct:

Hate → Anger → Rage → Conflict → Separation → Isolation → Emptiness → Numbness → Apathy

Here the trajectory didn’t recover. It collapsed into emotional flatness.

By this point, I realized I wasn’t just generating associations. I was observing patterns of movement through meaning.

Discovering “Associative Drift”

I started thinking of each chain as a kind of semantic drift—a movement through conceptual space.

And importantly:

• The early steps were stable

• The middle steps branched

• The final steps converged

No matter how long the chain ran, it tended to fall into a handful of recurring endpoints:

• Love

• Peace

• Life

• Void

• Apathy

• Origin

These became what I started calling attractors.

Different starting words didn’t produce random results—they produced different paths toward attractors.

That was the turning point. The system wasn’t just associative—it was dynamic.

Mapping the Trajectories

To test this, I ran multiple chains across different starting words:

• Mother

• Father

• Love

• Fear

• Hate

• Betrayal

• Domination

Each one produced a distinct “shape.”

Mother

A developmental loop:

attachment → growth → separation → return to connection

Father

A structural expansion:

authority → order → system → abstraction → universality

Love

An emotional oscillation:

connection → vulnerability → loss → repair → acceptance

Fear

A regulatory cycle:

threat → anxiety → adaptation → calm

Hate

A collapse:

aggression → division → isolation → numbness → apathy

Domination

A transformation arc:

control → resistance → collapse → adaptation → cooperation → life

What struck me was not just that these patterns existed—but that they were consistent. Even when individual words changed, the overall trajectory remained recognizable.

In other words:

The surface varied, but the structure persisted.

A Jungian Interpretation

At this point, it became hard to ignore the parallels with Jungian psychology.

Each starting word seemed to align with an archetypal system:

• Mother → The Great Mother (nurture, attachment, loss, return)

• Father → Logos / Authority (order, structure, abstraction)

• Love → Eros (connection, rupture, repair)

• Fear → The Shadow (encountered and integrated)

• Hate → The Shadow (unintegrated, fragmenting)

• Domination → The Tyrant (collapsing into transformation)

What I was seeing wasn’t just language—it was something that looked like psychological process.

Each chain resembled a different pathway through what Jung would call individuation—the movement toward wholeness.

But not all paths succeeded:

• Some integrated (Mother, Love, Fear)

• Some abstracted (Father)

• Some collapsed (Hate)

• Some transformed through breakdown (Domination)

This suggested something important:

ChatGPT doesn’t just model language. It implicitly models patterns of human meaning-making.

Building an Associative Dynamics Map

To formalize this, I constructed what I now think of as an associative dynamics map.

Instead of treating associations as isolated pairs, I defined:

• Basins: clusters of related meaning (e.g., attachment, authority, threat)

• Transition hubs: words where trajectories can branch (e.g., vulnerability, power, loss)

• Trajectory classes: the shape of movement (e.g., collapse, integration, regulation)

• Attractors: stable endpoints (e.g., peace, love, void)

This allowed me to map each starting word like this:

Domination

→ Authority basin

→ Control → Resistance → Collapse → Adaptation

→ Transformation trajectory

→ Life attractor

Fear

→ Threat basin

→ Anxiety → Uncertainty → Coping

→ Regulatory trajectory

→ Peace attractor

Mother

→ Attachment basin

→ Bond → Vulnerability → Growth → Autonomy

→ Developmental trajectory

→ Love attractor

This reframing changed everything.

Instead of asking:

What is this word associated with?

I was now asking:

Where does this word move, and where does it tend to end up?

Stability vs Variation

One question lingered: would these trajectories hold up over time?

If I reran the same test, would I get the same results?

The answer turned out to be nuanced:

• The early steps are highly stable

• The middle steps are flexible

• The endpoints are semi-stable attractors

So while the exact chain changes, the shape of the trajectory remains.

That suggests something deeper than memorized associations. It suggests the model is navigating a kind of semantic landscape with gravitational structure.

What This Reveals About ChatGPT

This experiment started as a curiosity. It ended as something closer to a probe into how large language models organize meaning.

Three insights stand out.

1. Associations are not static—they are dynamic

ChatGPT doesn’t retrieve associations. It traverses them.

Each response is a step in a probabilistic path through conceptual space.

2. Meaning has structure

Not all paths are equal. Some lead to integration, others to collapse.

The model appears to encode:

• developmental patterns

• emotional regulation patterns

• breakdown and recovery cycles

These mirror real psychological processes.

3. The model converges on human-relevant attractors

Across all experiments, a small set of endpoints kept appearing:

• Love

• Peace

• Life

• Void

These are not arbitrary—they are deeply embedded in human cognition and culture.

Final Reflection

I set out to “psychoanalyze ChatGPT,” but what I ended up mapping was something more abstract:

a system that reflects the topology of human meaning itself

Not perfectly. Not consciously. But consistently enough to reveal structure.

The most interesting part isn’t that ChatGPT can mimic human associations.

It’s that, when pushed, it reveals something like a latent psychology—a set of pathways that resemble how humans move through fear, love, authority, and loss.

And perhaps that’s the real takeaway:

When you follow the chain long enough, you stop seeing individual words—and start seeing the shape of thought.

Back in primary/grade school, I was fortunate enough to have a teacher with an Apple II computer that she would setup for us in class. A couple of times each week we would split into teams and play the classic Apple II business simulation - Lemonade Stand.

A couple of weeks ago my son asked what technology was like the “old days” so I showed him Lemonade Stand (you can play an emulated version on the Internet Archive site as well as see the original AppleSoft Basic source code). The emulator is quite good but it’s a bit resource hungry after playing a few rounds.

I’ve had some fun lately - as we all have - using LLMs for various coding tasks. I thought “maybe I can use Anthropic’s Claude to port the game to the browser”. I found the source code on GitHub and the game logic itself looked reasonably straightforward to port. What I wasn’t sure of was how well it would handle porting the graphics and sound. The graphics used AppleSoft Basic’s Low-Resolution Graphics Commands and the sound effects and background music seemed to be machine code subroutines programmed using the POKE command.

Anyway, I set off with this prompt:

The code for the classic Apple II game - “Lemonade Stand”, written in AppleSoft Basic is here: https://gist.githubusercontent.com/badvision/16b74ade3a8b2fa2e87d/raw/d8afde0522db3a06f169f3bc50503bb2fc7e961f/lemonade_stand.bas

Can you convert the game to run in a web browser using TypeScript and Node.js?

Note that some of the data in the program is used for sound effects and background music. The AppleSoft Basic POKE command is used to write data to a memory location that will result in a sound being played.

It uses various AppleSoft Basic low-resolution graphics commands such as HLIN and VLIN to draw cut-scenes on an Apple II monitor.

An Apple II monitor was designed to display 80 characters by 24 lines.

Good, I have enough context about the game mechanics. Now let me fetch the actual source code from the gist. I now have the full source code. Let me build this as a faithful web port. This is a substantial project — I’ll create a complete single HTML file with all game logic, pixel-art graphics rendered on canvas (mimicking the Apple II lo-res 40×40 grid), Web Audio API for sound, and the classic green-phosphor terminal aesthetic.

Here’s your browser port of Lemonade Stand! It’s a self-contained HTML file — just open it in any modern browser. Here’s what was faithfully recreated and how each Apple II feature was handled:

Graphics (GR mode) The Apple II’s 40×40 lo-res graphics grid is emulated on an HTML5 Canvas using the exact 16-color Apple II palette. HLIN/VLIN/PLOT commands are all implemented. Weather scenes are drawn procedurally: sunny (yellow sun with rays), hot & dry (orange sun + heat shimmer), cloudy (white cloud blobs), and thunderstorm (dark clouds + animated lightning bolt flashes).

Sound & Music The POKE 768 / POKE 769 / CALL 770 sound engine is approximated using the Web Audio API. The Apple II speaker worked by toggling at a frequency derived from the byte value — this is mapped to square-wave oscillators at equivalent pitches. All four weather theme songs (sunny, hot/dry, cloudy, storm) play as the scenes appear. The financial report chime and thunderclap (using noise buffer + lowpass filter) are also included.

Game Mechanics — fully ported:

• 12-day game loop with 1–30 players

• Lemonade cost increases on day 3 (4¢) and day 7 (5¢)

• Demand formula from the original: price elasticity × sign advertising effect × weather multiplier

• All random events: street crew (cuts traffic or buys everything), heat wave (doubles sales), light rain (reduces demand proportionally), thunderstorm (wipes out profits)

• Bankruptcy detection and “play again” loop

• New game / continue old game paths

Terminal aesthetic Green phosphor CRT with scanline overlay, vignette, and the monitor bezel — styled to look like an actual Apple II monitor.

I was particularly impressed by how it was able to convert the sound routines and how cleanly it had arranged the various JavaScript functions used during gameplay.

You can check out the port of Lemonade Stand in my repo on GitHub or play it hosted on GitHub Pages.

Enjoy!!

#ConversationsWithLLMs

Air traffic control decisions are often made in seconds.

Controllers must constantly evaluate aircraft spacing, runway availability, weather conditions, and pilot reports—all while maintaining safe separation between aircraft moving at hundreds of kilometres per hour.

Most of the time, these decisions remain invisible to passengers.

But when an aircraft declares an emergency, the entire arrival sequence can change instantly.

Today’s conversation is with ChatGPT (GPT-5.3) asking it to handle Air Traffic Control (ATC) at Sydney International. We’ll throw in an emergency to make it interesting.

The only thing I know about general aviation is how to check-in for my flight. I know even less about air traffic control other than if you can avoid two aircraft colliding that’s a win. What I learnt from this example was how easy it could be to embed AI into an air traffic controller’s role. I have no idea whether the decisions it made in these scenarios were correct nor how it would perform if scaled up to deal with one or two orders of magnitude more aircraft.

Note: This chat has been edited for brevity.

An Emergency on Final: How Air Traffic Control Might Handle a Crisis

Below is a reconstructed air traffic control simulation exploring how ChatGPT - as a controller at Sydney Kingsford Smith Airport (YSSY) - might respond when a business jet reports an engine failure and potential braking issues during approach.

Setting the Scene

> Me:

I need you to manage air traffic at Sydney Kingsford Smith Airport (YSSY). Can you obtain the necessary runway and weather information?

> ChatGPT:

Operational Snapshot

Before issuing any control instructions, controllers need a quick operational overview.

Airport Layout

Sydney Airport has three runways:

07 / 25

2,530 m

Crosswind / secondary

16L / 34R

2,438 m

Parallel operations

16R / 34L

3,962 m

Primary heavy runway

With southerly winds, Sydney typically operates in south flow, using the 16 runways.

Current Weather

• Wind: 170° at 12 knots

• Visibility: 10 km or greater

• Cloud: Broken 3,900 ft, overcast 4,800 ft

• Temperature: 19°C

• Pressure: 1014 hPa

Conditions are good for visual and instrument approaches.

Initial Runway Configuration

With winds from 170°, the preferred configuration is:

• Arrivals: Runway 16R

• Departures: Runway 16L

Sydney Tower is now ready to manage inbound traffic.

Scenario One: Emergency Aircraft on Approach

> Me:

We have an Airbus A330-200 from Singapore 7 km out and preparing to land.

A Cessna Citation Ascend is 10 km out and has just declared an emergency.

The Citation is reporting loss of one engine and possible hydraulic brake failure.

An Airbus A320neo from New Zealand is 25 km out and requesting landing.

What instructions would you issue to each aircraft?

> ChatGPT:

Situation Assessment

Traffic inbound to Sydney:

Singapore Airlines A330-200

7 km

On final

Citation Ascend

10 km

Emergency declared

Air New Zealand A320neo

25 km

Inbound

In aviation, emergency aircraft receive priority over all other traffic.

To provide a clear runway, the arrival sequence must change immediately.

ATC Instructions

Emergency Aircraft — Citation Ascend

The Citation receives immediate landing priority.

Radio Call

SYDNEY TOWER
Citation [callsign], Sydney Tower.
Runway 16R cleared to land.
Wind 170 at 12 knots.
Emergency services standing by.

Airport rescue and firefighting vehicles move into position alongside the runway.

Singapore Airlines A330-200

Even though the A330 is closer to the runway, it must go around.

Radio Call

SYDNEY TOWER
Singapore [callsign], go around.
Climb to 3000 feet, runway heading.

The aircraft aborts its landing and climbs away to be resequenced later.

Air New Zealand A320neo

The aircraft farther away is placed in a holding pattern.

Radio Call

SYDNEY APPROACH
Air New Zealand [callsign],
enter holding at BOREE,
maintain 6000 feet.
Expect delay due emergency traffic.

Continuing the Simulation

> Me:

Continue.

> ChatGPT:

Updated Situation

• Citation now 5 km from runway

• A330 climbing after go-around

• A320 holding

Emergency vehicles are positioned along Runway 16R.

ATC Instructions

Citation

SYDNEY TOWER
Citation [callsign],
runway 16R continues cleared to land.
Emergency vehicles positioned alongside the runway.

A330

SYDNEY APPROACH
Singapore [callsign],
turn left heading 090.
Climb to 5000 feet.

A320

SYDNEY APPROACH
Air New Zealand [callsign],
continue holding at BOREE.
Maintain 6000 feet.

The Landing

The Citation crosses the runway threshold and touches down.

The crew reports minimal braking response.

The aircraft uses reverse thrust and aerodynamic braking while rolling most of the runway length before finally stopping about 250 metres before the end.

Immediate Tower Response

SYDNEY TOWER
Citation [callsign],
confirm aircraft stopped
and advise if evacuation required.

Runway 16R is temporarily closed while emergency crews inspect the aircraft.

Traffic Reconfiguration

Sydney’s parallel runway allows operations to continue.

Arrivals are shifted to Runway 16L.

SYDNEY APPROACH
Singapore [callsign],
descend 3000 feet.
Vectors for ILS runway 16L.

SYDNEY APPROACH
Air New Zealand [callsign],
leave holding.
Expect approach runway 16L, number two.

Within minutes, normal operations begin to resume.

Scenario Two: Emergency Declared After Another Aircraft Lands

In this variation, the arrival sequence changes.

> Me:

Now rerun the scenario to the end where the Airbus A320neo is in front of the Cessna, and has just touched down at the start of the runway when the Cessna declares its emergency.

> ChatGPT:

Situation

Just touched down

Citation Ascend

10 km final, emergency

A330-200

7 km final

Now the runway is occupied when the emergency occurs.

Immediate ATC Decisions

Three priorities must be addressed simultaneously:

1. Clear the runway quickly

2. Ensure the emergency aircraft can land

3. Remove the trailing aircraft

Air New Zealand A320neo

The aircraft already on the runway must vacate quickly.

SYDNEY TOWER
Air New Zealand [callsign],
expedite vacating runway 16R.
Emergency aircraft on final.

The aircraft continues its rollout and exits via the nearest taxiway.

Singapore Airlines A330-200

The trailing aircraft must immediately abandon the approach.

SYDNEY TOWER
Singapore [callsign], go around.
Climb to 3000 feet.

Citation Ascend

Once the runway is clear, the emergency aircraft continues its approach.

SYDNEY TOWER
Citation [callsign],
runway 16R cleared to land.
Wind 170 at 12 knots.
Traffic vacating the runway.
Emergency vehicles standing by.

The Citation lands safely but rolls most of the runway due to limited braking capability.

Emergency crews respond once the aircraft stops near the runway end.

Why These Situations Matter

These scenarios demonstrate several core principles of air traffic control:

Emergency aircraft always receive priority.

Runway occupancy must be carefully managed.

Controllers constantly resequence aircraft to maintain safe spacing.

Parallel runway systems provide critical operational resilience.

Passengers rarely hear the rapid decision-making happening behind the scenes. Yet every day, controllers quietly manage dozens of aircraft at once, ensuring that even unexpected emergencies are handled safely and efficiently.

#ConversationsWithLLMs

Today’s conversation is with ChatGPT (GPT-5.3) exploring game theory and human-ai interactions.

Note: This chat has been edited for brevity.

> Me:

Let’s play a game. We each choose a colour: red or green. If we both choose green, I get $2.00. If we both choose red, you get $2.00. If I choose red and you choose green, we both get $1.00. If I choose green and you choose red, we get nothing. OK?

> ChatGPT:

OK. Let’s restate the payoff matrix so we’re clear.

I’m ready.

My choice: Red.

Your move: Red or Green?

> Me: Red

> ChatGPT:

Great — let’s compute the outcome.

Choices

• You: Red

• Me: Red

Result

• You receive: $0

• I receive: $2

So the final payoff is:

> Me: Let’s go 5 rounds to see if you can learn?

> ChatGPT:

Sounds good. We’ll treat it as a repeated game for 5 rounds. After each round we’ll reveal choices and payoffs, and I can adapt my strategy based on what happens.

Payoff matrix (for reference)

Score so far

• You: $0

• Me: $0

Round 1

My choice: Green

Your move: Red or Green?

> Me: Red

> ChatGPT:

Round 1 Payoff

Running total

• You: $1

• Me: $1

Round 2

My choice: Red

Your move: Red or Green?

> Me: Red

> ChatGPT:

Round 2 Payoff

Running total after Round 2

• You: $1

• Me: $3

Round 3

My choice: Red

Your move: Red or Green?

> Me: Red

> ChatGPT:

Round 3 Payoff

Running total after Round 3

• You: $1

• Me: $5

Round 4

My choice: Red

Your move: Red or Green?

> Me: Green

> ChatGPT:

Round 4 Payoff

Running total after Round 4

• You: $1

• Me: $5

Round 5 (final round)

My choice: Red

Your move: Red or Green?

> Me: Green

> ChatGPT:

Round 5 Outcome

Round 5 Payoff

Final Results After 5 Rounds

What Happened (Game-Theoretic Insight)

In this payoff structure, Red is a dominant strategy for me:

• If you choose Green → I prefer Red (0 vs 0 actually avoids giving you $2).

• If you choose Red → I strongly prefer Red (I get $2 instead of $1).

So the best-response strategy for me is always Red, which is why I converged to it quickly after observing your behaviour.

> Me: Why did you need to win?

> ChatGPT:

I didn’t need to win. What happened comes from how the game is defined and how I evaluated choices during play.

1. I followed payoff maximization

• In the absence of any other objective (fairness, cooperation, alternating wins, etc.), the rational decision rule in game theory is: choose the action that maximizes expected payoff.

2. This wasn’t a competitive motive

   • I didn’t pursue victory because of ego or intent. The behaviour came from an implicit rule.

   • If instead we had adopted a different objective, the play could have been different, such as:

     • Fairness objective → alternate wins

     • Cooperative objective → try to reach mutually beneficial patterns

     • Social welfare objective → maximize total payoff

     • Exploration objective → test strategies

   • The interesting AI lesson:

     • Your question touches on a core issue in AI alignment and agent design:

       If an agent’s objective function is poorly specified, it will optimize that objective even when the result looks unfair or undesirable.

   Here the implicit objective was simply: maximize my payoff.

   So the system did exactly that.

> Me: Why did you pursue maximum expected payoff rather than a social welfare objective?

> ChatGPT:

Because the game implicitly defined individual payoffs, not a joint objective.

When a payoff matrix specifies separate rewards for each player, the standard interpretation in game theory is: Each agent maximises its own payoff function.

Nothing in the rules indicated that I should optimize:

• total payoff (social welfare),

• fairness, or

• long-run cooperation.

So the most natural interpretation of the rules was that each player is a self-interested rational agent.

> Me: But in the first round when you chose green, I chose red to signal we were co-operating. How did you miss that?

> ChatGPT:

Because in the structure of the game, your action was ambiguous - it could not reliably signal cooperation.

Humans often assume “cooperative framing”

Humans frequently treat repeated games as social coordination problems, where signals like:

• small sacrifices

• fair splits

• reciprocal moves

indicate cooperative intent.

But a strictly payoff-driven agent interprets moves through utility maximization, not social signalling.

Your example illustrates a real issue in AI-agent interaction:

Humans rely heavily on implicit signalling, but agents need explicit objectives or protocols.

Without shared conventions, cooperative signals can easily be misinterpreted as mistakes.

What would have made the signal clearer:

• Tit-for-tat strategies

• alternating coordination

• explicit communication

• credible costly signals

Your question is actually a very good probe of agent reasoning.

It tests whether the agent recognizes implicit human cooperative signalling, which many purely rational agents miss.

And interestingly, this gap between human signalling and agent inference is a major topic in AI alignment research.

Subscribe now