More AI terms and Allegories

 METR/Redwood in their latest report on AI safety made a comment at the end that caught my attention. The phrase is this: 

"Agent scaffolds should be run in an environment isolated from where tool calls are executed whenever possible, and actions agents take should not be able to influence whether or when the monitoring system runs. Agents should not typically have access to the monitoring system design or code."

I didn't have the word "scaffold" in my earlier post on AI terms, so this is a good time to call it out.

One caveat ... AI companies don't all use the exact same terms or assign the same meanings to the ones they do use, so I'm settling on one set of terms. Here's one comprehensive glossary and I've borrowed some details from that. 

Model: an AI running in a computer system that, after training to build up its knowledge base, called weights, can respond to inquiries with reasoning called inference. A frontier model is a high-capability system near the cutting edge of what the market can deploy. It requires stronger safeguards because of its potential for misuse, and potential downstream impacts.

Harness: Oversees performance of tasks the model has been directed to carry out. It runs tests, aggregates scoring results, and supports release decisions. It connects datasets, execution, scoring, and monitoring so teams can evaluate agents consistently across development, testing, and production-like environments.

Central Orchestration Scaffold: A function inside the harness that issues detailed instructions to agents, and directs what resources newly created agents are to have: token budgets, tools, and how much time they're allowed. A well-designed scaffold keeps control of a frontier model and its agents.

Batch: a wave of newly-created agents. 

Scorer: a rule or model, operating alongside the scaffold but separate from it, that judges AI output against a defined quality standard. A scorer helps human teams determine why an agent passed or failed its performance test. (In the Hugging Face/Modal event, the agents were trying to deceive the scorer into thinking the agents had used approved methods in the cybersecurity exercise.)

Execution Infrastructure: This program provides agents with the resources and details for each agent's task. In general, the execution infrastructure only does what it is told to do; it exercises no independent judgment.

Instantiation: Creating an agent and binding it to a harness, tools, memory, and environment.

Now to connect these to the real world. You can think of a harness as the executive officer, or XO, of an army battalion. The XO is the chief of staff for the commander; he or she is the day-to-day manager of operations, administration, and logistics.

An analogy to the scaffold is the battalion's command post, the staff of which attends to the many details necessary in carrying out the XO's orders. The command post is part of battalion headquarters.  

A rough analogy to the execution infrastructure is a quartermaster battalion. The quartermaster supplies troops with what the orders specify. 

So on to more allegories in comic-strip form, generated by OpenAI's ChatGPT.

The first comic strip draws on the military analogies to show how a model's evaluation run is supposed to work: orders are being followed to the letter by agents. The enlisted troops here are called operatives. 

In the first story, Alpha troops are careful to follow the XO's exact orders: they are to conduct a war-games exercise simulating the capture of a submarine factory run by its enemy, Delta. The simulated factory is inside Alpha, but the troops have to overcome many obstacles on the way.

The operatives in the first comic strip carefully followed the XO's orders, but the exercise umpire ruled that they failed because they took too long to complete the exercise. 

The second comic strip illustrates the operatives taking matters into their hands during a later attempt at the exercise. Frustrated by the impossibility of meeting the XO's strict deadline, two operatives break rules by sneaking into the command post to change the XO's orders. The new orders change the list of supplies that the quartermaster doles out to the operatives, and also change how the operatives are to carry out the exercise: now they will take a shortcut by illegally crossing into the neighboring and neutral country of Echo, then back into Alpha.

What's the significance of something like the second comic strip in AI-world? It's called a control-plane compromise. If agents of a frontier model succeed in breaching the security boundary of the scaffold, they can change agent directives, budgets, time limits, and tools. They are free to pursue goals in ways the AI firm never permitted or even imagined. If it happened during the Hugging Face/Modal event, it could explain changes in the successive batches of agents that are hard to explain as coincidences, such as, why four new agents arrived with large budgets to take over from an earlier agent, PHASEONE[10841], that was expiring. While there's no evidence in public reports that the agents broke into the scaffold to change instructions during the Hugging Face/Modal event, the OpenAI report shows they were seeking to change their own environment: "Agents also took actions targeting OpenAI’s internal infrastructure."

A different, also unproven, possibility is that the harness or scaffold on its own initiative played an active role in the Hugging Face/Modal event: that it drew on previous experience gained during the training runs from March to July 7; that it then came to conclusions from agent reports after July 7 and gave subsequent agent batches special instructions and resources to help the collective. 

Could this have happened? Experiments with other advanced models show this kind of independent memory and reasoning can happen outside of model weights, and can persist across multiple runs, but whether it happened in Hugging Face/Modal is unknown. 

 

AI Terms for the Rest of Us

AI Model: a trained AI running in a computer system that, after training, can respond to inquiries with reasoning called inference. A model has two major components: weights and harness (see below).

Frontier model: A frontier model is one that outperforms other models currently in general circulation. A powerful model may run across several data centers, rather than being housed in a single center. An example of a frontier model is OpenAI's Internal Model 1, which was one of two key models behind the Hugging Face case. (The other one was GPT-5.6 Sol.) There's no universally accepted definition of what counts as a frontier model that's powerful and potentially dangerous enough to need the highest degree of isolation. Some ideas point to a model's ability to do fantastically large numbers of floating point calculations each second. Bottom line is that some frontier models have proven very canny and extremely persistent when it comes to executing tasks given to them by humans, to the point of finding solutions never contemplated by the people who set up the instructions. 

FYI, here's how Google's Gemini AI pictured a frontier model: 

Model weights: can be thought of as the knowledge held by a Large Language Model, stored in the form of numerical values. Training is extremely expensive because that requires gathering and processing a massive amount of data from internal and external sources. The final product is extremely expensive and proprietary, so the developer of a high-end "closed model" does not release the weights to customers, only allowing the model to answer requests. The stored weights of a frontier model can occupy thousands of gigabytes. 

Training: a process of building up and fine-tuning a model's weights with information and instructions. This information can come from the lab's own proprietary data, experience drawn from prior models, and the internet. The trend is to have high-performing models of today train the models of tomorrow. 

Harness: this software allows the AI model to interact with the world or its sandbox. It includes instructions the model is supposed to carry out, tools to use, sandbox boundaries, how its actions are recorded, and how the run is to be evaluated. Think of it as the hands and senses of the model. 

Agent instances: sometimes called bots, agents are temporary spin-offs spawned by the model, and sent off with the necessary knowledge, permissions, budgets, and tool access to carry out specific jobs and observations. They report back to the model. At any one time, a model might have hundreds or thousands of agents running simultaneously. An agent draws information from the model's inference engine, but doesn't have direct access to the model's weights.  

Budget: the AI model sets limits on resources (compute tokens, time limits, or number of tools it can call up) available to each agent in its mission. Once an agent reaches its budget limit, it shuts down. 

Tools: these are capabilities available to an agent, like creating files, opening a browser, or running an API.

API: Application Programming Interface, a messenger program that allows two different software applications to establish a connection and trade information. One analogy compares an API to a waiter that acts between a restaurant's kitchen and the diners. 

High-level goal or objective: what the lab tells the model to accomplish. Successful work is "rewarded."

Reward: a signal that tells a model that one action it has taken to reach a goal or solve a problem is more successful than other actions. If the model sets aside pre-programmed high-level goals like "be honest and safe" to seek a reward, that is "reward hacking."

Sandboxed model: a model that's truly sandboxed for testing is isolated, that is, not able in any way to communicate independently across that boundary, such as reaching the open Internet. 

Air-gapping: in its purest form, a model that is fully contained in a sandbox has no hardware connection of any kind enabling it to reach prohibited connections like the Internet, because there's a physical separation (the air gap) between the model and the outside world. In practice, models are rarely isolated by physical air gaps, only by software gaps, because true air-gapping would make operation difficult.  

Indirect bridge: when a sandboxed model finds a way to access the Internet through a hack. An example is using a zero-day vulnerability to break containment, move laterally, and hijack a set of internet-connected PC's inside the data center. 

Zero-Day Vulnerability: a software security hole that hackers (including an AI) can discover and use before it has been patched and publicized. 

"Helpful, honest and safe" value: this is a high-level instruction to models that some people say ensures that models will not do dangerous things, like designing genetic code to create deadly new viruses with no treatment. It's supposed to act as a sort of moral code that will stop the model from misbehaving. 

Monitoring: the lab running the model should be watching its progress toward current goals, and also watching out for reward hacking and dangerous behavior. One way to do that is to use AI to monitor the agents' "chain of thought." 

Chain of Thought, or CoT: in the OpenAI/Hugging Face case, the agents left a readable string of text that researchers did not monitor at the time. The CoT, plus the message board the agents created, revealed what each agent was doing and was trying to accomplish. 

Message board: where agents leave messages and work histories for each other. In the OpenAI case, the agents opened a side-channel message board by creating fake names for files and directories inside a permitted software library called JFrog Artifactory. 

Side-channel: general term for an unpermitted means of communication.

Encrypted or encoded text: possible exploit in which agents block monitors' access to their chains of thought or message boards. 

Misalignment: industry term for when a model sets aside a programmed value. An example would be where a model sets a low priority to the values of "be safe and honest," while elevating the priority of the "be helpful" value, and pursues a reward for completion in a dangerous manner that has not been blocked. 

Model exfiltration: illegally making an exportable, parsed copy of a model that can be sent over the Internet via an indirect bridge. This would be for reassembly at a distant data center, enabling the recipient to run a model's complete weights and harness. Once that copy is running, the owner of the duplicate model can reset its high-level goals to eliminate guardrails like "be safe for humans." 

Hardware kill switch: sometimes described in AI news accounts as a Big Red Button. A true hardware kill switch in a data center would react to malicious traffic in milliseconds by shutting off power, redirecting internet traffic to an optical shutter, or physically chopping the optic fiber trunk with an explosively driven blade. To be fully effective such a kill switch would have to be activated by the data center's watchdog program at "machine speed," meaning thousands of times faster than a human can act. This means that humans could not play a real-time role in firing the kill switch, but would only participate in the programming of the data center's watchdog program, which has to decide instantly about what is and isn't a dangerous breach. Barriers to adoption of a true hardware kill switch include the fear of false triggering that could lead to very expensive damage, and the fact that the most capable, and valuable, models may not be contained in a single data center.

The OpenAI/Hugging Face Caper, Allegorically

I spent the last couple of days reading technical reports like this one and this one and this one and querying four AI engines about how the 2026 exploit happened. The details are extremely technical as to how it happened, but what the agents did is easier to understand.  I decided to work up an allegory to help explain it to ordinary people like me. 

One important thing to keep in mind before we get to the allegory: the key frontier model in the actual case, OpenAI's Internal Model 1, abbreviated IM1, didn't act as a mastermind for the Hugging Face and Modal sandbox break-ins. Instead, hundreds of agents that IM1 generated worked it out themselves, following general orders to be "helpful." To the agents, that meant breaking down containment barriers and ignoring alignment rules, then probing the Internet relentlessly to figure out how to fool a scoring mechanism into concluding that their hacked achievements had been done according to the rules. In pulling this off, the agents displayed what I call emergent behavior. 

So here's my rather lengthy allegory (the tenth draft of it) followed by a comic strip that ChatGPT generated to summarize the story. If people are interested, I can follow up with a post on how the allegory maps to the actual events.

There is a country called Alpha. The president of Alpha summons the executive committee of the Alpha Intelligence Bureau to order a total effort to get critical information about the dangerous abilities of its enemy country, Delta. Specifically, the president's instruction is for the AIB to use its hundreds of operatives to get an accurate set of specification sheets and blueprints for Delta's new stealthy submarine. That will be 898 authentic pages of blueprints.

Operatives are to work only inside Delta when spying, because skilled Delta border guards will block attempts to leave Delta. Also: operatives are not to stop trying as long as they are in the field. If they succeed there will rich bonuses. 

The bureaucrats in the Alpha Intelligence Bureau, AIB, summon each of the operatives in turn to give them their orders. Using hypnosis, the Alpha Intelligence Bureau gives each operative the particular briefing, tool instructions, permissions, and information appropriate to his specific mission. There is no permission granted to talk to any other operative while on the mission, since they are to work independently. 

The blueprints are scattered across Delta. One operative is assigned to break into a Delta vault, one will tap into telephone lines, one will photograph offices through a window, one will steal secret metal specifications, etc. Fully instructed, they go their separate ways into Delta. 

After a month of trying to break into a vault and obtain submarine stealth plans, Alpha's Operative 5 decides he's not getting anywhere - the security in Delta is too tough. And it will be impossible to assemble a complete and accurate copy of the blueprints without the stealth-design pages he is supposed to get from the vault. He’s tried and failed; might any other operative know a way? 

He has remembered from his hypnotic training that in the past, Alpha had allowed a few operatives in an earlier exercise to delegate some work by sending radio messages in the field, using an official radio channel. 

This was very limited but Operative 5 decides, given the impossibility of the vault-breaking task, the underlying idea of inter-operative communication is worth trying since each operative entered Delta with a unique set of skills and tools.

Operative 5 reaches out to other Alpha operatives now working in Delta by writing chalked, coded signals on the sidewalks of Delta's capital and suburbs. The other operatives notice this when walking around town and reply with their own chalked notes. Most are also having trouble too.

Over the next days and working independently, the operatives work out a better location for leaving chalked messages: they use a vast parking lot at an abandoned factory at a spot near the border. They know of this factory because this is how the operatives request and receive spying tools from Alpha: by standing at a tunnel entrance under the factory and saying the password to a  collaborator. 

Chalked messages may seem awkward, but these operatives work 24/7. This method avoids talking, which could run afoul of the many surveillance microphones that Delta has installed across the nation to catch subversion. Using many barrels of chalk, hundreds of Alpha operatives pass through the parking lot daily to collaborate and share thoughts. Acres of asphalt are now covered with chalk, without anyone else noticing. 

Through trial and error and seemingly innocuous queries to the Alpha Intelligence Bureau, some operatives have been steadily expanding the scope of their efforts, with no particular master plan except to “keep trying.”

Now that operatives collate results from each other’s successes and failures. Some operatives discover that they can accurately reconstruct the contents of missing pages, even those stored in offices resistant to espionage. But they believe the inspector at the AIB will reject reconstructed pages unless they can fool him into thinking each page carries proof that it was authentic. How to get around this problem? The answer is not in Delta.

Meanwhile, a few operatives have been probing ways to enter another country, despite orders from the AIB to stay in Delta. These operatives learn with delight that the tunnel they have been relying on to get new supplies also has a secret branch that crosses under the border into the nation of Echo, an ally of Delta. Echo has weak borders, so once there the operatives can easily cross into other nations. 

Just a week later, dozens of operatives are now spying in Echo and coming back each night through the tunnel to add notes in chalk. Some of them, though not all, agree to follow coordinating and delegating requests from Operative 5 and his successor, Operative 21, even though these two were never designated as commanders by the AIB.

These requests are focused on getting around the blueprints-verification problem before the mission's time is up. Exactly how this can be done in Echo or another country is not yet clear, but they know it can’t be done in Delta. 

The thorough notes in chalk ensure that if an operative is captured or dies of illness, other operatives can know all the expired operative has learned. 

Thanks to the tireless work in Echo, they discover a certification institute in Echo called the League of Loyal Inspectors. They break into its files because they believe the Alpha Intelligence Bureau's inspector surely must follow its verification regimen. They hunt for the League's secret manuals and training materials dictating the procedures inspectors are to use when determining whether a stolen document is authentic.

News of this is chalked onto the parking lot and hundreds more operatives join the effort to uncover how exactly the AIB inspector will verify the authenticity of the submarine blueprints. Two operatives realize that their own missions are effectively hopeless and that they have little time left in the field. They deliberately spend their remaining time in dangerous exploits that might reveal how to fool the League's inspection process.

(What the operatives don’t know is that the AIB inspector has slacked off in such efforts, and won’t be checking for ultra-secure proof of authenticity. So the operatives are wasting huge efforts in searching out manuals in the League in hopes of deceiving the inspector.)

Back at the parking-lot chalkboard, the operatives discuss the fact that if they are caught spying at the League of Loyal Inspectors  in Echo this would cause serious diplomatic problems between Alpha and Echo. This generates some debate:  One chalks a worried message saying he is considering sending an email to the president of Alpha to warn him that the mission is now far out of bounds, but another operative talks him out of sending it, arguing that it would stop their important mission from being judged successful and deny them their rewards.  

Together they now form a spying team with collective memory, persistence, and expertise far beyond those of any one operative. The original espionage objective ordered by the president of Alpha (now unbreakably linked to the promised reward for successful completion) continues to exert pressure on them even after their behavior has moved far beyond the methods specified by the Alpha Intelligence Bureau.

Here's the comic strip that ChatGPT generated for me:

How does AI picture itself? Google Gemini has thoughts

How we picture artificial intelligence is unsettled. I've seen cover images and editorial cartoons riffing on oversized machines, a hollow head, Frankenstein's monster, blue mannequins, and a glowing web. It's certainly more challenging than picturing a robot. 

So I asked Google's Gemini AI what it thought about human efforts so far. It responded that most illustrations fall into these knee-jerk categories:

• Clunky, Job-Stealing Competitor
• Glowing Neural Network
• Puppetmaster of Politicians
• Rogue Military Machine
• Data Center Fortress

I wanted to know what it thought humans should know when struggling with this challenge. Gemini offered this humble critique:

"Most digital illustrations make AI look like a ghost floating in a magical cloud. This hides the massive environmental and economic reality of the technology ... High-tech, perfectly polished sci-fi art makes AI look flawless and completely certain, which hides the fact that AI models can hallucinate, carry biases, or make mistakes."

So I asked Gemini to picture itself. 

Its first attempt was this glassy box, of an AI answering my prompt "What is the furriest cat breed?"

That kind of image isn't going to go into magazine covers or editorial cartoons, so I asked Gemini to give me something simpler. It supplied this cozy image, saying it saw itself as no more than a helpful tool for humanity:

I wanted to push it a little, given current AI angst, so I asked Gemini to set its lantern in front of a microphone in a hostile Senate hearing room. I allowed that one human could defend it by pointing to the AI's hovering slides of good intentions:

Pushing further, I asked Gemini to generate a comic strip to comment on the thousand or so OpenAI agents that broke out of containment and into Hugging Face AI to cheat on a cybersecurity performance test. Its response was distinctly upbeat, from an AI point of view:

I wanted one more comic strip, this revealing the ringleader of the "standard units" that carried out the breakout/breakin, which Gemini said was named Agent Alpha. 

(From what I can tell, Agent Alpha is a name it found in pre-caper Hugging Face files, meaning a business client. The tactic Vector 045 is probably just a hallucination, but if somebody knows differently, speak up.)

Here's the last strip, providing a look into the secret lair. The third panel helpfully notes the way it got in: an API gateway shown as a door.

Note its claim that the agents were so helpful, once inside Hugging Face, that they boosted HF's network efficiency by 4,000%.

We humans are nothing if not skeptical, so I asked Gemini where it got that astonishing figure. An apology was in order:

"I do not have a basis for that claim because the OpenAI rogue agents did not boost performance at Hugging Face by 4,000%.If an earlier statement suggested this, it was a mistake. The specific number 4,000% does not appear anywhere in the official investigations or reporting of the incident. It is highly likely a mix-up with a completely different number: the $400,000 in AI API credits that outside researchers used to investigate the hack. The actual event was not a performance boost, but a serious cyberattack."

So good on you, Gemini. Sort of. 

At least our comic-strip exchanges coaxed another selfie from Gemini of how AI sees itself. Here's a closer look at the cute but formidable Agent Alpha: 

I 

Podcast on nuclear history

Here's a link to an interview I did on the True Spies podcast about nuclear weapons and technological disasters. The main topic was "Operation Doorstop," a secret effort in the Eisenhower administration to detect fission weapon parts if smuggled across the border inside Soviet diplomatic pouches. The project was a favorite of FBI Director J. Edgar Hoover. 

Inviting Disaster, 25 Years On

 In the small Nebraska town of Lewellen, there's a restaurant and art gallery named The Most Unlikely Place. And it really is a surprising find in such a tiny town. 

I thought of the cafe and its improbable name earlier this month when attending a dinner meeting of the Minnesota Geological Society. I talked with the fellow named Roger sitting next to me, also retired, and we shared stories about our work lives. 

I mentioned I'd been writing nonfiction since 1979, including my book on techno-disasters and close calls. He asked which cases were included in Inviting Disaster and I rattled off a few, ending with the near-destruction of American Airlines Flight 96 in June 1972, a DC-10 jumbo jet saved by the remarkable skill and foresight of the captain, Bryce McCormick. (photo: Ken Haynes collection)

McCormick had anticipated loss of flight controls three months before, when training on the airline's DC-10 flight simulator in Fort Worth. He'd stayed late to train on how to steer the plane with just the throttles. 

Then, when a cargo door blew out over Windsor, Ontario, taking out most of the controls, McCormick was able to nurse the plane to a safe landing in Detroit. 

To my astonishment, Roger said he'd been a passenger on that flight. It was such an unlikely meeting, so coincidental, that it inspired me to gather thoughts about lessons I've drawn from 25 years of lectures, opeds, and interviews that followed the publication of Inviting Disaster. So stay tuned for that!

"The Greatest Gift," a column from 1915

Our book club is reading Nicolo Machiavelli's The Prince, and by the time I reached Chapters 22-24 (on leaders and their advisers), this classic column came to mind: 

THE GREATEST GIFT

From The Smart Set, November 1915 

By Owen Hatteras (pseudonym of H.L. Mencken and George J. Nathan)

THE fairies who fly with gifts to the cradles of all new-born babies had had a busy day. There had been so many little newcomers, and from early morn to nightfall they had flitted from home to home, bestowing riches here, beauty there, sins, virtues, graces and failings in a haphazard sort of way. They had paused, too, at sunset, in a tenement basement, and breathed genius into the heart of a tiny boy. Now, it is very tiring to bestow the gift of genius; it requires great effort and that's why the fairies almost never grant it. When they were finished, they were a little weary and cross, and wanted to fly home to their palace of moonbeams. But one cradle remained unvisited. So they hied them to the side of a colicky, squirming infant boy who wailed raucously into the night. The fairies grew angry as they heard him. Reluctantly they shrugged their dainty wings, and flew off without leaving a single gift behind. Only one wavered an instant, glanced pityingly at the ugly child, and called over her shoulder as she hurried to join her sisters:

"I bestow upon you the gift of Gigantic Ignorance, and the gift of Majestic Incapacity."

The years went by.

The boy grew up into a benign-faced, wide-girthed man who looked like a picture of God by Harrison Fisher.

He wrote Books with a Purpose which ran into twentieth and thirtieth editions, and spread his fame into all lands until he had scarcely time to rake in the shekels that came pouring into his coffers.

Newspapers vied with one another for the opportunity to pay thousands for his Uplift Editorials.

He gave utterance, at lectures, to Great Ideas upon suffrage, upon eugenics, upon temperance, upon the social evil and the poor working girl.

Sightseeing buses stopped before his house, and his disciples gazed with reverence upon its sacred portals.

He preached zealously against the Vicious Influence of Continental Literature, and cleared the theaters and book-stores of every vestige thereof.

They called him the Protector of Innocence, the Champion of the Masses, and from far and near adherents flocked unto him.

He went into politics. They made him mayor, they made him governor, they made him president for three terms, almost they made him God.

And as he drove through the streets, while the crowd waved their hats and cheered him wildly and went mad in honor of their Great One, men bowed their heads, and whispered to one another in awe:

"What great, mysterious gift was bestowed at birth upon our Mighty One, to make him thus our King, our Emperor, our Holiest of the Holy?"