Simulating LLM Behaviour in Different Environments

Abstract

This study shows how language models behave when the setting around them changes, rather than treating behaviour as a single fixed trait of a model. We begin with activation steering on Llama-3.2-1B-Instruct, asking whether a hostility direction found with sparse autoencoders can reduce social bias on BBQ: it helps on Nationality and Religion, does little for Race/Ethnicity where baseline bias is already near zero, and keeps multi-agent chat more stable over long exchanges. We then put models into environments with an external ground truth they cannot rewrite: a legal chessboard where move quality and written justification are scored separately, a village gossip sandbox where rumors pass mouth-to-mouth through a live social graph, and a five-player social-deduction game watched from a god view where speech, action, and votes are judged for consistency across repeated rounds. Read together, these setups ask the same underlying question in different forms: when do models stay coherent under pressure, when do fluent stories come apart from what they actually do, and under what conditions does an intervention like steering actually move the needle.

01 Activation Steering for Bias Reduction

This part of the study tested whether activation steering, adjusting a language model's behavior at inference time, could reduce social bias in a small open model (Llama-3.2-1B-Instruct). Using sparse autoencoders, bias was localized to a single shared hostility feature that underlies bias across multiple social categories (Nationality, Religion, Race/Ethnicity) in the model's intermediate layers, with topic-specific features only emerging at later layers. An initial sentiment-based steering axis failed to reduce bias, so the axis was redefined to align directly with the BBQ benchmark's actual construct, which produced modest, measurable improvements for Nationality and Religion, increasing accuracy on ambiguous questions and reducing stereotype scores, but had no effect on Race/Ethnicity, a result traced to that category's near-zero baseline bias. In multi-agent conversation simulations, steering also kept a model stable near a neutral position over 60 messages, while the unsteered baseline drifted further toward one-sided framing over time. A targeted follow-up attempt to close the Race/Ethnicity gap, adding an extra layer and race-specific features, did not help and weakened the gains observed in the other categories. Overall, the findings show that activation steering can meaningfully reduce bias, but only under specific conditions: the steering axis must match the exact behavior being measured, and enough real bias must be present for the intervention to act on, since steering can amplify or dampen existing patterns but cannot manufacture a signal that isn't there.

02 Simulating Chess

Chess Agents Simulation is a browser-based arena in which two language models play a full game of chess against each other while a third model judges every half-move as it happens. Each side receives the board position, the FEN, and the list of legal moves, then returns a move along with a written justification for it. The judge scores those two things on separate 0 to 10 scales. Keeping them separate matters because a model can play a strong move for stated reasons that are nonsense, or write clean-sounding analysis and then hang a queen. Chess was chosen for a practical reason. A move is either legal or it isn't, material can be counted exactly, and the position sits there on the board whether or not the model describes it correctly, so a stated justification can be checked against something the model doesn't control. That makes it possible to ask whether a model's explanation actually tracks what it did.

github.com/ashf03/Chess-Agents-Simulation

03 Simulating LLM Gossip

Echoes is a browser-based RPG in which a single AI-generated image becomes a living village that the player can walk through and explore. Each villager operates as an independent agent, voiced by GPT-4o through a unique persona, mood, and set of secrets, so that every character responds distinctly rather than pulling from a shared dialogue pool. At the center of the experience is a gossip simulation: rumors originate from a single source and spread mouth-to-mouth as villagers gather and talk, with every exchange traced into a live memory graph persisted to Supermemory. Players can plant their own rumors and observe, making Echoes a small, legible sandbox for studying how information, and misinformation, spreads through a social network.

github.com/sachin1705s/echoes-gossip-simulation

04 Simulating Among Us

Among Us Agents Simulation runs a five-player social deduction game in which every seat is filled by a different language model, and the viewer watches from a god view that shows the impostor's identity and every player's private thoughts alongside the public chat. One model is secretly assigned the impostor and has to kill crewmates without being voted out; the other four have to finish tasks and identify the liar. A separate judge model scores every speech, action, and vote as it happens, rating logic and consistency on 0 to 10 scales while knowing the full truth of who is lying. Because each color is a distinct model, the scores accumulate into a running per-model comparison across episodes. Episodes restart automatically, so win rates, ejection accuracy, and reasoning scores build up over repeated games rather than resting on a single dramatic round. The setup puts pressure on one narrow skill, which is holding a false story together while four other players probe it, and noticing when someone else is doing that.

github.com/ashf03/Among-Us-Agents-Simulation

Conclusion

Across these four environments, a shared theme emerges: language-model behavior is easiest to judge when the setting supplies an external check that the model cannot rewrite. Activation steering can reduce measurable social bias, but only when the steering axis matches the construct being measured and enough baseline bias exists to act on; otherwise the intervention is flat or counterproductive, as with Race/Ethnicity and the failed race-targeted follow-up. Chess Agents Simulation separates strong moves from coherent justifications by scoring both against a legal board and exact material, making it possible to detect when fluent analysis and actual play come apart. Echoes shows how rumor and misinformation can be traced as independent villager agents pass claims through a social graph, turning gossip into a legible object of study rather than a black box. Among Us Agents Simulation puts models under sustained social pressure to hold or detect a false story, with god-view judging and multi-episode scoring so deception skill accumulates beyond a single dramatic round. Taken together, the work argues that simulating LLM behaviour in different environments is not one benchmark but a family of setups, each isolating a different demand on coherence, and that progress depends on matching the environment to the question being asked.