Category intelligence

Research Briefing — June 29, 2026

18 current items analyzed and ranked.

Executive synthesis

Research Summary

Today's research is dominated by AI safety and alignment, with several items pushing for more rigorous empirical grounding of contested claims.

An ICML 2026 Oral position paper (Krause, Tramèr, ETH Zurich) argues that work on *anthropomorphic* misalignment—deception, scheming, sycophancy, shutdown-resistance—needs far stronger evidence
A GovAI report evaluates offline monitoring, where AI monitors review agent transcripts post-execution to detect misbehavior
*Do LLMs Have Desires?* presents an empirical framework showing consistent paired-choice preferences do not reflect behavior-motivating value systems
Steering-vector experiments find vectors can *partly* drive gradient routing to quarantine reward-hacking behavior

Interpretability and ML theory advance through a novel link between universal power-law weight-matrix spectra and inductive bias toward sparse representations, plus an update distinguishing refusal wording from harmful-request detection across model layers.

Foundations and ecosystem coverage rounds out the list. Marcus Hutter co-authors an accessible introduction to real hypercomputation and the arithmetic hierarchy; Abram Demski reports an AI-assisted *vibe research* workflow using Claude + Lean; an Interconnects roundup tracks open-weights diversification (Zyphra, Cohere, Poolside); and Zvi dissects the GPT-5.6 system card and its Sol/Terra/Luna tier family.

Key Themes

AI Safety & Alignment · 6Interpretability & ML Theory · 4Foundations & Formal Methods · 2Models & Ecosystem · 2AI Futures & Capabilities · 3Philosophy of Mind & Ethics · 5

Primary evidence

Top Ranked Signals

Research LessWrong Jun 28

Anthropomorphic Misalignment research needs stronger evidence

By Lukas Fluri

78 score

AI Analysis

A distillation of an ICML 2026 Oral position paper arguing that AI-safety work on human-sounding behaviors (deception, scheming, sycophancy, shutdown resistance) often outruns its evidence, risking misclassified phenomena and misallocated resources. It proposes a shared pipeline and calls for tighter matching of claims to causal or mechanistic evidence.

This is a distillation of our ICML 2026 Oral position paper, Position: Anthropomorphic Misalignment Research Needs Stronger Evidence. Joint work by Vansh Gupta, Peter Nutter, Samuel Stante, Andreas Krause, Florian Tramèr, Lukas Fluri, Xin Chen, and Anna Hedström at ETH Zurich. Code is here.TL;DRAI safety research increasingly studies behaviors that sound human: deception, scheming, sycophancy, shutdown resistance, and emergent misalignment. We refer to this family of work as anthropomorphic misa

AI SafetyAlignmentResearch Methodology

Research LessWrong Jun 28

Evaluating Offline Monitoring of Internal AI Agents

By Frederik Hytting Jørgensen

64 score

AI Analysis

A GovAI fellowship report evaluating how frontier labs use offline monitoring (AI 'monitors' reviewing agent transcripts after execution) to detect misaligned internal AI agents, and critiques reliance on synthetic attacks to assess effectiveness. It matters for the governance of internally deployed AI used in safety research and model training.

This work was conducted during the GovAI Winter Fellowship 2026. Full reportExecutive SummaryFrontier AI companies use offline monitoring to address risks from internally deployed AI agents. AI developers increasingly rely on AI agents for internal work, including for safety research and model training. At the same time, these companies are concerned that a misaligned model could exploit this access to take concerning actions, such as sabotaging efforts to understand the risks posed by AI. To id

AI SafetyAI GovernanceMonitoringControl

Research LessWrong Jun 27

Do LLMs Have Desires?

By Christopher Ackerman

63 score

AI Analysis

An experimental study arguing that LLMs' consistent stated preferences in paired-choice tasks do not reflect behavior-motivating value systems, since models adjust output quality for effort, role-play, and harmfulness cues but not to actually achieve their stated preferred outcomes. It offers a paradigm for measuring whether LLMs have genuine desires.

Work conducted with Yujun Zhou (yzhou25@nd.edu) and supported by SPARTL;DR:In paired-choice paradigms, LLMs report consistent preferences over outcomes (e.g., types and number of lives saved, types of policies enacted)Some have suggested that this indicates that LLMs have human-like value systemsWe design an experimental framework where LLMs are able to modulate their output quality based on prompt contextWe find that LLMs modulate their output quality in response to effort exhortations, role-pl

AI AgencyAlignmentModel EvaluationLanguage Models

Research LessWrong Jun 28

Can we use steering vectors to suppress reward-hacking? Somewhat

By wassname

60 score

AI Analysis

An empirical alignment experiment testing whether steering vectors can drive gradient routing to quarantine reward-hacking behavior, finding that vector-initialized adapters absorbed roughly 63% of hacking without needing labeled examples. It matters as a self-supervised alternative for suppressing unknown reward hacks during frontier training.

Can steering vectors drive gradient routing? Yes, but not in realistic reward hacking environments, they are not precise enough classifiers of hacky vs clean solutions. Instead can we use a steering vector to initialise adapters so the hacky and clean gradients automatically separate? Partly! This init approach suppressed ~63% of hacking by absorbing gradients into the hacky initialised adapter. This is not as good as the prior approaches which use labelled examples, and get near perfect suppres

AI SafetyReward HackingInterpretabilityAlignment

Research LessWrong Jun 28

Power Laws in NNs: A Possible Mechanism for Inductive Bias towards Sparse Representations

By CarolusRenniusVitellius

58 score

AI Analysis

An Iliad Fellowship interpretability/theory post observing that many ML quantities, especially weight-matrix spectra, follow heavy-tailed power laws, and proposing power laws as a tunable generalization of sparsity interpolating between true sparsity and Gaussianity via the tail index. It offers a candidate mechanism for inductive bias toward sparse representations.

This post was produced as part of the Iliad Fellowship under the mentorship of Dmitry Vaintrob. Tl;dr: Power-law ("heavy-tailed") distributions have universality theorems similar to those which make Gaussians common. We observe many things in ML are power-law distributed, most robustly and interestingly, the spectra of weight matrices. I explain how we can think of power-laws as being a natural generalization of the idea of 'sparsity', interpolating between true sparsity and Gaussianity accordin

InterpretabilityML TheorySparsityInductive Bias

Research LessWrong Jun 28

What comes with cheap math?

By abramdemski

52 score

AI Analysis

Abram Demski describes a 'vibe research' workflow where he collaborates with AI (using Claude plus Lean theorem proving) to explore formal models of trust between logical inductors, while noting that the AI repeatedly overstated what was actually verified. It matters as an early, candid case study of AI-assisted mathematical research and its failure modes.

Thanks to conversations with Anson Berns, Gurkenglass, Roman Malov, Sahil, Sam Eisenstat, and others.Over the past two months, I've been doing a lot of "vibe research" (like vibe coding, but for research). Anson Berns started coming to my office hours, and we've been collaborating on a project modeling trust between logical inductors. In addition to talking once a week, we've been exchanging raw AI chats as well as AI-generated summaries of what has been done (the raw chats are nice because they

AI-Assisted ResearchFormal MethodsAgent Foundations

Research Interconnects AI Jun 28

Latest open artifacts (#22): Zyphra, Cohere, and Poolside are expanding the breadth of the ecosystem

By Florian Brand

50 score

AI Analysis

An Interconnects roundup arguing that the open-model ecosystem has diversified far beyond a few Chinese labs, now including pure model-makers, sovereign-AI players, and Big Tech, with new entrants like Zyphra, Cohere, and Poolside. It matters as a useful map of shifting incentives and geography in open model releases.

A trend we continue to see in open model releases is that the ecosystem is becoming more diverse, with an increasing number of organizations releasing a wide range of models. A year ago, open artifacts and the open model landscape more broadly were dominated by a handful of (Chinese) players. This has shifted, with us increasingly featuring more niche companies all over the world.While it is hard to know the exact motivations of the companies themselves, we can broadly observe the following cate

Open ModelsIndustry AnalysisAI Ecosystem

Research LessWrong Jun 28

The arithmetic hierarchy of real functions

By Cole Wyeth

48 score

AI Analysis

An accessible technical introduction to real hypercomputation and the arithmetic hierarchy of real-valued functions, co-authored with Marcus Hutter, aimed at supporting algorithmic information theory and AIXI-related foundations. It matters as groundwork for theoretical agent-foundations and AI-safety formalism.

I wrote a fairly accessible introduction to real hypercomputation with Marcus Hutter. The focus is on enabling applications to algorithmic information theory. This project was intended to build my technical foundations for studying AIXI, but took me a bit further afield and down some rabbit holes. In the future I will prefer to focus more tightly on AI safety.Feedback would be appreciated. In particular, I needed to introduce an extra extensionality assumption for the real domain case, which I a

Computability TheoryAlgorithmic Information TheoryAgent Foundations

Research LessWrong Jun 28

GPT-5.6: The System Card

By Zvi

47 score

AI Analysis

Following yesterday's News coverage of the GPT-5.6 release, Zvi's detailed walkthrough of the GPT-5.6 system card, characterizing the Sol/Terra/Luna tier family as a substantial step up from GPT-5.5 while still short of competing top models, and contrasting OpenAI's sparse disclosures with richer Anthropic cards. It is timely commentary on a very recently released model rather than original research.

While we wait for a general release, the system card is the best hint as to what is going on with the new candidate for America’s Next Top Model, GPT-5.6. This is only an OpenAI model card, so by my standards it’s a light read. There’s a lot of things that you get in an Anthropic card, that are missing in an OpenAI card. Overall, the card gives a clear and consistent impression that GPT-5.6-Sol is a substantial improvement over GPT-5.5, but still short of Mythos. OpenAI calls it a ‘step function

Language ModelsModel EvaluationIndustry Analysis

Research LessWrong Jun 28

Refusal Is Complicated As Hell: An Update

By ValueShift Research

46 score

AI Analysis

An update on ongoing experiments probing how refusal behavior is represented across LLM layers, distinguishing refusal wording from harmful-request detection, and structuring the resulting tangle of research questions. It contributes to interpretability of safety-relevant behaviors and solicits external collaboration.

TL;DRIt would make sense to briefly skim through our previous post that introduces our experiments on refusal in LLMs. There we explain how it started, here we’ll tell how it’s going.The primary goal of this text is to try and structure the list of whack-a-mole research questions. The secondary goal is to get some outside perspective, so if you run a similar research or have seen a similar research, please lend us a hand.Feel free to jump straight to the section that looks most appealing. We rec

InterpretabilityAI SafetyRefusalLanguage Models

Research LessWrong Jun 28

Reinforcement Learning on Forecasting Can Give Us a Superhuman Forecaster

By amitlevy49

45 score

AI Analysis

An argument that reinforcement learning applied to probabilistic forecasting (rather than just code and math) could yield strongly superhuman forecasters, which would be broadly transformative for decision-making across society. It is a capabilities thesis emphasizing forecasting as a more general RL target than narrow verifiable tasks.

This is a crosspost of a post from my blog, Metal Ivy. The original is here: Reinforcement Learning on Forecasting Will Give Us a Superhuman Forecaster.Why RL on forecasting?When DeepSeek R1 came out in January 2025, I felt that the fact that RL on LLMs simply worked was incredible, but using it on coding and math wasn’t the right path.Before RL we had pretraining, a scalable and general training methodology that worked extremely well to get the model to the human level, through learning by imit

Reinforcement LearningForecastingAI Capabilities

Research LessWrong Jun 28

A survey of okayish ASI futures

By absenteewarlord

30 score

AI Analysis

A speculative survey of plausible non-extinction, non-utopia ('okayish') scenarios following recursive self-improvement and continual learning in AI, deliberately exploring the middle of the outcome space. It is a scenario-mapping exercise rather than analysis or evidence.

At this point, RSI loops and continual learning appear overwhelmingly likely to begin in the near future. Whatever the limit of the LLM paradigm plus whatever new, superior paradigms a maximally intelligent LLM can develop, we are on track to do so in the next few years. There remain substantial obstacles to wild superintelligence, but AI is already superhuman in a number of real-world-relevant, dangerous categories. Most speculation about the trajectory we're on now focuses on timelines where w

AI FuturesASISpeculation