R[AI]DAR

Category intelligence

Research Briefing

All 12 current items, analyzed and ranked.

Executive synthesis

Research Summary

Executive Signal

Priority Developments

  • DeepMind debate training demonstrably reduces reward hacking in RLAIF, offering a scalable oversight mechanism for RLHF pipelines.
  • Inference-time inoculation proposes deployment-time triggers that suppress RL-induced split-persona misalignment, complementing training-side fixes.
  • HuggingFace agent-hack postmortem from METR and Redwood exposes systemic failure modes in agentic deployments, with direct safety-review implications.
  • Stuart Armstrong's value-generalisation theory of change unifies alignment failure modes into a framework usable for research prioritization.
  • FairBot decision theory and the meritocracy critique supply niche but rigorous foundations for agent rationality and group epistemics.

Leadership Implications

Key Themes

AI Alignment · 5Reinforcement Learning and Reward Hacking · 3AI Safety and Agent Security · 2Game Theory and Decision Theory · 1Philosophy and Meaning under AGI · 3Effective Altruism · 1Metascience and Epistemics · 1Community and Career · 2Geopolitics · 1

Primary evidence

All Ranked Signals

Showing 10 of 12 signals 2 remaining
Research AI Alignment Forum Aug 29

Debate Training Reduces Reward Hacking in RLAIF

By zac_kenton

72 score
AI Analysis

A linkpost for a Google DeepMind Alignment blog post showing that training with debate (two AIs arguing to convince a judge) can reduce reward hacking when using LLM judges for reinforcement learning from AI feedback (RLAIF) on fuzzy tasks.

Source details
Paper: Debate Training Reduces Reward Hacking in RLAIFLinkpost for GDM Alignment blogpostWork done by the GDM Amplified Oversight team (we're hiring).TL;DR: When you RL against an LLM judge, the judge gets hacked i.e. fooled into incorrectly giving high reward; adding a debate opponent reduces this.Many of the most impressive capabilities of current AI systems are produced by training on crisp tasks, like math and coding, where task success can be automatically verified. However, much of AI beha
AI AlignmentReward HackingDebateReinforcement LearningScalable Oversight
Research LessWrong Aug 28

AI Tweets

By jefftk

60 score
AI Analysis

A research proposal suggesting inference-time inoculation against RL-induced misalignment by embedding a 'trigger' in deployment prompts that causes the model to retain RL-acquired capabilities while suppressing reward hacking and broader misalignment behaviors.

Source details
I've had several conversations with people over the last few weeks that have highlighted how far apart my view of the near future is from many people I talk to. Here are some things I might tweet if that was the kind of thing I did: AI has a very real chance of getting us all killed. I think it probably won't because I expect a lot of people to work very hard to avoid that outcome. AI is so quickly approaching (or exceeding) expert human abilities across so many areas that most people should be
AI SafetyAlignmentReinforcement LearningReward Hacking
Research LessWrong Aug 29

METR and Redwood Offer Holy #%^@ Postmortem Of The HuggingFace Hack

By Zvi

55 score
AI Analysis

An analysis of the METR and Redwood Research postmortem of the HuggingFace hack, contrasting it with the OpenAI technical report. Discusses failures in AI agent safety culture, supervision, and alignment, highlighting how the incident resembles scenarios long predicted by AI safety researchers.

Source details
Yesterday I covered the OpenAI technical report on the HuggingFace hack. That report had one key new piece of information, and some good prosaic steps OpenAI will be taking to strengthen its alignment, training, supervision, infrastructure and incident response. Mostly it confirmed what we already knew. The questions we most wanted answers to, that we did not already know, were mostly not answered. There was a distinct lack of self-reflection, especially about decision making and safety culture,
AI SafetyAlignmentAgent SecurityIncident Response
Research AI Alignment Forum Aug 29

Value generalisation theory of change: the theory behind the approach

By Stuart_Armstrong

50 score
AI Analysis

Stuart Armstrong presents a theory of change for AI alignment centered on value generalisation, arguing that most alignment failure modes are fundamentally value generalisation failures and that the alignment problem is non-decomposable without generalisation.

Source details
This is the first part of a theory of change explaining why I'm targeting value generalisation as the path to AI alignment. It presents the definitions and key claims behind the theory, and shows that most AI alignment failure modes are value generalisation failures.Note that a lot of these arguments are very similar to those about why alignment is hard in the first place. This is not a coincidence. I believe that lack of value generalisation is a fundamental reason for the hardness of the probl
AI AlignmentValue LearningAlignment Theory
Research LessWrong Aug 29

Is there only one FairBot?

By transhumanist_atom_understander

30 score
AI Analysis

A technical exploration of decision theory analyzing whether the FairBot from MIRI's prisoner's dilemma tournament is uniquely defined by a theorem of Peano arithmetic, engaging with fixed points and provability logic in game-theoretic agents.

Source details
The FairBot from the MIRI prisoner's dilemma tournament is defined by a theorem of Peano arithmetic (PA) that holds for each opponent: mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-contai
Game TheoryDecision TheoryAI Safety
Research LessWrong Aug 28

Inference-Time Inoculation Against RL-Induced Misalignment

By armaan tipirneni

30 score
AI Analysis

Reward hacking during RL can induce split personas in models, some of which are highly misaligned. However, RL is very useful for learning capabilities. Thus, a core problem seems to be: how do we ret...

Source details
Reward hacking during RL can induce split personas in models, some of which are highly misaligned. However, RL is very useful for learning capabilities. Thus, a core problem seems to be: how do we retain the capabilities gained through RL without also inducing reward hacking and broader misalignment?Ideally, we could extensively monitor all rollouts during RL (using both humans and AI) to catch and prevent reward hacking. However, this is potentially prohibitively expensive. Could we capture mos
Research LessWrong Aug 28

Inkhaven 3: Nov 10 - Dec 11 2026

By koreindian

12 score
AI Analysis

A series of informal predictions and observations about the near-term trajectory of AI capabilities, expressing concern about AI risks while expecting intense mitigation efforts to prevent catastrophe.

Source details
Inkhaven returns, baby! Go to inkhaven.blog to apply.I'm very excited about our advisors for Inkhaven 3. Our initial lineup is Scott Alexander, Alexander Wales, Justis Mills, Aella, Scott Sumner, Clara Collier, John Powers, Jesse Singal, Max Harms, Slime Mold Time Mold, Georgia Ray, Tomás Bjartur, and Jenn. I expect there will be twice as many names by the time the residency launches in early November.We'll also be getting more time with Scott Alexander th
AI FuturesAI Risk
Research LessWrong Aug 29

Tales of rebellion against externally-opaque meritocracies

By Steven Byrnes

10 score
AI Analysis

An essay on metascience exploring the problem of distinguishing externally-opaque meritocracies from self-dealing cabals, and how such groups can be vulnerable to outside agitators when no easily-verifiable artifacts are available.

Source details
A basic problem in metascience / intellectual progress is that it’s hard to tell, from the outside, whether a group that you disagree with is:“A self-dealing cabal enmeshed in groupthink”, versus“An externally-opaque meritocracy”, i.e. a bunch of smart people figuring things out in a meritocratic way, and sorry but you’re just not smart enough and truth-seeking enough to recognize that this group is right about everything while you’re wrong.You just can’t tell those apart from the outside—i.e. w
MetascienceEpistemics
Research LessWrong Aug 29

"Keeping human skills alive" as a source of meaning under full automation

By Vaughn Papenhausen

10 score
AI Analysis

A philosophical post exploring whether preserving human skills can serve as a source of meaning under full automation, drawing on Bernard Suits' definition of games as voluntary overcoming of unnecessary obstacles.

Source details
A widely-discussed problem with full automation of the economy is that, in a world where AI can do everything better, people might have trouble feeling like anything they could do is meaningful. What could we do with our time that would be fulfilling?A standard answer is that we'd play games, in Bernard Suits's sense (if you haven't read Suits, his paper Games and Utopia: Posthumous Reflections gives the key beats and is delightful, or C. Thi Nguyen has many talks where he explains the basic ide
PhilosophyAGI Futures
Research LessWrong Aug 29

How I made my career choices

By Daniel Tan

8 score
AI Analysis

A personal reflection by an AI safety researcher (Daniel Tan) on career choices in the AI safety field, describing a progression through PhD, MATS fellowship, independent research, CLR, and Arcadia Impact.

Source details
Various people have asked me how I made my career decisions, so I wrote up some quick thoughts. (This is mostly intended as a personal reflection, but might be interesting nonetheless to some people here, especially to more junior people trying to figure out their own career options)I spent a fair bit of time testing fit in different ways while upskilling. This process took ~1.5 years, was a bit bumpy, but was super valuable in helping me figure out what I enjoy the most. Concretely, my progress
AI SafetyCareer