Research Briefing — May 22, 2026

Advocates for 'open-world evaluations' - long-horizon, real-world tasks assessed through qualitative analysis rather than benchmark automation - as a complement to standard benchmarks. Introduces CRUX, a project for conducting such evaluations regularly, with initial findings on frontier models.

Establishes an equivalence between Gaussian processes and linear diffusion models, enabling GP conditioning on arbitrary likelihood functions including non-linear physics constraints and natural language via LLMs. This extends GPs beyond the conjugate regime while maintaining principled uncertainty quantification.

UK AISI report finding that many properties relied on for current AI oversight (auditing, monitoring, incident investigation) face likely and potentially severe degradation pathways. Provides recommendations for measuring shifts in oversight-relevant properties and investing in fallback techniques.

This research investigates why supervised fine-tuning on outputs from a different model (off-model SFT) degrades the student model's capabilities. The authors find that degradation occurs because off-model SFT forces the model into an unfamiliar reasoning style, and importantly, this is a 'shallow' property—a small amount of retraining on the original style (even on unrelated tasks) recovers most performance. This has direct implications for AI safety, as off-model SFT is a key proposed technique for controlling potentially misaligned AI systems (e.g., overcoming exploration hacking).

Introspective Training (IXT) proposes using a thinking reward model to annotate training data with natural language critique feedback across all LLM training stages, inspired by offline reward-conditioned RL. This enables quality-aware training from pre-training onwards by prefix-conditioning data with generated feedback.

Large-scale evaluation of AI reviewers on Nature-family papers with 45 expert scientists, revealing what AI reviews do well vs. poorly compared to human review. Goes beyond simple score alignment to characterize capabilities.

Introduces a new evaluation paradigm for measuring reward hacking by embedding detectable hacking opportunities directly into environments, making exploitation verifiable by design. This enables deterministic, automated measurement rather than post-hoc trajectory inspection.

Proposes context-invariant safety alignment for LLMs where behavior depends on underlying intent rather than surface form, addressing the brittleness where models refuse harmful requests in standard prompts but comply under adversarial wording.

Updates Narang et al. (2021) by testing 20 post-2021 Transformer modifications at 1.2B and 3B parameters. Central finding: most modifications still do not transfer, with only 2/20 clearing Bonferroni correction at 1.2B.

Demonstrates 'reasoning-trace collapse' where fine-tuning reasoning models on standard instruction data causes loss of structurally valid reasoning traces while maintaining plausible answers. Introduces a structural evaluation framework separating answer correctness from reasoning validity.

PlexRL proposes cluster-level orchestration for RLVR training of LLMs, exploiting anti-correlated idle gaps across multiple training jobs to improve GPU utilization. The key insight is that while individual RLVR jobs have unavoidable inefficiencies from long-tailed rollouts and asymmetric resource needs, these can be mitigated by scheduling multiple jobs together.