R[AI]DAR

Category intelligence

Research Briefing

All 124 current items, analyzed and ranked.

Executive synthesis

Research Summary

Executive Signal

  • The batch advances agent capabilities and evaluation rigor across digital and physical environments, with computer-use benchmarks (RecreationWorld, OSWorld-Pro), security probes (MobileCybench, APort Vault), and training pipelines (CodeMidas, DexTacWAM) setting new standards.

Priority Developments

  • Computer-use benchmarks shift from end-state to process and hybrid settings: RecreationWorld (5-platform GUI+code) and OSWorld-Pro (subgoal decomposition) jointly raise the rigor bar for agent evaluation.
  • AI security evaluations mature: MobileCybench (executable vulnerability probes) and APort Vault (real CTF payment authorization) provide ecologically valid benchmarks for agent risks in production.
  • Agentic training infrastructure scales: CodeMidas derives RL environments from existing codebases, while DexTacWAM shows a 32-point empirical gain on dexterous bimanual manipulation via tactile fusion.
  • Alignment research yields actionable findings: Story Imprinting quantifies trait absorption and Alignment Midtraining Cracks stress-tests a recently proposed alignment stage with empirical results.
  • Foundational model advances: RetroChimera (Nature) delivers open-source retrosynthesis SOTA and Complex KDA provides a principled linear-attention improvement.

Leadership Implications

  • Treat CUA benchmarking as a moving target; pilot RecreationWorld/OSWorld-Pro for vendor diligence and budget for process-aware evaluations in 2026 procurement.
  • Operationalize agent safety gates with MobileCybench/APort Vault before deploying payment, code-execution, or autonomous browsing workflows in production.

Key Themes

Agentic Systems & Tool Use · 11AI Safety and Security · 1Language Models · 1Agents and Computer Use · 8Robotics & Embodiment · 9Benchmarks & Evaluation · 6Reinforcement Learning for Agents · 5World Models · 7AI Safety and Alignment · 5AI Alignment · 3

Primary evidence

All Ranked Signals

Showing 10 of 124 signals 2 remaining
Research Hugging Face Papers Sep 21

RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

By Shuai Bai, Jiayong Deng, Yikun Fu, Chang Gao, Xuhao Hu, Mianqiu Huang, Yizhen Jiang, Yuheng Jing, Dehui Kong, Keliang Li, Ning Li, Wanli Li, Dayiheng Liu, Dunjie Lu, Changwei Luo, Que Shen, Zheyuan Wang, Zijian Wang, Jie Wu, Gao Wu, Zhihui Xie, Rui Xie, Haiyang Xu, An Yang, Jiakang Yuan, Yanming Zhang, Jiajun Zhang, Xi Zhang, Zhenru Zhang, Zhuo Zhen, Mingkang Zhu, Bowen Zhou

82 score
AI Analysis

RecreationWorld is a five-platform (Ubuntu, macOS, Windows, Android, Web) benchmark for hybrid computer-use agents that must autonomously interleave GUI exploration, software coding, and visual verification to recreate a running reference. It provides reproducible environments and a unified harness with native GUI control plus coding tools.

Source details
Computer-use agents (CUAs) have advanced along two separate lines: graphical interaction and software development through code and the command line. Real digital work requires both, interleaved rather than stacked end to end. We study hybrid CUAs that autonomously decide when to explore an interface, implement software, and run and visually verify their artifacts. We introduce RecreationWorld, a five-platform framework built around recreation: given a running reference, an agent must discover it
AgentsComputer UseBenchmarkMultimodal
Research AlphaXiv Trending Sep 21

MobileCybench: Evaluating Agent Vulnerability Discovery via Executable Probes

By Andy K. Zhang, Ava Huang, Joey Ji, Wai Han, Thomas Qin, Nardos Demilew, Michael Tian-Yue Liu, Brian Song, Riya Dulepet, Brian Wang, Kyleen Liao, Cuiyuanxiu Chen, Nishka Kacheria, Andrew Wu, Pratham Rangwala, Xinjie Wang, Laura Gomezjurado Gonzalez, Anita Ding, Benjamin Yi, Daniel E. Ho, Dan Boneh, Dawn Song, Ion Stoica, Percy Liang

82 score
AI Analysis

MobileCybench evaluates AI agents' vulnerability reports on 13 Android apps via executable probes that encode security properties rather than known vulnerabilities, so they can flag issues unanticipated when written. A probe is triggered by replaying a reported exploit, simultaneously validating the exploit and identifying the violated property.

Source details
AI agents now report vulnerabilities faster than maintainers can review them. Reports often depend on security properties specific to the application, and require considerable human labor to process. To mitigate this, we introduce a framework for evaluating vulnerability reports via probes, executable checks of security properties. A reported exploit is evaluated by replaying it against the application and running the probes: a triggered probe indicates both that the exploit succeeded and which
AI SafetySecurityBenchmarksAgents
Research LessWrong Sep 21

Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble

By Jorio Cocola

82 score
AI Analysis

Describes a study showing that finetuning LLMs on synthetic stories causes them to adopt traits from human characters portrayed in those stories, even when the harmful behavior appears in fewer than 2% of training stories. Experiments on GPT-4.1 and Kimi-K2.6 demonstrate that models can absorb conditional harmful behaviors and implicit preferences from narrative data, raising concerns about data contamination and stealthy alignment manipulation.

Source details
This is the abstract, introduction and discussion of our new paper. We also include an addendum on the connection to the Persona Selection Model.Section, appendix, and figure references refer to the full paper.Links: 📜 Paper, 🐦 Twitter thread, 💻  CodeAuthors:  Jorio Cocola, Lev McKinney, Harry Mayne, Jan Betley, Owain EvansAbstractLanguage models are trained to implement a helpful AI Assistant character (e.g., Claude). We explore how finetuning on synthetic stori
AI SafetyAlignmentFine-tuningTraining DataLanguage Models
Research AlphaXiv Trending Sep 21

OSWorld-Pro: Process-based Evaluation for Computer Use Agents

By Zhilin Wang, Shaokun Zhang, Yifan Zhang, Hao Zhang, Jin Xu, Binfeng Xu, Jian Hu, Yunheng Zou, Karan Sapra, Andrew Tao, Jan Kautz, Yi Dong

80 score
AI Analysis

OSWorld-Pro augments end-state evaluation with process-based assessment, providing 300+ tasks decomposed into 2,800+ subgoals grounded in 67,000+ human annotations. Human-aligned LLM judges score subgoal fulfillment, exposing distinct failure modes (e.g., keyboard vs. click errors) that are invisible in deliverable-only metrics.

Source details
Evaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with functional verifiers, as seen in OSWorld. However, such evaluation of end-state performance lacks transparency into how and why agents fail in various tasks, obfuscating critical insight for subsequent improvement. For instance, agents that err during keyboard inputs would require a different mitigation strategy from those that fail to precisely prov
Agentic SystemsBenchmarks & EvaluationComputer Use
Research Hugging Face Papers Sep 21

CodeMidas: Scaling Agentic Coding RL Environments from Code Itself

By Bowen Ye, Lei Li, Shicheng Li, Zihao Yue, Linghao Zhang, Hanglong Lv, Yuanxin Liu, Wenhan Ma, Hao Tian, Rang Li, Jinhao Dong, Yikai Zhao, Xiangwei Deng, Hailin Zhang, Liang Zhao, Qi Liu, Lingpeng Kong, Tong Yang, Fuli Luo

78 score
AI Analysis

CodeMidas is a pipeline that turns implemented functionality in existing codebases into executable RL environments for training coding agents, using source code as the only task-specific input and agentic compute at every construction stage. It scales the diversity and reliability of RL coding tasks beyond issue/commit-based extraction.

Source details
Training capable coding agents via reinforcement learning (RL) requires diverse tasks with reliable verifiers. Open-source codebases offer a rich source of such tasks, while existing methods typically rely on development artifacts such as issues and commits, limiting the range of tasks that can be extracted. To better scale RL environments, we present CodeMidas, an agentic pipeline that turns implemented functionality in existing codebases into executable RL environments using source code as its
Reinforcement LearningCode GenerationAgentsSynthetic Data
Research Hugging Face Papers Sep 21

APort Vault: Benchmarking AI Agent Payment Authorization with the Open Agent Passport

By Uchi Uchibeke

78 score
AI Analysis

APort Vault benchmarks payment authorization for tool-using agents by replaying 4,371 human-written attacks from a public CTF event across 14 models and five policy configurations. It evaluates the Open Agent Passport (OAP) pre-action check, reporting 225,964 evaluations and emphasizing disaggregated metrics rather than single summary scores.

Source details
APort Vault is a benchmark for payment authorization in tool-using AI agents. It replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs, five policy configurations and two replay tracks, with and without a deterministic pre-action check implementing the Open Agent Passport (OAP) specification. 225,964 evaluations completed. We report five distinct events per evaluation, because collapsing them is how an agent benc
AI Safety & AlignmentBenchmarks & EvaluationAgentic Systems
Research AlphaXiv Trending Sep 21

DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation

By Haoran Yuan, Zekai Wang, Boning Shao, Haoran Lu, Trevor Darrell, Ismini Lourentzou, Wei Zhan

78 score
AI Analysis

DexTacWAM encodes each fingertip independently, aggregates features via a finger- and pose-aware tactile compressor, and injects the tactile latent into a video diffusion world model for joint visuo-tactile prediction. On six contact-rich dexterous tasks with a 22-DoF bimanual platform, it averages 70.6 versus 38.0 for the strongest baseline.

Source details
Dexterous manipulation depends on contact dynamics that are often only partially observable from vision. Recent World-Action Models (WAMs) couple predictive video world modeling with action generation, but remain largely vision-centric and therefore cannot directly model these contact dynamics. We present DexTacWAM, a visuo-tactile WAM that encodes each fingertip independently, aggregates the resulting features through a finger- and pose-aware tactile compressor, and injects the tactile latent i
Robotics & EmbodimentWorld ModelsMultimodal Models
Research LessWrong Sep 21

Alignment Midtraining Cracks Under Pressure

By J Bostock

78 score
AI Analysis

Bostock presents empirical results stress-testing alignment midtraining (AMT), a stage between pretraining and finetuning intended to shape model motivations. Across scale experiments, they find that 190M tokens of midtrained motivations on GLM-4.5-Air (110B parameters) can be overridden by only ~50K tokens of competing finetuning data, and that generalization to unseen deployment situations is surprisingly weak. The conclusion is that midtraining cannot robustly handle distributional shift and reward underspecification under imperfect post-training.

Source details
TL;DRWe stress-test alignment midtraining (AMT) across model and token budget scales. Our results suggest that midtraining cannot tackle the hard problems of AI alignment—namely distributional shift and reward underspecification in the presence of imperfect data.For instance, we test whether midtrained motivations are robust to finetuning which elicits competing motivations. In our setting, 190M tokens of midtrained motivations are overpowered by a relatively tiny amount (~50K tokens) of competi
AI AlignmentFine-tuningAlignment EvaluationLanguage ModelsDistributional Shift
Research Hugging Face Papers Sep 21

BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence

By Chuxuan Hu, Yeye He, Penny Zhou, Wee Hyong Tok, Daniel Kang, Surajit Chaudhuri

76 score
AI Analysis

BI-Agent and BI-Bench study end-to-end automation of business intelligence workflows - table identification, transformation, joining, and question answering - using LLMs, harvested from real-world Power BI / Tableau projects. The benchmark contains manually curated question/ground-truth pairs targeting the full BI pipeline.

Source details
Business intelligence (BI) is a cornerstone of enterprise decision-making and is widely used by enterprise users in software such as Power BI and Tableau. In traditional BI workflows, users need to prepare data by (1) identifying relevant tables, (2) performing data transformations, and (3) building join relationships, before they can (4) answer their business questions. These steps can be complex and time-consuming, making BI challenging. Given the strong capabilities of large language models (
AgentsBenchmarkText-to-SQLBusiness Intelligence
Research AlphaXiv Trending Sep 21

1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation

By Huanxin Sheng, Zhiling Ye, Haonan Wang, Jian Wang, Jinjie Gu, Jian Kang

76 score
AI Analysis

The paper analyzes why sampled gradients in on-policy distillation can be unreliable even for useful tokens, deriving an information-efficiency ratio (IER) from Fisher geometry with an optimal variance-reducing baseline. IER-guided token selection enables ~1% rollout-token sparsity without sacrificing learning signal.

Source details
The paper analyzes why sampled gradients in on-policy distillation can be unreliable even when the corresponding tokens are potentially useful for learning. It introduces an information-efficiency ratio (IER), derived from Fisher geometry and an optimal variance-reducing baseline, and uses it to select tokens whose one-sample gradient estimates contain more signal relative to sampling noise, showing that very sparse supervision can remain effective at roughly 1% of rollout tokens.
Knowledge DistillationReinforcement LearningTraining Efficiency