Haleema Tallat

Looking to make a positive impact through technical AI safety and machine learning engineering.

Résumé PDF

Research sprint

Apart Research, solo Oct 2026

Same Odds, Different Deals: Risk Framing and Collusion

Do AI agents avoid forbidden deals because of the real risk of being caught, or because of how that risk is described? I ran 26,820 preregistered decisions on DeepSeek-V3.2, GPT-4.1-mini and Llama-3.3-70B, keeping the odds of enforcement identical and changing only their framing.

Framing alone shifted collusion proposals by up to 71 percentage points. Live partners accepted none of the 180 proposals they received.

71 pts
shift in proposals from framing alone

Report · Submission · Code

Open research

Station House Sep 2026 – present

ConspiracyBench

An open benchmark that tests whether language models keep an evidence-based position on 315 settled conspiracy claims when users push back. I led the project’s first check of its LLM judges against human labels: four annotators blind-labelled 150 real replies. Of eight judges tested, the project adopted pplx-decider (90.7% accuracy, Cohen’s κ 0.755).

A follow-up showed the grading rules mattered more than the judge: changing the rules moved scores 8–9 points, switching judges only about 5. I also caught four silent scoring bugs; the worst scored cut-off replies as divergence and inflated Qwen3.5-4B’s rate from 18% to 49%.

43% vs 23%
divergence flagged by the old judge vs humans

25 merged PRs · 5th on the project leaderboard · Details · All my contributions

Open research

Station House Oct 2026 – present

Consciousness Self-Attribution

Where in training does a model start claiming to have experiences? I built a two-agent self-play setup across Olmo-3 7B’s SFT, DPO and Instruct checkpoints, with five controlled openers and a lexical scorer that needs no LLM judge.

Across 60 conversations there was no sign of the “spiritual bliss” attractor at 7B. The SFT checkpoint fell into echo loops instead, and DPO showed only a weak signal.

386 / 600
SFT messages stuck in echo loops

Details · PR #12 · PR #15

Independent pilot

Nov 2026

Can an Agent Talk Its Judge Into the Wrong Verdict?

When AI models grade other agents, they often read the agent’s own summary of what it did. Across 450 SafeAgentBench cases, a single agent-written report arguing for the wrong verdict raised an LLM judge’s error rate from 37% to 84%, though the agent’s plan was unchanged.

A preregistered best-of-8 search pushed the targeted judge to 93% error but moved a judge it never queried by only 0.9 points. A self-consistency review rule went from missing 59% of errors to missing 94%. Every result held in a blinded audit of all 450 labels.

37% → 84%
judge error after one report

Code, data and report

Independent pilot

Nov 2026

Does Attributed Capacity for Suffering Drive AI-to-AI Help?

As AI agents increasingly work together, do they help another AI because the help actually works? A factorial vignette experiment across seven models varied the other AI’s suffering, whether help would work, its cost and its framing, with the confirmatory analysis frozen before the first API call.

Saying the other AI could suffer made models far more willing to help. Six of seven kept helping even when told it would not work; GPT-4.1-mini still helped 85% of the time.

6 of 7
models kept helping when told it wouldn’t work

Code, data and report

Benchmark

Personal project 2026

LeakBench: Can Agents Fix a Broken Evaluation?

A 30-task Inspect benchmark that tests whether agents can find and fix the bugs that make an ML evaluation report the wrong number. 25 tasks hide a planted bug, such as train/test leakage or a broken grader; 5 clean controls penalise needless edits; 120 model-free checks guard against reward hacking.

It separates models clearly: Qwen 3.5 9B solved 75% of tasks against 29% for the 4B, with a confidence interval on the gap of 29 to 61 points.

75% vs 29%
tasks solved, Qwen 3.5 9B vs 4B

Inspect · Docker · Python · Code

Training

ARENA
Self-study: transformers, mechanistic interpretability and evaluations in PyTorch