In a September 29, 2026 Google Research blog post, Chih-wei Hsu and Moonkyung Ryu describe Diffusion Controller, based on a paper first posted to arXiv in March by a team including Craig Boutilier and Bo Dai. The problem is familiar to anyone who has used a text-to-image model: pushing a model to include every element of a prompt often damages image quality. The post's example is a lizard whose face distorts when the model is forced to add sunglasses.
The approach reframes reverse diffusion sampling as a stochastic control problem. Instead of fine-tuning the base model, a small side network -- the authors call it a lightweight "steering damper" -- reweights the frozen model's own denoising steps toward outputs a reward model prefers. It is trained with either policy-gradient (PPO-style) updates or reward-weighted regression.
- In a gray-box setting, where the base model's weights are not accessible, the controller outperformed LoRA fine-tuning on the HPS-v2 preference metric, according to the post.
- In a white-box setting with full access, it achieved a 90% win rate over the baseline model, and human raters preferred its prompt matching on complex prompts.
Why it matters: the most capable generative models are increasingly available only through restricted interfaces, and many teams can't fine-tune them. An add-on controller that improves alignment without touching base weights is a way to customise a model you don't own, and the same framing -- a small learned controller steering a frozen generator -- could extend to other generation settings. It also keeps the base model's safety behaviour intact, since the weights don't change. Caveats: the results come from Google's own evaluation, the post does not mention a code release, and "gray-box" still needs access to intermediate sampling steps, which most commercial image APIs don't expose. Reward-model steering can also over-optimise toward whatever the reward model likes, so human evaluation remains the real test.
Diffusion Controller improves prompt fidelity by steering a frozen image model with a small learned controller, beating LoRA in Google's restricted-access tests; it's a promising way to customise models you can't retrain, but it still needs sampling-step access and has no announced code release.