DPO can be extended to diffusion models by contrastively optimizing the probability of Gaussian noise.

https://arxiv.org/abs/2311.12908

It is used in SD 3

By Sora, I am referring to the version used in the 2024 Feb publication.

Resolve NA if I cannot figure out by EOY 2025