Isn't that one of the reasons why KL-divergence is used, at least in DPO/RL for LLM? Otherwise the model can effectively cheat and mode collapse. For pre-training against a 1-hot label the KL-divergence should be equivalent to cross-entropy anyway.