Intuitively, regex or json grammar have a much lower "semantic dimension" than what today LLMs allow. Maybe the observed performance gains result from such lower dimensionality.