Sometimes the metric you use for the loss function is not the best metric for eval. Think BLEU for translation, for instance (not that BLEU is particularly great).