A possible solution would be to add a large penalty term to your loss function based on this constraint. While this does not guarantee that the predictions will satisfy the constraint over many training iteration fewer predictions will break it as the loss is minimized.