Changed is not improved. Without a labelled eval set, 'improved naming' just means 'changed naming'. You can't ship a classifier you can't score.

Rename rate is a signal. Start with rename, remove and hide rates as the live signal, plus a small labelled set from consenting testers. That's how every classifier feature bootstraps.