Judge the model by the business metric, not the validation set.

Most machine-learning projects are judged by a number the business never sees: accuracy, F1, RMSE on a held-out set. Those numbers are useful for engineers. They are a poor way to decide whether a project worked.

Start with the decision

Every model exists to change a decision someone makes — how much to order, which case to review first, when to send a technician. We write that decision down before any data work begins, along with how it is made today.

Pick the number that moves when the decision improves

For a forecast it might be stock-outs and waste. For a triage model it might be time-to-resolution for the cases that matter. That number becomes the measure the model is judged by, and it goes into the written scope.

“A model is good when it moves the number you care about, not when it tops a leaderboard.”

Keep the offline scores — but as a guide

Validation scores still help us choose between models. But a model that scores slightly worse offline and moves the business number further wins, every time.

If you are about to start an AI project, ask one question first: which number, in the business, should change if this works? If nobody can answer it, that is the first piece of work.