Metrics are used to align an organization or team towards a shared goal and to track progress towards that goal. A successful metric is one that helps to inspire and coordinate efforts across many people. This means that, at a minimum, a metric should be easily understood by people across the organization. Understandability becomes even more important as the number of people or the seniority of people who rely on it increases.
Consequently, when defining a new metric, an extremely effective tool for metric selection is Occam’s razor. When there are multiple possible metrics that could be used to track the same notion of business or model performance, the simplest metric is generally the most useful. This tends to be true for a few reasons:
- Simple metrics are usually pretty “accurate”. That is, they often achieve a good balance between bias and variance with respect to the true notion of performance
- Simple metrics feel transparent and thus, more trustworthy
- Simple metrics are robust: modifying them or complicating them slightly tends not to make much of a difference
- Simple metrics tend to have clear statistical properties
A common mistake especially of new data scientists is to favor complicated metrics that may have theoretical advantages or motivations but are impenetrable to non-DS. Complications are best hidden under the hood of a neural net, where they are expected to be.
Operationally, “simple” means that to the extent possible, the metric should have the following properties:
- Use only counting or basic arithmetic
- Has no parameters that need to be chosen
- Does not rely on an underlying statistical or theoretical model for its interpretation
- Has an understandable name or quick description
There are other desirable properties but simplicity is the most effective filter.
Data Science Principles 