Almost everyone starts tracking their communities with just a few metrics, but the number of metrics in use quickly grows to dozens and sometimes hundreds. Many metrics seem important enough to track them constantly and use for decision making. Problems begin almost immediately when, according to one important metric, the community is doing good, but according to another, it is doing bad. In addition to these two metrics, there are a dozen other important metrics and each of them tells a very unique story. Confirmation, as well as refutation, can be found for any hypothesis one likes. It is almost impossible to interpret metrics unambiguously in such a situation.
To prevent this from happening, it is necessary to approach metric design in a special way:
- Identify the independent subsystems.
- Create an hierarchy of metrics with different levels of detail.
- Define the priority of metrics within the levels.
This is a fragment of a draft of the book “Lessons Learned While Working On Stack Overflow”. Read the full book on kindle or the paperback version.