TL;DR;
If you want to assess performance of a community manager, a CM team or yourself from the business perspective, you can take a look at a few fundamental indicators with and without the CM, CM team or you working on the community. The indicators are:
- Inflow and outflow of users on the platform
- The number of posts by their serial number
- The number of posts by the engagement level
- The absolute number of posts and tenure with one time posters dropped
You should see some positive differences with and without someone working on the community. If there is no positive influence… well, you need to change the way you work with the community.
Let’s say you have a community manager or a whole community management team. How do you assess how well they are doing their job? Easy! Look at what the community has achieved with and without the subject managing the community.
For 13 years I have been working on / with Stack Overflow in Russian, the first four years as my own company and another 9 years as part of Stack Overflow. The project’s history has seen everything from an inactive launch to proactive fading. So let’s try to see how the community changed over different periods and make some conclusions from that.
Activities on Stack Overflow in Russian
What we will look at is the number of posts (questions + answers) on the site during four distinct periods:
- From the beginning, October 10th, 2010, up to November 1st, 2012. This is the time when we were actively working on the community.
- From November 1st, 2012 and up to April 1st, 2015. This is the period of time when we were doing whatever but not growing the community.
- From April 1st, 2015 up to mid-Jun, 2018. This is the period of time when I joined Stack Overflow and served as an owner of Stack Overflow in Russian.
- From mid-Jun, 2018 ‘till the end. In mid June, 2018 I moved to work on some other things for Stack Overflow.
We can consider period 1 and 3 as when the community was managed. Even though we supported users during periods 2 and 4 I think we can say that the community was unmanaged or better “under managed”. Also, periods 1 and 2 are from the HashCode era. Periods 3 and 4 are from the Stack Overflow era. The further analysis is based on those assumptions.
You can read the whole epic story of how my friends and I were growing HashCode in the first part of the book Lessons Learned While Working on Stack Overflow.
In order to compare periods of a community’s life, it is necessary to select the parameters by which we will compare them. In fact, this is the most difficult task because there are a lot of them. I selected four most fundamental and universal parameters which you can use to evaluate almost any community on your own.
User inflow and user outflow
One of the first parameters to pay attention to is the number of users who published their first post and the number of users who published their last post. This is one of the important parameters and clearly define the health of a community. The only problem with it is that the true value of this parameter can only be seen after some time. (That’s why I do not take into account the last year and a half in the graphs below.)
It’s obvious that there is a difference (and statistical significance agrees with our eyes!) During the periods when I was working on the community, the outflow to inflow ratio was noticeably lower (i.e. more people stayed in the community) than when the community was operating on its own.
Please let me repeat myself. The ratio of users leaving the community to those joining it is one of the most important indicators of the health of the community. After all, the community exists only as long as its members need it. A community grows when it acquires new members without losing the old ones. A community is dying when existing members depart and nobody replaces them.
Number of posts by their serial number
What is the difference between a marketing manager and a community manager? A marketing manager drives users to the site, i.e. works with the external audience where a community manager makes existing users stay. So one of the ways to see how a community manager performs is to look at how users progress over their engagement journey. To do this, let’s assign a serial number to every user’s post, then count the number of posts with the same serial number.
Here we can note a few things
-
The first posts are the bottom of the engagement funnel. Usually CMs do not have control over this number. It is all about marketing. When we started our HashCode community we did a lot of promotion. After joining the Stack Overflow family, we pump our traffic significantly over the first few months but after that the number of the first posts was flat almost all the time, even during “The Fall Of The Star” and mostly depended on the search traffic.
-
Even though the number of first posts matters a lot, most of the content gets created by regular users who created 10+ posts on the platform.
You do not need to do rocket science math here to judge whether your CM team performs well, just look at the amount of posts with high serial numbers on such a chart. A community will grow only if there are people who post regularly. Actually, let’s talk a bit more about it.
Number of posts by the level of participation
When I was working on the design of health metrics for Stack Overflow, one of the ideas was to look at distinct groups of users instead of looking at the whole community when we calculate engagement metrics. I usually group users based on the number of posts they create per period of time. For monthly stats the thresholds are:
- One time posters — 1 post per month
- Casual users — 2, 3 or 4 posts per month
- Regular users — from 5 to 12 posts per month
- Core users — more than 12 posts per month
(Please note that here we count the number of posts per user per month, not all time, as we did in the previous section. So if a user posts once per month they all the time will appear in the one time posters category.)
Usually, if you have a good CM team, you should see growth for all engagement levels. If the team does not have enough support or simply does not do what they are supposed to do, the first users to drop will be the core users. The reason seems to be straightforward. Any community has a very limited number of core users and each of them contributes a lot. When a few such users reduce their contribution we see it immediately.
Absolute number of posts and tenure
Usually, when someone says “let’s work on users’ engagement” what they really mean is “let’s make users post more or stay longer”. So, most of “engagement” activities can be measured through metrics that are based on the number of posts.
As you can see, during the periods when the community was managed we have a longer tail that represents users who posted a lot.
You see the bars of different heights since the periods have different lengths. To make the comparison fair (and be able to look at the absolute numbers), let’s take a look at users who joined the community over two intervals with the same duration and see how many posts on average users posted and what was the average tenure on the site. Let the first interval be the time when I was working on Stack Overflow in Russian and the second interval the time of 40 months after that. Here are the stats (with one time posters dropped):
The community with a community manager
- Mean days on the site 535.51, median 219.06
- Mean number of posts 16.80, median 4.00
The community without a community manager
- Mean days on the site 286.88, median 107.00
- Mean number of posts 7.90, median 3.00
The numbers differ by almost two times!
Conclusions
Yes, some online communities grow on their own with no no management… to some extent. But most communities do not. They need someone to organize the users and make them interested in the group and excited about what the group does. A good community manager is like a gardener: they know what to water, when to fertilize, how much sun each of the plants needs, they can grow a tropical palm tree for you in almost any climate. At the same time, not all CMs are the same and you can see it clearly if you look at the performance of the community with and without the CM working on our community.