CommunityData:Wikia rises and declines

This is a project by Nate to find out if impersonal governance (reverting, automated reverting, reverting without discussion) a. increases in with increasing activity by newcomers and b. increases the chances that a newcomer will drop out on Wikia and Wikipedias of many languages. Below is the outline. Task management is at CommunityData:wikia_rises_and_declines_tasks.

Outline

Rationale

• Commons based peer production (CBPP) communities are admired for their ability to coordinate work on complex goods by workers with diverse motivations, without reliance on formal hierarchy or market transaction (Benkler 2002). Understanding how peer production projects this is an important question for designing more “efficient and equitable” systems for cooperative work on expanding categories of goods (Benkler, Shaw, and Hill 2015).
• The number of active contributors to Wikipedia rose rapidly in 2005, but peaked in 2007 and began a gradual decline Halfaker et al. (2013). The decline is a source of concern for the long term success of peer production projects.
• CBPP systems are able to perform decentralized governance work to resolve disputes and manage resources (Forte, Larco, and Bruckman 2009).
• However, as communities grow, territorial and controlling senior members of the community can sometimes appropriate governance systems to centralize power (Shaw and Hill 2014).
• Wikipedia’s decline has been explained by process in which influxes of newcomers correspond with increasing strict or impersonal governance quality control and that these hurt newcomer retention (Halfaker et al. 2013). Halfaker et al. (2013) show that quality control mechanisms including contribution rejection, formal and calcified rules, and algorithmic tool are associated with newcomer dropout on Wikipedia.
• Kiene, Monroy-Hernández, and Hill (2016) similarly observe how an influx of newcomers lead an original horror fiction subreddit to develop stricter governance to preserve the community’s distinctive culture and collective identity.
• Halfaker et al. (2013) also hypothesize that Wikipedia increased impersonal governance to deal with the massive influx of newcomers caused by Wikipedia’s popularity.
• However, evidence for this as a theory about peer production systems in general rather than a phenomenon specific to Wikipedia requires observing many communities. It is unknown whether the mechanisms described by (Halfaker et al. 2013) generalize to other wikis.
• This is important because this theory is informing design interventions on Wikipedia that aim to mitigate the decline by promoting newcomer socialization (Farzan et al. 2012; Morgan et al. 2013; Narayan et al. 2017; Halfaker, Geiger, and Terveen 2014).
• Furthermore, it is informing the development of commons based peer production projects other than Wikipedia (Palen et al. 2015).
• If influxes of newcomers promote strict governance, and strict governance conversely decreases newcomer activity, complex dynamics may arise. Growth patterns are often bursty (though not on Wikipedia). Crises other than newcomer influxes might also promote strict governance. In an extreme case a community may experience many crisis periods each of which accompany increases in governance. If governance does not decrease during periods of non-growth, it will accumulate and in the long run the community will die as newcomer attention approaches zero.

Objectives

General Objectives

1. Contribute to understanding the relationships between governance mechanisms and contributor retention in commons based peer production communities by synthesizing theories of how crises such as influxes of newcomers increase the accumulation rules and impersonal governance, which can threaten the long term health of the communities, with supporting statistical evidence from a large number of Wikia wikis and Wikipedias.

Specific Objectives

1. Test hypotheses from Halfaker et al. (2013) on a population of wikis using an econometric model of newcomer retention given rules, automated regulation tools, contribution rejection, group size, and interactions.
2. Test the hypothesis that (controlling for damage) an increased rate of newcomer activity increases impersonal governance.

Meta Objective

Nate to practice swift execution of a straightforward, important, good, and interesting article.

Null Hypotheses

1. Impersonal governance is not negatively related to newcomer retention.
   1. [A] Rejected contributions do not have a negative relationship with newcomer survival.
   2. [B] (given we reject [A]) Talk page discussion following a rejected newcomer contribution is not related to a decrease in newcomer survival.
   3. (given we reject [B]) Use of a tool in rejecting a contribution is not related to a decrease in follow-up discussion in newcomer edits.
   4. The formalization and calcification of rules is not related to a decrease in newcomer survival.
2. Increasing newcomer activity is not positively related to impersonal governance.
   1. The rate of change in newcomer contributions by newcomers is not related to increased rejection rate of newcomer contributions.
   2. The rate of change in newcomer contributions is not related to increased rate of tool assisted rejection of newcomer contributions.
   3. The rate of change in newcomer contributions is not related to decreased discussion following rejected newcomer contributions.
   4. The rate of change in newcomer contributions has no relationship with rule accrual.

Conceptual Model/Diagram

Measures

The study will have 5 models. The first model is for hypothesis 1. The last 4 models are for hypothesis 2.

1. [mod.discrete] Discrete time survival model of newcomers.
2. Contribution rejection
3. Rule accrual
4. Automated tool accrual
5. Interaction with newcomers

Data will be the 2010 Wikia dumps. Inclusion criteria will be broad.

Update: It looks like norms on wikia about BRD are generally not the same as en WP. Messages and talk page use are kinda rare. We will also analyze data from all sufficiently large (non-english) Wikipedias.

The unit of analysis for model 1 is the newcomer. We will model random intercept variance terms for Wikis because partial pooling is more realistic than complete pooling and afford inclusion of the many small Wikis with activity levels and scales. Also Nate is comfortable working with lme4. (Halfaker et al. 2013) use fixed effects for year, which is fine, but we should also take a look at models with fixed effects for month as well.

For models 2-5 the unit of analysis will be the wiki. This model will have only two levels: Wiki and time. Again we will fit models with fixed effects for month and year.

For both models we will use heteroskedasticity robust standard errors.

For models 2-5 we will include autoregression terms (equivalent to adding lagged outcome variables to the RHS) for the depdendent variable (AR(1) or AR(2)).

Newcomer retention

Following Halfaker et al. (2013), a new contributor is a logged-in editor in their first edit-session. The dependent variable for model [mod.discrete] indicates whether a new contributor makes a subsequent edit within the next 2 months.

While Halfaker et al. (2013) sampled a set of desirable newcomers to distinguish them from spammers and vandals. Doing this for a large number of Wikis would be very labor intensive. Halfaker et al. (2013) results for their set of desirable newcomers point in the same direction as their results for all newcomers. We will analyze all newcomers without attempting to distinguish the desirable from the undesirable.

Newcomer controls

Again following Halfaker et al. (2013) we will include controls for the number of edits that a newcomer makes in their first session.

We will also include a count variable for the number of edits that newcomer has made on Wikia (or all language Wikipedias) overall and an indicator for whether the newcomer has edited any other Wikia wikis.

Rejected newcomer contributions

After identifying newcomers, we can easily identify contributions they have made which have been rejected. These are (a.) edits which are reverted. Halfacker et al also use new articles which are deleted as a measure of rejection, but this is not possible for us without direct access to the database.

Discussion following rejection

After identifying rejected edits we can identify whether there is follow-up discussion according. Following Halfaker et al. (2013) we will measure reciprocity in discussion, which is when the reverting editor posts to the talk page after the newcomer.

It is likely that the bold-revert-discuss (BRD) is not a strong norm on Wikia as it is on Wikipedia. Therefore we will also include indications of other forms of interaction between reverting editor and reverted newcomer (user talk page, talk page no matter who posts first, message wall).

Automated regulation

We will use Mako’s tool for scraping admin and bot data from Wikia to identify bot accounts and detect edits made by these tools.

Find out if methods following (Geiger et al. 2012) can easily identify tool use on Wikia.

Rules

To measure rules we will track activity in namespace 4. Again following Halfaker et al. (2013) we use the following variables for norm “formalization”.

• The number of total contributors who contributed to norms pages.
• The number of contributions to norms pages
• The change in page length in a norm category

And the following variables for norm “calcification”

• The time since the first edit of a norm page editor (slightly different from Halfaker et al. (2013).
• Wiki age.

We do not use the “Essay” category as it is Wikipedia specific.

Increasing newcomer activity

We will estimate the rate of change of newcomer activity according to the change in newcomer (the rate of editing by with accounts less than 2 months old) from one month to the next.

The outcome variables in models 2-5 are also rates of change and likewise we can model using first differences.

We will fit a number of alternative specifications using things like moving averages. We will aim to report the simplest model that makes a compelling and justified argument.

Dummy Tables

Regression table for hypothesis 1

Variable	Est	SE
(Intercept)	.	.
Time	.	.
Session Edits	.	.
Reverted	.	.
Messaged	.	.
Tool Revert	.	.
Reciprocal Discussion	.	.

Regression table for hypothesis 2

Variable	M2. Est	M2.SE	M3. Est	M3.SE	M4.Est	M4.SE	M5. Est	M5.SE
(Intercept)	.	.	.	.	.	.	.	.
Time	.	.	.	.	.	.	.	.
Active Editors	.	.	.	.	.	.	.	.
Rate of editing overall	.	.	.	.	.	.	.	.
Rate of newcomer edits	.	.	.	.	.	.	.	.
Number of articles	.	.	.	.	.	.	.	.
Total wiki length	.	.	.	.	.	.	.	.

Threats to Validity and Limitations

Measures

Our measures of “discussion following rejection” as an indication of the degree of impersonality in governance follows the prior work we aim to replicate. These measures are virtuous in their simplicity and clarity, but reduce complex interpersonal communication to counts of categories of interaction. This is a potential source of bias, noise, and a threat to construct and ecological validity.

We do not attempt to identify desirable vs undesirable newcomers. Halfaker et al. (2013) did not find substantive differences between the two groups. However if we fail to fully replicate their findings, this could be a reason.

Using first difference to measure rates of change is pretty freaking noisy! We hope to capture average trends, but it’s possible for oscillations and lags and other fun time series problems to impart bias. Using a smoothed average helps address such threats in exchange for new ones. We should fit multiple models with different specifications to make sure our results are robust to this decision.

Using namespace 4 as a proxy for rules is also noisy and possibly biased. Some wikis use namespace 4 for activities other than norm setting and rule making. We have not found a good systematic way to separate these activities from rules.

Causality

Our data are observational and not capable of offering strong evidence for causal claims.

Generalizability

We are studying a lot of Wikis. Our theories are of concern to other kinds of communities (e.g. Reddit, OSM). We don’t know if our results generalize to such settings.

Communities might make rules in response to other kinds of crises (consider instances of harassment). We don’t know if our results will generalize to rules created in response to other crises. Maybe governance strategies created in response to newcomers are more damaging to newcomers than governance systems targeted at managing other kinds of crises.

Theory

Our theory is incomplete. Maybe the feedback loop between newcomer influx and rule making dampens itself to a stable equilibria with a sustainable population and institutionalized rules. Maybe communities can identify when governance systems threaten growth and survival and scale them back. Our data and analysis don’t have much to say about that.

Schedule

The CHI abstract deadline is September 12.

The goal is to have a draft of this article written in 5 weeks, by July 31th. This should be doable because our measures are quite straight forward. 2010 Wikia data is already on Hyak and ready to go. The first draft of the paper can be short, but should hit the key points, and have a clear and attractive presentation of results. This will give us over a month to refine and revise.

Weekly milestones

Date	Milestone	Description
July 30	Build dataset (70% done)	Have a data.table with analytic variables. (It is ok if we don’t have great data on bots yet)
July 7	Finish building datasets and have preliminary results	Fit, interpret, plot (first attempt) models for 5 hypotheses.
July 14	Draft front matter	Read additional key citations, draft a CHI-style introduction and background section.
July 20	Draft for CDSC retreat	Circulate a rough draft for the CDSC collective.
July 31	Complete draft	Finish the rest of the draft. Turn over to Mako for feedback.

References

Benkler, Yochai. 2002. “Coase’s Penguin, or, Linux and the Nature of the Firm.” Yale Law Journal 112 (3): 369–446.

Benkler, Yochai, Aaron Shaw, and Benjamin Mako Hill. 2015. “Peer Production: A Form of Collective Intelligence.” In The Handbook of Collective Intelligence, edited by Michael Bernstein and Thomas Malone, 175–204. Cambridge, MA: MIT Press.

Farzan, Rosta, Robert Kraut, Aditya Pal, and Joseph Konstan. 2012. “Socializing Volunteers in an Online Community: A Field Experiment.” In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, 325–34. CSCW ’12. New York, NY, USA: ACM. doi:10.1145/2145204.2145256.

Forte, Andrea, Vanesa Larco, and Amy Bruckman. 2009. “Decentralization in Wikipedia Governance.” J. Manage. Inf. Syst. 26 (1): 49–72. doi:10.2753/MIS0742-1222260103.

Geiger, R. Stuart, Aaron Halfaker, Maryana Pinchuk, and Steven Walling. 2012. “Defense Mechanism or Socialization Tactic? Improving Wikipedia’s Notifications to Rejected Contributors.” In Sixth International AAAI Conference on Weblogs and Social Media, 122–29. Dublin, Ireland: AAAI Publications. http://www.aaai.org/ocs/index.php/ICWSM/ICWSM12/paper/view/4657.

Halfaker, Aaron, R. Stuart Geiger, and Loren G. Terveen. 2014. “Snuggle: Designing for Efficient Socialization and Ideological Critique.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 311–20. CHI ’14. New York, NY, USA: ACM. doi:10.1145/2556288.2557313.

Halfaker, Aaron, R. Stuart Geiger, Jonathan T. Morgan, and John Riedl. 2013. “The Rise and Decline of an Open Collaboration System How Wikipedia’s Reaction to Popularity Is Causing Its Decline.” American Behavioral Scientist 57 (5): 664–88. doi:10.1177/0002764212469365.

Kiene, Charles, Andrés Monroy-Hernández, and Benjamin Mako Hill. 2016. “Surviving an ‘Eternal September’ How an Online Community Managed a Surge of Newcomers.” In. San Jose, CA, USA. https://mako.cc/academic/kiene_monroy_hill-surving_eternal_september-CHI2016.pdf.

Morgan, Jonathan T., Siko Bouterse, Heather Walls, and Sarah Stierch. 2013. “Tea and Sympathy: Crafting Positive New User Experiences on Wikipedia.” In Proceedings of the 2013 Conference on Computer Supported Cooperative Work, 839–48. CSCW ’13. New York, NY, USA: ACM. doi:10.1145/2441776.2441871.

Narayan, Sneha, Jake Orlowitz, Jonathan Morgan, Benjamin Mako Hill, and Aaron Shaw. 2017. “The Wikipedia Adventure: Field Evaluation of an Interactive Tutorial for New Users.” In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, 1785–99. CSCW ’17. New York, NY, USA: ACM. doi:10.1145/2998181.2998307.

Palen, Leysia, Robert Soden, T. Jennings Anderson, and Mario Barrenechea. 2015. “Success & Scale in a Data-Producing Organization: The Socio-Technical Evolution of OpenStreetMap in Response to Humanitarian Events.” In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 4113–22. CHI ’15. New York, NY, USA: ACM. doi:10.1145/2702123.2702294.

Shaw, Aaron, and Benjamin M. Hill. 2014. “Laboratories of Oligarchy? How the Iron Law Extends to Peer Production.” Journal of Communication 64 (2): 215–38. doi:10.1111/jcom.12082.