Editing CommunityData:Dataset And Tools Release 2018

In summer 2018 [[People#Nathan_TeBlunthuis_.28University_of_Washington.29 | Nate]] is leading efforts to improve the code our research group uses to generate datasets from raw mediawiki dumps. The end goal is to release both the code and datasets generated on wikia and wikipedia wikis and to publish a data descriptor. This page documents these efforts. 

== Overview ==
  
There are 4 types of datasets we will support: 

# Wiki level edits: for each wiki, a table where each row corresponds to an edit.
# Wiki level edit weeks: edit data aggregated by each week. 
# [[User level edits | user level mediawiki datasets]]: for each user, a table where each row corresponds to an edit. 
# User level edit weeks: user level edits aggregated by week.

The program [[wikiq]] is used to generate wiki and (in the near future) user level edit datasets. Wikiq is a python script meant to be used through a command line interface.  It depends on functionality from [https://github.com/mediawiki-utilities Mediawiki Utilities]. 

The build_edit_weeks.R Rscript is used to generate edit weeks datasets from wikiq outputs. Currently this script is scattered across projects.

=== Goals ===

# Improve the reliability, usability, and maintainability of our software utilities for generating datasets. 
# Cover important and useful analytic variables in wikiq and build_edit_weeks. 
# Document the data sets with codebooks and example code. 
# Document the use and development of wikiq and build_edit_weeks to support future maintainers.
# Write a dataset descriptor to accompany the release of the code and data to publicize the release and explain the contribution.  

=== Tasks === 

# Talk to potential users of the code and datasets including the research teams at CDSC, UCM and Wikimedia to gather requirements. 
# Design user-level datasets (beginning with Jeremy and Kaylea's work).
# Add variables to wikiq according to requirements.
# Create unit test suits for wikiq and build edit weeks. 
# Refactor wikiq to produce user-level datasets.
# Refactor wikiq to migrate from python-mediawiki-utilities to the new mediawiki-utilities projects. 
# Refactor build edit weeks into the RCommunity data repository and support usability via CLI.
# Document release version of wikiq
# Document release version of build_edit_weeks. 
# Generate datasets from wikia dumps, wikipedia language editions. 
# Write data descriptor to accompany the release of the code and data to publicize the release and explain the contribution.
# Give presentations and tutorials to consumers of the work (UCM, WMF, CDSC).

=== Schedule === 
[[File:Gantt_schedule.svg]]


=== Ideas for New Wikiq Variables === 
# Regex language for building new variables.
# Variable indicating what sections were changed in the edit (Built around regex language)? 
# Variable indicate which if any Wikiprojects (for the wikipedia case) are associated with the page (Hard, probably requires 2 passes).


=== Requests === 

# Handle Default edits that initialize wikis.