Editing CommunityData:Wikia data (section)

== XML Dumps == 
So far our Wikia projects all use the XML dumps. These are stored on hyak at /gscratch/comdata/raw_data/wikia_dumps. 

=== Use the 2010 dumps ===

The 2010 wikia dumps were created by wikia and are the only ones that approach being complete or reliable. They include the full history of ~76K wikis. In general, if the questions you want to answer don't require newer data then use the 2010 dumps.
* While the process of creating the dumps is somewhat opaque, it appears that most of the dumps were created in early January 2010. The last dumps were created in early April 2010.

The next most useful dumps are from WikiTeam and were obtained from archive.org.
As of 5-23-2017, the most recent complete dumps [https://archive.org/download/wikia_dump_20141219] from Wikiteam were from December 2014. Mako found some missing data in these dumps and contacted them. They released a patch [https://archive.org/details/wikiteam-incomplete-2014-12], which we have yet to validate. Another release exists from 2020 [https://archive.org/details/wikia_dump_20200214], which is the most current as of 1-31-2021.

These, and all other dumps after 2010, have a few issues that make using them complicated:

* Wikis that never gain traction are deleted from the DB. The criteria for which wikis get deleted is not clear.
* There have been a number of technical changes to the site since 2010, such as forums, message walls, etc.
* Other dumps were created via various processes (e.g., through the API) and not by Wikia and have missing data, truncated data, etc.