Editing COVID-19 Digital Observatory
From CommunityData
The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then publish the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 3: | Line 3: | ||
[[File:Novel Coronavirus SARS-CoV-2.jpg|thumb|300px|right|Microscopy image of the virus which causes COVID-19.]] | [[File:Novel Coronavirus SARS-CoV-2.jpg|thumb|300px|right|Microscopy image of the virus which causes COVID-19.]] | ||
This page documents a ''digital observatory'' project that aims to collect, aggregate, distribute, and document public social data from digital communication platforms in relation to the [[:wikipedia:2019–20 coronavirus pandemic|2019–20 coronavirus pandemic]] | This page documents a ''digital observatory'' project that aims to collect, aggregate, distribute, and document public social data from digital communication platforms in relation to the [[:wikipedia:2019–20 coronavirus pandemic|2019–20 coronavirus pandemic]]. The project is being coordinated by the [[Community Data Science Collective]] and [https://pushshift.io/ Pushshift]. | ||
== Overview and objectives == | == Overview and objectives == | ||
As people struggle to make sense of the COVID-19 pandemic, many turn to social media and [[:wikipedia:Social_computing|social computing systems]] to share information, to understand what's happening, and to find new ways to support one another. As scholars, scientists, technologists, and concerned members of the public, we are building a digital observatory to understand where and how people are talking about COVID-19-related topics. The observatory collects, aggregates, and distributes social data related to how people are responding to the ongoing public health crisis of COVID-19. The public datasets and freely licensed tools created through this project will allow researchers, practitioners, and public health officials to more efficiently understand and act to improve these crucial sources of information during crises | As people struggle to make sense of the COVID-19 pandemic, many turn to social media and [[:wikipedia:Social_computing|social computing systems]] to share information, to understand what's happening, and to find new ways to support one another. As scholars, scientists, technologists, and concerned members of the public, we are building a digital observatory to understand where and how people are talking about COVID-19-related topics. The observatory collects, aggregates, and distributes social data related to how people are responding to the ongoing public health crisis of COVID-19. The public datasets and freely licensed tools, techniques, and knowledge created through this project will allow researchers, practitioners, and public health officials to more efficiently gather, analyze, understand, and act to improve these crucial sources of information during crises. | ||
Everything here is a work in progress as we get the project running, create communication channels, and start releasing datasets. Learn how you can stay connected, use our resources as we produce them, and get involved below. | Everything here is a work in progress as we get the project running, create communication channels, and start releasing datasets. Learn how you can stay connected, use our resources as we produce them, and get involved below. | ||
Line 18: | Line 16: | ||
The email list will contain occasional updates, information about new datasets, partnerships, and so on. We will not use the list or email addresses for other purposes. | The email list will contain occasional updates, information about new datasets, partnerships, and so on. We will not use the list or email addresses for other purposes. | ||
== | ==Access Data and Code== | ||
The | The best way to find the data is to visit [https://covid19.communitydata.science/datasets/ covid19.comuunhitydata.science/datasets/]. The <code>search_results</code> directory contains compressed raw data generated by Nick Vincent's [https://github.com/nickmvincent/LinkCoordMin SERP scraping project]. The <code>wikipedia_views</code> directory has view counts for Wikipedia pages of COVID19-related articles in <code>.json</code> and <code>.tsv</code> format. The <code>keywords</code> directory has <code>.csv</code> files with COVID-19 related keywords translated into many languages and associated Wikidata item identifiers. | ||
For code used to produce the data and get started with analysis we have a [https://github.com/CommunityDataScienceCollective/COVID-19_Digital_Observatory github repository]. If you want to get involved or start using our work please clone the repository! You'll find example analysis scripts that walk through downloading data directly into something like R and producing some minimal analysis to help you get started. | |||
More details on the different datasets and sources follow below. We hope that our initial data releases will be a good starting point for investigating social computing and social media content related COVID-19. At first, we will focus on providing static datasets (in raw text and structured formats like csv and JSON) from Wikipedia as well as search engine results pages (SERPs) for a set of searches on COVID-19 relevant terms. We are also releasing a list of keywords generated by daily Google search trends, Wikidata entities, and translations into many languages based on Wikidata entity links. We plan to expand these offerings with new material including data from Twitter, Reddit, and localized content specific to particular geographic regions. We also plan to build infrastructure to provide rapid and frequent updates of datasets in a variety of forms. | |||
===Search Engine Results Pages (SERP) Data=== | |||
The SERP data in our initial data release includes the first search result page from Google and Bing for a variety of COVID-19 related terms gathered from Google Trends and Google and Bing's autocomplete "search suggestions." Specifically, using a set of six "stem keywords" about COVID-19 and online communities ("coronavirus", "coronavirus reddit", coronavirus wiki", "covid 19", "covid 19 reddit", and "covid 19 wiki"), we collect related keywords from Google Trends (using open source software[https://www.npmjs.com/package/google-trends-api]) and autocomplete suggestions from Google and Bing (using open source software[https://github.com/gitronald/suggests]). In addition to COVID 19 keywords, we also collect SERP data for the top daily trending queries. Currently, the SERP data collection process does not specify location in its searches. Consequently, the default location used is the location of our machine, at Northwestern University's Evanston campus. We are working on collecting SERP data with location specified beyond the Chicago area (aka other 'localized' content). | |||
= | The SERP data is released as a series of compressed archives (7z), one archive per day, that follow the naming convention <code>covid_search_data-[YYYYMMDD].7z</code>. Within these compressed archives, there is a folder for each device emulated in the data collection (currently two: Chrome on Windows and iPhone X) which contains all of the respective SERP data. Per each device subdirectory, SERP data itself is organized into folders that are titled by the URL of the search query (e.g. <code>'https---www.google.com-search?q=Krispy Kreme'</code>), and each SERP folder contains three data files: | ||
* a PNG screenshot of the full first page of results, | |||
* an mhtml "snapshot" (https://github.com/puppeteer/puppeteer/issues/3658), | |||
* and a json file with a variety of metadata (e.g. date, the device emulated) and a list of every link (<a>) element in the page with its coordinates (top, left, bottom, right) in pixels. | |||
===Wikipedia article pages and edit history=== | |||
Our initial release provides exhaustive edit data for the list of English Wikipedia articles covered by [[Wikipedia:Wikiproject COVID-19|WikiProject Covid-19]]. | |||
=== | ===Wikipedia article pageviews=== | ||
Our initial release provides [[Wikipedia:Pageview_statistics|pageview data]] for Wikipedia articles related to the covid-19 pandemic, as identified by members of [[Wikipedia:WikiProject Covid-19|WikiProject Covid-19]]. | |||
===Keywords and search terms=== | |||
We currently use and provide three different types of keywords and search terms: | |||
* Article names/topics from Wikipedia's [[Wikipedia:WikiProject_COVID-19|WikiProject Covid-19]] | |||
* Wikidata entities generated via the "Main items" described by Wikidata's [https://www.wikidata.org/wiki/Wikidata:WikiProject_COVID-19 WikiProject COVID-19] | |||
* Top 25 daily trending search terms from Google and Bing. | |||
==Get Help Using Data== | ==Get Help Using Data== | ||
(more coming soon) | |||
As we develop data collection resources and datasets, we will also provide simple example analysis scripts to demonstrate how you might access, import, and analyze small subsets of the data we produce. For instance, take a look at the [https://github.com/CommunityDataScienceCollective/COVID-19_Digital_Observatory/tree/master/wikipedia_views/analysis "analysis" subdirectory of the wikipedia views section of our Github project]. | |||
We plan develop tutorials and demos for the use of the data we release and particularly welcome contributions that help make these resources more easily usable by others. In some cases, the data sources are quite large and might not be suitable for analysis on your personal computer. Wherever possible, we'll try to point you to useful tools to help make this feasible or easier. | |||
==Contribute== | ==Contribute== | ||
Line 70: | Line 62: | ||
* Want to contribute to our datasets and/or analysis code? Clone our [https://github.com/CommunityDataScienceCollective/COVID-19_Digital_Observatory github repository] and pitch in with pull requests. | * Want to contribute to our datasets and/or analysis code? Clone our [https://github.com/CommunityDataScienceCollective/COVID-19_Digital_Observatory github repository] and pitch in with pull requests. | ||
==Code of Conduct== | |||
We ask that all contributors adhere to the [https://www.contributor-covenant.org/ Contributor Covenant]. | |||
== | == Links to related projects == | ||
This is an incomplete list of related projects, several of which have additional and more comprehensive lists of related projects. Please add more! | This is an incomplete list of related projects, several of which have additional and more comprehensive lists of related projects. Please add more! | ||
Line 81: | Line 74: | ||
* [https://covid19obs.fbk.eu/ COVID-19 Infodemics Observatory] analyzing Twitter data. Created by the [https://comunelab.fbk.eu/CoMuNe Laboratory of the FBK research unit]. | * [https://covid19obs.fbk.eu/ COVID-19 Infodemics Observatory] analyzing Twitter data. Created by the [https://comunelab.fbk.eu/CoMuNe Laboratory of the FBK research unit]. | ||
* [https://www.archiveteam.org/index.php?title=Coronavirus Archive team listing of archive sites related to the Coronavirus] | * [https://www.archiveteam.org/index.php?title=Coronavirus Archive team listing of archive sites related to the Coronavirus] | ||