COVID-19 Digital Observatory: Difference between revisions

From CommunityData
 
(57 intermediate revisions by 12 users not shown)
Line 1: Line 1:
{{notice|This project page is currently under construction.}}
{{notice|This project and project page are under active development.}}


[[File:Novel Coronavirus SARS-CoV-2.jpg|thumb|300px|right|Microscopy image of the virus which causes COVID-19.]]
[[File:Novel Coronavirus SARS-CoV-2.jpg|thumb|300px|right|Microscopy image of the virus which causes COVID-19.]]


This page documents a ''digital observatory'' project that aims to collect, aggregate, distribute, and document public social data from digital communication platforms in relation to the [[:wikipedia:2019–20 coronavirus pandemic|2019–20 coronavirus pandemic  
This page documents a ''digital observatory'' project that aims to collect, aggregate, distribute, and document public social data from digital communication platforms in relation to the [[:wikipedia:2019–20 coronavirus pandemic|2019–20 coronavirus pandemic]]. The primary goal is to build on existing data collection efforts to make data analysis possible by a wider range of social, health, and computational scientists. The project is being coordinated by the [[Community Data Science Collective]] and [https://pushshift.io/ Pushshift].
]]. The project is being coordinated by the [[Community Data Science Collective]] and [https://pushshift.io/ Pushshift].


== Overview and objectives ==
== Overview and objectives ==
As people struggle to make sense of the COVID-19 pandemic, many of us turn to social media and [[:wikipedia:Social_computing|social computing systems]] to share information, to understand what's happening, and to find new ways to support one another. We are building a digital observatory to understand where and how people are talking about COVID-19-related topics. The observatory collects, aggregates, and distributes social data related to how people are responding to the ongoing public health crisis of COVID-19. The public datasets and freely licensed tools, techniques, and knowledge created through this project will allow researchers, practitioners, and public health officials to more efficiently gather, analyze, understand, and act to improve these crucial sources of information during crises.
As people struggle to make sense of the COVID-19 pandemic, many turn to social media and [[:wikipedia:Social_computing|social computing systems]] to share information, to understand what's happening, and to find new ways to support one another. As scholars, scientists, technologists, and concerned members of the public, we are building a digital observatory to understand where and how people are talking about COVID-19-related topics. The observatory collects, aggregates, and distributes social data related to how people are responding to the ongoing public health crisis of COVID-19. The public datasets and freely licensed tools created through this project will allow researchers, practitioners, and public health officials to more efficiently understand and act to improve these crucial sources of information during crises.
 
The public data we are focused on is available on public webpages and in public APIs but requires technical skills and computational resources that are less widely distributed than the ability to analyze data. In particular, we are attempting to make datasets that researchers can download and analyze on personal computers.  


Everything here is a work in progress as we get the project running, create communication channels, and start releasing datasets. Learn how you can stay connected, use our resources as we produce them, and get involved below.
Everything here is a work in progress as we get the project running, create communication channels, and start releasing datasets. Learn how you can stay connected, use our resources as we produce them, and get involved below.
Line 15: Line 16:
Subscribe to [https://communitydata.science/mailman3/postorius/lists/covid19-announce.communitydata.science/ our low traffic announcement mailing list]. You can fill out the form [https://communitydata.science/mailman3/postorius/lists/covid19-announce.communitydata.science/ on the list website] or email [mailto:covid19-announce-request@communitydata.science covid19-announce-request@communitydata.science] with the word 'help' in the subject or body (don't include the quotes). You will get back a message with instructions.
Subscribe to [https://communitydata.science/mailman3/postorius/lists/covid19-announce.communitydata.science/ our low traffic announcement mailing list]. You can fill out the form [https://communitydata.science/mailman3/postorius/lists/covid19-announce.communitydata.science/ on the list website] or email [mailto:covid19-announce-request@communitydata.science covid19-announce-request@communitydata.science] with the word 'help' in the subject or body (don't include the quotes). You will get back a message with instructions.


The email list will contain occasional updates, information about new datasets, partnerships, and so on.
The email list will contain occasional updates, information about new datasets, partnerships, and so on. We will not use the list or email addresses for other purposes.


==Access Data==
==Resources==


(coming soon)
The digital observatory data, code, and other resources will exist in a few locations, all linked from this page. More details on the different datasets and sources follow below.


Our initial data releases from this project provide a good starting point for investigating social computing and social media content related COVID-19. At first, we will focus on providing static datasets (in raw text / structured formats like csv and JSON) from Wikipedia and related Wikimedia projects as well as search engine results pages (SERPs) for a set of searches on COVID-19 relevant terms. Over time, we plan to include more sources of data (including international/non-English information), localized content specific to particular geographic regions, additional social media and social computing data sources, and searches from more keywords, all delivered more quickly.
Our initial releases should provide a good starting point for investigating social computing and social media content related COVID-19. We're currently releasing three types of material: code, keywords, and data.


In general, the best way to find the data is to look through our Github repository. If you're familiar with git and github, you can clone the repository and that will get you started. For each data source (e.g., SERPs, Wikipedia pageviews, etc.), you should be able to find a data directory and can download files through the Github web interface or your favorite analysis software. We are also working on providing short example analysis scripts that walk through downloading data directly into something like R and producing some minimal analysis to help you get started.
=== Code ===
For code used to produce the data and get started with analysis we have a [https://github.com/CommunityDataScienceCollective/COVID-19_Digital_Observatory github repository] where almost everything lives. If you want to get involved or start using our work please clone the repository! You'll find example analysis scripts that walk through downloading data directly into something like R and producing some minimal analysis to help you get started.


===Search Engine Results Pages (SERP) Data===
The code used to generate the search engine results pages (SERP) data come from Nick Vincent's [https://github.com/nickmvincent/LinkCoordMin SERP scraping project].  
The SERP data in our initial data release includes the first search result page from Google and Bing for a variety of COVID-19 related terms gathered from Google Trends and Google and Bing's autocomplete "search suggestions." Specifically, using a set of six "stem keywords" about COVID-19 and online communities ("coronavirus", "coronavirus reddit", coronavirus wiki", "covid 19", "covid 19 reddit", and "covid 19 wiki"), we collected related keywords from Google Trends (using open source software[https://www.npmjs.com/package/google-trends-api]) and autcomplete suggestions from Google and Bing (using open source software[https://github.com/gitronald/suggests]).


<TODO quick overview format/structure in which data is distributed> Within these compressed archives, each SERP has three data files: a PNG screenshot of the full first page of results, an mhtml "snapshot" (https://github.com/puppeteer/puppeteer/issues/3658), and a json file with a variety of metadata (e.g. date, the device emulated) and a list of every link (<a>) element in the page with its coordinates (top, left, bottom, right) in pixels.
====Keywords====


===Wikipedia article pages and edit history===
We currently use and provide three different types of keywords and search terms:
* Article names/topics from Wikipedia's [[:wikipedia:Wikipedia:WikiProject_COVID-19|WikiProject Covid-19]]
* Wikidata entities generated via the "Main items" described by Wikidata's [https://www.wikidata.org/wiki/Wikidata:WikiProject_COVID-19 WikiProject COVID-19]
* Top 25 daily trending search terms from Google and Bing.


Our initial release provides exhaustive edit data for the list of English Wikipedia articles covered by [[Wikipedia:Wikiproject COVID-19|WikiProject Covid-19]].
We also provide translations of keywords into many languages by collecting translations of labels from Wikidata related to the COVID-19 pandemic.  This is done by passing keywords and  trending Google "related searches" to the Wikidata search API. The resulting Wikidata items are tagged with labels and aliases in many languages.  We hope this provides a useful starting point for searches to discover pandemic related social information in languages beyond English.  Code for this part of the project, including examples for loading the data in Python and R, is under [https://github.com/CommunityDataScienceCollective/COVID-19_Digital_Observatory <code>keywords</code>] in our git repository.  Similarly, resultant data is under [https://covid19.communitydata.science/datasets/keywords/csv/ <code>keywords/csv</code>] on our server.


===Wikipedia pageviews===
===Data ===
The best way to find the data is to visit https://covid19.communitydata.science/datasets/. The <code>search_results</code> directory contains compressed raw data generated by Nick Vincent's [https://github.com/nickmvincent/LinkCoordMin SERP scraping project]. The <code>wikipedia</code> directory has view counts and revision histories for Wikipedia pages of COVID-19-related articles in <code>.json</code> and <code>.tsv</code> format. The <code>keywords</code> directory has <code>.csv</code> files with COVID-19 related keywords translated into many languages and associated Wikidata item identifiers.


Our initial release provides pageview data for the list of English Wikipedia articles covered by WikiProject Covid-19.
====Search Engine Results Pages (SERP) Data====
The SERP data in our initial data release includes the first search result page from Google and Bing for a variety of COVID-19 related terms gathered from Google Trends and Google and Bing's autocomplete "search suggestions." Specifically, using a set of six "stem keywords" about COVID-19 and online communities ("coronavirus", "coronavirus reddit", coronavirus wiki", "covid 19", "covid 19 reddit", and "covid 19 wiki"), we collect related keywords from Google Trends (using open source software[https://www.npmjs.com/package/google-trends-api]) and autocomplete suggestions from Google and Bing (using open source software[https://github.com/gitronald/suggests]). In addition to COVID-19 keywords, we also collect SERP data for the top daily trending queries. Currently, the SERP data collection process does not specify location in its searches. Consequently, the default location used is the location of our machine, at Northwestern University's Evanston campus. We are working on collecting SERP data with location specified beyond the Chicago area (aka other 'localized' content).  


===Keywords and search terms===
The SERP data is released as a series of compressed archives (7z), one archive per day, that follow the naming convention <code>covid_search_data-[YYYYMMDD].7z</code>. You will need a 7z extractor, "7z Opener" on windows worked well for me. Within these compressed archives, there is a folder for each device emulated in the data collection (currently two: Chrome on Windows and iPhone X) which contains all of the respective SERP data. Per each device subdirectory, SERP data itself is organized into folders that are titled by the URL of the search query (e.g. <code>'https---www.google.com-search?q=Krispy Kreme'</code>), and each SERP folder contains three data files:
* a PNG screenshot of the full first page of results,
* an mhtml "snapshot" (https://github.com/puppeteer/puppeteer/issues/3658),
* and a json file with a variety of metadata (e.g. date, the device emulated) and a list of every link (<a>) element in the page with its coordinates (top, left, bottom, right) in pixels.


We currently use and provide three different types of keywords and search terms:
====Wikipedia data====
* Article names/topics from Wikipedia's [[Wikipedia:WikiProject_COVID-19|WikiProject Covid-19]]
 
* Wikidata entities generated via the "Main items" described by Wikidata's [https://www.wikidata.org/wiki/Wikidata:WikiProject_COVID-19 WikiProject COVID-19]
Our initial release provides exhaustive edit and [[Wikipedia:Pageview_statistics|pageview data]] for the list of English Wikipedia articles covered by [[Wikipedia:Wikiproject COVID-19|WikiProject Covid-19]]. Please note that the edit JSON data of revisions include the full text of every revision made to articles in [[:wikipedia:Wikipedia:WikiProject COVID-19|English Wikipedia's Wikiproject COVID-19]]. They are highly compressed and and expand to between 20G and 200GB of data per day. Depending on the computer you use, it may not work to load them into memory all at once for analysis.
* Top 25 daily trending search terms from Google and Bing.
 
Each are updated daily and we are working to add historical data from all other language Wikipedia editions.


==Get Help Using Data==
==Get Help Using Data==


(coming soon)
As we develop data collection resources and datasets, we will also provide simple example analysis scripts to demonstrate how you might access, import, and analyze small subsets of the data we produce. For instance, take a look at the [https://github.com/CommunityDataScienceCollective/COVID-19_Digital_Observatory/tree/master/wikipedia/example_analysis "example analysis" subdirectory] of the wikipedia section of our Github project.
 
We plan to develop tutorials and demos for the use of the data we release and particularly welcome contributions that help make these resources more easily usable by others. In some cases, the data sources are quite large and might not be suitable for analysis on your personal computer. Wherever possible, we'll try to build examples that only ingest a small subset of data and/or point you to useful tools to help make larger scale analyses feasible or easier.


We will also develop tutorials and demos for the use of the data we release. In some cases, the data sources are quite large and might not be suitable for analysis on your personal computer. Wherever possible, we'll try to point you to useful tools to help make this feasible or easier.
[[SERP Tips | Tips for working with SERP data]]


==Contribute==
==Contribute==
Line 58: Line 70:
* Want to contribute to our datasets and/or analysis code? Clone our [https://github.com/CommunityDataScienceCollective/COVID-19_Digital_Observatory github repository] and pitch in with pull requests.
* Want to contribute to our datasets and/or analysis code? Clone our [https://github.com/CommunityDataScienceCollective/COVID-19_Digital_Observatory github repository] and pitch in with pull requests.


==Code of Conduct==
In terms of conduct, we expect all contributors adhere to the [https://www.contributor-covenant.org/ Contributor Covenant].
We ask that all contributors adhere to the [https://www.contributor-covenant.org/ Contributor Covenant].
 
== Related projects ==
 
This is an incomplete list of related projects, several of which have additional and more comprehensive lists of related projects. Please add more!
 
* [https://hub.jhu.edu/2020/03/27/mark-dredze-social-media-misinformation/?mc_cid=4a12581bb9&mc_eid=edb6743d32 Social media for public health] (Johns Hopkins University + University of Maryland + GWU). Includes a lengthy [http://www.socialmediaforpublichealth.org/covid-19/resources/ list of resources and related projects].
* [https://www.notion.so/977d5e5be0434bf996704ec361ad621d?v=aa8e0c75520a479ea48f56cb4c289b7e COVID-19 Containment measures data] from the COVID-19 Forecasting project (University of Oxford).
* [https://healthgeolab.net/resources/covid-19/ Health GeoLab Collaborative] listing of (mostly geospatial) data resources related to COVID-19.
* [https://covid19obs.fbk.eu/ COVID-19 Infodemics Observatory] analyzing Twitter data. Created by the [https://comunelab.fbk.eu/CoMuNe Laboratory of the FBK research unit]. 
* [https://www.archiveteam.org/index.php?title=Coronavirus Archive team listing of archive sites related to the Coronavirus]
* [https://github.com/nychealth/coronavirus-data Repository with data on COVID-19 from the NYC Department of Health and Mental Hygiene (DOHMH)]
* The Citizens and Technology Lab is [https://covid-algotracker.citizensandtech.org/ tracking COVID-related posts on the Reddit front page] (Cornell and J. Nathan Mattias).
* [https://covid-data.wmflabs.org/ General statistics about COVID-19 editing in Wikipedia projects] (Diego Saez from the Wikimedia Foundation)
* [https://github.com/nytimes/covid-19-data The New York Times data files] with cumulative counts of coronavirus cases in the United States, at the state and county level, over time.

Latest revision as of 15:39, 1 August 2023

This project and project page are under active development.
Microscopy image of the virus which causes COVID-19.

This page documents a digital observatory project that aims to collect, aggregate, distribute, and document public social data from digital communication platforms in relation to the 2019–20 coronavirus pandemic. The primary goal is to build on existing data collection efforts to make data analysis possible by a wider range of social, health, and computational scientists. The project is being coordinated by the Community Data Science Collective and Pushshift.

Overview and objectives[edit]

As people struggle to make sense of the COVID-19 pandemic, many turn to social media and social computing systems to share information, to understand what's happening, and to find new ways to support one another. As scholars, scientists, technologists, and concerned members of the public, we are building a digital observatory to understand where and how people are talking about COVID-19-related topics. The observatory collects, aggregates, and distributes social data related to how people are responding to the ongoing public health crisis of COVID-19. The public datasets and freely licensed tools created through this project will allow researchers, practitioners, and public health officials to more efficiently understand and act to improve these crucial sources of information during crises.

The public data we are focused on is available on public webpages and in public APIs but requires technical skills and computational resources that are less widely distributed than the ability to analyze data. In particular, we are attempting to make datasets that researchers can download and analyze on personal computers.

Everything here is a work in progress as we get the project running, create communication channels, and start releasing datasets. Learn how you can stay connected, use our resources as we produce them, and get involved below.

Stay connected[edit]

Subscribe to our low traffic announcement mailing list. You can fill out the form on the list website or email covid19-announce-request@communitydata.science with the word 'help' in the subject or body (don't include the quotes). You will get back a message with instructions.

The email list will contain occasional updates, information about new datasets, partnerships, and so on. We will not use the list or email addresses for other purposes.

Resources[edit]

The digital observatory data, code, and other resources will exist in a few locations, all linked from this page. More details on the different datasets and sources follow below.

Our initial releases should provide a good starting point for investigating social computing and social media content related COVID-19. We're currently releasing three types of material: code, keywords, and data.

Code[edit]

For code used to produce the data and get started with analysis we have a github repository where almost everything lives. If you want to get involved or start using our work please clone the repository! You'll find example analysis scripts that walk through downloading data directly into something like R and producing some minimal analysis to help you get started.

The code used to generate the search engine results pages (SERP) data come from Nick Vincent's SERP scraping project.

Keywords[edit]

We currently use and provide three different types of keywords and search terms:

  • Article names/topics from Wikipedia's WikiProject Covid-19
  • Wikidata entities generated via the "Main items" described by Wikidata's WikiProject COVID-19
  • Top 25 daily trending search terms from Google and Bing.

We also provide translations of keywords into many languages by collecting translations of labels from Wikidata related to the COVID-19 pandemic. This is done by passing keywords and trending Google "related searches" to the Wikidata search API. The resulting Wikidata items are tagged with labels and aliases in many languages. We hope this provides a useful starting point for searches to discover pandemic related social information in languages beyond English. Code for this part of the project, including examples for loading the data in Python and R, is under keywords in our git repository. Similarly, resultant data is under keywords/csv on our server.

Data[edit]

The best way to find the data is to visit https://covid19.communitydata.science/datasets/. The search_results directory contains compressed raw data generated by Nick Vincent's SERP scraping project. The wikipedia directory has view counts and revision histories for Wikipedia pages of COVID-19-related articles in .json and .tsv format. The keywords directory has .csv files with COVID-19 related keywords translated into many languages and associated Wikidata item identifiers.

Search Engine Results Pages (SERP) Data[edit]

The SERP data in our initial data release includes the first search result page from Google and Bing for a variety of COVID-19 related terms gathered from Google Trends and Google and Bing's autocomplete "search suggestions." Specifically, using a set of six "stem keywords" about COVID-19 and online communities ("coronavirus", "coronavirus reddit", coronavirus wiki", "covid 19", "covid 19 reddit", and "covid 19 wiki"), we collect related keywords from Google Trends (using open source software[1]) and autocomplete suggestions from Google and Bing (using open source software[2]). In addition to COVID-19 keywords, we also collect SERP data for the top daily trending queries. Currently, the SERP data collection process does not specify location in its searches. Consequently, the default location used is the location of our machine, at Northwestern University's Evanston campus. We are working on collecting SERP data with location specified beyond the Chicago area (aka other 'localized' content).

The SERP data is released as a series of compressed archives (7z), one archive per day, that follow the naming convention covid_search_data-[YYYYMMDD].7z. You will need a 7z extractor, "7z Opener" on windows worked well for me. Within these compressed archives, there is a folder for each device emulated in the data collection (currently two: Chrome on Windows and iPhone X) which contains all of the respective SERP data. Per each device subdirectory, SERP data itself is organized into folders that are titled by the URL of the search query (e.g. 'https---www.google.com-search?q=Krispy Kreme'), and each SERP folder contains three data files:

  • a PNG screenshot of the full first page of results,
  • an mhtml "snapshot" (https://github.com/puppeteer/puppeteer/issues/3658),
  • and a json file with a variety of metadata (e.g. date, the device emulated) and a list of every link (<a>) element in the page with its coordinates (top, left, bottom, right) in pixels.

Wikipedia data[edit]

Our initial release provides exhaustive edit and pageview data for the list of English Wikipedia articles covered by WikiProject Covid-19. Please note that the edit JSON data of revisions include the full text of every revision made to articles in English Wikipedia's Wikiproject COVID-19. They are highly compressed and and expand to between 20G and 200GB of data per day. Depending on the computer you use, it may not work to load them into memory all at once for analysis.

Each are updated daily and we are working to add historical data from all other language Wikipedia editions.

Get Help Using Data[edit]

As we develop data collection resources and datasets, we will also provide simple example analysis scripts to demonstrate how you might access, import, and analyze small subsets of the data we produce. For instance, take a look at the "example analysis" subdirectory of the wikipedia section of our Github project.

We plan to develop tutorials and demos for the use of the data we release and particularly welcome contributions that help make these resources more easily usable by others. In some cases, the data sources are quite large and might not be suitable for analysis on your personal computer. Wherever possible, we'll try to build examples that only ingest a small subset of data and/or point you to useful tools to help make larger scale analyses feasible or easier.

Tips for working with SERP data

Contribute[edit]

We are eager for collaborators and committed to working openly.

In terms of conduct, we expect all contributors adhere to the Contributor Covenant.

Related projects[edit]

This is an incomplete list of related projects, several of which have additional and more comprehensive lists of related projects. Please add more!