Zooniverse research (2016-2017 archive)

Playground for Jim, Darren, and Aaron to build up a project analyzing volunteer management in the Zooniverse.

Action items for next meeting

 * give feedback on draft planning document 1st page (JM)
 * continue to build out draft planning document w feedback (AS)
 * consider incorporating Q's/H's that use only metadata
 * keep fixing wiki page (JM & AS)
 * stay on top of panoptes data access (JM)
 * inquiries about other data attributes (JM):
 * where does the text of talk comments live?
 * final/correct classifications?
 * build preliminary variables/tables from ouroborous data (JM)
 * classifications (w/out final answers)
 * users (w/out content of talk)

Goals
Winter 2017
 * Start building data tables from existing Ouroboros data.
 * Build up theoretical framework and argumentation for models
 * Model data to analyze communication and productivity
 * Think about publishing to CSCW 2017 (April) as a stretch goal

Fall 2016
 * Aquire Panoptes data (either dump or API access from Zooniverse developers (Adam and Marten)

Spring 2016
 * Fully developed planning document including a thorough analysis plan.
 * A cleaned up dataset (variables and measures that we can model; not just raw database records).

Preliminary Project Overview
Understanding Effective Leadership and Volunteer Engagement in Citizen Science

Rationale:

Prior research investigating the mechanisms of effective crowd work and citizen science has focused on volunteer/worker motivations, algorithmic and computational techniques for quality control/improvement, and task/incentive design. Despite a number of findings indicating that the patterns of interaction between requesters and volunteers/workers shape participant experiences and work contributions in important ways, these interactions remain largely unexplored as a domain of analysis, design, and intervention. This is arguably an especially important design space in the context of citizen science, where "the crowd" consists entirely of volunteers and poor volunteer management or relations would likely result in complete project failure.

As task requesters in citizen science are scientists who perform important leadership functions for the project as a whole, we draw on prior literature analyzing effective leadership in volunteer online communities. Several studies by Haiyi Zhu and colleagues in the context of peer production show that "shared leadership" behaviors increase newcomer motivation and participation along multiple dimensions, but may adversely affect experienced participant motivation and participation.

This study will contribute a comparative analysis of requester-volunteer interactions in a sample of citizen science projects from within the Zooniverse platform. Building on literature on crowd work, citizen science, and studies of effective leadership in online communities and peer production, we construct models to test whether shared leadership behaviors lead to enhanced citizen scientist engagement.

Analytic approach:

Use observational data drawn from Zooniverse database records to build variables and multilevel models that test whether science team member behaviors and science team - volunteer interactions consistent with shared leadership predict increased volunteer engagement and higher quality contributions.

Dependent variables (all nested within projects):


 * Count of volunteer contributions
 * Median/total quality of contributions (using gold-standard data where available)
 * Volunteer survival rate: time on zooniverse platform, completed project (0/1)
 * Number of active volunteers throughout project?
 * Risk of highly negative interactions between volunteers and science team members (if we can identify these)
 * from my observations I have yet to find these.
 * these seems like an independent, not a dependent variable

Theory
shared leadership (from Zhu et al.)
 * transactional (positive feedback)
 * aversive (negative feedback)
 * directive (directive message)
 * person focused (social message)

 Group Identity (from Ren et al.)
 * identity attachment to group within the community
 * bond based attachment to individual members
 * attachment to the large community

 Network Exchange Patters (from Faraj & Johnson) 
 * direct reciporcity
 * indirect reciprocity
 * preferential attachment

social Q&A?

Folk-knowledge

 * building a great project
 * built in cooperation with volunteers.
 * discuss your research goals (especially on project page)
 * two way communication--volunteer testing
 * the launch rush
 * Write a newsletter.
 * have a promotion plan, recruit from outside zooniverse
 * use talk during initial launch
 * moderators for talk
 * in for the long hall
 * reminder emails
 * create community--more commitment from volunteers involved in talk
 * positive feedback, report on accomplishments

Observations

 * information seeking

Data and measures we want

 * Users for every project we study
 * Account creation date / project
 * Projects active in
 * Tags for various roles (participant, moderator, science team, zooniverse employee)
 * Dates roles acquired/dropped

Resources & Links
08-02-16

Jim's Notes from meeting w/ Zooniverse

08-01-16

Open Questions for Zooniverse Researchers

07-28-16

Project Proposal for Zooniverse Researchers

API endpoint: 'https://api.zooniverse.org/projects//status'

04-25-16

BYOR Planning Document

typology of leadership and volunteer mgmt practices

04-13-16

Zooniverse Best Practices

04-04-16 to 04-19-16

 * we should split between volunteer posts and scientist posts to forums
 * volunteers and scientists use the forums for different purposes
 * "folk-knowledge" only applies to scientists/admins
 * jungle rhythms uses talk to engage community, single researcher/moderator?
 * use of talk is somewhat task dependent?
 * small community, but still no social leadership
 * all the same researcher as point of contact? creates community?  form of social leadership?
 * different communities have different methods of linking researchers to volunteers
 * Shakespeare world has explicit thread in forum
 * Some use blogs

BYOR 04-18-16

 * do a better job explaining Zooniverse (the platform), the roles within the community, and the types of interaction on the community forums
 * refine research questions
 * first question is causal but the research design doesn't support causal inference
 * second question can't be answered by independent variables as currently defined. Consider using qualitative methods and/or NLP
 * define "success" in concrete terms (success of the project vs success of the interaction), success is actually a collective outcome
 * define measures of quality
 * control for individual level variables (one of the
 * relevant lit
 * community management
 * kickerstarter


 * Aaron's notes (many redundant)
 * Read syracuse folks
 * Talk to Adler
 * What is it? Explain Zooniverse a bit more.
 * Explain measures
 * Explain puzzle/interest x2
 * Explain how you are gathering qualitative data
 * First question seems to require causal identification
 * Second question seems like it needs text analysis
 * What does this communication look like? What roles do volunteers take? x2
 * Community mgmt literature?
 * Refine RQs
 * Explain variables in relation to invididual level (vs conversation level?)

04-04-16 to 04-13-16
Zooniverse Best Practices


 * building a great project
 * Some of our most successful projects in terms of engagement have been those built in cooperation with volunteers. (all)
 * You can (and should) discuss your research goals in depth on your project’s Research page. (directive)
 * two way communication--volunteer testing (including the formal Zooniverse "project review" stage) will reveal things to alter (directive, social)
 * the launch rush
 * importance of the launch period, most traffic during initial spike
 * Write a newsletter. (directive? informational?)
 * have a promotion plan, recruit from outside zooniverse
 * use talk during initial launch (all)
 * moderators for talk (all)
 * in for the long hall
 * reminder emails (all)
 * create community--more commitment from volunteers involved in talk (all, social)
 * positive feedback, report on accomplishments (transactional)

notes from observations (classification shared doc)


 * question level vs. board/discussion level
 * scientist feedback and volunteer feedback--supports shared leadership
 * most feedback isn't exactly "leadership". responding to requests for information, information sharing, not exactly directive leadership.
 * single response vs multi-response questions
 * questions from research team vs questions from public
 * most chat is response to questions
 * not much traditional "leadership" because instructions are pretty clear/explicit. most discussion above & beyond the project
 * only examples of transactional (and possibly directive) leadership
 * we might find examples of social leadership, but likely in smaller projects
 * could "questions asked" be an independent variable?


 * is an informational post directive leadership? is it directive leadership when it responds to a question?
 * information sharing is an important component of engaging volunteers, but isn't necessarily "leadership"
 * can we find lit on information sharing? communities of practice?
 * responses to questions might be a good indication of leadership
 * sometimes responses are just informational


 * types of contributors
 * volunteer
 * moderator
 * scientist
 * zooniverse employee
 * admin (spacewarps)

 independent vars 


 * levels: question, board, project
 * question lvl vars (possibly omit this level because it's too complicated and not that helpful):
 * is question: 0/1
 * response (is two way communication): 0/1
 * response time
 * poster role
 * answer role
 * leadership? (transactional, aversive, directive, person focused)
 * discussion lvl vars:
 * object/discussion
 * num posts
 * num contributors
 * num responses (in response to)
 * avg response time
 * num volunteers, num mods, num scientists, num zooniverse team
 * num leadership (transactional, aversive, directive, person focused)
 * project level vars
 * num volunteers, num mods, num scientists, num zooniverse team
 * num discussions
 * num objects
 * project length (time)
 * frequency of blog posts
 * length of blog posts

01-27-17
Jim and Aaron reviewed Aaron's revised project proposal and discussed data that still needs to be acquired. Next week Jim will pursue missing data, and Aaron will continue to work on the proposal. Jim (and hopefully Darren) will provide feedback on the proposal.

01-20-17
Jim and Aaron discussed project goals for the winter and spring quarters, including analytical goals, timelines, and publishing strategies. The project will analyze communication between scientists and volunteers, and the effect/association with productivity. Moving forward, Aaron will work up the theoretical framing, and Jim will wrangle the data.

08-11-16
Talked about API access, set up git repo and server access, and discussed future directions for paper based on Graham et al's work.

08-02-16
Met with Zooniverse Researchers to discuss project proposal and data acquisition. See Jim's Notes from meeting w/ Zooniverse

06-23-2016
JM DG & AS narrowed research focus to project outcomes associated with information seeking, specifically tracking users who have engaged in information seeking and judging performance before and after.

04-26-2016
Jim circulated the BYOR Planning Document and typology of leadership and volunteer mgmt practices, and concluded that "shared leadership" might not be the best conceptual framing. The group brainstormed updated RQs and methods of analysis.

04-19-16
Darren joined the meeting/project. Discussed more observations, recent readings, and responses to BYOR feedback. Set goals to build a leadership typology and plan revisions to planning document.

04-13-16
Discussed "triangulation" (theory, observations, folk-knowledge). Discussed observations from community talk and best practices and connections to literature. Created list of other relevant theory and literature.

04-04-16
Talked about predicted probabilities for unrelated projects. Revised and updated work goals for next week.

03-29-16
Laid out plans for the quarter. Set some preliminary work goals.