Editing Human Centered Data Science (Fall 2019)/Schedule

From CommunityData

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.

Latest revision Your text
Line 20: Line 20:
  
 
=== Week 1: September 26 ===
 
=== Week 1: September 26 ===
 +
[[HCDS_(Fall_2019)/Day_1_plan|Day 1 plan]]
 
<!--
 
<!--
 
[[:File:HCDS_2018_week_1_slides.pdf|Day 1 slides]]
 
[[:File:HCDS_2018_week_1_slides.pdf|Day 1 slides]]
Line 27: Line 28:
  
 
;Assignments due
 
;Assignments due
* Fill out the [https://docs.google.com/forms/d/e/1FAIpQLSffoC-Dd2eYtiWr00ZoRcaTc9eeaK_lySaAVDTX2ZTj_lHIFA/viewform?usp=sf_link pre-course survey]
+
* Fill out the pre-course survey
* Read ('''not graded'''): Provost, Foster, and Tom Fawcett. [http://online.liebertpub.com/doi/pdf/10.1089/big.2013.1508 ''Data science and its relationship to big data and data-driven decision making.''] Big Data 1.1 (2013): 51-59.  
+
* Read: Provost, Foster, and Tom Fawcett. [http://online.liebertpub.com/doi/pdf/10.1089/big.2013.1508 ''Data science and its relationship to big data and data-driven decision making.''] Big Data 1.1 (2013): 51-59.
  
 
;Agenda
 
;Agenda
* Syllabus review
+
{{:HCDS (Fall 2019)/Day 1 plan}}
* Pre-course survey results
 
* What do we mean by ''data science?''
 
* What do we mean by ''human centered?''
 
* How does human centered design relate to data science?
 
* In-class activity
 
* Intro to assignment 1: Data Curation
 
  
 
;Homework assigned
 
;Homework assigned
Line 46: Line 41:
  
 
;Resources
 
;Resources
* Princeton Dialogues on AI & Ethics: ''[https://aiethics.princeton.edu/case-studies/ Case studies]''
 
 
* Aragon, C. et al. (2016). [https://cscw2016hcds.files.wordpress.com/2015/10/cscw_2016_human-centered-data-science_workshop.pdf ''Developing a Research Agenda for Human-Centered Data Science.''] Human Centered Data Science workshop, CSCW 2016.
 
* Aragon, C. et al. (2016). [https://cscw2016hcds.files.wordpress.com/2015/10/cscw_2016_human-centered-data-science_workshop.pdf ''Developing a Research Agenda for Human-Centered Data Science.''] Human Centered Data Science workshop, CSCW 2016.
 
* Kling, Rob and Star, Susan Leigh. [https://scholarworks.iu.edu/dspace/bitstream/handle/2022/1798/wp97-04B.html ''Human Centered Systems in the Perspective of Organizational and Social Informatics.''] 1997.
 
* Kling, Rob and Star, Susan Leigh. [https://scholarworks.iu.edu/dspace/bitstream/handle/2022/1798/wp97-04B.html ''Human Centered Systems in the Perspective of Organizational and Social Informatics.''] 1997.
 
* Harford, T. (2014). ''[http://doi.org/10.1111/j.1740-9713.2014.00778.x Big data: A big mistake?]'' Significance, 11(5), 14–19.
 
* Harford, T. (2014). ''[http://doi.org/10.1111/j.1740-9713.2014.00778.x Big data: A big mistake?]'' Significance, 11(5), 14–19.
* Ideo.org [http://www.designkit.org/ ''The Field Guide to Human-Centered Design.''] 2015.
+
<!-- * Ideo.org [http://www.designkit.org/ ''The Field Guide to Human-Centered Design.''] 2015. -->
 +
 
 
<br/>
 
<br/>
 
<hr/>
 
<hr/>
Line 56: Line 51:
  
 
=== Week 2: October 3 ===
 
=== Week 2: October 3 ===
 +
[[HCDS_(Fall_2019)/Day_2_plan|Day 2 plan]]
 +
 
<!--
 
<!--
 
[[:File:HCDS_2019_week_2_slides.pdf|Day 2 slides]]
 
[[:File:HCDS_2019_week_2_slides.pdf|Day 2 slides]]
Line 67: Line 64:
  
 
;Agenda
 
;Agenda
* Reading reflection discussion
+
{{:HCDS (Fall 2019)/Day 2 plan}}
* Assignment 1 review & reflection
 
* A primer on copyright, licensing, and hosting for code and data
 
* Introduction to replicability, reproducibility, and open research
 
* In-class activity
 
* Intro to assignment 2: Bias in data
 
  
 
;Homework assigned
 
;Homework assigned
* Read and reflect: Duarte, N., Llanso, E., & Loup, A. (2018). ''[https://cdt.org/files/2017/12/FAT-conference-draft-2018.pdf Mixed Messages? The Limits of Automated Social Media Content Analysis].'' Proceedings of the 1st Conference on Fairness, Accountability and Transparency, 81, 106.
+
* Read and reflect: Olteanu, A., Castillo, C., Diaz, F., Kıcıman, E., & Kiciman, E. (2019). ''[https://www.frontiersin.org/articles/10.3389/fdata.2019.00013/pdf Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries].'' Frontiers in Big Data, 2, 13. https://doi.org/10.3389/fdata.2019.00013
 
* [[Human_Centered_Data_Science_(Fall_2019)/Assignments#A2:_Bias_in_data|A2: Bias in data]]
 
* [[Human_Centered_Data_Science_(Fall_2019)/Assignments#A2:_Bias_in_data|A2: Bias in data]]
  
 
;Resources
 
;Resources
 
* Hickey, Walt. [https://fivethirtyeight.com/features/the-bechdel-test-checking-our-work/ ''The Bechdel Test: Checking Our Work'']. FiveThirtyEight, 2014.
 
* Hickey, Walt. [https://fivethirtyeight.com/features/the-bechdel-test-checking-our-work/ ''The Bechdel Test: Checking Our Work'']. FiveThirtyEight, 2014.
* GroupLens, ''[https://grouplens.org/datasets/movielens/ MovieLens datasets]''
 
 
* J. Priem, D. Taraborelli, P. Groth, C. Neylon (2010), ''[http://altmetrics.org/manifesto Altmetrics: A manifesto]'', 26 October 2010.
 
* J. Priem, D. Taraborelli, P. Groth, C. Neylon (2010), ''[http://altmetrics.org/manifesto Altmetrics: A manifesto]'', 26 October 2010.
* Chapter 2 [https://www.practicereproducibleresearch.org/core-chapters/2-assessment.html "Assessing Reproducibility"] and Chapter 3 [https://www.practicereproducibleresearch.org/core-chapters/3-basic.html "The Basic Reproducible Workflow Template"] from ''The Practice of Reproducible Research'' University of California Press, 2018.
+
 
* Halfaker, A., Geiger, R. S., Morgan, J. T., & Riedl, J. (2013). ''[https://www-users.cs.umn.edu/~halfaker/publications/The_Rise_and_Decline/halfaker13rise-preprint.pdf The rise and decline of an open collaboration system: How Wikipedia’s reaction to popularity is causing its decline].'' American Behavioral Scientist, 57(5), 664-688
+
<!--  
 
* TeBlunthuis, N., Shaw, A., and Hill, B.M. (2018). Revisiting "The rise and decline" in a population of peer production projects. In ''Proceedings of the 2018 ACM Conference on Human Factors in Computing Systems (CHI '18)''. https://doi.org/10.1145/3173574.3173929
 
* TeBlunthuis, N., Shaw, A., and Hill, B.M. (2018). Revisiting "The rise and decline" in a population of peer production projects. In ''Proceedings of the 2018 ACM Conference on Human Factors in Computing Systems (CHI '18)''. https://doi.org/10.1145/3173574.3173929
 
* Press, Gil. [https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#2608257f6f63 ''Cleaning Big Data: Most Time-Consuming, Least Enjoyable Data Science Task, Survey Says.''] Forbes, 2016.
 
* Press, Gil. [https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#2608257f6f63 ''Cleaning Big Data: Most Time-Consuming, Least Enjoyable Data Science Task, Survey Says.''] Forbes, 2016.
 
* Christensen, Garret. [https://github.com/garretchristensen/BestPracticesManual/blob/master/Manual.pdf ''Manual of Best Practices in Transparent Social Science Research.''] 2016.
 
* Christensen, Garret. [https://github.com/garretchristensen/BestPracticesManual/blob/master/Manual.pdf ''Manual of Best Practices in Transparent Social Science Research.''] 2016.
<!--
 
 
* Aschwanden, Christie. [https://fivethirtyeight.com/features/science-isnt-broken/ ''Science Isn't Broken''] FiveThirtyEight, 2015.
 
* Aschwanden, Christie. [https://fivethirtyeight.com/features/science-isnt-broken/ ''Science Isn't Broken''] FiveThirtyEight, 2015.
 +
-->
 +
 +
;Assignment 1 [[Human_Centered_Data_Science_(Fall_2019)/Assignments#A1:_Data_curation|Data curation]] resources:
 +
 +
*Chapter 2 [https://www.practicereproducibleresearch.org/core-chapters/2-assessment.html "Assessing Reproducibility"] and Chapter 3 [https://www.practicereproducibleresearch.org/core-chapters/3-basic.html "The Basic Reproducible Workflow Template"] from ''The Practice of Reproducible Research'' University of California Press, 2018.
 
* sample code for API calls ([http://paws-public.wmflabs.org/paws-public/User:Jtmorgan/data512_a1_example.ipynb view the notebook], [http://paws-public.wmflabs.org/paws-public/User:Jtmorgan/data512_a1_example.ipynb?format=raw download the notebook]).
 
* sample code for API calls ([http://paws-public.wmflabs.org/paws-public/User:Jtmorgan/data512_a1_example.ipynb view the notebook], [http://paws-public.wmflabs.org/paws-public/User:Jtmorgan/data512_a1_example.ipynb?format=raw download the notebook]).
 
*''See [[Human_Centered_Data_Science/Datasets#Dataset_documentation_examples|the datasets page]] for examples of well-documented and not-so-well documented open datasets.''
 
*''See [[Human_Centered_Data_Science/Datasets#Dataset_documentation_examples|the datasets page]] for examples of well-documented and not-so-well documented open datasets.''
-->
+
 
 
<br/>
 
<br/>
 
<hr/>
 
<hr/>
Line 97: Line 92:
  
 
=== Week 3: October 10 ===
 
=== Week 3: October 10 ===
 +
[[HCDS_(Fall_2019)/Day_3_plan|Day 3 plan]]
 
<!--
 
<!--
 
[[:File:HCDS 2019 week 3 slides.pdf|Day 3 slides]]
 
[[:File:HCDS 2019 week 3 slides.pdf|Day 3 slides]]
Line 107: Line 103:
  
 
;Agenda
 
;Agenda
* Reading reflection review
+
{{:HCDS (Fall 2019)/Day 3 plan}}
* Sources and consequences of bias in data collection, processing, and re-use
+
 
* In-class activity
+
;Readings assigned (Read both, reflect on one)
 +
* Wang, Tricia. ''[https://medium.com/ethnography-matters/why-big-data-needs-thick-data-b4b3e75e3d7 Why Big Data Needs Thick Data]''. Ethnography Matters, 2016.
 +
* Shilad Sen, Margaret E. Giesel, Rebecca Gold, Benjamin Hillmann, Matt Lesicko, Samuel Naden, Jesse Russell, Zixiao (Ken) Wang, and Brent Hecht. 2015. ''[http://www-users.cs.umn.edu/~bhecht/publications/goldstandards_CSCW2015.pdf Turkers, Scholars, "Arafat" and "Peace": Cultural Communities and Algorithmic Gold Standards]''. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15)
  
 
;Homework assigned
 
;Homework assigned
* Read both, reflect on one:
+
* Reading reflection
:* Wang, Tricia. ''[https://medium.com/ethnography-matters/why-big-data-needs-thick-data-b4b3e75e3d7 Why Big Data Needs Thick Data]''. Ethnography Matters, 2016.
 
:* Kery, M. B., Radensky, M., Arya, M., John, B. E., & Myers, B. A. (2018). ''[https://marybethkery.com/projects/Verdant/Kery-The-Story-in-the-Notebook-Exploratory-Data-Science-using-a-Literate-Programming-Tool.pdf The Story in the Notebook: Exploratory Data Science using a Literate Programming Tool]''. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI’18, 1–11. https://doi.org/10.1145/3173574.3173748
 
  
 
;Resources
 
;Resources
 +
* Olteanu, A., Castillo, C., Diaz, F., & Kiciman, E. (2016). ''[http://kiciman.org/wp-content/uploads/2017/08/SSRN-id2886526.pdf Social data: Biases, methodological pitfalls, and ethical boundaries].
 +
* Brian N Larson. 2017. ''[http://www.ethicsinnlp.org/workshop/pdf/EthNLP04.pdf Gender as a Variable in Natural-Language Processing: Ethical Considerations]. EthNLP, 3: 30–40.
 
* Bender, E. M., & Friedman, B. (2018). [https://openreview.net/forum?id=By4oPeX9f Data Statements for NLP: Toward Mitigating System Bias and Enabling Better Science]. To appear in Transactions of the ACL.
 
* Bender, E. M., & Friedman, B. (2018). [https://openreview.net/forum?id=By4oPeX9f Data Statements for NLP: Toward Mitigating System Bias and Enabling Better Science]. To appear in Transactions of the ACL.
* Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumeé III, H., & Crawford, K. (2018). [https://www.fatml.org/media/documents/datasheets_for_datasets.pdf Datasheets for datasets]. arXiv preprint arXiv:1803.09010.
+
* Isaac L. Johnson, Yilun Lin, Toby Jia-Jun Li, Andrew Hall, Aaron Halfaker, Johannes Schöning, and Brent Hecht. 2016. ''[http://delivery.acm.org/10.1145/2860000/2858123/p13-johnson.pdf?ip=209.166.92.236&id=2858123&acc=CHORUS&key=4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E6D218144511F3437&__acm__=1539880715_eb477907771cea4ecaabc953094c3080 Not at Home on the Range: Peer Production and the Urban/Rural Divide].'' CHI '16. DOI: https://doi.org/10.1145/2858036.2858123
* Olteanu, A., Castillo, C., Diaz, F., Kıcıman, E., & Kiciman, E. (2019). ''[https://www.frontiersin.org/articles/10.3389/fdata.2019.00013/pdf Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries].'' Frontiers in Big Data, 2, 13. https://doi.org/10.3389/fdata.2019.00013
+
* Leo Graiden Stewart, Ahmer Arif, A. Conrad Nied, Emma S. Spiro, and Kate Starbird. 2017. ''[https://faculty.washington.edu/kstarbi/Stewart_Starbird_Drawing_the_Lines_of_Contention-final.pdf Drawing the Lines of Contention: Networked Frame Contests Within #BlackLivesMatter Discourse].'' Proc. ACM Hum.-Comput. Interact. 1, CSCW, Article 96 (December 2017), 23 pages. DOI: https://doi.org/10.1145/3134920
* Rose Eveleth ''[https://www.vox.com/the-highlight/2019/10/1/20887003/tech-technology-evolution-natural-inevitable-ethics The biggest lie tech people tell themselves — and the rest of us].'' October 8, 2019, Vox.com.
+
* Cristian Danescu-Niculescu-Mizil, Robert West, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. ''[https://web.stanford.edu/~jurafsky/pubs/linguistic_change_lifecycle.pdf No country for old members: user lifecycle and linguistic change in online communities].'' In Proceedings of the 22nd international conference on World Wide Web (WWW '13). ACM, New York, NY, USA, 307-318. DOI: https://doi.org/10.1145/2488388.2488416  
* Rani Molla ''[https://www.vox.com/2019/2/8/18211794/government-data-internet The government is using the wrong data to make crucial decisions about the internet].'' February 8, 2019, Vox.com.
+
<!-- * Astrid Mager. 2012. Algorithmic ideology: How capitalist society shapes search engines. Information, Communication & Society 15, 5: 769–787. http://doi.org/10.1080/1369118X.2012.676056 (in Canvas) -->
* Isaac L. Johnson, Yilun Lin, Toby Jia-Jun Li, Andrew Hall, Aaron Halfaker, Johannes Schöning, and Brent Hecht. 2016. Not at Home on the Range: Peer Production and the Urban/Rural Divide. DOI: https://doi.org/10.1145/2858036.2858123  
 
* Leo Graiden Stewart, Ahmer Arif, A. Conrad Nied, Emma S. Spiro, and Kate Starbird. 2017. Drawing the Lines of Contention: Networked Frame Contests Within #BlackLivesMatter Discourse. CSCW 2017. DOI: https://doi.org/10.1145/3134920
 
* Lada A. Adamic and Natalie Glance. 2005. The political blogosphere and the 2004 U.S. election: divided they blog. (LinkKDD '05). DOI=http://dx.doi.org/10.1145/1134271.1134277
 
* Cristian Danescu-Niculescu-Mizil, Robert West, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. No country for old members: user lifecycle and linguistic change in online communities. (WWW '13). DOI: https://doi.org/10.1145/2488388.2488416  
 
 
<br/>
 
<br/>
 
<hr/>
 
<hr/>
Line 131: Line 125:
  
 
=== Week 4: October 17 ===
 
=== Week 4: October 17 ===
 +
[[HCDS_(Fall_2019)/Day_4_plan|Day 4 plan]]
 +
 
<!--
 
<!--
 
[[:File:HCDS 2019 week 4 slides.pdf|Day 4 slides]]
 
[[:File:HCDS 2019 week 4 slides.pdf|Day 4 slides]]
 
-->
 
-->
  
;Introduction to qualitative and mixed-methods research: ''Big data vs thick data; integrating qualitative research methods into data science practice; crowdsourcing''
+
;Introduction to mixed-methods research: ''Big data vs thick data; integrating qualitative research methods into data science practice; crowdsourcing''
  
 
;Assignments due
 
;Assignments due
* Reading reflection
+
* Week 3 reading reflection
 
* A2: Bias in data
 
* A2: Bias in data
  
 
;Agenda
 
;Agenda
* Reading reflection reflection
+
{{:HCDS (Fall 2019)/Day 4 plan}}
* Overview of qualitative research
 
* Introduction to ethnography
 
* In-class activity: explaining art to aliens
 
* Mixed methods research and data science
 
* An introduction to crowdwork
 
* Overview of assignment 3: Crowdwork ethnography
 
  
 
;Homework assigned
 
;Homework assigned
* Read and reflect: Barocas, Solan and Nissenbaum, Helen. ''Big Data's End Run around Anonymity and Consent''. In ''Privacy, Big Data, and the Public Good''. 2014. ([https://canvas.uw.edu/courses/1319253/files/folder/Readings PDF available on Canvas])
+
* Read and reflect: Barocas, Solan and Nissenbaum, Helen. [https://www.nyu.edu/projects/nissenbaum/papers/BigDatasEndRun.pdf ''Big Data's End Run around Anonymity and Consent'']. In ''Privacy, Big Data, and the Public Good''. 2014.
 
* [[Human_Centered_Data_Science_(Fall_2019)/Assignments#A3:_Crowdwork_ethnography|A3: Crowdwork ethnography]]
 
* [[Human_Centered_Data_Science_(Fall_2019)/Assignments#A3:_Crowdwork_ethnography|A3: Crowdwork ethnography]]
  
;Resources
+
;Qualitative research methods resources
* Singer, P., Lemmerich, F., West, R., Zia, L., Wulczyn, E., Strohmaier, M., & Leskovec, J. (2017, April). ''[https://arxiv.org/pdf/1702.05379.pdf Why we read wikipedia]''. In Proceedings of the 26th International Conference on World Wide Web.
 
* [https://meta.wikimedia.org/wiki/Research:The_role_of_citations_in_how_readers_evaluate_Wikipedia_articles/Trust_taxonomy Taxonomy of reasons why people trust/distrust Wikipedia], Jonathan Morgan, Wikimedia Research report, May 2019.
 
 
* Ladner, S. (2016). ''[http://www.practicalethnography.com/ Practical ethnography: A guide to doing ethnography in the private sector]''. Routledge.
 
* Ladner, S. (2016). ''[http://www.practicalethnography.com/ Practical ethnography: A guide to doing ethnography in the private sector]''. Routledge.
 
* Spradley, J. P. (2016). ''[https://www.waveland.com/browse.php?t=688 The ethnographic interview]''. Waveland Press.
 
* Spradley, J. P. (2016). ''[https://www.waveland.com/browse.php?t=688 The ethnographic interview]''. Waveland Press.
* Spradley, J. P. (2016) ''[https://www.waveland.com/browse.php?t=689 Participant Observation]''. Waveland Press
+
* Spradley Participant Observation FIXME
 
* Eriksson, P., & Kovalainen, A. (2015). ''[http://study.sagepub.com/sites/default/files/Eriksson%20and%20Kovalainen.pdf Ch 12: Ethnographic Research]''. In Qualitative methods in business research: A practical guide to social research. Sage.
 
* Eriksson, P., & Kovalainen, A. (2015). ''[http://study.sagepub.com/sites/default/files/Eriksson%20and%20Kovalainen.pdf Ch 12: Ethnographic Research]''. In Qualitative methods in business research: A practical guide to social research. Sage.
* ''[http://www.wou.edu/~girodm/library/zork.pdf Qualitative research activity: categorizing student responses].'' Mark Girod, Western Oregon University
+
* Usability.gov, ''[https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html System usability scale]''.
* ''[https://cmci.colorado.edu/~palen/EmpiricalEpistemologiesforHCC-7.pdf Empirical    Epistemologies Applied to Human-­‐Centered Computing Research]'' Leysia Palen, University of Colorado Boulder, November 16 2014.
+
* Nielsen, Jakob (2000). ''[https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ Why you only need to test with five users]''. nngroup.com.
  
 +
;Wikipedia gender gap research resources
 
<!--
 
<!--
* Usability.gov, ''[https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html System usability scale]''.  
+
* Hill, B. M., & Shaw, A. (2013). ''[journals.plos.org/plosone/article?id=10.1371/journal.pone.0065782 The Wikipedia gender gap revisited: Characterizing survey response bias with propensity score estimation]''. PloS one, 8(6), e65782
* Nielsen, Jakob (2000). ''[https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ Why you only need to test with five users]''. nngroup.com.
+
* Shyong (Tony) K. Lam, Anuradha Uduwage, Zhenhua Dong, Shilad Sen, David R. Musicant, Loren Terveen, and John Riedl. 2011. ''[http://files.grouplens.org/papers/wp-gender-wikisym2011.pdf WP:clubhouse?: an exploration of Wikipedia's gender imbalance.]'' In Proceedings of the 7th International Symposium on Wikis and Open Collaboration (WikiSym '11). ACM, New York, NY, USA, 1-10. DOI=http://dx.doi.org/10.1145/2038558.2038560
 +
* Maximillian Klein. ''[http://whgi.wmflabs.org/gender-by-language.html Gender by Wikipedia Language]''. Wikidata Human Gender Indicators (WHGI), 2017.
 +
* Source: Wagner, C., Garcia, D., Jadidi, M., & Strohmaier, M. (2015, April). ''[https://www.aaai.org/ocs/index.php/ICWSM/ICWSM15/paper/viewFile/10585/10528 It's a Man's Wikipedia? Assessing Gender Inequality in an Online Encyclopedia]''. In ICWSM (pp. 454-463).
 +
 
 +
* Benjamin Collier and Julia Bear. ''[https://static1.squarespace.com/static/521c8817e4b0dca2590b4591/t/523745abe4b05150ff027a6e/1379354027662/2012+-+Collier%2C+Bear+-+Conflict%2C+confidence%2C+or+criticism+an+empirical+examination+of+the+gender+gap+in+Wikipedia.pdf Conflict, criticism, or confidence: an empirical examination of the gender gap in wikipedia contributions]''. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (CSCW '12). DOI: https://doi.org/10.1145/2145204.2145265
 +
* Christina Shane-Simpson, Kristen Gillespie-Lynch, Examining potential mechanisms underlying the Wikipedia gender gap through a collaborative editing task, In Computers in Human Behavior, Volume 66, 2017, https://doi.org/10.1016/j.chb.2016.09.043. (PDF on Canvas)
 +
* Amanda Menking and Ingrid Erickson. 2015. ''[https://upload.wikimedia.org/wikipedia/commons/7/77/The_Heart_Work_of_Wikipedia_Gendered,_Emotional_Labor_in_the_World%27s_Largest_Online_Encyclopedia.pdf The Heart Work of Wikipedia: Gendered, Emotional Labor in the World's Largest Online Encyclopedia]''. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). https://doi.org/10.1145/2702123.2702514
 
-->
 
-->
  
 +
;Crowdwork research resources
 +
* WeArDynamo contributors. ''[http://wiki.wearedynamo.org/index.php?title=Basics_of_how_to_be_a_good_requester How to be a good requester]'' and ''[http://wiki.wearedynamo.org/index.php?title=Guidelines_for_Academic_Requesters Guidelines for Academic Requesters]''. Wearedynamo.org
 
<br/>
 
<br/>
 
<hr/>
 
<hr/>
Line 174: Line 171:
  
 
=== Week 5: October 24 ===
 
=== Week 5: October 24 ===
 +
[[HCDS_(Fall_2019)/Day_5_plan|Day 5 plan]]
 
<!--
 
<!--
 
[[:File:HCDS 2019 week 5 slides.pdf|Day 5 slides]]
 
[[:File:HCDS 2019 week 5 slides.pdf|Day 5 slides]]
Line 181: Line 179:
  
 
;Assignments due
 
;Assignments due
*Reading reflection
+
*Week 4 reading reflection
  
 
;Agenda
 
;Agenda
* Reading reflection review
+
{{:HCDS (Fall 2019)/Day 5 plan}}
* Guest lecture
 
* A2 retrospective
 
* Final project deliverables and timeline
 
* A brief history of research ethics in the United States
 
 
 
  
 
;Homework assigned
 
;Homework assigned
* Read and reflect: Gray, M. L., & Suri, S. (2019). Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Eamon Dolan Books. ([https://canvas.uw.edu/courses/1319253/files/folder/Readings PDF available on Canvas])
+
* Read and reflect: Mary Gray, Ghost Work FIXME
 +
* Final project proposal FIXME
  
 
;Resources
 
;Resources
Line 199: Line 193:
 
* Javier Salido (2012). ''[http://download.microsoft.com/download/D/1/F/D1F0DFF5-8BA9-4BDF-8924-7816932F6825/Differential_Privacy_for_Everyone.pdf Differential Privacy for Everyone].'' Microsoft Corporation Whitepaper.
 
* Javier Salido (2012). ''[http://download.microsoft.com/download/D/1/F/D1F0DFF5-8BA9-4BDF-8924-7816932F6825/Differential_Privacy_for_Everyone.pdf Differential Privacy for Everyone].'' Microsoft Corporation Whitepaper.
 
* Markham, Annette and Buchanan, Elizabeth. [https://aoir.org/reports/ethics2.pdf ''Ethical Decision-Making and Internet Researchers.''] Association for Internet Research, 2012.
 
* Markham, Annette and Buchanan, Elizabeth. [https://aoir.org/reports/ethics2.pdf ''Ethical Decision-Making and Internet Researchers.''] Association for Internet Research, 2012.
* Kelley, P. G., Bresee, J., Cranor, L. F., & Reeder, R. W. (2009). ''[http://cups.cs.cmu.edu/soups/2009/proceedings/a4-kelley.pdf A “nutrition label” for privacy.]'' Proceedings of the 5th Symposium on Usable Privacy and Security - SOUPS ’09, 1990, 1. https://doi.org/10.1145/1572532.1572538
 
* Warncke-Wang, M., Cosley, D., & Riedl, J. (2013). ''[https://opensym.org/wsos2013/proceedings/p0202-warncke.pdf Tell me more: An actionable quality model for wikipedia].'' Proceedings of the 9th International Symposium on Open Collaboration, WikiSym + OpenSym 2013. https://doi.org/10.1145/2491055.2491063
 
<!--
 
 
* Hill, Kashmir. [https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook-manipulated-689003-users-emotions-for-science/#6a01653e197c ''Facebook Manipulated 689,003 Users' Emotions For Science.''] Forbes, 2014.
 
* Hill, Kashmir. [https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook-manipulated-689003-users-emotions-for-science/#6a01653e197c ''Facebook Manipulated 689,003 Users' Emotions For Science.''] Forbes, 2014.
 
* Adam D. I. Kramer, Jamie E. Guillory, and Jeffrey T. Hancock [http://www.pnas.org/content/111/24/8788.full ''Experimental evidence of massive-scale emotional contagion through social networks.''] PNAS 2014 111 (24) 8788-8790; published ahead of print June 2, 2014.
 
* Adam D. I. Kramer, Jamie E. Guillory, and Jeffrey T. Hancock [http://www.pnas.org/content/111/24/8788.full ''Experimental evidence of massive-scale emotional contagion through social networks.''] PNAS 2014 111 (24) 8788-8790; published ahead of print June 2, 2014.
Line 207: Line 198:
 
* Zetter, Kim. [https://www.wired.com/2012/06/wmw-arvind-narayanan/ ''Arvind Narayanan Isn’t Anonymous, and Neither Are You.''] WIRED, 2012.
 
* Zetter, Kim. [https://www.wired.com/2012/06/wmw-arvind-narayanan/ ''Arvind Narayanan Isn’t Anonymous, and Neither Are You.''] WIRED, 2012.
 
* Gray, Mary. [http://culturedigitally.org/2014/07/when-science-customer-service-and-human-subjects-research-collide-now-what/ ''When Science, Customer Service, and Human Subjects Research Collide. Now What?''] Culture Digitally, 2014.
 
* Gray, Mary. [http://culturedigitally.org/2014/07/when-science-customer-service-and-human-subjects-research-collide-now-what/ ''When Science, Customer Service, and Human Subjects Research Collide. Now What?''] Culture Digitally, 2014.
 +
* Tene, Omer and Polonetsky, Jules. [https://www.stanfordlawreview.org/online/privacy-paradox-privacy-and-big-data/ ''Privacy in the Age of Big Data.''] Stanford Law Review, 2012.
 
* Dwork, Cynthia. [https://www.microsoft.com/en-us/research/wp-content/uploads/2008/04/dwork_tamc.pdf ''Differential Privacy: A survey of results'']. Theory and Applications of Models of Computation , 2008.
 
* Dwork, Cynthia. [https://www.microsoft.com/en-us/research/wp-content/uploads/2008/04/dwork_tamc.pdf ''Differential Privacy: A survey of results'']. Theory and Applications of Models of Computation , 2008.
 
* Hsu, Danny. [http://blog.datasift.com/2015/04/09/techniques-to-anonymize-human-data/ ''Techniques to Anonymize Human Data.''] Data Sift, 2015.
 
* Hsu, Danny. [http://blog.datasift.com/2015/04/09/techniques-to-anonymize-human-data/ ''Techniques to Anonymize Human Data.''] Data Sift, 2015.
-->
 
  
 
<br/>
 
<br/>
Line 216: Line 207:
  
 
=== Week 6: October 31 ===
 
=== Week 6: October 31 ===
 +
[[HCDS_(Fall_2019)/Day_6_plan|Day 6 plan]]
 
<!--
 
<!--
 
[[:File:HCDS 2019 week 6 slides.pdf|Day 6 slides]]
 
[[:File:HCDS 2019 week 6 slides.pdf|Day 6 slides]]
Line 226: Line 218:
  
 
;Agenda
 
;Agenda
* Reading reflections
+
{{:HCDS (Fall 2019)/Day 7 plan}}
* Assignment 3 review
 
* Guest lecture: Stefania Druga
 
* In-class activity
 
* Introduction to assignment 4: Final project proposal
 
  
 
;Homework assigned
 
;Homework assigned
Line 236: Line 224:
 
:* Baumer, E. P. S. (2017). ''[http://journals.sagepub.com/doi/pdf/10.1177/2053951717718854 Toward human-centered algorithm design].'' Big Data & Society.
 
:* Baumer, E. P. S. (2017). ''[http://journals.sagepub.com/doi/pdf/10.1177/2053951717718854 Toward human-centered algorithm design].'' Big Data & Society.
 
:* Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014). ''[http://www.aaai.org/ojs/index.php/aimagazine/article/download/2513/2456 Power to the People: The Role of Humans in Interactive Machine Learning].'' AI Magazine, 35(4), 105.
 
:* Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014). ''[http://www.aaai.org/ojs/index.php/aimagazine/article/download/2513/2456 Power to the People: The Role of Humans in Interactive Machine Learning].'' AI Magazine, 35(4), 105.
* [[Human_Centered_Data_Science_(Fall_2019)/Assignments#A4:_Final project proposal|A4: Final project proposal]]
+
* [[Human_Centered_Data_Science_(Fall_2018)/Assignments#A4:_Final_project_plan|A4: Final project plan]]
  
 
;Resources
 
;Resources
 +
<!--
 +
* Neff, G., Tanweer, A., Fiore-Gartland, B., & Osburn, L. (2017). Critique and Contribute: A Practice-Based Framework for Improving Critical Data Studies and Data Science. Big Data, 5(2), 85–97. https://doi.org/10.1089/big.2016.0050
 
* Lilly C. Irani and M. Six Silberman. 2013. ''[https://escholarship.org/content/qt10c125z3/qt10c125z3.pdf Turkopticon: interrupting worker invisibility in amazon mechanical turk]''. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). DOI: https://doi.org/10.1145/2470654.2470742
 
* Lilly C. Irani and M. Six Silberman. 2013. ''[https://escholarship.org/content/qt10c125z3/qt10c125z3.pdf Turkopticon: interrupting worker invisibility in amazon mechanical turk]''. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). DOI: https://doi.org/10.1145/2470654.2470742
* Salehi, Niloufar, Lilly C. Irani, Michael S. Bernstein, Ali Alkhatib, Eva Ogbe, and Kristy Milland. ''[https://hci.stanford.edu/publications/2015/dynamo/DynamoCHI2015.pdf We are dynamo: Overcoming stalling and friction in collective action for crowd workers]''. In Proceedings of the 33rd annual ACM conference on human factors in computing systems, pp. 1621-1630. ACM, 2015.
+
* Bivens, R. and Haimson, O.L. 2016. ''[http://journals.sagepub.com/doi/pdf/10.1177/2056305116672486 Baking Gender Into Social Media Design: How Platforms Shape Categories for Users and Advertisers]''. Social Media + Society. 2, 4 (2016), 205630511667248. DOI:https://doi.org/10.1177/2056305116672486.
* Hill, B. M., Dailey, D., Guy, R. T., Lewis, B., Matsuzaki, M., & Morgan, J. T. (2017). ''[https://mako.cc/academic/hill_etal-cdsw_chapter-DRAFT.pdf Democratizing Data Science: The Community Data Science Workshops and Classes]''. In N. Jullien, S. A. Matei, & S. P. Goggins (Eds.), Big Data Factories: Scientific Collaborative approaches for virtual community data collection, repurposing, recombining, and dissemination. New York, New York: Springer Nature. https://doi.org/10.1007/978-3-319-59186-5_9
+
* Schlesinger, A. et al. 2017. ''[http://arischlesinger.com/wp-content/uploads/2017/03/chi2017-schlesinger-intersectionality.pdf Intersectional HCI: Engaging Identity through Gender, Race, and Class].'' Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17. (2017), 5412–5427. DOI:https://doi.org/10.1145/3025453.3025766.
* Ingold, David and Soper, Spencer. ''[https://www.bloomberg.com/graphics/2016-amazon-same-day/ Amazon Doesn’t Consider the Race of Its Customers. Should It?].'' Bloomberg, 2016.
+
-->
* Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner. ''[https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing Machine Bias: Risk Assessment in Criminal Sentencing]. Propublica, May 2018.
 
 
<br/>
 
<br/>
 
<hr/>
 
<hr/>
Line 249: Line 238:
  
 
=== Week 7: November 7 ===
 
=== Week 7: November 7 ===
 +
[[HCDS_(Fall_2019)/Day_7_plan|Day 7 plan]]
 
<!--
 
<!--
 
[[:File:HCDS 2019 week 7 slides.pdf|Day 7 slides]]
 
[[:File:HCDS 2019 week 7 slides.pdf|Day 7 slides]]
Line 259: Line 249:
  
 
;Agenda
 
;Agenda
* Reading reflection review
+
{{:HCDS (Fall 2019)/Day 6 plan}}
* Algorithmic transparency, interpretability, and accountability
 
* Auditing algorithms
 
* In-class activity
 
* Introduction to assignment 5: Final project proposal
 
  
 
;Homework assigned
 
;Homework assigned
* Read and reflect: Kocielnik, R., Amershi, S., & Bennett, P. N. (2019). ''[http://saleemaamershi.com/papers/chi2019.AI.Expectations.pdf Will You Accept an Imperfect AI?]'' Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems  - CHI ’19, 1–14. https://doi.org/10.1145/3290605.3300641
+
* Read and reflect: TBD
* [[Human_Centered_Data_Science_(Fall_2019)/Assignments#A5:_Final_project_plan|A5: Final project plan]]
+
* A5: Final project plan
  
 
;Resources
 
;Resources
 +
<!--
 
* Christian Sandvig, Kevin Hamilton, Karrie Karahalios, Cedric Langbort (2014/05/22) ''[http://www-personal.umich.edu/~csandvig/research/Auditing%20Algorithms%20--%20Sandvig%20--%20ICA%202014%20Data%20and%20Discrimination%20Preconference.pdf Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms].'' Paper presented to "Data and Discrimination: Converting Critical Concerns into Productive Inquiry," a preconference at the 64th Annual Meeting of the International Communication Association. May 22, 2014; Seattle, WA, USA.  
 
* Christian Sandvig, Kevin Hamilton, Karrie Karahalios, Cedric Langbort (2014/05/22) ''[http://www-personal.umich.edu/~csandvig/research/Auditing%20Algorithms%20--%20Sandvig%20--%20ICA%202014%20Data%20and%20Discrimination%20Preconference.pdf Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms].'' Paper presented to "Data and Discrimination: Converting Critical Concerns into Productive Inquiry," a preconference at the 64th Annual Meeting of the International Communication Association. May 22, 2014; Seattle, WA, USA.  
 +
* Shahriari, K., & Shahriari, M. (2017). ''[https://ethicsinaction.ieee.org/ IEEE standard review - Ethically aligned design: A vision for prioritizing human wellbeing with artificial intelligence and autonomous systems].'' Institute of Electrical and Electronics Engineers
 +
* ACM US Policy Council ''[https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf Statement on Algorithmic Transparency and Accountability].'' January 2017.
 +
* ''[https://futureoflife.org/ai-principles/ Asilomar AI Principles].'' Future of Life Institute, 2017.
 +
* Diakopoulos, N., Friedler, S., Arenas, M., Barocas, S., Hay, M., Howe, B., … Zevenbergen, B. (2018). ''[http://www.fatml.org/resources/principles-for-accountable-algorithms Principles for Accountable Algorithms and a Social Impact Statement for Algorithms].'' Fatml.Org 2018.
 
* Friedman, B., & Nissenbaum, H. (1996). ''[https://www.vsdesign.org/publications/pdf/64_friedman.pdf Bias in Computer Systems]''. ACM Trans. Inf. Syst., 14(3), 330–347.
 
* Friedman, B., & Nissenbaum, H. (1996). ''[https://www.vsdesign.org/publications/pdf/64_friedman.pdf Bias in Computer Systems]''. ACM Trans. Inf. Syst., 14(3), 330–347.
 +
* Diakopoulos, N. (2014). Algorithmic accountability reporting: On the investigation of black boxes. Tow Center for Digital Journalism, 1–33. https://doi.org/10.1002/ejoc.201200111
 
* Nate Matias, 2017. ''[https://medium.com/@natematias/how-anyone-can-audit-facebooks-newsfeed-b879c3e29015 How Anyone Can Audit Facebook's Newsfeed].'' Medium.com
 
* Nate Matias, 2017. ''[https://medium.com/@natematias/how-anyone-can-audit-facebooks-newsfeed-b879c3e29015 How Anyone Can Audit Facebook's Newsfeed].'' Medium.com
 
* Hill, Kashmir. ''[https://gizmodo.com/facebook-figured-out-my-family-secrets-and-it-wont-tel-1797696163 Facebook figured out my family secrets, and it won't tell me how].'' Engadget, 2017.
 
* Hill, Kashmir. ''[https://gizmodo.com/facebook-figured-out-my-family-secrets-and-it-wont-tel-1797696163 Facebook figured out my family secrets, and it won't tell me how].'' Engadget, 2017.
 
* Blue, Violet. ''[https://www.engadget.com/2017/09/01/google-perspective-comment-ranking-system/ Google’s comment-ranking system will be a hit with the alt-right].'' Engadget, 2017.
 
* Blue, Violet. ''[https://www.engadget.com/2017/09/01/google-perspective-comment-ranking-system/ Google’s comment-ranking system will be a hit with the alt-right].'' Engadget, 2017.
* Anderson, Carl. ''[https://medium.com/@leapingllamas/the-role-of-model-interpretability-in-data-science-703918f64330 The role of model interpretability in data science].'' Medium, 2016.
+
* Ingold, David and Soper, Spencer. ''[https://www.bloomberg.com/graphics/2016-amazon-same-day/ Amazon Doesn’t Consider the Race of Its Customers. Should It?].'' Bloomberg, 2016.
 +
* Paul Lamere. ''[https://musicmachinery.com/2011/05/14/how-good-is-googles-instant-mix/ How good is Google's Instant Mix?].'' Music Machinery, 2011.
 
* Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner. ''[https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing Machine Bias: Risk Assessment in Criminal Sentencing]. Propublica, May 2018.
 
* Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner. ''[https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing Machine Bias: Risk Assessment in Criminal Sentencing]. Propublica, May 2018.
* Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., … Gebru, T. (2019). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, 220–229. https://doi.org/10.1145/3287560.3287596
+
* [https://www.perspectiveapi.com/#/ Google's Perspective API]
* Hosseini, H., Kannan, S., Zhang, B., & Poovendran, R. (2017). Deceiving Google’s Perspective API Built for Detecting Toxic Comments. ArXiv:1702.08138 [Cs]. Retrieved from http://arxiv.org/abs/1702.08138
+
-->
* Binns, R., Veale, M., Van Kleek, M., & Shadbolt, N. (2017). Like trainer, like bot? Inheritance of bias in algorithmic content moderation. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10540 LNCS, 405–415. https://doi.org/10.1007/978-3-319-67256-4_32
 
* Borkan, D., Dixon, L., Sorensen, J., Thain, N., & Vasserman, L. (2019). Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification. 2, 491–500. https://doi.org/10.1145/3308560.3317593
 
* Zhang, J., Chang, J., Danescu-Niculescu-Mizil, C., Dixon, L., Hua, Y., Taraborelli, D., & Thain, N. (2019). Conversations Gone Awry: Detecting Early Signs of Conversational Failure. 1350–1361. https://doi.org/10.18653/v1/p18-1125
 
* Miriam Redi, Besnik Fetahu, Jonathan T. Morgan, and Dario Taraborelli. 2019. ''[https://arxiv.org/pdf/1902.11116.pdf Citation Needed a Taxonomy and Algorithmic Assessment of Wikipedia’s Verifiability].'' The Web Conference.
 
*[https://www.perspectiveapi.com/#/ Google's Perspective API]
 
  
 
<br/>
 
<br/>
Line 290: Line 278:
  
 
=== Week 8: November 14 ===
 
=== Week 8: November 14 ===
 +
[[HCDS_(Fall_2019)/Day_8_plan|Day 8 plan]]
 
<!--
 
<!--
 
[[:File:HCDS 2019 week 8 slides.pdf|Day 8 slides]]
 
[[:File:HCDS 2019 week 8 slides.pdf|Day 8 slides]]
Line 300: Line 289:
  
 
;Agenda
 
;Agenda
* ''coming soon''
+
{{:HCDS (Fall 2019)/Day 8 plan}}
<!--
 
* Final project overview & examples
 
* Reading reflections
 
* Human-centered algorithm design
 
:* design process
 
:* user-driven evaluation
 
:* design patterns & anti-patterns
 
-->
 
  
 
;Homework assigned
 
;Homework assigned
* Reading and reflect: Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé, III, Miro Dudik, and Hanna Wallach. 2019. ''[https://arxiv.org/pdf/1812.05239.pdf Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?]''. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, Paper 600, 16 pages. DOI: https://doi.org/10.1145/3290605.3300830
+
* Reading and reflect: TBD (data science ethics survey paper)
* [[Human_Centered_Data_Science_(Fall_2019)/Assignments#A6:_Final project presentation|A6: Final project presentation]]
+
* A6: Final project presentation
  
 
;Resources
 
;Resources
* Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. ''[http://files.grouplens.org/papers/mcnee-chi06-hri.pdf Making recommendations better: an analytic model for human-recommender interaction].'' In CHI '06 Extended Abstracts on Human Factors in Computing Systems (CHI EA '06).
+
* Ethical OS ''[https://ethicalos.org/wp-content/uploads/2018/08/Ethical-OS-Toolkit-2.pdf Toolkit]'' and ''[https://ethicalos.org/wp-content/uploads/2018/08/EthicalOS_Check-List_080618.pdf Risk Mitigation Checklist]''. EthicalOS.org.
* Sean M. McNee, Nishikant Kapoor, and Joseph A. Konstan. 2006. ''[http://files.grouplens.org/papers/p171-mcnee.pdf Don't look stupid: avoiding pitfalls when recommending research papers].'' In Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work (CSCW '06).
 
* Shahriari, K., & Shahriari, M. (2017). ''[https://ethicsinaction.ieee.org/ IEEE standard review - Ethically aligned design: A vision for prioritizing human wellbeing with artificial intelligence and autonomous systems].'' Institute of Electrical and Electronics Engineers
 
* ACM US Policy Council ''[https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf Statement on Algorithmic Transparency and Accountability].'' January 2017.
 
* Diakopoulos, N., Friedler, S., Arenas, M., Barocas, S., Hay, M., Howe, B., … Zevenbergen, B. (2018). ''[http://www.fatml.org/resources/principles-for-accountable-algorithms Principles for Accountable Algorithms and a Social Impact Statement for Algorithms].'' Fatml.Org 2018.  
 
 
* Morgan, J. 2016. ''[https://meta.wikimedia.org/wiki/Research:Evaluating_RelatedArticles_recommendations Evaluating Related Articles recommendations]''. Wikimedia Research.
 
* Morgan, J. 2016. ''[https://meta.wikimedia.org/wiki/Research:Evaluating_RelatedArticles_recommendations Evaluating Related Articles recommendations]''. Wikimedia Research.
 
* Morgan, J. 2017. ''[https://meta.wikimedia.org/wiki/Research:Comparing_most_read_and_trending_edits_for_Top_Articles_feature Comparing most read and trending edits for the top articles feature]''. Wikimedia Research.
 
* Morgan, J. 2017. ''[https://meta.wikimedia.org/wiki/Research:Comparing_most_read_and_trending_edits_for_Top_Articles_feature Comparing most read and trending edits for the top articles feature]''. Wikimedia Research.
 
*Michael D. Ekstrand, F. Maxwell Harper, Martijn C. Willemsen, and Joseph A. Konstan. 2014. ''[https://md.ekstrandom.net/research/pubs/listcmp/listcmp.pdf User perception of differences in recommender algorithms].'' In Proceedings of the 8th ACM Conference on Recommender systems (RecSys '14).
 
*Michael D. Ekstrand, F. Maxwell Harper, Martijn C. Willemsen, and Joseph A. Konstan. 2014. ''[https://md.ekstrandom.net/research/pubs/listcmp/listcmp.pdf User perception of differences in recommender algorithms].'' In Proceedings of the 8th ACM Conference on Recommender systems (RecSys '14).
 +
* Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. ''[http://files.grouplens.org/papers/mcnee-chi06-hri.pdf Making recommendations better: an analytic model for human-recommender interaction].'' In CHI '06 Extended Abstracts on Human Factors in Computing Systems (CHI EA '06).
 +
* Sean M. McNee, Nishikant Kapoor, and Joseph A. Konstan. 2006. ''[http://files.grouplens.org/papers/p171-mcnee.pdf Don't look stupid: avoiding pitfalls when recommending research papers].'' In Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work (CSCW '06).
 +
* Michael D. Ekstrand and Martijn C. Willemsen. 2016. ''[https://md.ekstrandom.net/research/pubs/behaviorism/BehaviorismIsNotEnough.pdf Behaviorism is Not Enough: Better Recommendations through Listening to Users].'' In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys '16).
 +
* Jess Holbrook. ''[https://medium.com/google-design/human-centered-machine-learning-a770d10562cd Human Centered Machine Learning].'' Google Design Blog. 2017.
 +
* Anderson, Carl. ''[https://medium.com/@leapingllamas/the-role-of-model-interpretability-in-data-science-703918f64330 The role of model interpretability in data science].'' Medium, 2016.
 +
 
<br/>
 
<br/>
 
<hr/>
 
<hr/>
Line 328: Line 311:
  
 
=== Week 9: November 21 ===
 
=== Week 9: November 21 ===
 +
[[HCDS_(Fall_2019)/Day_9_plan|Day 9 plan]]
  
;Data science in context: Doing human centered datascience in product organizations; communicating and collaborating across roles and disciplines; HCDS industry trends and trajectories
+
;Data science in organizations: TBD
  
 
;Assignments due
 
;Assignments due
Line 335: Line 319:
  
 
;Agenda
 
;Agenda
* Filling out course evaluation
+
{{:HCDS (Fall 2019)/Day 9 plan}}
* Week 8 in-class activity report out
 
* End of quarter logistics
 
* Final project presentations and reports
 
* Guest lecture: Rich Caruana, Microsoft Research
 
* In-class activity (InterpretML): Harsha Nori, Microsoft
 
 
 
  
 
;Homework assigned
 
;Homework assigned
* Read and reflect: Passi, S., & Jackson, S. J. (2018). ''[https://dl.acm.org/citation.cfm?doid=3290265.3274405 Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects].'' Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), 1–28. https://doi.org/10.1145/3274405 ([https://sjackson.infosci.cornell.edu/Passi&Jackson_TrustinDataScience(CSCW2018).pdf ACCESS PDF HERE])
+
* Read and reflect: TBD
* [[Human_Centered_Data_Science_(Fall_2019)/Assignments#A7:_Final_project_report|A7: Final project report]]
+
* A6: Final project presentation
 +
* A7: Final project report
  
 
;Resources
 
;Resources
* Rich Caruana, Harsha Nori, Samuel Jenkins, Paul Koch, Ester de Nicolas. 2019. ''InterpretML software toolkit'' ([https://github.com/interpretml/interpret github repo], [https://www.microsoft.com/en-us/research/blog/creating-ai-glass-boxes-open-sourcing-a-library-to-enable-intelligibility-in-machine-learning/ blog post])
+
<!--
* Partnership on AI, 2019 ''[https://www.partnershiponai.org/report-on-machine-learning-in-risk-assessment-tools-in-the-u-s-criminal-justice-system/ Report on Algorithmic Risk Assessment Tools in the U.S. Criminal Justice System].''
+
* Daniela Aiello, Lisa Bates, et al. [https://shelterforce.org/2018/08/22/eviction-lab-misses-the-mark/ Eviction Lab Misses the Mark], ShelterForce, August 2018.
* Morgan, J. T., 2019. ''[https://figshare.com/articles/Ethical_Human_Centered_AI/8044553 Ethical and Human-centered AI at Wikimedia]''. Wikimedia Research 2030​.
+
-->
  
 
<br/>
 
<br/>
Line 360: Line 339:
 
;Assignments due
 
;Assignments due
 
* Reading reflection
 
* Reading reflection
 +
 +
;Readings assigned
 +
* NONE
  
 
;Homework assigned
 
;Homework assigned
* Read and reflect: Barocas, S., & Boyd, D. (2017). ''Engaging the ethics of data science in practice.'' Communications of the ACM, 60(11), 23–25. https://doi.org/10.1145/3144172 ([https://canvas.uw.edu/courses/1319253/files/folder/Readings PDF available on Canvas])
+
* NONE
  
 
;Resources
 
;Resources
 +
<!--
 +
*Fabien Girardin. ''[https://medium.com/@girardin/experience-design-in-the-machine-learning-era-e16c87f4f2e2 Experience design in the machine learning era].'' Medium, 2016.
 +
* Xavier Amatriain and Justin Basilico. ''[https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429 Netflix Recommendations: Beyond the 5 stars].'' Netflix Tech Blog, 2012.
 +
* Jess Holbrook. ''[https://medium.com/google-design/human-centered-machine-learning-a770d10562cd Human Centered Machine Learning].'' Google Design Blog. 2017.
 +
* Bart P. Knijnenburg, Martijn C. Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. ''[https://pure.tue.nl/ws/files/3484177/724656348730405.pdf Explaining the user experience of recommender systems].'' User Modeling and User-Adapted Interaction 22, 4-5 (October 2012), 441-504. DOI=http://dx.doi.org/10.1007/s11257-011-9118-4
 +
* Patrick Austin, ''[https://gizmodo.com/facebook-google-and-microsoft-use-design-to-trick-you-1827168534 Facebook, Google, and Microsoft Use Design to Trick You Into Handing Over Your Data, New Report Warns].'' Gizmodo, 6/18/2018
 +
* Brown, A., Tuor, A., Hutchinson, B., & Nichols, N. (2018). ''[[https://arxiv.org/abs/1803.04967 Recurrent Neural Network Attention Mechanisms for Interpretable System Log Anomaly Detection].'' arXiv preprint arXiv:1803.04967.
 +
* Cremonesi, P., Elahi, M., & Garzotto, F. (2017). ''[https://core.ac.uk/download/pdf/74313597.pdf User interface patterns in recommendation-empowered content intensive multimedia applications].'' Multimedia Tools and Applications, 76(4), 5275-5309.
 
* Marilynn Larkin, ''[https://www.elsevier.com/connect/how-to-give-a-dynamic-scientific-presentation How to give a dynamic scientific presentation].'' Elsevier Connect, 2015.
 
* Marilynn Larkin, ''[https://www.elsevier.com/connect/how-to-give-a-dynamic-scientific-presentation How to give a dynamic scientific presentation].'' Elsevier Connect, 2015.
 
* Megan Risdal, ''[http://blog.kaggle.com/2016/06/29/communicating-data-science-a-guide-to-presenting-your-work/ Communicating data science: a guide to presenting your work].'' Kaggle blog, 2016.
 
* Megan Risdal, ''[http://blog.kaggle.com/2016/06/29/communicating-data-science-a-guide-to-presenting-your-work/ Communicating data science: a guide to presenting your work].'' Kaggle blog, 2016.
Line 370: Line 360:
 
* Megan Risdal, ''[http://blog.kaggle.com/2016/06/13/communicating-data-science-an-interview-with-a-storytelling-expert-tyler-byers/ Communicating data science: an interview with a storytelling expert].'' Kaggle blog, 2016.
 
* Megan Risdal, ''[http://blog.kaggle.com/2016/06/13/communicating-data-science-an-interview-with-a-storytelling-expert-tyler-byers/ Communicating data science: an interview with a storytelling expert].'' Kaggle blog, 2016.
 
* Brent Dykes, ''[https://www.forbes.com/sites/brentdykes/2016/03/31/data-storytelling-the-essential-data-science-skill-everyone-needs/ Data Storytelling: The Essential Data Science Skill Everyone Needs].'' Forbes, 2016.
 
* Brent Dykes, ''[https://www.forbes.com/sites/brentdykes/2016/03/31/data-storytelling-the-essential-data-science-skill-everyone-needs/ Data Storytelling: The Essential Data Science Skill Everyone Needs].'' Forbes, 2016.
 +
-->
 +
 
<br/>
 
<br/>
 
<hr/>
 
<hr/>

Please note that all contributions to CommunityData are considered to be released under the Attribution-Share Alike 3.0 Unported (see CommunityData:Copyrights for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource. Do not submit copyrighted work without permission!

To protect the wiki against automated edit spam, we kindly ask you to solve the following CAPTCHA:

Cancel Editing help (opens in new window)