Troubleshooting in Computational Research Design (UW COM 597 Winter 2026)

Mini-workshop: Troubleshooting in Computational Communication Research Design

Instructor:

• Yibin Fan / yibin115@uw.edu (#Office Hours)

Course Meetings: Fridays 3:30-4:20pm CMU 242

Are you currently working on a computational communication research project? Or, are you already finished on such a project? If the answer is yes, you probably have had a similar experience as me: there are numerous methodological decisions one needs to make to design and conduct a high-quality study, yet the rationales, tradeoffs, and implications of these decisions are often not articulated or documented in published research papers.

In other words, there is a (potentially big!) mismatch between what a research paper says it has done and what a researcher actually needs to know to conduct that study.

I call this hidden process troubleshooting in computational research design. This is tacit knowledge not usually made explicit in publications—but it is crucial for achieving valid and rigorous research findings. This mini-workshop discusses this tacit knowledge, based on real examples and cases in computational communication research.

Participants are expected to have experience in designing at least one computational communication research project, either ongoing or finished. They are encouraged to bring their own questions or puzzles related to troubleshooting for weekly discussion.

Learning Objectives

1. Evaluate a computational research project from a research design perspective.
2. Become familiar with common issues across genres of computational research and learn how to address these issues for validity.

Workshop Format

This workshop meets weekly for 50 minutes over ten weeks. Each week focuses on a dedicated research design topic, ranging from conceptualization and operationalization, GenAI use, textual analysis, behavioral analysis, network analysis, and statistical models, to other topics of shared interest.

For each week:

• The first half features discussion with the author of a completed computational study on the week’s topic, sharing key troubleshooting experiences. Authors may include the instructor, invited senior scholars, or enrolled students.
• The second half is open Q&A on the topic.

This workshop primarily serves CDSC-UW graduate students, but is open to other UW graduate students who meet the prerequisite of having designed one computational communication project. The discussions span across substantial subfields, including organizational communication, political communication, communication and technology, and health communication.

Grading

The grade consists of two components:

1. Class Participation (40%)

Students are expected to attend weekly sessions and engage in discussions.

2. Final Reflection Report (60%)

Students select one topic from the course and write a ~1000-word reflection report. The report should:

(a) Briefly introduce how the research design of one computational communication project (ongoing or completed) relates to the selected topic. (b) Identify two issues that emerged in the research design, explaining why they matter for validity. (c) For each issue, describe your solution(s) and explain your rationale (e.g., conceptual clarity, methodological rigor, empirical constraints).

A successful report should be strong enough to serve as part of a response to journal reviewers raising these concerns.

After this workshop, the instructor will organize a collaborative article on troubleshooting in computational communication research design, aimed as a book chapter in an edited volume. Submitted reflection reports may be included, and selected student authors will be invited as coauthors for revisions.

Weekly Topics

Week 1 Jan 9: Conceptualization and Operationalization

In the first session, we are going to briefly introduce the aim of this workshop, as well as how it is going to be organized. This introduction is followed by a statement from each participant talking about one of their recent projects. In this statement, one should clearly identify the conceptualization and operationalization (or at least their plans) in the research design.

Week 2 Jan 16: The Role of Generative AI Use in Research Design: Object, Agent, and/or Assistant

This week is dedicated to discussion on the role of Gen AI in research design. Based on the reading of recent literature on how Gen AI influences computational research in social science, I have summarized three approaches: Gen AI as (1) research object, (2) agent, and/or (3) assistant. We will read three example articles for each approach for better discussion:

Object: Zhan, X., Goyal, A., Chen, Y., Chandrasekharan, E., & Saha, K. (2025). SLM-mod: Small language models surpass LLMs at content moderation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 8774-8790). [Please feel free to skim the technique details]

Agent: Kozlowski, A. C., & Evans, J. (2025). Simulating Subjects: The Promise and Peril of Artificial Intelligence Stand-Ins for Social Agents and Interactions. Sociological Methods & Research, 00491241251337316.

Assistant: Li, L., Li, J., Chen, C., Gui, F., Yang, H., Yu, C., ... & Dong, Y. (2024). Political-llm: Large language models in political science. arXiv preprint https://arxiv.org/abs/2412.06864. [Please only read Sections 3., 4.1, and 4.2.]

With reading these pieces, please reflect on how your research might be assisted by Gen AI, and we will discuss on that in the class session.

Week 3 Jan 23: Computational Textual Analysis

Fan, Y., Hill, B. M., & Moy, P. (2026). Unintended politics: Partisan opinion expression and incivility in incidental political discussion. Manuscript under review for publication.

This week Yibin will talk about his paper on incidental political discussion on Reddit. The manuscript and supplemental materials can be found here. The expectation is that when you read these two files, please pay more attention to the Methods section and related details in supplemental materials, while feel free to skip literature review part (other than hypotheses).

Week 4 Jan 30: Network Analysis

Foote, J., Shaw, A., & Hill, B. M. (2023). Communication networks do not predict success in attempts at peer production. Journal of Computer-Mediated Communication, 28(3), zmad002.

This week, we will have Jeremy Foote to talk about his experiences with a paper published with network analysis in the first half of our workshop. Please read the above piece before the workshop starts!

Week 5 Feb 6: Behavioral Analysis

Hill, B. M., & Shaw, A. (2021). The hidden costs of requiring accounts: quasi-experimental evidence from peer production. Communication Research, 48(6), 771-795.

This week, we plan to have Mako as our guest speaker to talk on his paper on how a policy change (i.e., requiring accounts before contributing to wiki pages) brings impact on actual contribution. I would expect us to pay more attention on Introduction, hypotheses, Data and Methods than others sections when reading this paper.

Week 6 Feb 13: Mixed-method Research Design

Champion, K., & Hill, B. M. (2024). Life Histories of Taboo Knowledge Artifacts. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW2), 1-32.

Kaylea Champion would join us to talk about her project(s) on taboo knowledge, which substantially engaged with mixed-method design with Internet research. Please read the above paper before the workshop.

In addition, if you are interested, please check a quantitative paper coming before it in a row, which inspires the mixed-method paper during the troubleshooting process:

[Optional] Champion, K., & Hill, B. M. (2023). Taboo and Collaborative Knowledge Production: Evidence from Wikipedia. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW2), 1-25.