Troubleshooting in Computational Research Design: Report from a Workshop Series

stylized visualization of a messy process turning into a paper

Visualization of the messy process that turns into the polished paper. Image generated by claude.ai.

This winter quarter, a small group of CDSC students at the University of Washington participated in a series of workshops on Troubleshooting in Computational Research Design. The workshops were organized by Yibin Fan.

Research articles typically present a streamlined account of research design. How should a concept be operationalized? What counts as valid data? The process of producing those designs often involves a series of complex decisions that are rarely documented in detail. To address this gap, this workshop focused on the “troubleshooting” process that is central to computational communication research but is often omitted from published work: How did the authors arrive at particular methodological decisions? What challenges arose at different stages of the research design? How did they navigate the tradeoffs involved in choosing among alternative methods?

Each session of the workshop focused on a specific aspect of computational research design, including conceptualization and operationalization; the role of generative AI in research design; computational text analysis; network analysis; behavioral analysis; and mixed-methods research.

Our workshop sparked many interesting discussions, featuring guest speakers from the CDSC community who shared the behind-the-scenes decision-making processes of their work. Examples include:

Yibin Fan discussed his paper on incidental political discussion on Reddit (currently under review). He shared his experience using Large Language Models for text classification, highlighting the limits of this approach, and explaining when he opted for human-based content analysis when automated methods fell short.
Jeremy Foote discussed an article investigating whether communication network structures in early-stage peer production communities predict eventual success. He shared how he navigated the operationalization of difficult concepts like “success” and “integrative networks.”
Mako Hill discussed a paper on how account requirements affect contributions to wikis. He shared how he addressed model convergence issues caused by a large number of parameters using a novel two-step process that combined Bayesian and frequentist approaches.
Kaylea Champion discussed her mixed-methods research design for a project on knowledge production about taboo topics in Wikipedia. She explained how a qualitative study inspired another quantitative work, and how she navigated the unique challenges of researching sensitive or “taboo” topics.

For researchers working with computational methods, the workshop’s focus on troubleshooting offered a practical perspective on how rigorous research designs are developed in practice.

I’m writing this up, in part, because I think this might be a useful general model for other groups. Although the specifics vary, we found that asking computational researchers to bring their “real problems” to the table led to valuable conversations—especially for the early-career scholars. Yibin Fan is happy to have anybody reach out if they are interested in chatting about replicating the model at their own institution.