CommunityData:Quantitative paper milestones

This page lists milestones that a quantitative paper should hit on its way to submission. It is meant to help researchers seeking to create a timeline for a study by setting internal deadlines and identifying which parts of the work are still unfinished. This is meant to be used in conjunction with the CommunityData:Quantitative planning document.

Many of these milestones correspond to the production of an artifact like a document, a table, a plot, or a running pipeline. A useful way to use the list is to work backward from a target submission date to give each one a target date. Some of them (especially anything involving data, measures, or writing) will probably be revisited. And that's a very normal part of the research process.

Workshop feedback

The CDSC workshop can be a useful resource throughout the paper process. A great way to use it is for workshopping your planning document. Early feedback on questions, hypotheses, planned analytic approach, and planned measures can reshape the study before you have made major commitments and investments. Later workshops (up to and including full papers!) are valuable too.

Milestones

Note: A ★ marks a milestone where a check-in with your coauthors and/or advisor is probably a good idea.

Planning

1. Questions finalized ★—rationale, general and specific objectives, and hypotheses or propositions written and reviewed.
2. Planning document finalized ★—everything through the dummy tables and figures drafted; advisor sign-off.

Building the paper

3. Data access secured—e.g., downloaded, IRB approval received.
4. Intro and related work drafted—expand from the planning document's rationale; doesn't depend on data.
5. Dataset construction works on a sample—parsing, filtering, and aggregation. This will often happen on a small subset or sample so it can be completed entirely on your laptop.
6. Measures validated on the sample ★—an artifact showing each measure captures what you claim it does. This might involve manual review of cases, inter-rater reliability, comparison to an alternative operationalization, and so on.
7. Methods section drafted—expand the planning document's Data & Measures and Analytic Approach sections.
8. Dataset constructed at scale—construct your full analytic dataset. This will often happen on Hyak or some other HPC or server.
9. Descriptive stats and bivariate plots ★—descriptive statistics tables plus bivariate plots (e.g., boxplots, scatterplots) corresponding to every hypothesis.
10. Analysis pipeline runs on your "final" dataset ★—every planned table and figure is produced from your real data, even if the numbers and figures are still ugly.
11. Results section drafted—real tables and figures replace the dummies and you write prose that describes what the numbers show. Be sure to interpret your results, not just report them.
12. Robustness and sensitivity checks ★—robustness checks planned and run (deciding which ones matter is a judgment call worth discussing). This will typically give you a solid draft of your Limitations or Threats to Validity section.
13. Discussion drafted—this should reflect on interpretation, implications, future work, and a short (typically one-paragraph) conclusion.
14. Full first draft assembled ★—abstract, all sections, references cleaned, formatted for the target venue.

Review and submission

15. Coauthor review complete—Gather feedback from all your coauthors and address it! Typically you'll just add them to Overleaf. Give your coauthors the deadline and remind them.
16. Submission-ready ★—venue-formatted; cover letter, supplementary materials, and archive of data and code prepared.
17. Submitted!

See also

• CommunityData:Quantitative planning document
• CommunityData:Qualitative planning document