CommunityData:Automating and Streamlining Walkthrough

Welcome to the Automation and Streamlining Walkthrough!

This guide steps you through why and how you might like to adopt some of our tips and tricks around automating and streamlining your research workflow. Our questions and methods lead to a couple of challenges and complexities. These are strategies we use to keep away from certain kinds of annoyances, traps, and mistakes. Some of what's described here will be a *lot* easier if you have completed the CommunityData:Onboarding Checklist. CDSC members will want to make sure they have a fresh copy of the cdsc_examples git repository.

Staying Organized

The kid cartoon version of the scientific method describes a linear and rather sterile process from hypothesis to experiment to insight -- the reality looks a lot messier. We rummage around, scratch our heads, wander down dark alleys, think and re-think, scrape, crunch, gather....and then we look around at all the beautiful mess we've made and try to turn it into a paper: write and re-write, submit, revise, re-submit, re-revise, re-resubmit --- and then maybe a year or two later, we're announcing, releasing, publishing and presenting. Keeping track of the weird and wild ride can be tremendously helpful.

One strategy is taking notes for yourself as you go along -- think of it as keeping a lab notebook.

Another strategy is to make sure you don't let specific details fall through the cracks by developing a way to keep track of metadata.

Automating Updates

Automation can be extremely helpful, but it's an investment. You will not regret time spent on modest automation, in particular if you do computational work. You never want to be in the position of copy-pasting from R into LaTeX or Word: it is error prone and when you later find yourself needing to revise, you might not remember where you got that number from ....... instead, what you want is some automation magic, so that every time you run your R code, your new data and fresh visualizations land in your Overleaf. Example code for making this work is in the *cdsc_examples/R_examples/automation* git repository.

We have an overall guide for setting up this marvelous automation.

If you want to learn even more, there's a little tutorial on Knitr, and here's a more expanded guide.

Building from Prior Efforts

CommunityData:Zotero CommunityData:TeX — Installing our LaTeX templates CommunityData:LaTex Diff — For an R+R, it's often helpful to create a PDF that shows the changes made. Here's one way to do that. Latexdiff is available on CommunityData:Kibo.