CommunityData:Automating and Streamlining Walkthrough

Welcome to the Automation and Streamlining Walkthrough!

This guide steps you through why and how you might like to adopt some of our tips and tricks around automating and streamlining your research workflow. Our questions and methods lead to a couple of challenges and complexities. These are strategies we use to keep away from certain kinds of annoyances, traps, and mistakes. Some of what's described here will be a *lot* easier if you have completed the CommunityData:Onboarding Checklist. CDSC members will want to make sure they have a fresh copy of the cdsc_examples git repository.

Staying Organized

The kid cartoon version of the scientific method describes a linear and rather sterile process from hypothesis to experiment to insight -- the reality looks a lot messier. We rummage around, scratch our heads, wander down dark alleys, think and re-think, scrape, crunch, gather....and then we look around at all the beautiful mess we've made and try to turn it into a paper: write and re-write, submit, revise, re-submit, re-revise, re-resubmit --- and then maybe a year or two later, we're announcing, releasing, publishing and presenting. Keeping track of the weird and wild ride can be tremendously helpful.

1. ***Take notes*** for yourself as you go along -- think of it as keeping a lab notebook.
2. Don't let specific details fall through the cracks: keep track of metadata.

Automating Updates

Automation can be extremely helpful, but it's an investment. You will not regret time spent on modest automation, in particular if you do computational work. You never want to be in the position of copy-pasting from R into LaTeX or Word: it is error prone and when you later find yourself needing to revise, you might not remember where you got that number from ....... instead, what you want is some automation magic, so that every time you run your R code, your new data and fresh visualizations land in your Overleaf. Example code for making this work is in the **cdsc_examples/R_examples/automation** git repository.

We have an overall guide for setting up this marvelous automation.

If you want to learn even more, there's a little tutorial on Knitr, and here's a more expanded guide.

Building from Prior Efforts

You can save yourself a lot of time and confusion if you build from the work that others have done in the lab to streamline and automate their work.

You can build your bibliography using our shared Zotero --- if you have Dropbox wired up to your Overleaf, that means one or zero clicks to keep your bibliography up to date.

You can make nice-looking documents if you install and use our LaTeX templates.

Revise and resubmit makes papers better but can be a time crunch and if you have co-authors, it can get hard to tell what's been done or what certain revisions were trying to accomplish. When you get reviews back that ask you to make changes, you can save yourself a lot of annoyance if you follow this four-part method:

1. Keep a copy of your original submission, including the main.tex file (if you are using .Rtex file because you followed the above advice about automation....please note that the main.tex file is hidden in Overleaf -- click the little 'document' icon next to 'Recompile' (mouseover has a tooltip that says 'Logs and output files' -- then 'Other logs and files' at the bottom right -- note that the output.bbl file is there too, that's also useful because you'll want it for your upload to arXiv if you post a preprint). You will need this for the LaTeX Diff below.
2. Paste all your reviews into a Google Sheet -- feel free to use this as a template
3. Also paste all your reviews into a response to reviewers letter in the same Overleaf project where you wrote the paper. Place them below the /end{document} line so it won't show up.. As you revise to address the issue, move the comment up into the letter, quote it and then write what you did. It might feel a skosh redundant with the Google Sheet, and it is ... think of it as double-entry bookkeeping to make sure nothing gets dropped.
4. Show your work: some venues require (and reviewers often appreciate) a document that shows what was changed. If you use Word, that means turning on Track Changes. If you use LaTeX, latexdiff is a way to achieve this. Latexdiff is available on CommunityData:Kibo.