CommunityData:Git

Wondering why this is a topic to care about? Check this what is version control? article.

Install Git

To get started, you will need to install git. Doing so requires different steps depending on your operating system. Basic instructions available from the Git website.

You will also likely need to set it up so it knows what your name and email address is. You can do that like:

$ git config --global user.name "John Doe"
$ git config --global user.email johndoe@example.com

Note that RStudio also has Git integration now. Instructions and details available via RStudio support documentation.

Configuring Git for submodules

Once you've installed git, there are some configuration options which will make your life much easier. You can set them globally with the following commands:

git config --global alias.spull '!__git_spull() { git pull "$@" && git submodule sync --recursive && git submodule update --init --recursive; }; __git_spull'
git config --global status.submoduleSummary true

These two commands will ensure that git works a little better with submodules. Submodules are essentially git repositories that are buried inside other git repositories. For example, the wikiresearch repository currently uses the RCommunityData repository as a submodule. If you're working in a repository like this, you'll want to use git spull instead of just git pull which will also check for and pull changes made in any of your submodules.

Gitea

We have a private git server which uses Gitea to manage permissions for git repositories. It's like a private Github server that hosts our respositories, but just ours, and on our server. If you have a question about Gitea, please send a message in the #communitydata-it channel over in Matrix or look in Gitea's general documentation.

Creating an account

You can make a new Gitea account by reaching out to Matt or Carl on Matrix and providing your email address and what username you would like to use on Gitea. We are in the process of updating the registration procedure.

Adding SSH keys

Connect your local git to your new Gitea account by adding your SSH key ((usually ~/.ssh/id_rsa.pub) to "SSH/GPG Keys" in your user "Settings" page.

Cloning a repository

"Cloning" a repository downloads the files, as well as the history, of a repository. It also creates a new git instance in that directory, so that you can commit changes to the code.

To clone a repository, run the following command:

 git clone gitea@gitea.communitydata.science:$USERNAME/REPOSITORY_NAME$.git

Creating a new repository

To create a new Gitea repository, follow the instructions in the ``Create button in the top-left corner of the homepage, immediately left of the user profile icon.

Pushing data into a new repository on the server from a local git repository you already have

You could then go to wherever the files are that you would like to track, and add this repository as a remote, like so:

$ cd foo
$ git remote add origin gitea@gitea.communitydata.science:$USERNAME/REPOSITORY_NAME$.git
$ git push --set-upstream origin main

If this project already exists in git, then it's even easier. Just change the remote, and push it.

$ git remote set-url origin gitea@gitea.communitydata.science:$USERNAME/REPOSITORY_NAME$.git
$ git push

Giving other users access to your repository

To add new users, use the project Settings page from the home page of the repository.

Setting the default branch

Gitea sometimes has issues because it expects the default branch to be "main", but some repositories may have a different default branch name. To fix this (if it is a problem), change the default branch for the repositories under "Settings > Branches".

Using git-annex to manage large files in git

 Note: This is still experimental, and may go away. Don't put files in it without a backup.

Getting Set Up

Git is not a very good system for managing large files, which is a problem for us, since we often have large data files. Enter git-annex, a system that works in tandem with git and lets you store large files (but avoids using git as the data store). Our gitolite installation supports git-annex. To start using git-annex, install git-annex locally in your computer. Most GNU/Linux distributions have git-annex packages. If you're on a Mac, in Terminal.app, try the instructions from Homebrew: https://formulae.brew.sh/formula/git-annex.

Setting Up Your Repo To Use Annex

Then, in your existing git repository execute the following initialization command:

$ git annex init

This needs to be done only once. To add a file, in your repository, run the following commands:

$ mkdir data
$ cp ~/largedata.csv.bz2 data/

You should encrypt the file if the data is not public. You can use GNU Privacy Guard to do the encryption, and have all your collaborators as recipients for the file. Once encrypted, execute the following commands to include and push the file to the server.

$ git annex add data/largedata.csv.bz2.gpg
$ git commit -m "Added data file"
$ git push --all
$ git annex copy --to origin

Using an Existing Annex Repo

Once these commands are successful, your collaborators should be able to get the file with the following command (assuming that they have already run git annex init):

$ git annex get data/largedata.csv.bz2.gpg

Once you've encrypted non-public data, git-annex is easy to use using the webapp.

$ git annex webapp

Gitolite (deprecated)

Several older repositories are hosted in our old server at https://code.communitydata.science which we have deprecated in favor of Gitea.

If you need access to something in the old Gitolite Repository, you should ask on the #communitydata-it Matrix channel for access. If you need access to a specific repository only, mention which one. While you likely already know which repo you want access to, you can find the public ones on code.communitydata.science, and a complete list of all of them conf/gitolite.conf file in the gitolite-admin git repository. If you are a new CDSC member, mention that you need to be added to the @collective group in Gitolite. Anybody in the collective who uses the Git repository will be able add you.