CommunityData:Kibo

Kibo is a server that CDSC has which can be used for research. For an overview, check out CommunityData:Kibo Quick Reference. It's a pretty powerful computer (2x14 core processors, 12x32GB memory, 12x2.4TB storage). To use Kibo, you will need to have a NU NetID and often, the Northwestern VPN. Instructions are below.

Getting access to Kibo

The first step will be getting a NU NetID. You should contact [Aaron Shaw] for this. If you already have a NU NetID (e.g. you are a Northwestern student), you can skip this initial step, but you still need your NetID to be given access to kibo.

Once you have a NetID, you should contact to let Aaron know along with the NetID. He will contact the Northwestern IT folks and they will enable your NetID account to access the Kibo system.

Logging into Kibo

Now that you have access, you can log into kibo! If you're off the NU campus, you need to use the Northwestern VPN to connect. Instructions for the VPN are [here]. Once you have a NU NetID and the VPN set up:

1. Connect to Northwestern University using the VPN (GlobalConnect)
2. Open your terminal and ssh yourNetID@kibo.soc.northwestern.edu
3. The first time you connect it will ask you if you are sure you want to continue connecting. You are sure (presumably). Type yes and enter.
4. It will prompt you for yourNetID@kibo.soc.northwestern.edu's password. Enter your NU NetID password and you should now be connected.

Navigating Kibo

The structure of directory set-up on Kibo is simple. Once you log in, you will be at your home directory which is a directory named your NetID. If you ls ../, you'll see that there are multiple such directories labeled with NetIDs of other people who use Kibo. Generally, you will not have write access on those other directories nor root access, unless that access has been granted.

Email forwarding

As part of using kibo, the system will create and send you email about important system level things that happen. By default, this email will just queue up in your inbox kept on kibo (which you almost certainly didn't even realize you had) which means you will almost certainly never see it. For example, any jobs you have running from a cronjob will send their output to you when they are done which is important because they will often contains errors. In some cases, emails full of error logs have been queuing up on kibo for months without anybody knowing about them.

You can read this mail using a console-based mail client installed on kibo like mutt but it's almost certainly easier to setup email forwarding. Doing so is as simple as creating a file called ~/.forward that contains a single line with the email address you want your email forwarded to.

For example, Mako just ran:

echo "makohill@uw.edu" > ~/.forward

Storing Data

Kibo is set up to retrieve, store, and analyze lots of data. One use case is gathering data from the web, as we are doing for the COVID-19 project. If you are gathering data like that, you shouldn't use your home directory to store it.

You will need to create a space for it in /data and symlink to your home directory. Something like:

mkdir /data/users/my_user
mkdir /data/users/my_user/my_new_project
ln -s /data/users/my_user/my_new_project ~/my_new_project

Then, whatever you put into ~/my_new_project will be stored in /data, where we have lots of space.

If you've already been using Kibo and need to move things from your home directory into /data, set up your user directory as above (if you haven't already) and then use mv [current file location] [desired file location] to move the relevant files.

Viewing image files

We currently don't have anything set up to open and see the image files stored on the remote Kibo machine from the terminal. For now, one easy solution would be to enter scp -T -r yourNetID@kibo.soc.northwestern.edu:pathtofile . in a folder on your local machine. This securely copies the files to the current directory that you are in. You will have to enter your password as if you were logging into kibo.

If you use VS Code, there is a very nifty extension that can be used to easily access files (including images!) from the GUI of the editor, instructions and more information for which can be found here.

Publishing data

If you would like to share data, just create a directory called public_html in your home directory. Any file or directory in that directory will be available at a public URL with your NetID baked into it in the form of https://kibo.communitydata.science/~YOURNETID. For example, Mako is bmh1867 so his public URL would be:

https://kibo.communitydata.science/~bmh1867/

Keep in mind limited space in your home directories and be sure to symlink to any large files or subdirectories. As long as you are the owner, symlinks should work fine!

MySQL

Kibo runs a MySQL database server for the group. Details on that are on CommunityData:MySQL.