CommunityData:Kibo

From CommunityData
Revision as of 00:07, 11 April 2020 by Sohw (talk | contribs) (→‎Storing Data)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Kibo is a server that CDSC has which can be used for research. It's a pretty powerful computer (2x14 core processors, 12x32GB memory, 12x2.4TB storage). To use Kibo, you will need to have a NU NetID and often, the Northwestern VPN. Instructions are below.

Getting access to Kibo[edit]

The first step will be getting a NU NetID. You should contact [Aaron Shaw] for this. If you already have a NU NetID (e.g. you are a Northwestern student), you can skip this initial step, but you still need your NetID to be given access to kibo.

Once you have a NetID, you should contact to let Aaron know along with the NetID. He will contact the Northwestern IT folks and they will enable your NetID account to access the Kibo system.

Logging into Kibo[edit]

Now that you have access, you can log into kibo! If you're off the NU campus, you need to use the Northwestern VPN to connect. Instructions for the VPN are [here]. Once you have a NU NetID and the VPN set up:

  1. Connect to Northwestern University using the VPN (GlobalConnect)
  2. Open your terminal and ssh yourNetID@kibo.soc.northwestern.edu
  3. The first time you connect it will ask you if you are sure you want to continue connecting. You are sure (presumably). Type yes and enter.
  4. It will prompt you for yourNetID@kibo.soc.northwestern.edu's password. Enter your NU NetID password and you should now be connected.

Documentation and Next Steps[edit]

Navigating Kibo[edit]

The structure of directory set-up on Kibo is simple. Once you log in, you will be at your home directory which is a directory named your NetID. If you ls ../, you'll see that there are multiple such directories labeled with NetIDs of other people who use Kibo. Generally, you will not have write access on those other directories nor root access, unless that access has been granted.

Storing Data[edit]

Kibo is set up to retrieve, store, and analyze lots of data. One use case is gathering data from the web, as we are doing for the COVID-19 project. If you are gathering data like that, you shouldn't use your home directory to store it.

You will need to create a space for it in /data and symlink to your home directory. Something like:

mkdir /data/users/my_user
mkdir /data/users/my_user/my_new_project
ln -s /data/users/my_user/my_new_project ~/my_new_project

Then, whatever you put into ~/my_new_project will be stored in /data, where we have lots of space.

If you've already been using Kibo and need to move things from your home directory into /data, set up your user directory as above (if you haven't already) and then use mv [current file location] [desired file location] to move the relevant files.

Viewing image files[edit]

We currently don't have anything set up to open and see the image files stored on the remote Kibo machine from the terminal. For now, one easy solution would be to enter scp -T -r yourNetID@kibo.soc.northwestern.edu:pathtofile . in a folder on your local machine. This securely copies the files to the current directory that you are in. You will have to enter your password as if you were logging into kibo.

If you use VS Code, there is a very nifty extension that can be used to easily access files (including images!) from the GUI of the editor, instructions and more information for which can be found here.