DS4UX (Spring 2016)/Reading and writing files

One of the most common operations you will perform when you use Python for data analysis will be reading and writing data to and from flat files. These are somtimes referred to as flat file databases—not because the files themselves are a special kind of code, but because they hold multiple pieces of data in a simple, structured format that facilitates common data processing operations like querying, overwriting, appending, and ingesting.

Probably to most basic type of "flat file database" is a plain text file with a different piece of data on each line. To process a file like this in Python, you generally follow these steps:
 * 1) Open the file and save it in working memory (also sometimes referred to as "buffer") as a file object.
 * 2) Read through the file object line-by-line and do something with it.
 * 3) Close the file object (whether or not you have changed it, or just read it and done something else with what your read).

Plain text flat files with one piece of data per line (for example, a list of names) are very easy to create and can hold a lot of data. In the simplest case, the new line in the file (defined by an invisible newline character  at the end of each line) serves as the delimiter between different pieces of data. This delimiter allows Python to know where one piece of data ends and the next begins, when it reads through the file.

However, if you want to store more complex data (such as a list of names, genders, and # of people of that gender with that name), you need to come up with a way to store associated datapoints together. To do this, you need to define a new delimiter so that you can separate different associated datapoints in each row. Theoretically, you could use any character as a delimiter, but two of the most common ones are a tab  and a comma.

Now that you have two delimiters, one that delimits vertically and the other that delimits horizontally, you have a two-dimensional matrix, a way of structuring data that is so common that we don't generally even use it's technical name—instead, we generally use the name for the kind of software application we often use to display matrix data: a spreadsheet.

Goals
In the exercises below, you will learn the basic syntax for reading common types of file into Python, and transforming them into other types of files. We will only cover a couple of the most basic types of file conversation you might want to do here:


 * 1) reading a raw (and messy!) comma-separated text file into Python with the generic   method, parsing through it line by line to clean it up, and output a much cleaner matrix version with newlines and tabs as delimiters.
 * 2) reading that file with Python's   library, which provides you with additional options for reading, formatting, and writing files with datapoints that are separated by newlines (vertically) and commas (horizontally).

Instructions

 * 1) If you haven't done so already, download the ZIPPED code and data file called , unzip it, and navigate to it in your Terminal or Powershell.
 * 2) Open the code and datafiles in that directory in TextWrangler as well, so you can see the code while we walk through it.

Outputting a TSV (tab-separated value) file

 * 1) open the input file   in TextWrangler. From the drop-down menu, select View->Text display->Show Invisibles. What do you see?
 * 2) run  . Look at the output printed on the terminal. Now examine the output file itself. What has changed?

Outputting a CSV (comma-separated value) file

 * 1) run , which takes the same messy input file as the previous script. Look at the output printed on the terminal. Now examine the output file itself. How is this file different from the .tsv file we created earlier?

Important syntax concepts

 * 1) using  will almost always be the best way to open a file (for reading or writing). This saves you some steps, and makes sure that the file is properly closed (removed from working memory) once you're done with it. There are other patterns for opening files, but I suggest you get used to this pattern before you explore any others.
 * 2) when you specify  after the filename in   you're saying whether you want to open the file to READ it; when you specify , you are saying you want to open the file (or create it, if it doesn't exist) so that you can write to it—add data into the file to save. There are some other options for reading and writing, such as   and  , which are used in different circumstances and for different types of data. We'll cover these in less detail next week.
 * 3)  vs  : both of these are functions that make it easy to read through a file object line-by-line once you have opened the file.   provides you with additional parameters that allow you to control how the data from the file is interpreted, which often comes in handy when dealing with messy data, but to use it you have to   first. The same goes for , used to write the .tsv file in the first script, and  , used to write the .csv file in the second script.