Commit 155b2a8e by Greg Wilson

### Modified build to generate Markdown files for IPython Notebooks in the source...

`Modified build to generate Markdown files for IPython Notebooks in the source directories so that the Jekyll build will put all the files in the right places.  (Having a separate CACHED directory is cleaner, but then we have to move or patch files after the fact to make links resolve.)`
parent 8df52db8
 01-numpy.md 02-func.md 03-loop.md 04-cond.md 05-defensive.md 06-cmdline.md
pages/01-numpy.md 0 → 100644
This diff is collapsed.
pages/02-func.md 0 → 100644
This diff is collapsed.
pages/03-loop.md 0 → 100644
 --- layout: lesson root: ../.. --- ## Analyzing Multiple Data Sets

We have created a function called analyze that creates graphs of the minimum, average, and maximum daily inflammation rates for a single data set:

%matplotlib inline

import numpy as np
from matplotlib import pyplot as plt

def analyze(filename):

plt.figure(figsize=(10.0, 3.0))

plt.subplot(1, 3, 1)
plt.ylabel('average')
plt.plot(data.mean(0))

plt.subplot(1, 3, 2)
plt.ylabel('max')
plt.plot(data.max(0))

plt.subplot(1, 3, 3)
plt.ylabel('min')
plt.plot(data.min(0))

plt.tight_layout()
plt.show()

analyze('inflammation-01.csv')

We can use it to analyze other data sets one by one:

analyze('inflammation-02.csv')

but we have a dozen data sets right now and more on the way. We want to create plots for all our data sets with a single statement. To do that, we'll have to teach the computer how to repeat things.

Objectives

• Explain what a for loop does.
• Correctly write for loops to repeat simple calculations.
• Trace changes to a loop variable as the loop runs.
• Trace changes to other variables as they are updated by a for loop.
• Explain what a list is.
• Create and index lists of simple values.
• Use a library function to get a list of filenames that match a simple wildcard pattern.
• Use a for loop to process multiple files.
### For Loops

Suppose we want to print each character in the word "lead" on a line of its own. One way is to use four print statements:

def print_characters(element):
print element[0]
print element[1]
print element[2]
print element[3]

l
e
a
d

but that's a bad approach for two reasons:

1. It doesn't scale: if we want to print the characters in a string that's hundreds of letters long, we'd be better off just typing them in.

2. It's fragile: if we give it a longer string, it only prints part of the data, and if we give it a shorter one, it produces an error because we're asking for characters that don't exist.

print_characters('tin')
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-13-5bc7311e0bf3> in <module>()
----> 1 print_characters('tin')

<ipython-input-12-11460561ea56> in print_characters(element)
3     print element[1]
4     print element[2]
----> 5     print element[3]
6

IndexError: string index out of ranget
i
n

Here's a better approach:

def print_characters(element):
for char in element:
print char

This is shorter---certainly shorter than something that prints every character in a hundred-letter string---and more robust as well:

print_characters('oxygen')

The improved version of print_characters uses a for loop to repeat an operation---in this case, printing---once for each thing in a collection. The general form of a loop is:

for variable in collection:
do things with variable

We can call the loop variable anything we like, but there must be a colon at the end of the line starting the loop, and we must indent the body of the loop.

Here's another loop that repeatedly updates a variable:

length = 0
for vowel in 'aeiou':
length = length + 1
print 'There are', length, 'vowels'

It's worth tracing the execution of this little program step by step. Since there are five characters in 'aeiou', the statement on line 3 will be executed five times. The first time around, length is zero (the value assigned to it on line 1) and vowel is 'a'. The statement adds 1 to the old value of length, producing 1, and updates length to refer to that new value. The next time around, vowel is 'e' and length is 1, so length is updated to be 2. After three more updates, length is 5; since there is nothing left in 'aeiou' for Python to process, the loop finishes and the print statement on line 4 tells us our final answer.

Note that a loop variable is just a variable that's being used to record progress in a loop. It still exists after the loop is over, and we can re-use variables previously defined as loop variables as well:

letter = 'z'
for letter in 'abc':
print letter
print 'after the loop, letter is', letter

Note also that finding the length of a string is such a common operation that Python actually has a built-in function to do it called len:

print len('aeiou')

len is much faster than any function we could write ourselves, and much easier to read than a two-line loop; it will also give us the length of many other things that we haven't met yet, so we should always use it when we can.

Challenges

1. Python has a built-in function called range that creates a list of numbers: range(3) produces [0, 1, 2], range(2, 5) produces [2, 3, 4], and range(2, 10, 3) produces [2, 5, 8]. Using range, write a function that prints the \(N\) natural numbers: ~python print_N(3) 1 2 3 ~

2. Exponentiation is built into Python: ~python print 2**4 16 ~ It also has a function called pow that calculates the same value. Write a function called expo that uses a loop to calculate the same result.

3. Python's strings have methods, just like NumPy's arrays. One of these is called reverse: ~python print 'Newton'.reverse() notweN ~ Write a function called rev that does the same thing: ~python print rev('Newton') notweN ~ As always, be sure to include a docstring.

### Lists

Just as a for loop is a way to do operations many times, a list is a way to store many values. Unlike NumPy arrays, there are built into the language. We create a list by putting values inside square brackets:

odds = [1, 3, 5, 7]
print 'odds are:', odds

We select individual elements from lists by indexing them:

print 'first and last:', odds[0], odds[-1]

and if we loop over a list, the loop variable is assigned elements one at a time:

for number in odds:
print number

There is one important difference between lists and strings: we can change the values in a list, but we cannot change the characters in a string. For example:

names = ['Newton', 'Darwing', 'Turing'] # typo in Darwin's name
print 'names is originally:', names
names[1] = 'Darwin' # correct the name
print 'final value of names:', names

works, but:

name = 'Bell'
name[0] = 'b'

does not.

Ch-Ch-Ch-Changes

Data that can be changed is called mutable, while data that cannot be is called immutable. Like strings, numbers are immutable: there's no way to make the number 0 have the value 1 or vice versa (at least, not in Python—there actually are languages that will let people do this, with predictably confusing results). Lists and arrays, on the other hand, are mutable: both can be modified after they have been created.

Programs that modify data in place can be harder to understand than ones that don't because readers may have to mentally sum up many lines of code in order to figure out what the value of something actually is. On the other hand, programs that modify data in place instead of creating copies that are almost identical to the original every time they want to make a small change are much more efficient.

There are many ways to change the contents of in lists besides assigning to elements:

odds.append(11)
print 'odds after adding a value:', odds
del odds[0]
print 'odds after removing the first element:', odds
odds.reverse()
print 'odds after reversing:', odds

Challenges

1. Write a function called total that calculates the sum of the values in a list. (Python has a built-in function called sum that does this for you. Please don't use it for this exercise.)
### Processing Multiple Files

We now have almost everything we need to process all our data files. The only thing that's missing is a library with a rather unpleasant name:

import glob

The glob library contains a single function, also called glob, that finds files whose names match a pattern. We provide those patterns as strings: the character * matches zero or more characters, while ? matches any one character. We can use this to get the names of all the IPython Notebooks we have created so far:

print glob.glob('*.ipynb')
['01-numpy.ipynb', '02-func.ipynb', '03-loop.ipynb', '04-cond.ipynb', '05-defensive.ipynb', '06-cmdline.ipynb', 'spatial-intro.ipynb']

or to get the names of all our CSV data files:

print glob.glob('*.csv')
['inflammation-01.csv', 'inflammation-02.csv', 'inflammation-03.csv', 'inflammation-04.csv', 'inflammation-05.csv', 'inflammation-06.csv', 'inflammation-07.csv', 'inflammation-08.csv', 'inflammation-09.csv', 'inflammation-10.csv', 'inflammation-11.csv', 'inflammation-12.csv', 'small-01.csv', 'small-02.csv', 'small-03.csv', 'swc_bc_coords.csv']

As these examples show, glob.glob's result is a list of strings, which means we can loop over it to do something with each filename in turn. In our case, the "something" we want is our analyze function. Let's test it by analyzing the first three files in the list:

filenames = glob.glob('*.csv')
filenames = filenames[0:3]
for f in filenames:
print f
analyze(f)
inflammation-01.csv
inflammation-02.csv
inflammation-03.csv

Sure enough, the maxima of these data sets show exactly the same ramp as the first, and their minima show the same staircase structure.

Challenges

1. Write a function called analyze_all that takes a filename pattern as its sole argument and runs analyze for each file whose name matches the pattern.

Key Points

• Use for variable in collection to process the elements of a collection one at a time.
• The body of a for loop must be indented.
• Use len(thing) to determine the length of something that contains other values.
• [value1, value2, value3, ...] creates a list.
• Lists are indexed and sliced in the same way as strings and arrays.
• Lists are mutable (i.e., their values can be changed in place).
• Strings are immutable (i.e., the characters in them cannot be changed).
• Use glob.glob(pattern) to create a list of files whose names match a pattern.
• Use * in a pattern to match zero or more characters, and ? to match any single character.

Next Steps

We have now solved our original problem: we can analyze any number of data files with a single command. More importantly, we have met two of the most important ideas in programming:

1. Use functions to make code easier to re-use and easier to understand.
2. Use lists and arrays to store related values, and loops to repeat operations on them.

We have one more big idea to introduce, and then we will be able to go back and create a heat map like the one we initially used to display our first data set.

pages/04-cond.md 0 → 100644
This diff is collapsed.
This diff is collapsed.
 --- layout: lesson root: ../.. --- ## Command-Line Programs

The IPython Notebook and other interactive tools are great for prototyping code and exploring data, but sooner or later we will want to use our program in a pipeline or run it in a shell script to process thousands of data files. In order to do that, we need to make our programs work like other Unix command-line tools. For example, we may want a program that reads a data set and prints the average inflammation per patient:

5.45
5.425
6.1
...
6.4
7.05
5.9

but we might also want to look at the minimum of the first four lines

or the maximum inflammations in several files one after another:

Our overall requirements are:

1. If no filename is given on the command line, read data from standard input.
2. If one or more filenames are given, read data from them and report statistics for each file separately.
3. Use the --min, --mean, or --max flag to determine what statistic to print.

To make this work, we need to know how to handle command-line arguments in a program, and how to get at standard input. We'll tackle these questions in turn below.

Objectives

• Use the values of command-line arguments in a program.
• Handle flags and files separately in a command-line program.
• Read data from standard input in a program so that it can be used in a pipeline.
### Command-Line Arguments

Using the text editor of your choice, save the following in a text file:

!cat sys-version.py
import sys
print 'version is', sys.version

The first line imports a library called sys, which is short for "system". It defines values such as sys.version, which describes which version of Python we are running. We can run this script from within the IPython Notebook like this:

%run sys-version.py
version is 2.7.5 |Anaconda 1.8.0 (x86_64)| (default, Oct 24 2013, 07:02:20)
[GCC 4.0.1 (Apple Inc. build 5493)]

or like this:

!ipython sys-version.py
version is 2.7.5 |Anaconda 1.8.0 (x86_64)| (default, Oct 24 2013, 07:02:20)
[GCC 4.0.1 (Apple Inc. build 5493)]

The first method, %run, uses a special command in the IPython Notebook to run a program in a .py file. The second method is more general: the exclamation mark ! tells the Notebook to run a shell command, and it just so happens that the command we run is ipython with the name of the script.

Here's another script that does something more interesting:

!cat argv-list.py
import sys
print 'sys.argv is', sys.argv

The strange name argv stands for "argument values". Whenever Python runs a program, it takes all of the values given on the command line and puts them in the list sys.argv so that the program can determine what they were. If we run this program with no arguments:

!ipython argv-list.py
sys.argv is ['/Users/gwilson/s/bc/python/novice/argv-list.py']

the only thing in the list is the full path to our script, which is always sys.argv[0]. If we run it with a few arguments, however:

!ipython argv-list.py first second third
sys.argv is ['/Users/gwilson/s/bc/python/novice/argv-list.py', 'first', 'second', 'third']

then Python adds each of those arguments to that magic list.

With this in hand, let's build a version of readings.py that always prints the per-patient mean of a single data file. The first step is to write a function that outlines our implementation, and a placeholder for the function that does the actual work. By convention this function is usually called main, though we can call it whatever we want:

import sys
import numpy as np

def main():
script = sys.argv[0]
filename = sys.argv[1]
for m in data.mean(axis=1):
print m

This function gets the name of the script from sys.argv[0], because that's where it's always put, and the name of the file to process from sys.argv[1]. Here's a simple test: