01-numpy.md 33.7 KB
Newer Older
1
---
2
3
4
5
title: Analyzing Patient Data
teaching: 30
exercises: 0
questions:
Greg Wilson's avatar
Greg Wilson committed
6
- "How can I process tabular data files in Python?"
7
8
objectives:
- "Explain what a library is, and what libraries are used for."
9
- "Import a Python library and use the functions it contains."
10
11
12
13
- "Read tabular data from a file into a program."
- "Assign values to variables."
- "Select individual values and subsections from data."
- "Perform operations on arrays of data."
14
- "Plot simple graphs from data."
15
keypoints:
Greg Wilson's avatar
Greg Wilson committed
16
17
18
19
20
21
- "Import a library into a program using `import libraryname`."
- "Use the `numpy` library to work with arrays in Python."
- "Use `variable = value` to assign a value to a variable in order to record it in memory."
- "Variables are created on demand whenever a value is assigned to them."
- "Use `print(something)` to display the value of `something`."
- "The expression `array.shape` gives the shape of an array."
22
- "Use `array[x, y]` to select a single element from a 2D array."
Greg Wilson's avatar
Greg Wilson committed
23
- "Array indices start at 0, not 1."
Dustin Lang's avatar
Dustin Lang committed
24
- "Use `low:high` to specify a `slice` that includes the indices from `low` to `high-1`."
Greg Wilson's avatar
Greg Wilson committed
25
26
27
28
29
- "All the indexing and slicing that works on arrays also works on strings."
- "Use `# some kind of explanation` to add comments to programs."
- "Use `numpy.mean(array)`, `numpy.max(array)`, and `numpy.min(array)` to calculate simple statistics."
- "Use `numpy.mean(array, axis=0)` or `numpy.mean(array, axis=1)` to calculate statistics across the specified axis."
- "Use the `pyplot` library from `matplotlib` for creating simple visualizations."
30
---
31
In this lesson we will learn how to manipulate the inflammation dataset with Python. But before we discuss how to deal with many data points, we will show how to store a single value on the computer.
32
33


34
The line below [assigns](reference.html#assignment) the value `55` to a [variable](reference.html#variable) `weight_kg`:
35

36
~~~
37
weight_kg = 55
Greg Wilson's avatar
Greg Wilson committed
38
~~~
39
40
{: .python}

41
A variable is just a name for a value,
42
such as `x_val`, `current_temperature`, or `subject_id`.
43
Python's variables must begin with a letter and are [case sensitive](reference.html#case-sensitive).
Kyler Brown's avatar
Kyler Brown committed
44
We can create a new variable by assigning a value to it using `=`.
45
46
When we are finished typing and press Shift+Enter,
the notebook runs our command.
47

48
Once a variable has a value, we can print it to the screen:
49

50
~~~
51
print(weight_kg)
Greg Wilson's avatar
Greg Wilson committed
52
~~~
53
54
55
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
56
57
55
~~~
58
{: .output}
59

60
and do arithmetic with it:
61

62
~~~
63
print('weight in pounds:', 2.2 * weight_kg)
Greg Wilson's avatar
Greg Wilson committed
64
~~~
65
66
67
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
68
69
weight in pounds: 121.0
~~~
70
{: .output}
71

jstapleton's avatar
jstapleton committed
72
73
74
As the example above shows,
we can print several things at once by separating them with commas.

75
We can also change a variable's value by assigning it a new one:
76

77
~~~
Greg Wilson's avatar
Greg Wilson committed
78
weight_kg = 57.5
79
print('weight in kilograms is now:', weight_kg)
Greg Wilson's avatar
Greg Wilson committed
80
~~~
81
82
83
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
84
85
weight in kilograms is now: 57.5
~~~
86
{: .output}
87

88
89
If we imagine the variable as a sticky note with a name written on it,
assignment is like putting the sticky note on a particular value:
90

91
![Variables as Sticky Notes](../fig/python-sticky-note-variables-01.svg)
92

93
94
95
This means that assigning a value to one variable does *not* change the values of other variables.
For example,
let's store the subject's weight in pounds in a variable:
96

97
~~~
Greg Wilson's avatar
Greg Wilson committed
98
weight_lb = 2.2 * weight_kg
99
print('weight in kilograms:', weight_kg, 'and in pounds:', weight_lb)
Greg Wilson's avatar
Greg Wilson committed
100
~~~
101
102
103
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
104
105
weight in kilograms: 57.5 and in pounds: 126.5
~~~
106
{: .output}
107

108
![Creating Another Variable](../fig/python-sticky-note-variables-02.svg)
109

110
and then change `weight_kg`:
111

112
~~~
Greg Wilson's avatar
Greg Wilson committed
113
weight_kg = 100.0
114
print('weight in kilograms is now:', weight_kg, 'and weight in pounds is still:', weight_lb)
Greg Wilson's avatar
Greg Wilson committed
115
~~~
116
117
118
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
119
120
weight in kilograms is now: 100.0 and weight in pounds is still: 126.5
~~~
121
{: .output}
122

123
![Updating a Variable](../fig/python-sticky-note-variables-03.svg)
124

125
126
127
128
Since `weight_lb` doesn't "remember" where its value came from,
it isn't automatically updated when `weight_kg` changes.
This is different from the way spreadsheets work.

129
> ## Who's Who in Memory
Benjamin Laken's avatar
Benjamin Laken committed
130
>
131
132
133
> You can use the `%whos` command at any time to see what
> variables you have created and what modules you have loaded into the computer's memory.
> As this is an IPython command, it will only work if you are in an IPython terminal or the Jupyter Notebook.
Benjamin Laken's avatar
Benjamin Laken committed
134
>
135
> ~~~
136
137
> %whos
> ~~~
138
139
140
> {: .python}
>
> ~~~
141
142
143
144
145
146
> Variable    Type       Data/Info
> --------------------------------
> numpy       module     <module 'numpy' from '/Us<...>kages/numpy/__init__.py'>
> weight_kg   float      100.0
> weight_lb   float      126.5
> ~~~
147
148
> {: .output}
{: .callout}
Benjamin Laken's avatar
Benjamin Laken committed
149

devendra1810's avatar
devendra1810 committed
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
Words are useful,
but what's more useful are the sentences and stories we build with them.
Similarly,
while a lot of powerful, general tools are built into languages like Python,
specialized tools built up from these basic units live in [libraries](reference.html#library)
that can be called upon when needed.

In order to load our inflammation data,
we need to access ([import](reference.html#import) in Python terminology)
a library called [NumPy](http://docs.scipy.org/doc/numpy/ "NumPy Documentation").
In general you should use this library if you want to do fancy things with numbers,
especially if you have matrices or arrays.
We can import NumPy using:

~~~
import numpy
~~~
{: .python}

Importing a library is like getting a piece of lab equipment out of a storage locker and setting it up on the bench.
Libraries provide additional functionality to the basic Python package,
171
172
much like a new piece of equipment adds functionality to a lab space. Just like in the lab, importing too many libraries
can sometimes complicate and slow down your programs - so we only import what we need for each program. 
173
Once we've imported the library,
devendra1810's avatar
devendra1810 committed
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
we can ask the library to read our data file for us:

~~~
numpy.loadtxt(fname='inflammation-01.csv', delimiter=',')
~~~
{: .python}

~~~
array([[ 0.,  0.,  1., ...,  3.,  0.,  0.],
       [ 0.,  1.,  2., ...,  1.,  0.,  1.],
       [ 0.,  1.,  1., ...,  2.,  1.,  1.],
       ...,
       [ 0.,  1.,  1., ...,  1.,  1.,  1.],
       [ 0.,  0.,  0., ...,  0.,  2.,  0.],
       [ 0.,  0.,  1., ...,  1.,  1.,  0.]])
~~~
{: .output}

The expression `numpy.loadtxt(...)` is a [function call](reference.html#function-call)
that asks Python to run the [function](reference.html#function) `loadtxt` which belongs to the `numpy` library.
This [dotted notation](reference.html#dotted-notation) is used everywhere in Python
to refer to the parts of things as `thing.component`.

`numpy.loadtxt` has two [parameters](reference.html#parameter):
the name of the file we want to read,
and the [delimiter](reference.html#delimiter) that separates values on a line.
These both need to be character strings (or [strings](reference.html#string) for short),
so we put them in quotes.

Since we haven't told it to do anything else with the function's output,
the notebook displays it.
In this case,
that output is the data we just loaded.
By default,
only a few rows and columns are shown
(with `...` to omit elements when displaying big arrays).
To save space,
Python displays numbers as `1.` instead of `1.0`
when there's nothing interesting after the decimal point.

Our call to `numpy.loadtxt` read our file,
but didn't save the data in memory.
To do that,
217
we need to assign the array to a variable. Just as we can assign a single value to a variable, we can also assign an array of values
Johnny Lin's avatar
Johnny Lin committed
218
to a variable using the same syntax.  Let's re-run `numpy.loadtxt` and save its result:
219

220
~~~
Greg Wilson's avatar
Greg Wilson committed
221
222
data = numpy.loadtxt(fname='inflammation-01.csv', delimiter=',')
~~~
223
{: .python}
224

225
226
227
This statement doesn't produce any output because assignment doesn't display anything.
If we want to check that our data has been loaded,
we can print the variable's value:
228

229
~~~
230
print(data)
Greg Wilson's avatar
Greg Wilson committed
231
~~~
232
233
234
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
235
[[ 0.  0.  1. ...,  3.  0.  0.]
236
237
 [ 0.  1.  2. ...,  1.  0.  1.]
 [ 0.  1.  1. ...,  2.  1.  1.]
238
 ...,
239
240
241
 [ 0.  1.  1. ...,  1.  1.  1.]
 [ 0.  0.  0. ...,  0.  2.  0.]
 [ 0.  0.  1. ...,  1.  1.  0.]]
Greg Wilson's avatar
Greg Wilson committed
242
~~~
243
{: .output}
244

245
246
247
Now that our data is in memory,
we can start doing things with it.
First,
248
let's ask what [type]({{ page.root }}/reference/#type) of thing `data` refers to:
249

250
~~~
251
print(type(data))
Greg Wilson's avatar
Greg Wilson committed
252
~~~
253
254
255
{: .python}

~~~
256
<class 'numpy.ndarray'>
Greg Wilson's avatar
Greg Wilson committed
257
~~~
258
{: .output}
259

260
261
262
263
264
265
The output tells us that `data` currently refers to
an N-dimensional array created by the NumPy library.
These data correspond to arthritis patients' inflammation.
The rows are the individual patients and the columns
are their daily inflammation measurements.

266
> ## Data Type
267
268
269
270
>
> A Numpy array contains one or more elements
> of the same type. `type` will only tell you that
> a variable is a NumPy array.
Valentina Staneva's avatar
Valentina Staneva committed
271
> We can also find out the type
272
273
> of the data contained in the NumPy array.
>
274
> ~~~
275
276
> print(data.dtype)
> ~~~
277
278
279
> {: .python}
>
> ~~~
280
281
> dtype('float64')
> ~~~
282
> {: .output}
283
284
>
> This tells us that the NumPy array's elements are
285
> [floating-point numbers]({{ page.root }}/reference/#floating-point number).
286
{: .callout}
287

Bree Norlander's avatar
Bree Norlander committed
288
With this command we can see the array's [shape]({{ page.root }}/reference/#shape):
289

290
~~~
291
print(data.shape)
Greg Wilson's avatar
Greg Wilson committed
292
~~~
293
294
295
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
296
297
(60, 40)
~~~
298
{: .output}
299

300
301
This tells us that `data` has 60 rows and 40 columns. When we created the
variable `data` to store our arthritis data, we didn't just create the array, we also
302
created information about the array, called [members]({{ page.root }}/reference/#member) or
303
304
attributes. This extra information describes `data` in
the same way an adjective describes a noun.
Bartosz T's avatar
Bartosz T committed
305
`data.shape` is an attribute  of `data` which describes the dimensions of `data`.
306
We use the same dotted notation for the attributes of variables
307
308
that we use for the functions in libraries
because they have the same part-and-whole relationship.
309

310
If we want to get a single number from the array,
311
we must provide an [index]({{ page.root }}/reference/#index) in square brackets,
Dustin Lang's avatar
Dustin Lang committed
312
just as we do in math when referring to an element of a matrix.  Our inflammation data has two dimensions, so we will need to use two indices to refer to a value:
313

314
~~~
315
print('first value in data:', data[0, 0])
Greg Wilson's avatar
Greg Wilson committed
316
~~~
317
318
319
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
320
321
first value in data: 0.0
~~~
322
{: .output}
323

324
~~~
325
print('middle value in data:', data[30, 20])
Greg Wilson's avatar
Greg Wilson committed
326
~~~
327
328
329
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
330
331
middle value in data: 13.0
~~~
332
{: .output}
333

334
335
The expression `data[30, 20]` may not surprise you,
but `data[0, 0]` might.
336
Programming languages like Fortran, MATLAB and R start counting at 1,
337
338
because that's what human beings have done for thousands of years.
Languages in the C family (including C++, Java, Perl, and Python) count from 0
339
340
341
342
343
because it represents an offset from the first value in the array (the second
value is offset by one index from the first value). This is closer to the way
that computers represent arrays (if you are interested in the historical
reasons behind counting indices from zero, you can read
[Mike Hoye's blog post](http://exple.tive.org/blarg/2013/10/22/citation-needed/)).
344
As a result,
Greg Wilson's avatar
Greg Wilson committed
345
if we have an M×N array in Python,
346
347
348
349
350
351
its indices go from 0 to M-1 on the first axis
and 0 to N-1 on the second.
It takes a bit of getting used to,
but one way to remember the rule is that
the index is how many steps we have to take from the start to get the item we want.

352
353
![Zero Index](../fig/python-zero-index.png)

354
> ## In the Corner
355
356
357
358
359
360
>
> What may also surprise you is that when Python displays an array,
> it shows the element with index `[0, 0]` in the upper left corner
> rather than the lower left.
> This is consistent with the way mathematicians draw matrices,
> but different from the Cartesian coordinates.
361
> The indices are (row, column) instead of (column, row) for the same reason,
362
> which can be confusing when plotting data.
363
{: .callout}
364
365
366
367
368

An index like `[30, 20]` selects a single element of an array,
but we can select whole sections as well.
For example,
we can select the first ten days (columns) of values
369
for the first four patients (rows) like this:
370

371
~~~
372
print(data[0:4, 0:10])
Greg Wilson's avatar
Greg Wilson committed
373
~~~
374
375
376
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
377
[[ 0.  0.  1.  3.  1.  2.  4.  7.  8.  3.]
378
379
380
 [ 0.  1.  2.  1.  2.  1.  3.  2.  2.  6.]
 [ 0.  1.  1.  3.  3.  2.  6.  2.  5.  9.]
 [ 0.  0.  2.  0.  4.  2.  2.  1.  6.  7.]]
Greg Wilson's avatar
Greg Wilson committed
381
~~~
382
{: .output}
383

384
The [slice]({{ page.root }}/reference/#slice) `0:4` means,
385
386
387
388
389
390
"Start at index 0 and go up to, but not including, index 4."
Again,
the up-to-but-not-including takes a bit of getting used to,
but the rule is that the difference between the upper and lower bounds is the number of values in the slice.

We don't have to start slices at 0:
391

392
~~~
393
print(data[5:10, 0:10])
Greg Wilson's avatar
Greg Wilson committed
394
~~~
395
396
397
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
398
[[ 0.  0.  1.  2.  2.  4.  2.  1.  6.  4.]
399
400
401
402
 [ 0.  0.  2.  2.  4.  2.  2.  5.  5.  8.]
 [ 0.  0.  1.  2.  3.  1.  2.  3.  5.  3.]
 [ 0.  0.  0.  3.  1.  5.  6.  5.  5.  8.]
 [ 0.  1.  1.  2.  1.  3.  5.  3.  5.  8.]]
Greg Wilson's avatar
Greg Wilson committed
403
~~~
404
{: .output}
405

406
407
408
409
410
411
412
413
We also don't have to include the upper and lower bound on the slice.
If we don't include the lower bound,
Python uses 0 by default;
if we don't include the upper,
the slice runs to the end of the axis,
and if we don't include either
(i.e., if we just use ':' on its own),
the slice includes everything:
414

415
~~~
Greg Wilson's avatar
Greg Wilson committed
416
small = data[:3, 36:]
417
418
print('small is:')
print(small)
Greg Wilson's avatar
Greg Wilson committed
419
~~~
420
421
422
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
423
small is:
424
425
426
[[ 2.  3.  0.  0.]
 [ 1.  1.  0.  1.]
 [ 2.  2.  1.  1.]]
Greg Wilson's avatar
Greg Wilson committed
427
~~~
428
{: .output}
429

430
Arrays also know how to perform common mathematical operations on their values.
Greg Wilson's avatar
Greg Wilson committed
431
432
433
434
435
The simplest operations with data are arithmetic:
add, subtract, multiply, and divide.
 When you do such operations on arrays,
the operation is done on each individual element of the array.
Thus:
436

437
~~~
Greg Wilson's avatar
Greg Wilson committed
438
439
doubledata = data * 2.0
~~~
440
{: .python}
441

Greg Wilson's avatar
Greg Wilson committed
442
443
will create a new array `doubledata`
whose elements have the value of two times the value of the corresponding elements in `data`:
444

445
~~~
446
447
448
449
print('original:')
print(data[:3, 36:])
print('doubledata:')
print(doubledata[:3, 36:])
Greg Wilson's avatar
Greg Wilson committed
450
~~~
451
452
453
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
454
original:
455
456
457
458
459
460
461
[[ 2.  3.  0.  0.]
 [ 1.  1.  0.  1.]
 [ 2.  2.  1.  1.]]
doubledata:
[[ 4.  6.  0.  0.]
 [ 2.  2.  0.  2.]
 [ 4.  4.  2.  2.]]
Greg Wilson's avatar
Greg Wilson committed
462
~~~
463
{: .output}
464

Greg Wilson's avatar
Greg Wilson committed
465
466
If,
instead of taking an array and doing arithmetic with a single value (as above)
467
you did the arithmetic operation with another array of the same shape,
Greg Wilson's avatar
Greg Wilson committed
468
469
the operation will be done on corresponding elements of the two arrays.
Thus:
470

471
~~~
Greg Wilson's avatar
Greg Wilson committed
472
473
tripledata = doubledata + data
~~~
474
{: .python}
475

476
477
478
will give you an array where `tripledata[0,0]` will equal `doubledata[0,0]` plus `data[0,0]`,
and so on for all other elements of the arrays.

479
~~~
480
481
print('tripledata:')
print(tripledata[:3, 36:])
Greg Wilson's avatar
Greg Wilson committed
482
~~~
483
484
485
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
486
tripledata:
487
488
489
[[ 6.  9.  0.  0.]
 [ 3.  3.  0.  3.]
 [ 6.  6.  3.  3.]]
Greg Wilson's avatar
Greg Wilson committed
490
~~~
491
{: .output}
492

493
Often, we want to do more than add, subtract, multiply, and divide values of data.
494
NumPy knows how to do more complex operations on arrays.
495
496
If we want to find the average inflammation for all patients on all days,
for example,
497
we can ask NumPy to compute `data`'s mean value:
498

499
~~~
500
print(numpy.mean(data))
Greg Wilson's avatar
Greg Wilson committed
501
~~~
502
503
504
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
505
506
6.14875
~~~
507
{: .output}
508

509
510
`mean` is a [function]({{ page.root }}/reference/#function) that takes
an array as an [argument]({{ page.root }}/reference/#argument).
511
512
513
If variables are nouns, functions are verbs:
they do things with variables.

514
> ## Not All Functions Have Input
515
516
517
>
> Generally, a function uses inputs to produce outputs.
> However, some functions produce outputs without
518
519
> needing any input. For example, checking the current time
> doesn't require any input.
520
>
521
> ~~~
522
523
> import time
> print(time.ctime())
524
> ~~~
525
526
527
> {: .python}
>
> ~~~
528
> 'Sat Mar 26 13:07:33 2016'
529
> ~~~
530
> {: .output}
531
532
533
534
>
> For functions that don't take in any arguments,
> we still need parentheses (`()`)
> to tell Python to go and do something for us.
535
{: .callout}
536
537
538

NumPy has lots of useful functions that take an array as input.
Let's use three of those functions to get some descriptive values about the dataset.
539
540
We'll also use multiple assignment,
a convenient Python feature that will enable us to do this all in one line.
541

542
~~~
543
maxval, minval, stdval = numpy.max(data), numpy.min(data), numpy.std(data)
544

Alistair Walsh's avatar
Alistair Walsh committed
545
546
547
print('maximum inflammation:', maxval)
print('minimum inflammation:', minval)
print('standard deviation:', stdval)
Greg Wilson's avatar
Greg Wilson committed
548
~~~
549
550
551
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
552
maximum inflammation: 20.0
553
554
minimum inflammation: 0.0
standard deviation: 4.61383319712
Greg Wilson's avatar
Greg Wilson committed
555
~~~
556
{: .output}
557

558
> ## Mystery Functions in IPython
559
>
560
561
> How did we know what functions NumPy has and how to use them?
> If you are working in the IPython/Jupyter Notebook there is an easy way to find out.
Dustin Lang's avatar
Dustin Lang committed
562
> If you type the name of something followed by a dot, then you can use tab completion
563
564
565
566
> (e.g. type `numpy.` and then press tab)
> to see a list of all functions and attributes that you can use. After selecting one you
> can also add a question mark (e.g. `numpy.cumprod?`) and IPython will return an
> explanation of the method! This is the same as doing `help(numpy.cumprod)`.
567
{: .callout}
568
569

When analyzing data, though,
570
571
572
we often want to look at partial statistics,
such as the maximum value per patient
or the average value per day.
573
One way to do this is to create a new temporary array of the data we want,
574
then ask it to do the calculation:
575

576
~~~
Dustin Lang's avatar
Dustin Lang committed
577
patient_0 = data[0, :] # 0 on the first axis (rows), everything on the second (columns)
578
print('maximum inflammation for patient 0:', patient_0.max())
Greg Wilson's avatar
Greg Wilson committed
579
~~~
580
581
582
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
583
584
maximum inflammation for patient 0: 18.0
~~~
585
{: .output}
586

587
Everything in a line of code following the '#' symbol is a
588
[comment]({{ page.root }}/reference/#comment) that is ignored by Python.
589
Comments allow programmers to leave explanatory notes for other
jstapleton's avatar
jstapleton committed
590
591
programmers or their future selves.

592
We don't actually need to store the row in a variable of its own.
593
Instead, we can combine the selection and the function call:
594

595
~~~
596
print('maximum inflammation for patient 2:', numpy.max(data[2, :]))
Greg Wilson's avatar
Greg Wilson committed
597
~~~
598
599
600
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
601
602
maximum inflammation for patient 2: 19.0
~~~
603
{: .output}
604

605
What if we need the maximum inflammation for each patient over all days (as in the
606
607
608
next diagram on the left), or the average for each day (as in the
diagram on the right)? As the diagram below shows, we want to perform the
operation across an axis:
609

610
![Operations Across Axes](../fig/python-operations-across-axes.png)
611

612
To support this,
613
most array functions allow us to specify the axis we want to work on.
614
If we ask for the average across axis 0 (rows in our 2D example),
615
we get:
616

617
~~~
618
print(numpy.mean(data, axis=0))
Greg Wilson's avatar
Greg Wilson committed
619
~~~
620
621
622
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
623
[  0.           0.45         1.11666667   1.75         2.43333333   3.15
624
625
626
627
628
629
630
   3.8          3.88333333   5.23333333   5.51666667   5.95         5.9
   8.35         7.73333333   8.36666667   9.5          9.58333333
  10.63333333  11.56666667  12.35        13.25        11.96666667
  11.03333333  10.16666667  10.           8.66666667   9.15         7.25
   7.33333333   6.58333333   6.06666667   5.95         5.11666667   3.6
   3.3          3.56666667   2.48333333   1.5          1.13333333
   0.56666667]
Greg Wilson's avatar
Greg Wilson committed
631
~~~
632
{: .output}
633

634
635
As a quick check,
we can ask this array what its shape is:
636

637
~~~
638
print(numpy.mean(data, axis=0).shape)
Greg Wilson's avatar
Greg Wilson committed
639
~~~
640
641
642
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
643
644
(40,)
~~~
645
{: .output}
646

Greg Wilson's avatar
Greg Wilson committed
647
The expression `(40,)` tells us we have an N×1 vector,
648
so this is the average inflammation per day for all patients.
649
If we average across axis 1 (columns in our 2D example), we get:
650

651
~~~
652
print(numpy.mean(data, axis=1))
Greg Wilson's avatar
Greg Wilson committed
653
~~~
654
655
656
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
657
[ 5.45   5.425  6.1    5.9    5.55   6.225  5.975  6.65   6.625  6.525
658
659
660
661
662
  6.775  5.8    6.225  5.75   5.225  6.3    6.55   5.7    5.85   6.55
  5.775  5.825  6.175  6.1    5.8    6.425  6.05   6.025  6.175  6.55
  6.175  6.35   6.725  6.125  7.075  5.725  5.925  6.15   6.075  5.75
  5.975  5.725  6.3    5.9    6.75   5.925  7.225  6.15   5.95   6.275  5.7
  6.1    6.825  5.975  6.725  5.7    6.25   6.4    7.05   5.9  ]
Greg Wilson's avatar
Greg Wilson committed
663
~~~
664
{: .output}
665

666
667
668
669
670
which is the average inflammation per patient across all days.

The mathematician Richard Hamming once said,
"The purpose of computing is insight, not numbers,"
and the best way to develop insight is often to visualize data.
Valentina Staneva's avatar
Valentina Staneva committed
671
Visualization deserves an entire lecture (of course) of its own,
672
but we can explore a few features of Python's `matplotlib` library here.
Greg Wilson's avatar
Greg Wilson committed
673
674
While there is no "official" plotting library,
this package is the de facto standard.
675
676
677
First,
we will import the `pyplot` module from `matplotlib`
and use two of its functions to create and display a heat map of our data:
678

679
~~~
680
import matplotlib.pyplot
681
image = matplotlib.pyplot.imshow(data)
682
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
683
~~~
684
{: .python}
685

686
![Heatmap of the Data](../fig/01-numpy_71_0.png)
687

688
689
690
Blue regions in this heat map are low values, while red shows high values.
As we can see,
inflammation rises and falls over a 40-day period.
691

692
> ## Some IPython Magic
693
694
695
696
>
> If you're using an IPython / Jupyter notebook,
> you'll need to execute the following command
> in order for your matplotlib images to appear
Damien Irving's avatar
Damien Irving committed
697
> in the notebook when `show()` is called:
Damien Irving's avatar
Damien Irving committed
698
>
699
> ~~~
700
> %matplotlib inline
Damien Irving's avatar
Damien Irving committed
701
> ~~~
702
> {: .python}
703
>
704
705
> The `%` indicates an IPython magic function -
> a function that is only valid within the notebook environment.
706
> Note that you only have to execute this function once per notebook.
707
{: .callout}
708

709
Let's take a look at the average inflammation over time:
710

711
~~~
712
ave_inflammation = numpy.mean(data, axis=0)
713
ave_plot = matplotlib.pyplot.plot(ave_inflammation)
714
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
715
~~~
716
{: .python}
717

718
![Average Inflammation Over Time](../fig/01-numpy_73_0.png)
719

720
721
Here,
we have put the average per day across all patients in the variable `ave_inflammation`,
722
then asked `matplotlib.pyplot` to create and display a line graph of those values.
723
724
725
726
727
The result is roughly a linear rise and fall,
which is suspicious:
based on other studies,
we expect a sharper rise and slower fall.
Let's have a look at two other statistics:
728

729
~~~
730
max_plot = matplotlib.pyplot.plot(numpy.max(data, axis=0))
731
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
732
~~~
733
{: .python}
734

735
![Maximum Value Along The First Axis](../fig/01-numpy_75_1.png)
Greg Wilson's avatar
Greg Wilson committed
736

737
~~~
738
min_plot = matplotlib.pyplot.plot(numpy.min(data, axis=0))
739
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
740
~~~
741
{: .python}
742

743
![Minimum Value Along The First Axis](../fig/01-numpy_75_3.png)
744

745
746
747
748
749
The maximum value rises and falls perfectly smoothly,
while the minimum seems to be a step function.
Neither result seems particularly likely,
so either there's a mistake in our calculations
or something is wrong with our data.
750
This insight would have been difficult to reach by
jstapleton's avatar
jstapleton committed
751
examining the data without visualization tools.
752

753
You can group similar plots in a single figure using subplots.
754
This script below uses a number of new commands. The function `matplotlib.pyplot.figure()`
755
creates a space into which we will place all of our plots. The parameter `figsize`
756
tells Python how big to make this space. Each subplot is placed into the figure using
757
its `add_subplot` [method]({{ page.root }}/reference/#method). The `add_subplot` method takes 3 parameters. The first denotes
758
how many total rows of subplots there are, the second parameter refers to the
valiseverywhere's avatar
valiseverywhere committed
759
total number of subplot columns, and the final parameter denotes which subplot
760
761
762
your variable is referencing (left-to-right, top-to-bottom). Each subplot is stored in a
different variable (`axes1`, `axes2`, `axes3`). Once a subplot is created, the axes can
be titled using the `set_xlabel()` command (or `set_ylabel()`).
763
Here are our three plots side by side:
764

765
~~~
766
767
import numpy
import matplotlib.pyplot
768

769
data = numpy.loadtxt(fname='inflammation-01.csv', delimiter=',')
770

771
fig = matplotlib.pyplot.figure(figsize=(10.0, 3.0))
772

773
774
775
axes1 = fig.add_subplot(1, 3, 1)
axes2 = fig.add_subplot(1, 3, 2)
axes3 = fig.add_subplot(1, 3, 3)
776

777
axes1.set_ylabel('average')
778
axes1.plot(numpy.mean(data, axis=0))
779

780
axes2.set_ylabel('max')
781
axes2.plot(numpy.max(data, axis=0))
782

783
axes3.set_ylabel('min')
784
axes3.plot(numpy.min(data, axis=0))
785
786

fig.tight_layout()
787

788
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
789
~~~
790
{: .python}
791

792
![The Previous Plots as Subplots](../fig/01-numpy_80_0.png)
793

794
The [call]({{ page.root }}/reference/#function-call) to `loadtxt` reads our data,
795
796
and the rest of the program tells the plotting library
how large we want the figure to be,
797
that we're creating three subplots,
798
799
800
what to draw for each one,
and that we want a tight layout.
(Perversely,
801
if we leave out that call to `fig.tight_layout()`,
802
the graphs will actually be squeezed together more closely.)
803

804
> ## Scientists Dislike Typing
805
806
807
808
809
810
811
>
> We will always use the syntax `import numpy` to import NumPy.
> However, in order to save typing, it is
> [often suggested](http://www.scipy.org/getting-started.html#an-example-script)
> to make a shortcut like so: `import numpy as np`.
> If you ever see Python code online using a NumPy function with `np`
> (for example, `np.loadtxt(...)`), it's because they've used this shortcut.
812
> When working with other people, it is important to agree on a convention of how common libraries are imported.
valiseverywhere's avatar
valiseverywhere committed
813
{: .callout}
814
> ## Check Your Understanding
mboisson's avatar
mboisson committed
815
816
817
>
> Draw diagrams showing what variables refer to what values after each statement in the following program:
>
818
> ~~~
mboisson's avatar
mboisson committed
819
820
821
822
823
> mass = 47.5
> age = 122
> mass = mass * 2.0
> age = age - 20
> ~~~
824
825
> {: .python}
{: .challenge}
mboisson's avatar
mboisson committed
826

827
> ## Sorting Out References
mboisson's avatar
mboisson committed
828
829
830
>
> What does the following program print out?
>
831
> ~~~
mboisson's avatar
mboisson committed
832
833
> first, second = 'Grace', 'Hopper'
> third, fourth = second, first
834
> print(third, fourth)
mboisson's avatar
mboisson committed
835
> ~~~