01-numpy.md 30.8 KB
Newer Older
1
---
2
3
4
5
title: Analyzing Patient Data
teaching: 30
exercises: 0
questions:
Greg Wilson's avatar
Greg Wilson committed
6
- "How can I process tabular data files in Python?"
7
8
objectives:
- "Explain what a library is, and what libraries are used for."
9
- "Import a Python library and use the functions it contains."
10
11
12
13
- "Read tabular data from a file into a program."
- "Assign values to variables."
- "Select individual values and subsections from data."
- "Perform operations on arrays of data."
14
- "Plot simple graphs from data."
15
keypoints:
Greg Wilson's avatar
Greg Wilson committed
16
17
18
19
20
21
22
23
24
25
26
27
28
29
- "Import a library into a program using `import libraryname`."
- "Use the `numpy` library to work with arrays in Python."
- "Use `variable = value` to assign a value to a variable in order to record it in memory."
- "Variables are created on demand whenever a value is assigned to them."
- "Use `print(something)` to display the value of `something`."
- "The expression `array.shape` gives the shape of an array."
- "Use `array[x, y]` to select a single element from an array."
- "Array indices start at 0, not 1."
- "Use `low:high` to specify a slice that includes the indices from `low` to `high-1`."
- "All the indexing and slicing that works on arrays also works on strings."
- "Use `# some kind of explanation` to add comments to programs."
- "Use `numpy.mean(array)`, `numpy.max(array)`, and `numpy.min(array)` to calculate simple statistics."
- "Use `numpy.mean(array, axis=0)` or `numpy.mean(array, axis=1)` to calculate statistics across the specified axis."
- "Use the `pyplot` library from `matplotlib` for creating simple visualizations."
30
31
---

32
Words are useful,
33
but what's more useful are the sentences and stories we build with them.
34
Similarly,
jstapleton's avatar
jstapleton committed
35
while a lot of powerful, general tools are built into languages like Python,
36
specialized tools built up from these basic units live in [libraries]({{ page.root }}/reference/#library)
jstapleton's avatar
jstapleton committed
37
that can be called upon when needed.
38
39

In order to load our inflammation data,
40
we need to access ([import]({{ page.root }}/reference/#import) in Python terminology)
Trevor Bekolay's avatar
Trevor Bekolay committed
41
a library called [NumPy](http://docs.scipy.org/doc/numpy/ "NumPy Documentation").
Greg Wilson's avatar
Greg Wilson committed
42
In general you should use this library if you want to do fancy things with numbers,
43
especially if you have matrices or arrays.
Trevor Bekolay's avatar
Trevor Bekolay committed
44
We can import NumPy using:
45

46
~~~
Greg Wilson's avatar
Greg Wilson committed
47
48
import numpy
~~~
49
{: .python}
50

51
52
53
Importing a library is like getting a piece of lab equipment out of a storage locker and setting it up on the bench.
Libraries provide additional functionality to the basic Python package,
much like a new piece of equipment adds functionality to a lab space.
Dustin Lang's avatar
Dustin Lang committed
54
Once we've imported the library,
55
we can ask the library to read our data file for us:
56

57
~~~
Peter Cock's avatar
Peter Cock committed
58
numpy.loadtxt(fname='inflammation-01.csv', delimiter=',')
Greg Wilson's avatar
Greg Wilson committed
59
~~~
60
61
62
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
63
array([[ 0.,  0.,  1., ...,  3.,  0.,  0.],
64
65
       [ 0.,  1.,  2., ...,  1.,  0.,  1.],
       [ 0.,  1.,  1., ...,  2.,  1.,  1.],
66
       ...,
67
68
       [ 0.,  1.,  1., ...,  1.,  1.,  1.],
       [ 0.,  0.,  0., ...,  0.,  2.,  0.],
Greg Wilson's avatar
Greg Wilson committed
69
70
       [ 0.,  0.,  1., ...,  1.,  1.,  0.]])
~~~
71
{: .output}
72

73
74
75
The expression `numpy.loadtxt(...)` is a [function call]({{ page.root }}/reference/#function-call)
that asks Python to run the [function]({{ page.root }}/reference/#function) `loadtxt` which belongs to the `numpy` library.
This [dotted notation]({{ page.root }}/reference/#dotted-notation) is used everywhere in Python
76
to refer to the parts of things as `thing.component`.
77

78
`numpy.loadtxt` has two [parameters]({{ page.root }}/reference/#parameter):
79
the name of the file we want to read,
80
81
and the [delimiter]({{ page.root }}/reference/#delimiter) that separates values on a line.
These both need to be character strings (or [strings]({{ page.root }}/reference/#string) for short),
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
so we put them in quotes.

When we are finished typing and press Shift+Enter,
the notebook runs our command.
Since we haven't told it to do anything else with the function's output,
the notebook displays it.
In this case,
that output is the data we just loaded.
By default,
only a few rows and columns are shown
(with `...` to omit elements when displaying big arrays).
To save space,
Python displays numbers as `1.` instead of `1.0`
when there's nothing interesting after the decimal point.

Our call to `numpy.loadtxt` read our file,
but didn't save the data in memory.
To do that,
100
we need to [assign]({{ page.root }}/reference/#assignment) the array to a [variable]({{ page.root }}/reference/#variable).
101
A variable is just a name for a value,
Greg Wilson's avatar
Greg Wilson committed
102
such as `x`, `current_temperature`, or `subject_id`.
103
Python's variables must begin with a letter and are [case sensitive]({{ page.root }}/reference/#case-sensitive).
Kyler Brown's avatar
Kyler Brown committed
104
We can create a new variable by assigning a value to it using `=`.
Greg Wilson's avatar
Greg Wilson committed
105
106
107
108
As an illustration,
let's step back and instead of considering a table of data,
consider the simplest "collection" of data,
a single value.
109
The line below assigns the value `55` to a variable `weight_kg`:
Greg Wilson's avatar
Greg Wilson committed
110

111
~~~
Greg Wilson's avatar
Greg Wilson committed
112
113
weight_kg = 55
~~~
114
{: .python}
115

116
Once a variable has a value, we can print it to the screen:
117

118
~~~
119
print(weight_kg)
Greg Wilson's avatar
Greg Wilson committed
120
~~~
121
122
123
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
124
125
55
~~~
126
{: .output}
127

128
and do arithmetic with it:
129

130
~~~
131
print('weight in pounds:', 2.2 * weight_kg)
Greg Wilson's avatar
Greg Wilson committed
132
~~~
133
134
135
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
136
137
weight in pounds: 121.0
~~~
138
{: .output}
139

jstapleton's avatar
jstapleton committed
140
141
142
As the example above shows,
we can print several things at once by separating them with commas.

143
We can also change a variable's value by assigning it a new one:
144

145
~~~
Greg Wilson's avatar
Greg Wilson committed
146
weight_kg = 57.5
147
print('weight in kilograms is now:', weight_kg)
Greg Wilson's avatar
Greg Wilson committed
148
~~~
149
150
151
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
152
153
weight in kilograms is now: 57.5
~~~
154
{: .output}
155

156
157
If we imagine the variable as a sticky note with a name written on it,
assignment is like putting the sticky note on a particular value:
158

159
![Variables as Sticky Notes](../fig/python-sticky-note-variables-01.svg)
160

161
162
163
This means that assigning a value to one variable does *not* change the values of other variables.
For example,
let's store the subject's weight in pounds in a variable:
164

165
~~~
Greg Wilson's avatar
Greg Wilson committed
166
weight_lb = 2.2 * weight_kg
167
print('weight in kilograms:', weight_kg, 'and in pounds:', weight_lb)
Greg Wilson's avatar
Greg Wilson committed
168
~~~
169
170
171
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
172
173
weight in kilograms: 57.5 and in pounds: 126.5
~~~
174
{: .output}
175

176
![Creating Another Variable](../fig/python-sticky-note-variables-02.svg)
177

178
and then change `weight_kg`:
179

180
~~~
Greg Wilson's avatar
Greg Wilson committed
181
weight_kg = 100.0
182
print('weight in kilograms is now:', weight_kg, 'and weight in pounds is still:', weight_lb)
Greg Wilson's avatar
Greg Wilson committed
183
~~~
184
185
186
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
187
188
weight in kilograms is now: 100.0 and weight in pounds is still: 126.5
~~~
189
{: .output}
190

191
![Updating a Variable](../fig/python-sticky-note-variables-03.svg)
192

193
194
195
196
Since `weight_lb` doesn't "remember" where its value came from,
it isn't automatically updated when `weight_kg` changes.
This is different from the way spreadsheets work.

197
> ## Who's Who in Memory
Benjamin Laken's avatar
Benjamin Laken committed
198
>
199
200
201
> You can use the `%whos` command at any time to see what
> variables you have created and what modules you have loaded into the computer's memory.
> As this is an IPython command, it will only work if you are in an IPython terminal or the Jupyter Notebook.
Benjamin Laken's avatar
Benjamin Laken committed
202
>
203
> ~~~
204
205
> %whos
> ~~~
206
207
208
> {: .python}
>
> ~~~
209
210
211
212
213
214
> Variable    Type       Data/Info
> --------------------------------
> numpy       module     <module 'numpy' from '/Us<...>kages/numpy/__init__.py'>
> weight_kg   float      100.0
> weight_lb   float      126.5
> ~~~
215
216
> {: .output}
{: .callout}
Benjamin Laken's avatar
Benjamin Laken committed
217

Johnny Lin's avatar
Johnny Lin committed
218
219
Just as we can assign a single value to a variable, we can also assign an array of values
to a variable using the same syntax.  Let's re-run `numpy.loadtxt` and save its result:
220

221
~~~
Greg Wilson's avatar
Greg Wilson committed
222
223
data = numpy.loadtxt(fname='inflammation-01.csv', delimiter=',')
~~~
224
{: .python}
225

226
227
228
This statement doesn't produce any output because assignment doesn't display anything.
If we want to check that our data has been loaded,
we can print the variable's value:
229

230
~~~
231
print(data)
Greg Wilson's avatar
Greg Wilson committed
232
~~~
233
234
235
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
236
[[ 0.  0.  1. ...,  3.  0.  0.]
237
238
 [ 0.  1.  2. ...,  1.  0.  1.]
 [ 0.  1.  1. ...,  2.  1.  1.]
239
 ...,
240
241
242
 [ 0.  1.  1. ...,  1.  1.  1.]
 [ 0.  0.  0. ...,  0.  2.  0.]
 [ 0.  0.  1. ...,  1.  1.  0.]]
Greg Wilson's avatar
Greg Wilson committed
243
~~~
244
{: .output}
245

246
247
248
Now that our data is in memory,
we can start doing things with it.
First,
249
let's ask what [type]({{ page.root }}/reference/#type) of thing `data` refers to:
250

251
~~~
252
print(type(data))
Greg Wilson's avatar
Greg Wilson committed
253
~~~
254
255
256
{: .python}

~~~
257
<class 'numpy.ndarray'>
Greg Wilson's avatar
Greg Wilson committed
258
~~~
259
{: .output}
260

261
262
263
264
265
266
The output tells us that `data` currently refers to
an N-dimensional array created by the NumPy library.
These data correspond to arthritis patients' inflammation.
The rows are the individual patients and the columns
are their daily inflammation measurements.

267
> ## Data Type
268
269
270
271
>
> A Numpy array contains one or more elements
> of the same type. `type` will only tell you that
> a variable is a NumPy array.
Valentina Staneva's avatar
Valentina Staneva committed
272
> We can also find out the type
273
274
> of the data contained in the NumPy array.
>
275
> ~~~
276
277
> print(data.dtype)
> ~~~
278
279
280
> {: .python}
>
> ~~~
281
282
> dtype('float64')
> ~~~
283
> {: .output}
284
285
>
> This tells us that the NumPy array's elements are
286
> [floating-point numbers]({{ page.root }}/reference/#floating-point number).
287
{: .callout}
288

289
We can see what the array's [shape]({{ page.root }}/reference/#shape) is like this:
290

291
~~~
292
print(data.shape)
Greg Wilson's avatar
Greg Wilson committed
293
~~~
294
295
296
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
297
298
(60, 40)
~~~
299
{: .output}
300

301
302
This tells us that `data` has 60 rows and 40 columns. When we created the
variable `data` to store our arthritis data, we didn't just create the array, we also
303
created information about the array, called [members]({{ page.root }}/reference/#member) or
304
305
attributes. This extra information describes `data` in
the same way an adjective describes a noun.
Bartosz T's avatar
Bartosz T committed
306
`data.shape` is an attribute  of `data` which describes the dimensions of `data`.
307
We use the same dotted notation for the attributes of variables
308
309
that we use for the functions in libraries
because they have the same part-and-whole relationship.
310

311
If we want to get a single number from the array,
312
we must provide an [index]({{ page.root }}/reference/#index) in square brackets,
Dustin Lang's avatar
Dustin Lang committed
313
just as we do in math when referring to an element of a matrix.  Our inflammation data has two dimensions, so we will need to use two indices to refer to a value:
314

315
~~~
316
print('first value in data:', data[0, 0])
Greg Wilson's avatar
Greg Wilson committed
317
~~~
318
319
320
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
321
322
first value in data: 0.0
~~~
323
{: .output}
324

325
~~~
326
print('middle value in data:', data[30, 20])
Greg Wilson's avatar
Greg Wilson committed
327
~~~
328
329
330
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
331
332
middle value in data: 13.0
~~~
333
{: .output}
334

335
336
337
338
339
The expression `data[30, 20]` may not surprise you,
but `data[0, 0]` might.
Programming languages like Fortran and MATLAB start counting at 1,
because that's what human beings have done for thousands of years.
Languages in the C family (including C++, Java, Perl, and Python) count from 0
340
341
342
343
344
because it represents an offset from the first value in the array (the second
value is offset by one index from the first value). This is closer to the way
that computers represent arrays (if you are interested in the historical
reasons behind counting indices from zero, you can read
[Mike Hoye's blog post](http://exple.tive.org/blarg/2013/10/22/citation-needed/)).
345
As a result,
Greg Wilson's avatar
Greg Wilson committed
346
if we have an M×N array in Python,
347
348
349
350
351
352
its indices go from 0 to M-1 on the first axis
and 0 to N-1 on the second.
It takes a bit of getting used to,
but one way to remember the rule is that
the index is how many steps we have to take from the start to get the item we want.

353
> ## In the Corner
354
355
356
357
358
359
>
> What may also surprise you is that when Python displays an array,
> it shows the element with index `[0, 0]` in the upper left corner
> rather than the lower left.
> This is consistent with the way mathematicians draw matrices,
> but different from the Cartesian coordinates.
360
> The indices are (row, column) instead of (column, row) for the same reason,
361
> which can be confusing when plotting data.
362
{: .callout}
363
364
365
366
367

An index like `[30, 20]` selects a single element of an array,
but we can select whole sections as well.
For example,
we can select the first ten days (columns) of values
368
for the first four patients (rows) like this:
369

370
~~~
371
print(data[0:4, 0:10])
Greg Wilson's avatar
Greg Wilson committed
372
~~~
373
374
375
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
376
[[ 0.  0.  1.  3.  1.  2.  4.  7.  8.  3.]
377
378
379
 [ 0.  1.  2.  1.  2.  1.  3.  2.  2.  6.]
 [ 0.  1.  1.  3.  3.  2.  6.  2.  5.  9.]
 [ 0.  0.  2.  0.  4.  2.  2.  1.  6.  7.]]
Greg Wilson's avatar
Greg Wilson committed
380
~~~
381
{: .output}
382

383
The [slice]({{ page.root }}/reference/#slice) `0:4` means,
384
385
386
387
388
389
"Start at index 0 and go up to, but not including, index 4."
Again,
the up-to-but-not-including takes a bit of getting used to,
but the rule is that the difference between the upper and lower bounds is the number of values in the slice.

We don't have to start slices at 0:
390

391
~~~
392
print(data[5:10, 0:10])
Greg Wilson's avatar
Greg Wilson committed
393
~~~
394
395
396
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
397
[[ 0.  0.  1.  2.  2.  4.  2.  1.  6.  4.]
398
399
400
401
 [ 0.  0.  2.  2.  4.  2.  2.  5.  5.  8.]
 [ 0.  0.  1.  2.  3.  1.  2.  3.  5.  3.]
 [ 0.  0.  0.  3.  1.  5.  6.  5.  5.  8.]
 [ 0.  1.  1.  2.  1.  3.  5.  3.  5.  8.]]
Greg Wilson's avatar
Greg Wilson committed
402
~~~
403
{: .output}
404

405
406
407
408
409
410
411
412
We also don't have to include the upper and lower bound on the slice.
If we don't include the lower bound,
Python uses 0 by default;
if we don't include the upper,
the slice runs to the end of the axis,
and if we don't include either
(i.e., if we just use ':' on its own),
the slice includes everything:
413

414
~~~
Greg Wilson's avatar
Greg Wilson committed
415
small = data[:3, 36:]
416
417
print('small is:')
print(small)
Greg Wilson's avatar
Greg Wilson committed
418
~~~
419
420
421
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
422
small is:
423
424
425
[[ 2.  3.  0.  0.]
 [ 1.  1.  0.  1.]
 [ 2.  2.  1.  1.]]
Greg Wilson's avatar
Greg Wilson committed
426
~~~
427
{: .output}
428

429
Arrays also know how to perform common mathematical operations on their values.
Greg Wilson's avatar
Greg Wilson committed
430
431
432
433
434
The simplest operations with data are arithmetic:
add, subtract, multiply, and divide.
 When you do such operations on arrays,
the operation is done on each individual element of the array.
Thus:
435

436
~~~
Greg Wilson's avatar
Greg Wilson committed
437
438
doubledata = data * 2.0
~~~
439
{: .python}
440

Greg Wilson's avatar
Greg Wilson committed
441
442
will create a new array `doubledata`
whose elements have the value of two times the value of the corresponding elements in `data`:
443

444
~~~
445
446
447
448
print('original:')
print(data[:3, 36:])
print('doubledata:')
print(doubledata[:3, 36:])
Greg Wilson's avatar
Greg Wilson committed
449
~~~
450
451
452
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
453
original:
454
455
456
457
458
459
460
[[ 2.  3.  0.  0.]
 [ 1.  1.  0.  1.]
 [ 2.  2.  1.  1.]]
doubledata:
[[ 4.  6.  0.  0.]
 [ 2.  2.  0.  2.]
 [ 4.  4.  2.  2.]]
Greg Wilson's avatar
Greg Wilson committed
461
~~~
462
{: .output}
463

Greg Wilson's avatar
Greg Wilson committed
464
465
If,
instead of taking an array and doing arithmetic with a single value (as above)
466
you did the arithmetic operation with another array of the same shape,
Greg Wilson's avatar
Greg Wilson committed
467
468
the operation will be done on corresponding elements of the two arrays.
Thus:
469

470
~~~
Greg Wilson's avatar
Greg Wilson committed
471
472
tripledata = doubledata + data
~~~
473
{: .python}
474

475
476
477
will give you an array where `tripledata[0,0]` will equal `doubledata[0,0]` plus `data[0,0]`,
and so on for all other elements of the arrays.

478
~~~
479
480
print('tripledata:')
print(tripledata[:3, 36:])
Greg Wilson's avatar
Greg Wilson committed
481
~~~
482
483
484
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
485
tripledata:
486
487
488
[[ 6.  9.  0.  0.]
 [ 3.  3.  0.  3.]
 [ 6.  6.  3.  3.]]
Greg Wilson's avatar
Greg Wilson committed
489
~~~
490
{: .output}
491

492
Often, we want to do more than add, subtract, multiply, and divide values of data.
493
NumPy knows how to do more complex operations on arrays.
494
495
If we want to find the average inflammation for all patients on all days,
for example,
496
we can ask NumPy to compute `data`'s mean value:
497

498
~~~
499
print(numpy.mean(data))
Greg Wilson's avatar
Greg Wilson committed
500
~~~
501
502
503
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
504
505
6.14875
~~~
506
{: .output}
507

508
509
`mean` is a [function]({{ page.root }}/reference/#function) that takes
an array as an [argument]({{ page.root }}/reference/#argument).
510
511
512
If variables are nouns, functions are verbs:
they do things with variables.

513
> ## Not All Functions Have Input
514
515
516
>
> Generally, a function uses inputs to produce outputs.
> However, some functions produce outputs without
517
518
> needing any input. For example, checking the current time
> doesn't require any input.
519
>
520
> ~~~
521
522
> import time
> print(time.ctime())
523
> ~~~
524
525
526
> {: .python}
>
> ~~~
527
> 'Sat Mar 26 13:07:33 2016'
528
> ~~~
529
> {: .output}
530
531
532
533
>
> For functions that don't take in any arguments,
> we still need parentheses (`()`)
> to tell Python to go and do something for us.
534
{: .callout}
535
536
537

NumPy has lots of useful functions that take an array as input.
Let's use three of those functions to get some descriptive values about the dataset.
538
539
We'll also use multiple assignment,
a convenient Python feature that will enable us to do this all in one line.
540

541
~~~
542
maxval, minval, stdval = numpy.max(data), numpy.min(data), numpy.std(data)
543

Alistair Walsh's avatar
Alistair Walsh committed
544
545
546
print('maximum inflammation:', maxval)
print('minimum inflammation:', minval)
print('standard deviation:', stdval)
Greg Wilson's avatar
Greg Wilson committed
547
~~~
548
549
550
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
551
maximum inflammation: 20.0
552
553
minimum inflammation: 0.0
standard deviation: 4.61383319712
Greg Wilson's avatar
Greg Wilson committed
554
~~~
555
{: .output}
556

557
> ## Mystery Functions in IPython
558
>
559
560
> How did we know what functions NumPy has and how to use them?
> If you are working in the IPython/Jupyter Notebook there is an easy way to find out.
Dustin Lang's avatar
Dustin Lang committed
561
> If you type the name of something followed by a dot, then you can use tab completion
562
563
564
565
> (e.g. type `numpy.` and then press tab)
> to see a list of all functions and attributes that you can use. After selecting one you
> can also add a question mark (e.g. `numpy.cumprod?`) and IPython will return an
> explanation of the method! This is the same as doing `help(numpy.cumprod)`.
566
{: .callout}
567
568

When analyzing data, though,
569
570
571
we often want to look at partial statistics,
such as the maximum value per patient
or the average value per day.
572
One way to do this is to create a new temporary array of the data we want,
573
then ask it to do the calculation:
574

575
~~~
Dustin Lang's avatar
Dustin Lang committed
576
patient_0 = data[0, :] # 0 on the first axis (row), everything on the second (column)
577
print('maximum inflammation for patient 0:', patient_0.max())
Greg Wilson's avatar
Greg Wilson committed
578
~~~
579
580
581
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
582
583
maximum inflammation for patient 0: 18.0
~~~
584
{: .output}
585

586
Everything in a line of code following the '#' symbol is a
587
[comment]({{ page.root }}/reference/#comment) that is ignored by the computer.
588
Comments allow programmers to leave explanatory notes for other
jstapleton's avatar
jstapleton committed
589
590
programmers or their future selves.

591
We don't actually need to store the row in a variable of its own.
592
Instead, we can combine the selection and the function call:
593

594
~~~
595
print('maximum inflammation for patient 2:', numpy.max(data[2, :]))
Greg Wilson's avatar
Greg Wilson committed
596
~~~
597
598
599
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
600
601
maximum inflammation for patient 2: 19.0
~~~
602
{: .output}
603

604
What if we need the maximum inflammation for each patient over all days (as in the
605
606
607
next diagram on the left), or the average for each day (as in the
diagram on the right)? As the diagram below shows, we want to perform the
operation across an axis:
608

609
![Operations Across Axes](../fig/python-operations-across-axes.png)
610

611
To support this,
612
most array functions allow us to specify the axis we want to work on.
613
If we ask for the average across axis 0 (rows in our 2D example),
614
we get:
615

616
~~~
617
print(numpy.mean(data, axis=0))
Greg Wilson's avatar
Greg Wilson committed
618
~~~
619
620
621
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
622
[  0.           0.45         1.11666667   1.75         2.43333333   3.15
623
624
625
626
627
628
629
   3.8          3.88333333   5.23333333   5.51666667   5.95         5.9
   8.35         7.73333333   8.36666667   9.5          9.58333333
  10.63333333  11.56666667  12.35        13.25        11.96666667
  11.03333333  10.16666667  10.           8.66666667   9.15         7.25
   7.33333333   6.58333333   6.06666667   5.95         5.11666667   3.6
   3.3          3.56666667   2.48333333   1.5          1.13333333
   0.56666667]
Greg Wilson's avatar
Greg Wilson committed
630
~~~
631
{: .output}
632

633
634
As a quick check,
we can ask this array what its shape is:
635

636
~~~
637
print(numpy.mean(data, axis=0).shape)
Greg Wilson's avatar
Greg Wilson committed
638
~~~
639
640
641
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
642
643
(40,)
~~~
644
{: .output}
645

Greg Wilson's avatar
Greg Wilson committed
646
The expression `(40,)` tells us we have an N×1 vector,
647
so this is the average inflammation per day for all patients.
648
If we average across axis 1 (columns in our 2D example), we get:
649

650
~~~
651
print(numpy.mean(data, axis=1))
Greg Wilson's avatar
Greg Wilson committed
652
~~~
653
654
655
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
656
[ 5.45   5.425  6.1    5.9    5.55   6.225  5.975  6.65   6.625  6.525
657
658
659
660
661
  6.775  5.8    6.225  5.75   5.225  6.3    6.55   5.7    5.85   6.55
  5.775  5.825  6.175  6.1    5.8    6.425  6.05   6.025  6.175  6.55
  6.175  6.35   6.725  6.125  7.075  5.725  5.925  6.15   6.075  5.75
  5.975  5.725  6.3    5.9    6.75   5.925  7.225  6.15   5.95   6.275  5.7
  6.1    6.825  5.975  6.725  5.7    6.25   6.4    7.05   5.9  ]
Greg Wilson's avatar
Greg Wilson committed
662
~~~
663
{: .output}
664

665
666
667
668
669
which is the average inflammation per patient across all days.

The mathematician Richard Hamming once said,
"The purpose of computing is insight, not numbers,"
and the best way to develop insight is often to visualize data.
Valentina Staneva's avatar
Valentina Staneva committed
670
Visualization deserves an entire lecture (of course) of its own,
671
but we can explore a few features of Python's `matplotlib` library here.
Greg Wilson's avatar
Greg Wilson committed
672
673
While there is no "official" plotting library,
this package is the de facto standard.
674
675
676
First,
we will import the `pyplot` module from `matplotlib`
and use two of its functions to create and display a heat map of our data:
677

678
~~~
679
import matplotlib.pyplot
680
image = matplotlib.pyplot.imshow(data)
681
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
682
~~~
683
{: .python}
684

685
![Heatmap of the Data](../fig/01-numpy_71_0.png)
686

687
688
689
Blue regions in this heat map are low values, while red shows high values.
As we can see,
inflammation rises and falls over a 40-day period.
690

691
> ## Some IPython Magic
692
693
694
695
>
> If you're using an IPython / Jupyter notebook,
> you'll need to execute the following command
> in order for your matplotlib images to appear
Damien Irving's avatar
Damien Irving committed
696
> in the notebook when `show()` is called:
Damien Irving's avatar
Damien Irving committed
697
>
698
> ~~~
Damien Irving's avatar
Damien Irving committed
699
700
> % matplotlib inline
> ~~~
701
> {: .python}
702
>
703
704
> The `%` indicates an IPython magic function -
> a function that is only valid within the notebook environment.
705
> Note that you only have to execute this function once per notebook.
706
{: .callout}
707

708
Let's take a look at the average inflammation over time:
709

710
~~~
711
ave_inflammation = numpy.mean(data, axis=0)
712
ave_plot = matplotlib.pyplot.plot(ave_inflammation)
713
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
714
~~~
715
{: .python}
716

717
![Average Inflammation Over Time](../fig/01-numpy_73_0.png)
718

719
720
Here,
we have put the average per day across all patients in the variable `ave_inflammation`,
721
then asked `matplotlib.pyplot` to create and display a line graph of those values.
722
723
724
725
726
The result is roughly a linear rise and fall,
which is suspicious:
based on other studies,
we expect a sharper rise and slower fall.
Let's have a look at two other statistics:
727

728
~~~
729
max_plot = matplotlib.pyplot.plot(numpy.max(data, axis=0))
730
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
731
~~~
732
{: .python}
733

734
![Maximum Value Along The First Axis](../fig/01-numpy_75_1.png)
Greg Wilson's avatar
Greg Wilson committed
735

736
~~~
737
min_plot = matplotlib.pyplot.plot(numpy.min(data, axis=0))
738
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
739
~~~
740
{: .python}
741

742
![Minimum Value Along The First Axis](../fig/01-numpy_75_3.png)
743

744
745
746
747
748
The maximum value rises and falls perfectly smoothly,
while the minimum seems to be a step function.
Neither result seems particularly likely,
so either there's a mistake in our calculations
or something is wrong with our data.
749
This insight would have been difficult to reach by
jstapleton's avatar
jstapleton committed
750
examining the data without visualization tools.
751

752
You can group similar plots in a single figure using subplots.
753
This script below uses a number of new commands. The function `matplotlib.pyplot.figure()`
754
creates a space into which we will place all of our plots. The parameter `figsize`
755
tells Python how big to make this space. Each subplot is placed into the figure using
756
its `add_subplot` [method]({{ page.root }}/reference/#method). The `add_subplot` method takes 3 parameters. The first denotes
757
how many total rows of subplots there are, the second parameter refers to the
valiseverywhere's avatar
valiseverywhere committed
758
total number of subplot columns, and the final parameter denotes which subplot
759
760
761
your variable is referencing (left-to-right, top-to-bottom). Each subplot is stored in a
different variable (`axes1`, `axes2`, `axes3`). Once a subplot is created, the axes can
be titled using the `set_xlabel()` command (or `set_ylabel()`).
762
Here are our three plots side by side:
763

764
~~~
765
766
import numpy
import matplotlib.pyplot
767

768
data = numpy.loadtxt(fname='inflammation-01.csv', delimiter=',')
769

770
fig = matplotlib.pyplot.figure(figsize=(10.0, 3.0))
771

772
773
774
axes1 = fig.add_subplot(1, 3, 1)
axes2 = fig.add_subplot(1, 3, 2)
axes3 = fig.add_subplot(1, 3, 3)
775

776
axes1.set_ylabel('average')
777
axes1.plot(numpy.mean(data, axis=0))
778

779
axes2.set_ylabel('max')
780
axes2.plot(numpy.max(data, axis=0))
781

782
axes3.set_ylabel('min')
783
axes3.plot(numpy.min(data, axis=0))
784
785

fig.tight_layout()
786

787
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
788
~~~
789
{: .python}
790

791
![The Previous Plots as Subplots](../fig/01-numpy_80_0.png)
792

793
The [call]({{ page.root }}/reference/#function-call) to `loadtxt` reads our data,
794
795
and the rest of the program tells the plotting library
how large we want the figure to be,
796
that we're creating three subplots,
797
798
799
what to draw for each one,
and that we want a tight layout.
(Perversely,
800
if we leave out that call to `fig.tight_layout()`,
801
the graphs will actually be s