01-numpy.md 25.2 KB
Newer Older
1
---
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
title: Analyzing Patient Data
teaching: 30
exercises: 0
questions:
- "FIXME"
objectives:
- "Explain what a library is, and what libraries are used for."
- "Import a Python library and use the things it contains."
- "Read tabular data from a file into a program."
- "Assign values to variables."
- "Select individual values and subsections from data."
- "Perform operations on arrays of data."
- "Display simple graphs."
keypoints:
- "FIXME"
17
18
---

19
Words are useful,
20
but what's more useful are the sentences and stories we build with them.
21
Similarly,
jstapleton's avatar
jstapleton committed
22
while a lot of powerful, general tools are built into languages like Python,
23
specialized tools built up from these basic units live in [libraries](reference.html#library)
jstapleton's avatar
jstapleton committed
24
that can be called upon when needed.
25
26

In order to load our inflammation data,
Trevor Bekolay's avatar
Trevor Bekolay committed
27
28
we need to access ([import](reference.html#import) in Python terminology)
a library called [NumPy](http://docs.scipy.org/doc/numpy/ "NumPy Documentation").
Greg Wilson's avatar
Greg Wilson committed
29
In general you should use this library if you want to do fancy things with numbers,
30
especially if you have matrices or arrays.
Trevor Bekolay's avatar
Trevor Bekolay committed
31
We can import NumPy using:
32

33
~~~
Greg Wilson's avatar
Greg Wilson committed
34
35
import numpy
~~~
36
{: .python}
37

38
39
40
Importing a library is like getting a piece of lab equipment out of a storage locker and setting it up on the bench.
Libraries provide additional functionality to the basic Python package,
much like a new piece of equipment adds functionality to a lab space.
41
Once you've imported the library,
42
we can ask the library to read our data file for us:
43

44
~~~
Peter Cock's avatar
Peter Cock committed
45
numpy.loadtxt(fname='inflammation-01.csv', delimiter=',')
Greg Wilson's avatar
Greg Wilson committed
46
~~~
47
48
49
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
50
array([[ 0.,  0.,  1., ...,  3.,  0.,  0.],
51
52
       [ 0.,  1.,  2., ...,  1.,  0.,  1.],
       [ 0.,  1.,  1., ...,  2.,  1.,  1.],
53
       ...,
54
55
       [ 0.,  1.,  1., ...,  1.,  1.,  1.],
       [ 0.,  0.,  0., ...,  0.,  2.,  0.],
Greg Wilson's avatar
Greg Wilson committed
56
57
       [ 0.,  0.,  1., ...,  1.,  1.,  0.]])
~~~
58
{: .output}
59

Greg Wilson's avatar
Greg Wilson committed
60
The expression `numpy.loadtxt(...)` is a [function call](reference.html#function-call)
61
that asks Python to run the [function](reference.html#function) `loadtxt` which belongs to the `numpy` library.
Greg Wilson's avatar
Greg Wilson committed
62
This [dotted notation](reference.html#dotted-notation) is used everywhere in Python
63
to refer to the parts of things as `thing.component`.
64

Greg Wilson's avatar
Greg Wilson committed
65
`numpy.loadtxt` has two [parameters](reference.html#parameter):
66
the name of the file we want to read,
Greg Wilson's avatar
Greg Wilson committed
67
68
and the [delimiter](reference.html#delimiter) that separates values on a line.
These both need to be character strings (or [strings](reference.html#string) for short),
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
so we put them in quotes.

When we are finished typing and press Shift+Enter,
the notebook runs our command.
Since we haven't told it to do anything else with the function's output,
the notebook displays it.
In this case,
that output is the data we just loaded.
By default,
only a few rows and columns are shown
(with `...` to omit elements when displaying big arrays).
To save space,
Python displays numbers as `1.` instead of `1.0`
when there's nothing interesting after the decimal point.

Our call to `numpy.loadtxt` read our file,
but didn't save the data in memory.
To do that,
Greg Wilson's avatar
Greg Wilson committed
87
we need to [assign](reference.html#assignment) the array to a [variable](reference.html#variable).
88
A variable is just a name for a value,
Greg Wilson's avatar
Greg Wilson committed
89
such as `x`, `current_temperature`, or `subject_id`.
90
Python's variables must begin with a letter and are [case sensitive](reference.html#case-sensitive).
Kyler Brown's avatar
Kyler Brown committed
91
We can create a new variable by assigning a value to it using `=`.
Greg Wilson's avatar
Greg Wilson committed
92
93
94
95
As an illustration,
let's step back and instead of considering a table of data,
consider the simplest "collection" of data,
a single value.
96
The line below assigns the value `55` to a variable `weight_kg`:
Greg Wilson's avatar
Greg Wilson committed
97

98
~~~
Greg Wilson's avatar
Greg Wilson committed
99
100
weight_kg = 55
~~~
101
{: .python}
102

103
Once a variable has a value, we can print it to the screen:
104

105
~~~
106
print(weight_kg)
Greg Wilson's avatar
Greg Wilson committed
107
~~~
108
109
110
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
111
112
55
~~~
113
{: .output}
114

115
and do arithmetic with it:
116

117
~~~
118
print('weight in pounds:', 2.2 * weight_kg)
Greg Wilson's avatar
Greg Wilson committed
119
~~~
120
121
122
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
123
124
weight in pounds: 121.0
~~~
125
{: .output}
126

jstapleton's avatar
jstapleton committed
127
128
129
As the example above shows,
we can print several things at once by separating them with commas.

130
We can also change a variable's value by assigning it a new one:
131

132
~~~
Greg Wilson's avatar
Greg Wilson committed
133
weight_kg = 57.5
134
print('weight in kilograms is now:', weight_kg)
Greg Wilson's avatar
Greg Wilson committed
135
~~~
136
137
138
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
139
140
weight in kilograms is now: 57.5
~~~
141
{: .output}
142

143
144
If we imagine the variable as a sticky note with a name written on it,
assignment is like putting the sticky note on a particular value:
145

146
![Variables as Sticky Notes]({{ site.root }}/fig/python-sticky-note-variables-01.svg)
147

148
149
150
This means that assigning a value to one variable does *not* change the values of other variables.
For example,
let's store the subject's weight in pounds in a variable:
151

152
~~~
Greg Wilson's avatar
Greg Wilson committed
153
weight_lb = 2.2 * weight_kg
154
print('weight in kilograms:', weight_kg, 'and in pounds:', weight_lb)
Greg Wilson's avatar
Greg Wilson committed
155
~~~
156
157
158
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
159
160
weight in kilograms: 57.5 and in pounds: 126.5
~~~
161
{: .output}
162

163
![Creating Another Variable]({{ site.root }}/fig/python-sticky-note-variables-02.svg)
164

165
and then change `weight_kg`:
166

167
~~~
Greg Wilson's avatar
Greg Wilson committed
168
weight_kg = 100.0
169
print('weight in kilograms is now:', weight_kg, 'and weight in pounds is still:', weight_lb)
Greg Wilson's avatar
Greg Wilson committed
170
~~~
171
172
173
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
174
175
weight in kilograms is now: 100.0 and weight in pounds is still: 126.5
~~~
176
{: .output}
177

178
![Updating a Variable]({{ site.root }}/fig/python-sticky-note-variables-03.svg)
179

180
181
182
183
Since `weight_lb` doesn't "remember" where its value came from,
it isn't automatically updated when `weight_kg` changes.
This is different from the way spreadsheets work.

184
> ## Who's Who in Memory
Benjamin Laken's avatar
Benjamin Laken committed
185
>
186
187
188
> You can use the `%whos` command at any time to see what
> variables you have created and what modules you have loaded into the computer's memory.
> As this is an IPython command, it will only work if you are in an IPython terminal or the Jupyter Notebook.
Benjamin Laken's avatar
Benjamin Laken committed
189
>
190
> ~~~
191
192
> %whos
> ~~~
193
194
195
> {: .python}
>
> ~~~
196
197
198
199
200
201
> Variable    Type       Data/Info
> --------------------------------
> numpy       module     <module 'numpy' from '/Us<...>kages/numpy/__init__.py'>
> weight_kg   float      100.0
> weight_lb   float      126.5
> ~~~
202
203
> {: .output}
{: .callout}
Benjamin Laken's avatar
Benjamin Laken committed
204

Johnny Lin's avatar
Johnny Lin committed
205
206
Just as we can assign a single value to a variable, we can also assign an array of values
to a variable using the same syntax.  Let's re-run `numpy.loadtxt` and save its result:
207

208
~~~
Greg Wilson's avatar
Greg Wilson committed
209
210
data = numpy.loadtxt(fname='inflammation-01.csv', delimiter=',')
~~~
211
{: .python}
212

213
214
215
This statement doesn't produce any output because assignment doesn't display anything.
If we want to check that our data has been loaded,
we can print the variable's value:
216

217
~~~
218
print(data)
Greg Wilson's avatar
Greg Wilson committed
219
~~~
220
221
222
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
223
[[ 0.  0.  1. ...,  3.  0.  0.]
224
225
 [ 0.  1.  2. ...,  1.  0.  1.]
 [ 0.  1.  1. ...,  2.  1.  1.]
226
 ...,
227
228
229
 [ 0.  1.  1. ...,  1.  1.  1.]
 [ 0.  0.  0. ...,  0.  2.  0.]
 [ 0.  0.  1. ...,  1.  1.  0.]]
Greg Wilson's avatar
Greg Wilson committed
230
~~~
231
{: .output}
232

233
234
235
Now that our data is in memory,
we can start doing things with it.
First,
Greg Wilson's avatar
Greg Wilson committed
236
let's ask what [type](reference.html#type) of thing `data` refers to:
237

238
~~~
239
print(type(data))
Greg Wilson's avatar
Greg Wilson committed
240
~~~
241
242
243
{: .python}

~~~
244
<class 'numpy.ndarray'>
Greg Wilson's avatar
Greg Wilson committed
245
~~~
246
{: .output}
247

248
249
250
251
252
253
The output tells us that `data` currently refers to
an N-dimensional array created by the NumPy library.
These data correspond to arthritis patients' inflammation.
The rows are the individual patients and the columns
are their daily inflammation measurements.

254
> ## Data Type
255
256
257
258
259
260
261
>
> A Numpy array contains one or more elements
> of the same type. `type` will only tell you that
> a variable is a NumPy array.
> We can also find the out the type
> of the data contained in the NumPy array.
>
262
> ~~~
263
264
> print(data.dtype)
> ~~~
265
266
267
> {: .python}
>
> ~~~
268
269
> dtype('float64')
> ~~~
270
> {: .output}
271
272
273
>
> This tells us that the NumPy array's elements are
> [floating-point numbers](reference.html#floating-point number).
274
{: .callout}
275

jstapleton's avatar
jstapleton committed
276
We can see what the array's [shape](reference.html#shape) is like this:
277

278
~~~
279
print(data.shape)
Greg Wilson's avatar
Greg Wilson committed
280
~~~
281
282
283
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
284
285
(60, 40)
~~~
286
{: .output}
287

288
289
This tells us that `data` has 60 rows and 40 columns. When we created the
variable `data` to store our arthritis data, we didn't just create the array, we also
Trevor Bekolay's avatar
Trevor Bekolay committed
290
created information about the array, called [members](reference.html#member) or
291
292
attributes. This extra information describes `data` in
the same way an adjective describes a noun.
Bartosz T's avatar
Bartosz T committed
293
`data.shape` is an attribute  of `data` which describes the dimensions of `data`.
294
We use the same dotted notation for the attributes of variables
295
296
that we use for the functions in libraries
because they have the same part-and-whole relationship.
297

298
If we want to get a single number from the array,
Greg Wilson's avatar
Greg Wilson committed
299
we must provide an [index](reference.html#index) in square brackets,
300
just as we do in math:
301

302
~~~
303
print('first value in data:', data[0, 0])
Greg Wilson's avatar
Greg Wilson committed
304
~~~
305
306
307
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
308
309
first value in data: 0.0
~~~
310
{: .output}
311

312
~~~
313
print('middle value in data:', data[30, 20])
Greg Wilson's avatar
Greg Wilson committed
314
~~~
315
316
317
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
318
319
middle value in data: 13.0
~~~
320
{: .output}
321

322
323
324
325
326
The expression `data[30, 20]` may not surprise you,
but `data[0, 0]` might.
Programming languages like Fortran and MATLAB start counting at 1,
because that's what human beings have done for thousands of years.
Languages in the C family (including C++, Java, Perl, and Python) count from 0
327
328
329
because that's more convenient when indices are computed rather than constant
(see [Mike Hoye's blog post](http://exple.tive.org/blarg/2013/10/22/citation-needed/)
for historical details).
330
331
332
333
334
335
336
337
As a result,
if we have an M&times;N array in Python,
its indices go from 0 to M-1 on the first axis
and 0 to N-1 on the second.
It takes a bit of getting used to,
but one way to remember the rule is that
the index is how many steps we have to take from the start to get the item we want.

338
> ## In the Corner
339
340
341
342
343
344
>
> What may also surprise you is that when Python displays an array,
> it shows the element with index `[0, 0]` in the upper left corner
> rather than the lower left.
> This is consistent with the way mathematicians draw matrices,
> but different from the Cartesian coordinates.
345
> The indices are (row, column) instead of (column, row) for the same reason,
346
> which can be confusing when plotting data.
347
{: .callout}
348
349
350
351
352

An index like `[30, 20]` selects a single element of an array,
but we can select whole sections as well.
For example,
we can select the first ten days (columns) of values
353
for the first four patients (rows) like this:
354

355
~~~
356
print(data[0:4, 0:10])
Greg Wilson's avatar
Greg Wilson committed
357
~~~
358
359
360
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
361
[[ 0.  0.  1.  3.  1.  2.  4.  7.  8.  3.]
362
363
364
 [ 0.  1.  2.  1.  2.  1.  3.  2.  2.  6.]
 [ 0.  1.  1.  3.  3.  2.  6.  2.  5.  9.]
 [ 0.  0.  2.  0.  4.  2.  2.  1.  6.  7.]]
Greg Wilson's avatar
Greg Wilson committed
365
~~~
366
{: .output}
367

Greg Wilson's avatar
Greg Wilson committed
368
The [slice](reference.html#slice) `0:4` means,
369
370
371
372
373
374
"Start at index 0 and go up to, but not including, index 4."
Again,
the up-to-but-not-including takes a bit of getting used to,
but the rule is that the difference between the upper and lower bounds is the number of values in the slice.

We don't have to start slices at 0:
375

376
~~~
377
print(data[5:10, 0:10])
Greg Wilson's avatar
Greg Wilson committed
378
~~~
379
380
381
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
382
[[ 0.  0.  1.  2.  2.  4.  2.  1.  6.  4.]
383
384
385
386
 [ 0.  0.  2.  2.  4.  2.  2.  5.  5.  8.]
 [ 0.  0.  1.  2.  3.  1.  2.  3.  5.  3.]
 [ 0.  0.  0.  3.  1.  5.  6.  5.  5.  8.]
 [ 0.  1.  1.  2.  1.  3.  5.  3.  5.  8.]]
Greg Wilson's avatar
Greg Wilson committed
387
~~~
388
{: .output}
389

390
391
392
393
394
395
396
397
We also don't have to include the upper and lower bound on the slice.
If we don't include the lower bound,
Python uses 0 by default;
if we don't include the upper,
the slice runs to the end of the axis,
and if we don't include either
(i.e., if we just use ':' on its own),
the slice includes everything:
398

399
~~~
Greg Wilson's avatar
Greg Wilson committed
400
small = data[:3, 36:]
401
402
print('small is:')
print(small)
Greg Wilson's avatar
Greg Wilson committed
403
~~~
404
405
406
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
407
small is:
408
409
410
[[ 2.  3.  0.  0.]
 [ 1.  1.  0.  1.]
 [ 2.  2.  1.  1.]]
Greg Wilson's avatar
Greg Wilson committed
411
~~~
412
{: .output}
413

414
Arrays also know how to perform common mathematical operations on their values.
Greg Wilson's avatar
Greg Wilson committed
415
416
417
418
419
The simplest operations with data are arithmetic:
add, subtract, multiply, and divide.
 When you do such operations on arrays,
the operation is done on each individual element of the array.
Thus:
420

421
~~~
Greg Wilson's avatar
Greg Wilson committed
422
423
doubledata = data * 2.0
~~~
424
{: .python}
425

Greg Wilson's avatar
Greg Wilson committed
426
427
will create a new array `doubledata`
whose elements have the value of two times the value of the corresponding elements in `data`:
428

429
~~~
430
431
432
433
print('original:')
print(data[:3, 36:])
print('doubledata:')
print(doubledata[:3, 36:])
Greg Wilson's avatar
Greg Wilson committed
434
~~~
435
436
437
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
438
original:
439
440
441
442
443
444
445
[[ 2.  3.  0.  0.]
 [ 1.  1.  0.  1.]
 [ 2.  2.  1.  1.]]
doubledata:
[[ 4.  6.  0.  0.]
 [ 2.  2.  0.  2.]
 [ 4.  4.  2.  2.]]
Greg Wilson's avatar
Greg Wilson committed
446
~~~
447
{: .output}
448

Greg Wilson's avatar
Greg Wilson committed
449
450
If,
instead of taking an array and doing arithmetic with a single value (as above)
451
you did the arithmetic operation with another array of the same shape,
Greg Wilson's avatar
Greg Wilson committed
452
453
the operation will be done on corresponding elements of the two arrays.
Thus:
454

455
~~~
Greg Wilson's avatar
Greg Wilson committed
456
457
tripledata = doubledata + data
~~~
458
{: .python}
459

460
461
462
will give you an array where `tripledata[0,0]` will equal `doubledata[0,0]` plus `data[0,0]`,
and so on for all other elements of the arrays.

463
~~~
464
465
print('tripledata:')
print(tripledata[:3, 36:])
Greg Wilson's avatar
Greg Wilson committed
466
~~~
467
468
469
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
470
tripledata:
471
472
473
[[ 6.  9.  0.  0.]
 [ 3.  3.  0.  3.]
 [ 6.  6.  3.  3.]]
Greg Wilson's avatar
Greg Wilson committed
474
~~~
475
{: .output}
476

477
Often, we want to do more than add, subtract, multiply, and divide values of data.
478
NumPy knows how to do more complex operations on arrays.
479
480
If we want to find the average inflammation for all patients on all days,
for example,
481
we can ask NumPy to compute `data`'s mean value:
482

483
~~~
484
print(numpy.mean(data))
Greg Wilson's avatar
Greg Wilson committed
485
~~~
486
487
488
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
489
490
6.14875
~~~
491
{: .output}
492

493
494
495
496
497
`mean` is a [function](reference.html#function) that takes
an array as an [argument](reference.html#argument).
If variables are nouns, functions are verbs:
they do things with variables.

498
> ## Not All Functions Have Input
499
500
501
>
> Generally, a function uses inputs to produce outputs.
> However, some functions produce outputs without
502
503
> needing any input. For example, checking the current time
> doesn't require any input.
504
>
505
> ~~~
506
507
> import time
> print(time.ctime())
508
> ~~~
509
510
511
> {: .python}
>
> ~~~
512
> 'Sat Mar 26 13:07:33 2016'
513
> ~~~
514
> {: .output}
515
516
517
518
>
> For functions that don't take in any arguments,
> we still need parentheses (`()`)
> to tell Python to go and do something for us.
519
{: .callout}
520
521
522

NumPy has lots of useful functions that take an array as input.
Let's use three of those functions to get some descriptive values about the dataset.
523
524
We'll also use multiple assignment,
a convenient Python feature that will enable us to do this all in one line.
525

526
~~~
527
maxval, minval, stdval = numpy.max(data), numpy.min(data), numpy.std(data)
528

Alistair Walsh's avatar
Alistair Walsh committed
529
530
531
print('maximum inflammation:', maxval)
print('minimum inflammation:', minval)
print('standard deviation:', stdval)
Greg Wilson's avatar
Greg Wilson committed
532
~~~
533
534
535
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
536
maximum inflammation: 20.0
537
538
minimum inflammation: 0.0
standard deviation: 4.61383319712
Greg Wilson's avatar
Greg Wilson committed
539
~~~
540
{: .output}
541

542
> ## Mystery Functions in IPython
543
>
544
545
546
547
548
549
550
> How did we know what functions NumPy has and how to use them?
> If you are working in the IPython/Jupyter Notebook there is an easy way to find out.
> If you type the name of something with a full-stop then you can use tab completion
> (e.g. type `numpy.` and then press tab)
> to see a list of all functions and attributes that you can use. After selecting one you
> can also add a question mark (e.g. `numpy.cumprod?`) and IPython will return an
> explanation of the method! This is the same as doing `help(numpy.cumprod)`.
551
{: .callout}
552
553

When analyzing data, though,
554
555
556
we often want to look at partial statistics,
such as the maximum value per patient
or the average value per day.
557
One way to do this is to create a new temporary array of the data we want,
558
then ask it to do the calculation:
559

560
~~~
Greg Wilson's avatar
Greg Wilson committed
561
patient_0 = data[0, :] # 0 on the first axis, everything on the second
562
print('maximum inflammation for patient 0:', patient_0.max())
Greg Wilson's avatar
Greg Wilson committed
563
~~~
564
565
566
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
567
568
maximum inflammation for patient 0: 18.0
~~~
569
{: .output}
570

571
572
573
Everything in a line of code following the '#' symbol is a
[comment](reference.html#comment) that is ignored by the computer.
Comments allow programmers to leave explanatory notes for other
jstapleton's avatar
jstapleton committed
574
575
programmers or their future selves.

576
We don't actually need to store the row in a variable of its own.
577
Instead, we can combine the selection and the function call:
578

579
~~~
580
print('maximum inflammation for patient 2:', numpy.max(data[2, :]))
Greg Wilson's avatar
Greg Wilson committed
581
~~~
582
583
584
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
585
586
maximum inflammation for patient 2: 19.0
~~~
587
{: .output}
588

589
590
591
592
What if we need the maximum inflammation for *all* patients (as in the
next diagram on the left), or the average for each day (as in the
diagram on the right)? As the diagram below shows, we want to perform the
operation across an axis:
593

594
![Operations Across Axes]({{ site.root }}/fig/python-operations-across-axes.png)
595

596
To support this,
597
most array functions allow us to specify the axis we want to work on.
598
If we ask for the average across axis 0 (rows in our 2D example),
599
we get:
600

601
~~~
602
print(numpy.mean(data, axis=0))
Greg Wilson's avatar
Greg Wilson committed
603
~~~
604
605
606
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
607
[  0.           0.45         1.11666667   1.75         2.43333333   3.15
608
609
610
611
612
613
614
   3.8          3.88333333   5.23333333   5.51666667   5.95         5.9
   8.35         7.73333333   8.36666667   9.5          9.58333333
  10.63333333  11.56666667  12.35        13.25        11.96666667
  11.03333333  10.16666667  10.           8.66666667   9.15         7.25
   7.33333333   6.58333333   6.06666667   5.95         5.11666667   3.6
   3.3          3.56666667   2.48333333   1.5          1.13333333
   0.56666667]
Greg Wilson's avatar
Greg Wilson committed
615
~~~
616
{: .output}
617

618
619
As a quick check,
we can ask this array what its shape is:
620

621
~~~
622
print(numpy.mean(data, axis=0).shape)
Greg Wilson's avatar
Greg Wilson committed
623
~~~
624
625
626
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
627
628
(40,)
~~~
629
{: .output}
630

631
632
The expression `(40,)` tells us we have an N&times;1 vector,
so this is the average inflammation per day for all patients.
633
If we average across axis 1 (columns in our 2D example), we get:
634

635
~~~
636
print(numpy.mean(data, axis=1))
Greg Wilson's avatar
Greg Wilson committed
637
~~~
638
639
640
{: .python}

~~~
Greg Wilson's avatar
Greg Wilson committed
641
[ 5.45   5.425  6.1    5.9    5.55   6.225  5.975  6.65   6.625  6.525
642
643
644
645
646
  6.775  5.8    6.225  5.75   5.225  6.3    6.55   5.7    5.85   6.55
  5.775  5.825  6.175  6.1    5.8    6.425  6.05   6.025  6.175  6.55
  6.175  6.35   6.725  6.125  7.075  5.725  5.925  6.15   6.075  5.75
  5.975  5.725  6.3    5.9    6.75   5.925  7.225  6.15   5.95   6.275  5.7
  6.1    6.825  5.975  6.725  5.7    6.25   6.4    7.05   5.9  ]
Greg Wilson's avatar
Greg Wilson committed
647
~~~
648
{: .output}
649

650
651
652
653
654
655
which is the average inflammation per patient across all days.

The mathematician Richard Hamming once said,
"The purpose of computing is insight, not numbers,"
and the best way to develop insight is often to visualize data.
Visualization deserves an entire lecture (or course) of its own,
656
but we can explore a few features of Python's `matplotlib` library here.
Greg Wilson's avatar
Greg Wilson committed
657
658
While there is no "official" plotting library,
this package is the de facto standard.
659
660
661
First,
we will import the `pyplot` module from `matplotlib`
and use two of its functions to create and display a heat map of our data:
662

663
~~~
664
665
import matplotlib.pyplot
image  = matplotlib.pyplot.imshow(data)
666
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
667
~~~
668
{: .python}
669

670
![Heatmap of the Data]({{ site.root }}/fig/01-numpy_71_0.png)
671

672
673
674
Blue regions in this heat map are low values, while red shows high values.
As we can see,
inflammation rises and falls over a 40-day period.
675

676
> ## Some IPython Magic
677
678
679
680
>
> If you're using an IPython / Jupyter notebook,
> you'll need to execute the following command
> in order for your matplotlib images to appear
Damien Irving's avatar
Damien Irving committed
681
> in the notebook when `show()` is called:
Damien Irving's avatar
Damien Irving committed
682
>
683
> ~~~
Damien Irving's avatar
Damien Irving committed
684
685
> % matplotlib inline
> ~~~
686
> {: .python}
687
>
688
689
> The `%` indicates an IPython magic function -
> a function that is only valid within the notebook environment.
690
> Note that you only have to execute this function once per notebook.
691
{: .callout}
692

693
Let's take a look at the average inflammation over time:
694

695
~~~
696
ave_inflammation = numpy.mean(data, axis=0)
697
ave_plot = matplotlib.pyplot.plot(ave_inflammation)
698
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
699
~~~
700
{: .python}
701

702
![Average Inflammation Over Time]({{ site.root }}/fig/01-numpy_73_0.png)
703

704
705
Here,
we have put the average per day across all patients in the variable `ave_inflammation`,
706
then asked `matplotlib.pyplot` to create and display a line graph of those values.
707
708
709
710
711
The result is roughly a linear rise and fall,
which is suspicious:
based on other studies,
we expect a sharper rise and slower fall.
Let's have a look at two other statistics:
712

713
~~~
714
max_plot = matplotlib.pyplot.plot(numpy.max(data, axis=0))
715
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
716
~~~
717
{: .python}
718

719
![Maximum Value Along The First Axis]({{ site.root }}/fig/01-numpy_75_1.png)
Greg Wilson's avatar
Greg Wilson committed
720

721
~~~
722
min_plot = matplotlib.pyplot.plot(numpy.min(data, axis=0))
723
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
724
~~~
725
{: .python}
726

727
![Minimum Value Along The First Axis]({{ site.root }}/fig/01-numpy_75_3.png)
728

729
730
731
732
733
The maximum value rises and falls perfectly smoothly,
while the minimum seems to be a step function.
Neither result seems particularly likely,
so either there's a mistake in our calculations
or something is wrong with our data.
734
This insight would have been difficult to reach by
jstapleton's avatar
jstapleton committed
735
examining the data without visualization tools.
736

737
You can group similar plots in a single figure using subplots.
738
This script below uses a number of new commands. The function `matplotlib.pyplot.figure()`
739
creates a space into which we will place all of our plots. The parameter `figsize`
740
tells Python how big to make this space. Each subplot is placed into the figure using
741
its `add_subplot` [method](reference.html#method). The `add_subplot` method takes 3 parameters. The first denotes
742
how many total rows of subplots there are, the second parameter refers to the
valiseverywhere's avatar
valiseverywhere committed
743
total number of subplot columns, and the final parameter denotes which subplot
744
745
746
your variable is referencing (left-to-right, top-to-bottom). Each subplot is stored in a
different variable (`axes1`, `axes2`, `axes3`). Once a subplot is created, the axes can
be titled using the `set_xlabel()` command (or `set_ylabel()`).
747
Here are our three plots side by side:
748

749
~~~
750
751
import numpy
import matplotlib.pyplot
752

753
data = numpy.loadtxt(fname='inflammation-01.csv', delimiter=',')
754

755
fig = matplotlib.pyplot.figure(figsize=(10.0, 3.0))
756

757
758
759
axes1 = fig.add_subplot(1, 3, 1)
axes2 = fig.add_subplot(1, 3, 2)
axes3 = fig.add_subplot(1, 3, 3)
760

761
axes1.set_ylabel('average')
762
axes1.plot(numpy.mean(data, axis=0))
763

764
axes2.set_ylabel('max')
765
axes2.plot(numpy.max(data, axis=0))
766

767
axes3.set_ylabel('min')
768
axes3.plot(numpy.min(data, axis=0))
769
770

fig.tight_layout()
771

772
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
773
~~~
774
{: .python}
775

776
![The Previous Plots as Subplots]({{ site.root }}/fig/01-numpy_80_0.png)
777

778
The [call](reference.html#function-call) to `loadtxt` reads our data,
779
780
and the rest of the program tells the plotting library
how large we want the figure to be,
781
that we're creating three subplots,
782
783
784
what to draw for each one,
and that we want a tight layout.
(Perversely,
785
if we leave out that call to `fig.tight_layout()`,
786
the graphs will actually be squeezed together more closely.)
787

788
> ## Scientists Dislike Typing
789
790
791
792
793
794
795
>
> We will always use the syntax `import numpy` to import NumPy.
> However, in order to save typing, it is
> [often suggested](http://www.scipy.org/getting-started.html#an-example-script)
> to make a shortcut like so: `import numpy as np`.
> If you ever see Python code online using a NumPy function with `np`
> (for example, `np.loadtxt(...)`), it's because they've used this shortcut.
796
{: .callout}
797

798
> ## Check Your Understanding
mboisson's avatar
mboisson committed
799
800
801
>
> Draw diagrams showing what variables refer to what values after each statement in the following program:
>
802
> ~~~
mboisson's avatar
mboisson committed
803
804
805
806
807
> mass = 47.5
> age = 122
> mass = mass * 2.0
> age = age - 20
> ~~~
808
809
> {: .python}
{: .challenge}
mboisson's avatar
mboisson committed
810

811
> ## Sorting Out References
mboisson's avatar
mboisson committed
812
813
814
>
> What does the following program print out?
>
815
> ~~~
mboisson's avatar
mboisson committed
816
817
> first, second = 'Grace', 'Hopper'
> third, fourth = second, first
818
> print(third, fourth)
mboisson's avatar
mboisson committed
819
> ~~~
820
821
> {: .python}
{: .challenge}
mboisson's avatar
mboisson committed
822

823
> ## Slicing Strings
mboisson's avatar
mboisson committed
824
825
826
827
>
> A section of an array is called a [slice](reference.html#slice).
> We can take slices of character strings as well:
>
Greg Wilson's avatar