01-numpy.md 34.6 KB
Newer Older
1
---
2
3
4
5
title: Analyzing Patient Data
teaching: 30
exercises: 0
questions:
Greg Wilson's avatar
Greg Wilson committed
6
- "How can I process tabular data files in Python?"
7
objectives:
Brian Jackson's avatar
Brian Jackson committed
8
- "Explain what a library is and what libraries are used for."
9
- "Import a Python library and use the functions it contains."
10
11
12
13
- "Read tabular data from a file into a program."
- "Assign values to variables."
- "Select individual values and subsections from data."
- "Perform operations on arrays of data."
14
- "Plot simple graphs from data."
15
keypoints:
Greg Wilson's avatar
Greg Wilson committed
16
17
18
19
20
21
- "Import a library into a program using `import libraryname`."
- "Use the `numpy` library to work with arrays in Python."
- "Use `variable = value` to assign a value to a variable in order to record it in memory."
- "Variables are created on demand whenever a value is assigned to them."
- "Use `print(something)` to display the value of `something`."
- "The expression `array.shape` gives the shape of an array."
22
- "Use `array[x, y]` to select a single element from a 2D array."
Greg Wilson's avatar
Greg Wilson committed
23
- "Array indices start at 0, not 1."
Dustin Lang's avatar
Dustin Lang committed
24
- "Use `low:high` to specify a `slice` that includes the indices from `low` to `high-1`."
Greg Wilson's avatar
Greg Wilson committed
25
26
27
28
29
- "All the indexing and slicing that works on arrays also works on strings."
- "Use `# some kind of explanation` to add comments to programs."
- "Use `numpy.mean(array)`, `numpy.max(array)`, and `numpy.min(array)` to calculate simple statistics."
- "Use `numpy.mean(array, axis=0)` or `numpy.mean(array, axis=1)` to calculate statistics across the specified axis."
- "Use the `pyplot` library from `matplotlib` for creating simple visualizations."
30
---
Brian Jackson's avatar
Brian Jackson committed
31
In this lesson, we will learn how to manipulate the inflammation dataset with Python, but before we discuss how to deal with many data points, we will show how to store a single value on the computer.
32
33


Trevor Bekolay's avatar
Trevor Bekolay committed
34
The line below [assigns]({{ page.root }}/reference/#assign) the value `55` to a [variable]({{ page.root }}/reference/#variable) `weight_kg`:
35

36
~~~
37
weight_kg = 55
Greg Wilson's avatar
Greg Wilson committed
38
~~~
39
{: .language-python}
40

41
A variable is just a name for a value,
42
such as `x_val`, `current_temperature`, or `subject_id`.
Trevor Bekolay's avatar
Trevor Bekolay committed
43
Python's variables must begin with a letter and are [case sensitive]({{ page.root }}/reference/#case-sensitive).
Kyler Brown's avatar
Kyler Brown committed
44
We can create a new variable by assigning a value to it using `=`.
45
46
When we are finished typing and press Shift+Enter,
the notebook runs our command.
47

48
Once a variable has a value, we can print it to the screen:
49

50
~~~
51
print(weight_kg)
Greg Wilson's avatar
Greg Wilson committed
52
~~~
53
{: .language-python}
54
55

~~~
Greg Wilson's avatar
Greg Wilson committed
56
57
55
~~~
58
{: .output}
59

60
and do arithmetic with it:
61

62
~~~
Brian Jackson's avatar
Brian Jackson committed
63
#There are 2.2 pounds per kilogram.
64
print('weight in pounds:', 2.2 * weight_kg)
Greg Wilson's avatar
Greg Wilson committed
65
~~~
66
{: .language-python}
67
68

~~~
Greg Wilson's avatar
Greg Wilson committed
69
70
weight in pounds: 121.0
~~~
71
{: .output}
72

jstapleton's avatar
jstapleton committed
73
74
75
As the example above shows,
we can print several things at once by separating them with commas.

76
We can also change a variable's value by assigning it a new one:
77

78
~~~
Greg Wilson's avatar
Greg Wilson committed
79
weight_kg = 57.5
80
print('weight in kilograms is now:', weight_kg)
Greg Wilson's avatar
Greg Wilson committed
81
~~~
82
{: .language-python}
83
84

~~~
Greg Wilson's avatar
Greg Wilson committed
85
86
weight in kilograms is now: 57.5
~~~
87
{: .output}
88

89
90
If we imagine the variable as a sticky note with a name written on it,
assignment is like putting the sticky note on a particular value:
91

92
![Variables as Sticky Notes](../fig/python-sticky-note-variables-01.svg)
93

94
95
96
This means that assigning a value to one variable does *not* change the values of other variables.
For example,
let's store the subject's weight in pounds in a variable:
97

98
~~~
Brian Jackson's avatar
Brian Jackson committed
99
#There are 2.2 pounds per kilogram.
Greg Wilson's avatar
Greg Wilson committed
100
weight_lb = 2.2 * weight_kg
101
print('weight in kilograms:', weight_kg, 'and in pounds:', weight_lb)
Greg Wilson's avatar
Greg Wilson committed
102
~~~
103
{: .language-python}
104
105

~~~
Greg Wilson's avatar
Greg Wilson committed
106
107
weight in kilograms: 57.5 and in pounds: 126.5
~~~
108
{: .output}
109

110
![Creating Another Variable](../fig/python-sticky-note-variables-02.svg)
111

112
and then change `weight_kg`:
113

114
~~~
Greg Wilson's avatar
Greg Wilson committed
115
weight_kg = 100.0
116
print('weight in kilograms is now:', weight_kg, 'and weight in pounds is still:', weight_lb)
Greg Wilson's avatar
Greg Wilson committed
117
~~~
118
{: .language-python}
119
120

~~~
Greg Wilson's avatar
Greg Wilson committed
121
122
weight in kilograms is now: 100.0 and weight in pounds is still: 126.5
~~~
123
{: .output}
124

125
![Updating a Variable](../fig/python-sticky-note-variables-03.svg)
126

Brian Jackson's avatar
Brian Jackson committed
127
Since `weight_lb` doesn't remember where its value came from,
128
129
130
it isn't automatically updated when `weight_kg` changes.
This is different from the way spreadsheets work.

131
> ## Who's Who in Memory
Benjamin Laken's avatar
Benjamin Laken committed
132
>
133
134
135
> You can use the `%whos` command at any time to see what
> variables you have created and what modules you have loaded into the computer's memory.
> As this is an IPython command, it will only work if you are in an IPython terminal or the Jupyter Notebook.
Benjamin Laken's avatar
Benjamin Laken committed
136
>
137
> ~~~
138
139
> %whos
> ~~~
140
> {: .language-python}
141
142
>
> ~~~
143
144
145
146
147
> Variable    Type       Data/Info
> --------------------------------
> weight_kg   float      100.0
> weight_lb   float      126.5
> ~~~
148
149
> {: .output}
{: .callout}
Benjamin Laken's avatar
Benjamin Laken committed
150

devendra1810's avatar
devendra1810 committed
151
152
153
154
Words are useful,
but what's more useful are the sentences and stories we build with them.
Similarly,
while a lot of powerful, general tools are built into languages like Python,
Trevor Bekolay's avatar
Trevor Bekolay committed
155
specialized tools built up from these basic units live in [libraries]({{ page.root }}/reference/#library)
devendra1810's avatar
devendra1810 committed
156
157
158
that can be called upon when needed.

In order to load our inflammation data,
Trevor Bekolay's avatar
Trevor Bekolay committed
159
we need to access ([import]({{ page.root }}/reference/#import) in Python terminology)
devendra1810's avatar
devendra1810 committed
160
161
162
163
164
165
166
167
a library called [NumPy](http://docs.scipy.org/doc/numpy/ "NumPy Documentation").
In general you should use this library if you want to do fancy things with numbers,
especially if you have matrices or arrays.
We can import NumPy using:

~~~
import numpy
~~~
168
{: .language-python}
devendra1810's avatar
devendra1810 committed
169
170
171

Importing a library is like getting a piece of lab equipment out of a storage locker and setting it up on the bench.
Libraries provide additional functionality to the basic Python package,
172
much like a new piece of equipment adds functionality to a lab space. Just like in the lab, importing too many libraries
Trevor Bekolay's avatar
Trevor Bekolay committed
173
can sometimes complicate and slow down your programs - so we only import what we need for each program.
174
Once we've imported the library,
devendra1810's avatar
devendra1810 committed
175
176
177
178
179
we can ask the library to read our data file for us:

~~~
numpy.loadtxt(fname='inflammation-01.csv', delimiter=',')
~~~
180
{: .language-python}
devendra1810's avatar
devendra1810 committed
181
182
183
184
185
186
187
188
189
190
191
192

~~~
array([[ 0.,  0.,  1., ...,  3.,  0.,  0.],
       [ 0.,  1.,  2., ...,  1.,  0.,  1.],
       [ 0.,  1.,  1., ...,  2.,  1.,  1.],
       ...,
       [ 0.,  1.,  1., ...,  1.,  1.,  1.],
       [ 0.,  0.,  0., ...,  0.,  2.,  0.],
       [ 0.,  0.,  1., ...,  1.,  1.,  0.]])
~~~
{: .output}

Trevor Bekolay's avatar
Trevor Bekolay committed
193
194
The expression `numpy.loadtxt(...)` is a [function call]({{ page.root }}/reference/#function-call)
that asks Python to run the [function]({{ page.root }}/reference/#function) `loadtxt` which belongs to the `numpy` library.
Brian Jackson's avatar
Brian Jackson committed
195
196
197
198
199
This [dotted notation]({{ page.root }}/reference/#dotted-notation) is used everywhere in Python:
the thing that appears before the dot contains the thing that appears after. 
As an example, John Smith is the John that belongs to the Smith family,
We could use the dot notation to write his name `smith.john`, 
just as `loadtxt` is a function that belongs to the `numpy` library.
devendra1810's avatar
devendra1810 committed
200

Trevor Bekolay's avatar
Trevor Bekolay committed
201
`numpy.loadtxt` has two [parameters]({{ page.root }}/reference/#parameter):
devendra1810's avatar
devendra1810 committed
202
the name of the file we want to read,
Trevor Bekolay's avatar
Trevor Bekolay committed
203
204
and the [delimiter]({{ page.root }}/reference/#delimiter) that separates values on a line.
These both need to be character strings (or [strings]({{ page.root }}/reference/#string) for short),
devendra1810's avatar
devendra1810 committed
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
so we put them in quotes.

Since we haven't told it to do anything else with the function's output,
the notebook displays it.
In this case,
that output is the data we just loaded.
By default,
only a few rows and columns are shown
(with `...` to omit elements when displaying big arrays).
To save space,
Python displays numbers as `1.` instead of `1.0`
when there's nothing interesting after the decimal point.

Our call to `numpy.loadtxt` read our file,
but didn't save the data in memory.
To do that,
221
we need to assign the array to a variable. Just as we can assign a single value to a variable, we can also assign an array of values
Johnny Lin's avatar
Johnny Lin committed
222
to a variable using the same syntax.  Let's re-run `numpy.loadtxt` and save its result:
223

224
~~~
Greg Wilson's avatar
Greg Wilson committed
225
226
data = numpy.loadtxt(fname='inflammation-01.csv', delimiter=',')
~~~
227
{: .language-python}
228

229
230
231
This statement doesn't produce any output because assignment doesn't display anything.
If we want to check that our data has been loaded,
we can print the variable's value:
232

233
~~~
234
print(data)
Greg Wilson's avatar
Greg Wilson committed
235
~~~
236
{: .language-python}
237
238

~~~
Greg Wilson's avatar
Greg Wilson committed
239
[[ 0.  0.  1. ...,  3.  0.  0.]
240
241
 [ 0.  1.  2. ...,  1.  0.  1.]
 [ 0.  1.  1. ...,  2.  1.  1.]
242
 ...,
243
244
245
 [ 0.  1.  1. ...,  1.  1.  1.]
 [ 0.  0.  0. ...,  0.  2.  0.]
 [ 0.  0.  1. ...,  1.  1.  0.]]
Greg Wilson's avatar
Greg Wilson committed
246
~~~
247
{: .output}
248

249
250
251
Now that our data is in memory,
we can start doing things with it.
First,
252
let's ask what [type]({{ page.root }}/reference/#type) of thing `data` refers to:
253

254
~~~
255
print(type(data))
Greg Wilson's avatar
Greg Wilson committed
256
~~~
257
{: .language-python}
258
259

~~~
260
<class 'numpy.ndarray'>
Greg Wilson's avatar
Greg Wilson committed
261
~~~
262
{: .output}
263

264
265
266
267
268
269
The output tells us that `data` currently refers to
an N-dimensional array created by the NumPy library.
These data correspond to arthritis patients' inflammation.
The rows are the individual patients and the columns
are their daily inflammation measurements.

270
> ## Data Type
271
272
273
274
>
> A Numpy array contains one or more elements
> of the same type. `type` will only tell you that
> a variable is a NumPy array.
Valentina Staneva's avatar
Valentina Staneva committed
275
> We can also find out the type
276
277
> of the data contained in the NumPy array.
>
278
> ~~~
279
280
> print(data.dtype)
> ~~~
281
> {: .language-python}
282
283
>
> ~~~
284
285
> dtype('float64')
> ~~~
286
> {: .output}
287
288
>
> This tells us that the NumPy array's elements are
289
> [floating-point numbers]({{ page.root }}/reference/#floating-point number).
290
{: .callout}
291

Bree Norlander's avatar
Bree Norlander committed
292
With this command we can see the array's [shape]({{ page.root }}/reference/#shape):
293

294
~~~
295
print(data.shape)
Greg Wilson's avatar
Greg Wilson committed
296
~~~
297
{: .language-python}
298
299

~~~
Greg Wilson's avatar
Greg Wilson committed
300
301
(60, 40)
~~~
302
{: .output}
303

304
305
This tells us that `data` has 60 rows and 40 columns. When we created the
variable `data` to store our arthritis data, we didn't just create the array, we also
306
created information about the array, called [members]({{ page.root }}/reference/#member) or
307
308
attributes. This extra information describes `data` in
the same way an adjective describes a noun.
Bartosz T's avatar
Bartosz T committed
309
`data.shape` is an attribute  of `data` which describes the dimensions of `data`.
310
We use the same dotted notation for the attributes of variables
311
312
that we use for the functions in libraries
because they have the same part-and-whole relationship.
313

314
If we want to get a single number from the array,
315
we must provide an [index]({{ page.root }}/reference/#index) in square brackets,
Dustin Lang's avatar
Dustin Lang committed
316
just as we do in math when referring to an element of a matrix.  Our inflammation data has two dimensions, so we will need to use two indices to refer to a value:
317

318
~~~
319
print('first value in data:', data[0, 0])
Greg Wilson's avatar
Greg Wilson committed
320
~~~
321
{: .language-python}
322
323

~~~
Greg Wilson's avatar
Greg Wilson committed
324
325
first value in data: 0.0
~~~
326
{: .output}
327

328
~~~
329
print('middle value in data:', data[30, 20])
Greg Wilson's avatar
Greg Wilson committed
330
~~~
331
{: .language-python}
332
333

~~~
Greg Wilson's avatar
Greg Wilson committed
334
335
middle value in data: 13.0
~~~
336
{: .output}
337

338
339
The expression `data[30, 20]` accesses the element at row 30, column 20. While this expression may not surprise you,
 `data[0, 0]` might.
340
Programming languages like Fortran, MATLAB and R start counting at 1,
341
342
because that's what human beings have done for thousands of years.
Languages in the C family (including C++, Java, Perl, and Python) count from 0
343
344
345
346
347
because it represents an offset from the first value in the array (the second
value is offset by one index from the first value). This is closer to the way
that computers represent arrays (if you are interested in the historical
reasons behind counting indices from zero, you can read
[Mike Hoye's blog post](http://exple.tive.org/blarg/2013/10/22/citation-needed/)).
348
As a result,
Greg Wilson's avatar
Greg Wilson committed
349
if we have an M×N array in Python,
350
351
352
353
354
355
its indices go from 0 to M-1 on the first axis
and 0 to N-1 on the second.
It takes a bit of getting used to,
but one way to remember the rule is that
the index is how many steps we have to take from the start to get the item we want.

356
357
![Zero Index](../fig/python-zero-index.png)

358
> ## In the Corner
359
360
361
362
363
364
>
> What may also surprise you is that when Python displays an array,
> it shows the element with index `[0, 0]` in the upper left corner
> rather than the lower left.
> This is consistent with the way mathematicians draw matrices,
> but different from the Cartesian coordinates.
365
> The indices are (row, column) instead of (column, row) for the same reason,
366
> which can be confusing when plotting data.
367
{: .callout}
368
369
370
371
372

An index like `[30, 20]` selects a single element of an array,
but we can select whole sections as well.
For example,
we can select the first ten days (columns) of values
373
for the first four patients (rows) like this:
374

375
~~~
376
print(data[0:4, 0:10])
Greg Wilson's avatar
Greg Wilson committed
377
~~~
378
{: .language-python}
379
380

~~~
Greg Wilson's avatar
Greg Wilson committed
381
[[ 0.  0.  1.  3.  1.  2.  4.  7.  8.  3.]
382
383
384
 [ 0.  1.  2.  1.  2.  1.  3.  2.  2.  6.]
 [ 0.  1.  1.  3.  3.  2.  6.  2.  5.  9.]
 [ 0.  0.  2.  0.  4.  2.  2.  1.  6.  7.]]
Greg Wilson's avatar
Greg Wilson committed
385
~~~
386
{: .output}
387

388
The [slice]({{ page.root }}/reference/#slice) `0:4` means,
389
390
391
392
393
394
"Start at index 0 and go up to, but not including, index 4."
Again,
the up-to-but-not-including takes a bit of getting used to,
but the rule is that the difference between the upper and lower bounds is the number of values in the slice.

We don't have to start slices at 0:
395

396
~~~
397
print(data[5:10, 0:10])
Greg Wilson's avatar
Greg Wilson committed
398
~~~
399
{: .language-python}
400
401

~~~
Greg Wilson's avatar
Greg Wilson committed
402
[[ 0.  0.  1.  2.  2.  4.  2.  1.  6.  4.]
403
404
405
406
 [ 0.  0.  2.  2.  4.  2.  2.  5.  5.  8.]
 [ 0.  0.  1.  2.  3.  1.  2.  3.  5.  3.]
 [ 0.  0.  0.  3.  1.  5.  6.  5.  5.  8.]
 [ 0.  1.  1.  2.  1.  3.  5.  3.  5.  8.]]
Greg Wilson's avatar
Greg Wilson committed
407
~~~
408
{: .output}
409

410
411
412
413
414
415
416
417
We also don't have to include the upper and lower bound on the slice.
If we don't include the lower bound,
Python uses 0 by default;
if we don't include the upper,
the slice runs to the end of the axis,
and if we don't include either
(i.e., if we just use ':' on its own),
the slice includes everything:
418

419
~~~
Greg Wilson's avatar
Greg Wilson committed
420
small = data[:3, 36:]
421
422
print('small is:')
print(small)
Greg Wilson's avatar
Greg Wilson committed
423
~~~
424
{: .language-python}
425
426

~~~
Greg Wilson's avatar
Greg Wilson committed
427
small is:
428
429
430
[[ 2.  3.  0.  0.]
 [ 1.  1.  0.  1.]
 [ 2.  2.  1.  1.]]
Greg Wilson's avatar
Greg Wilson committed
431
~~~
432
{: .output}
433

434
Arrays also know how to perform common mathematical operations on their values.
Greg Wilson's avatar
Greg Wilson committed
435
436
437
438
439
The simplest operations with data are arithmetic:
add, subtract, multiply, and divide.
 When you do such operations on arrays,
the operation is done on each individual element of the array.
Thus:
440

441
~~~
Greg Wilson's avatar
Greg Wilson committed
442
443
doubledata = data * 2.0
~~~
444
{: .language-python}
445

Greg Wilson's avatar
Greg Wilson committed
446
447
will create a new array `doubledata`
whose elements have the value of two times the value of the corresponding elements in `data`:
448

449
~~~
450
451
452
453
print('original:')
print(data[:3, 36:])
print('doubledata:')
print(doubledata[:3, 36:])
Greg Wilson's avatar
Greg Wilson committed
454
~~~
455
{: .language-python}
456
457

~~~
Greg Wilson's avatar
Greg Wilson committed
458
original:
459
460
461
462
463
464
465
[[ 2.  3.  0.  0.]
 [ 1.  1.  0.  1.]
 [ 2.  2.  1.  1.]]
doubledata:
[[ 4.  6.  0.  0.]
 [ 2.  2.  0.  2.]
 [ 4.  4.  2.  2.]]
Greg Wilson's avatar
Greg Wilson committed
466
~~~
467
{: .output}
468

Greg Wilson's avatar
Greg Wilson committed
469
470
If,
instead of taking an array and doing arithmetic with a single value (as above)
471
you did the arithmetic operation with another array of the same shape,
Greg Wilson's avatar
Greg Wilson committed
472
473
the operation will be done on corresponding elements of the two arrays.
Thus:
474

475
~~~
Greg Wilson's avatar
Greg Wilson committed
476
477
tripledata = doubledata + data
~~~
478
{: .language-python}
479

480
481
482
will give you an array where `tripledata[0,0]` will equal `doubledata[0,0]` plus `data[0,0]`,
and so on for all other elements of the arrays.

483
~~~
484
485
print('tripledata:')
print(tripledata[:3, 36:])
Greg Wilson's avatar
Greg Wilson committed
486
~~~
487
{: .language-python}
488
489

~~~
Greg Wilson's avatar
Greg Wilson committed
490
tripledata:
491
492
493
[[ 6.  9.  0.  0.]
 [ 3.  3.  0.  3.]
 [ 6.  6.  3.  3.]]
Greg Wilson's avatar
Greg Wilson committed
494
~~~
495
{: .output}
496

497
Often, we want to do more than add, subtract, multiply, and divide values of data.
498
NumPy knows how to do more complex operations on arrays.
499
500
If we want to find the average inflammation for all patients on all days,
for example,
501
we can ask NumPy to compute `data`'s mean value:
502

503
~~~
504
print(numpy.mean(data))
Greg Wilson's avatar
Greg Wilson committed
505
~~~
506
{: .language-python}
507
508

~~~
Greg Wilson's avatar
Greg Wilson committed
509
510
6.14875
~~~
511
{: .output}
512

513
514
`mean` is a [function]({{ page.root }}/reference/#function) that takes
an array as an [argument]({{ page.root }}/reference/#argument).
515
516
517
If variables are nouns, functions are verbs:
they do things with variables.

518
> ## Not All Functions Have Input
519
520
521
>
> Generally, a function uses inputs to produce outputs.
> However, some functions produce outputs without
522
523
> needing any input. For example, checking the current time
> doesn't require any input.
524
>
525
> ~~~
526
527
> import time
> print(time.ctime())
528
> ~~~
529
> {: .language-python}
530
531
>
> ~~~
532
> 'Sat Mar 26 13:07:33 2016'
533
> ~~~
534
> {: .output}
535
536
537
538
>
> For functions that don't take in any arguments,
> we still need parentheses (`()`)
> to tell Python to go and do something for us.
539
{: .callout}
540
541
542

NumPy has lots of useful functions that take an array as input.
Let's use three of those functions to get some descriptive values about the dataset.
543
544
We'll also use multiple assignment,
a convenient Python feature that will enable us to do this all in one line.
545

546
~~~
547
maxval, minval, stdval = numpy.max(data), numpy.min(data), numpy.std(data)
548

Alistair Walsh's avatar
Alistair Walsh committed
549
550
551
print('maximum inflammation:', maxval)
print('minimum inflammation:', minval)
print('standard deviation:', stdval)
Greg Wilson's avatar
Greg Wilson committed
552
~~~
553
{: .language-python}
554
555

~~~
Greg Wilson's avatar
Greg Wilson committed
556
maximum inflammation: 20.0
557
558
minimum inflammation: 0.0
standard deviation: 4.61383319712
Greg Wilson's avatar
Greg Wilson committed
559
~~~
560
{: .output}
561

562
> ## Mystery Functions in IPython
563
>
564
565
> How did we know what functions NumPy has and how to use them?
> If you are working in the IPython/Jupyter Notebook there is an easy way to find out.
Dustin Lang's avatar
Dustin Lang committed
566
> If you type the name of something followed by a dot, then you can use tab completion
567
568
569
570
> (e.g. type `numpy.` and then press tab)
> to see a list of all functions and attributes that you can use. After selecting one you
> can also add a question mark (e.g. `numpy.cumprod?`) and IPython will return an
> explanation of the method! This is the same as doing `help(numpy.cumprod)`.
571
{: .callout}
572
573

When analyzing data, though,
574
575
576
we often want to look at partial statistics,
such as the maximum value per patient
or the average value per day.
577
One way to do this is to create a new temporary array of the data we want,
578
then ask it to do the calculation:
579

580
~~~
Dustin Lang's avatar
Dustin Lang committed
581
patient_0 = data[0, :] # 0 on the first axis (rows), everything on the second (columns)
582
print('maximum inflammation for patient 0:', patient_0.max())
Greg Wilson's avatar
Greg Wilson committed
583
~~~
584
{: .language-python}
585
586

~~~
Greg Wilson's avatar
Greg Wilson committed
587
588
maximum inflammation for patient 0: 18.0
~~~
589
{: .output}
590

591
Everything in a line of code following the '#' symbol is a
592
[comment]({{ page.root }}/reference/#comment) that is ignored by Python.
593
Comments allow programmers to leave explanatory notes for other
jstapleton's avatar
jstapleton committed
594
595
programmers or their future selves.

596
We don't actually need to store the row in a variable of its own.
597
Instead, we can combine the selection and the function call:
598

599
~~~
600
print('maximum inflammation for patient 2:', numpy.max(data[2, :]))
Greg Wilson's avatar
Greg Wilson committed
601
~~~
602
{: .language-python}
603
604

~~~
Greg Wilson's avatar
Greg Wilson committed
605
606
maximum inflammation for patient 2: 19.0
~~~
607
{: .output}
608

609
What if we need the maximum inflammation for each patient over all days (as in the
610
611
612
next diagram on the left), or the average for each day (as in the
diagram on the right)? As the diagram below shows, we want to perform the
operation across an axis:
613

614
![Operations Across Axes](../fig/python-operations-across-axes.png)
615

616
To support this,
617
most array functions allow us to specify the axis we want to work on.
618
If we ask for the average across axis 0 (rows in our 2D example),
619
we get:
620

621
~~~
622
print(numpy.mean(data, axis=0))
Greg Wilson's avatar
Greg Wilson committed
623
~~~
624
{: .language-python}
625
626

~~~
Greg Wilson's avatar
Greg Wilson committed
627
[  0.           0.45         1.11666667   1.75         2.43333333   3.15
628
629
630
631
632
633
634
   3.8          3.88333333   5.23333333   5.51666667   5.95         5.9
   8.35         7.73333333   8.36666667   9.5          9.58333333
  10.63333333  11.56666667  12.35        13.25        11.96666667
  11.03333333  10.16666667  10.           8.66666667   9.15         7.25
   7.33333333   6.58333333   6.06666667   5.95         5.11666667   3.6
   3.3          3.56666667   2.48333333   1.5          1.13333333
   0.56666667]
Greg Wilson's avatar
Greg Wilson committed
635
~~~
636
{: .output}
637

638
639
As a quick check,
we can ask this array what its shape is:
640

641
~~~
642
print(numpy.mean(data, axis=0).shape)
Greg Wilson's avatar
Greg Wilson committed
643
~~~
644
{: .language-python}
645
646

~~~
Greg Wilson's avatar
Greg Wilson committed
647
648
(40,)
~~~
649
{: .output}
650

Greg Wilson's avatar
Greg Wilson committed
651
The expression `(40,)` tells us we have an N×1 vector,
652
so this is the average inflammation per day for all patients.
653
If we average across axis 1 (columns in our 2D example), we get:
654

655
~~~
656
print(numpy.mean(data, axis=1))
Greg Wilson's avatar
Greg Wilson committed
657
~~~
658
{: .language-python}
659
660

~~~
Greg Wilson's avatar
Greg Wilson committed
661
[ 5.45   5.425  6.1    5.9    5.55   6.225  5.975  6.65   6.625  6.525
662
663
664
665
666
  6.775  5.8    6.225  5.75   5.225  6.3    6.55   5.7    5.85   6.55
  5.775  5.825  6.175  6.1    5.8    6.425  6.05   6.025  6.175  6.55
  6.175  6.35   6.725  6.125  7.075  5.725  5.925  6.15   6.075  5.75
  5.975  5.725  6.3    5.9    6.75   5.925  7.225  6.15   5.95   6.275  5.7
  6.1    6.825  5.975  6.725  5.7    6.25   6.4    7.05   5.9  ]
Greg Wilson's avatar
Greg Wilson committed
667
~~~
668
{: .output}
669

670
671
672
673
674
which is the average inflammation per patient across all days.

The mathematician Richard Hamming once said,
"The purpose of computing is insight, not numbers,"
and the best way to develop insight is often to visualize data.
Valentina Staneva's avatar
Valentina Staneva committed
675
Visualization deserves an entire lecture (of course) of its own,
676
but we can explore a few features of Python's `matplotlib` library here.
Greg Wilson's avatar
Greg Wilson committed
677
678
While there is no "official" plotting library,
this package is the de facto standard.
679
680
681
First,
we will import the `pyplot` module from `matplotlib`
and use two of its functions to create and display a heat map of our data:
682

683
~~~
684
import matplotlib.pyplot
685
image = matplotlib.pyplot.imshow(data)
686
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
687
~~~
688
{: .language-python}
689

690
![Heatmap of the Data](../fig/01-numpy_71_0.png)
691

692
693
694
Blue regions in this heat map are low values, while red shows high values.
As we can see,
inflammation rises and falls over a 40-day period.
695

696
> ## Some IPython Magic
697
698
699
700
>
> If you're using an IPython / Jupyter notebook,
> you'll need to execute the following command
> in order for your matplotlib images to appear
Damien Irving's avatar
Damien Irving committed
701
> in the notebook when `show()` is called:
Damien Irving's avatar
Damien Irving committed
702
>
703
> ~~~
704
> %matplotlib inline
Damien Irving's avatar
Damien Irving committed
705
> ~~~
706
> {: .language-python}
707
>
708
709
> The `%` indicates an IPython magic function -
> a function that is only valid within the notebook environment.
710
> Note that you only have to execute this function once per notebook.
711
{: .callout}
712

713
Let's take a look at the average inflammation over time:
714

715
~~~
716
ave_inflammation = numpy.mean(data, axis=0)
717
ave_plot = matplotlib.pyplot.plot(ave_inflammation)
718
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
719
~~~
720
{: .language-python}
721

722
![Average Inflammation Over Time](../fig/01-numpy_73_0.png)
723

724
725
Here,
we have put the average per day across all patients in the variable `ave_inflammation`,
726
then asked `matplotlib.pyplot` to create and display a line graph of those values.
727
728
729
730
731
The result is roughly a linear rise and fall,
which is suspicious:
based on other studies,
we expect a sharper rise and slower fall.
Let's have a look at two other statistics:
732

733
~~~
734
max_plot = matplotlib.pyplot.plot(numpy.max(data, axis=0))
735
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
736
~~~
737
{: .language-python}
738

739
![Maximum Value Along The First Axis](../fig/01-numpy_75_1.png)
Greg Wilson's avatar
Greg Wilson committed
740

741
~~~
742
min_plot = matplotlib.pyplot.plot(numpy.min(data, axis=0))
743
matplotlib.pyplot.show()
Greg Wilson's avatar
Greg Wilson committed
744
~~~
745
{: .language-python}
746

747
![Minimum Value Along The First Axis](../fig/01-numpy_75_3.png)
748

749
750
751
752
753
The maximum value rises and falls perfectly smoothly,
while the minimum seems to be a step function.
Neither result seems particularly likely,
so either there's a mistake in our calculations
or something is wrong with our data.
754
This insight would have been difficult to reach by
jstapleton's avatar
jstapleton committed
755
examining the data without visualization tools.
756

757
You can group similar plots in a single figure using subplots.
758
This script below uses a number of new commands. The function `matplotlib.pyplot.figure()`
759
creates a space into which we will place all of our plots. The parameter `figsize`
760
tells Python how big to make this space. Each subplot is placed into the figure using
761
its `add_subplot` [method]({{ page.root }}/reference/#method). The `add_subplot` method takes 3 parameters. The first denotes
762
how many total rows of subplots there are, the second parameter refers to the
valiseverywhere's avatar
valiseverywhere committed
763
total number of subplot columns, and the final parameter denotes which subplot
764
765
766
your variable is referencing (left-to-right, top-to-bottom). Each subplot is stored in a
different variable (`axes1`, `axes2`, `axes3`). Once a subplot is created, the axes can
be titled using the `set_xlabel()` command (or `set_ylabel()`).
767
Here are our three plots side by side:
768

769
~~~
770
771
import numpy
import matplotlib.pyplot
772

773
data = numpy.loadtxt(fname='inflammation-01.csv', delimiter=',')
774

775
fig = matplotlib.pyplot.figure(figsize=(10.0, 3.0))
776

777
778
779
axes1 = fig.add_subplot(1, 3, 1)
axes2 = fig.add_subplot(1, 3, 2)
axes3 = fig.add_subplot(1, 3, 3)
780

781
axes1.set_ylabel('average')
782
axes1.plot(numpy.mean(data, axis=0))