README.rst 8.28 KB
Newer Older
Daniel Scheffler's avatar
Daniel Scheffler committed
1
2
3
=========================================================================================
gms_preprocessing - Spatial and spectral homogenization of satellite remote sensing data.
=========================================================================================
4

Daniel Scheffler's avatar
Daniel Scheffler committed
5
6
7
The goal of the gms_preprocessing Python library is to provide a fully automatic
pre-precessing pipeline for spatial and spectral fusion (i.e., homogenization)
of multispectral satellite image data. Currently it offers compatibility to
Daniel Scheffler's avatar
Daniel Scheffler committed
8
Landsat-5, Landsat-7, Landsat-8, Sentinel-2A and Sentinel-2B.
9

Daniel Scheffler's avatar
Daniel Scheffler committed
10
* Free software: GNU General Public License v3 (`license details <https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/blob/master/LICENSE>`_)
11
* Documentation: https://geomultisens.gitext-pages.gfz-potsdam.de/gms_preprocessing/doc/
Daniel Scheffler's avatar
Daniel Scheffler committed
12
* Code history: Release notes for the current and earlier versions of gms_preprocessing can be found `here <./HISTORY.rst>`_.
Daniel Scheffler's avatar
Daniel Scheffler committed
13
* OS compatibility: Linux
Daniel Scheffler's avatar
Daniel Scheffler committed
14
15
16
17
18


Status
------

19
20
.. .. image:: https://img.shields.io/travis/geomultisens/gms_preprocessing.svg
        :target: https://travis-ci.org/geomultisens/gms_preprocessing
21

22
23
.. .. image:: https://readthedocs.org/projects/gms_preprocessing/badge/?version=latest
        :target: https://gms_preprocessing.readthedocs.io/en/latest/?badge=latest
24
25
        :alt: Documentation Status

26
27
.. .. image:: https://pyup.io/repos/github/geomultisens/gms_preprocessing/shield.svg
     :target: https://pyup.io/repos/github/geomultisens/gms_preprocessing/
28
29
     :alt: Updates

Daniel Scheffler's avatar
Daniel Scheffler committed
30
.. image:: https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/badges/master/pipeline.svg
31
32
        :target: https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/commits/master
.. image:: https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/badges/master/coverage.svg
33
        :target: https://geomultisens.gitext-pages.gfz-potsdam.de/gms_preprocessing/coverage/
Daniel Scheffler's avatar
Daniel Scheffler committed
34
.. .. image:: https://img.shields.io/pypi/v/gms_preprocessing.svg
Daniel Scheffler's avatar
Daniel Scheffler committed
35
        :target: https://pypi.python.org/pypi/gms_preprocessing
Daniel Scheffler's avatar
Daniel Scheffler committed
36
.. .. image:: https://img.shields.io/pypi/l/gms_preprocessing.svg
Daniel Scheffler's avatar
Daniel Scheffler committed
37
        :target: https://gitext.gfz-potsdam.de/danschef/gms_preprocessing/blob/master/LICENSE
Daniel Scheffler's avatar
Daniel Scheffler committed
38
.. .. image:: https://img.shields.io/pypi/pyversions/gms_preprocessing.svg
Daniel Scheffler's avatar
Daniel Scheffler committed
39
        :target: https://img.shields.io/pypi/pyversions/gms_preprocessing.svg
40

41
See also the latest coverage_ report and the nosetests_ HTML report.
42
43
44
45

Features
--------

Daniel Scheffler's avatar
Daniel Scheffler committed
46
47
48
49
50
Level-1 processing:
^^^^^^^^^^^^^^^^^^^

* data import and  metadata homogenization (compatibility: Landsat-5/7/8, Sentinel-2A/2B)
* equalization of acquisition- and illumination geometry
Daniel Scheffler's avatar
Daniel Scheffler committed
51
52
* atmospheric correction (using `SICOR <https://gitext.gfz-potsdam.de/EnMAP/sicor>`_)
* correction of geometric errors (using `AROSICS <https://gitext.gfz-potsdam.de/danschef/arosics>`_)
Daniel Scheffler's avatar
Daniel Scheffler committed
53
54
55
56
57

Level-2 processing:
^^^^^^^^^^^^^^^^^^^

* spatial homogenization
58
* spectral homogenization (using `SpecHomo <https://gitext.gfz-potsdam.de/geomultisens/spechomo>`_)
Daniel Scheffler's avatar
Daniel Scheffler committed
59
60
61
62
63
64
65
66
67
68
69
* estimation of accuracy layers

=> application oriented analysis dataset


Getting started
---------------

Usage via WebApp
^^^^^^^^^^^^^^^^

Daniel Scheffler's avatar
Daniel Scheffler committed
70
71
72
73
The recommended way to use gms_preprocessing is to setup the WebApp (see the
gms-vis_ repository) providing a UI for GeoMultiSens. Using this UI, existing
satellite data can be explored, filtered and selected for processing. New data
homogenization jobs (using gms_preprocessing) can be defined and started. All
Daniel Scheffler's avatar
Daniel Scheffler committed
74
75
76
configuration parameters of gms_preprocessing are accessible in the UI.


Daniel Scheffler's avatar
Fix.    
Daniel Scheffler committed
77
.. image:: ./docs/images/webapp_screenshot_900x497.png
Daniel Scheffler's avatar
Daniel Scheffler committed
78
79
80
    :width: 900 px
    :height: 497 px
    :scale: 100 %
Daniel Scheffler's avatar
Daniel Scheffler committed
81
82
83
84
85


Usage via console interface
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Daniel Scheffler's avatar
Daniel Scheffler committed
86
Homogenization jobs can also be created and started using the command line
87
interface. Documentation can be found `here <https://geomultisens.gitext-pages.gfz-potsdam.de/gms_preprocessing/doc/usage.html#gms-preprocessing-command-line-interface>`_.
Daniel Scheffler's avatar
Daniel Scheffler committed
88

Daniel Scheffler's avatar
Daniel Scheffler committed
89
90
91
Here is a small example:

.. code:: console
Daniel Scheffler's avatar
Daniel Scheffler committed
92

Daniel Scheffler's avatar
Daniel Scheffler committed
93
94
    # start the job with the ID 123456 and override default configuration with the given one.
    >>> run_gms.py jobid 123456 --json_config /path/to/my/config.json
Daniel Scheffler's avatar
Daniel Scheffler committed
95
96
97

There is a default configuration file, called `options_default.json <https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/blob/master/gms_preprocessing/options/options_default.json>`_.
This file contains the documentation for all the available configuration
Daniel Scheffler's avatar
Daniel Scheffler committed
98
99
100
101
102
103
parameters.


Usage via Python API
^^^^^^^^^^^^^^^^^^^^

Daniel Scheffler's avatar
Daniel Scheffler committed
104
There is also a Python API that allows to setup and start homogenization jobs
Daniel Scheffler's avatar
Daniel Scheffler committed
105
106
107
108
109
110
111
by a Python function call.

This is an example:

.. code:: python

    from gms_preprocessing import ProcessController
Daniel Scheffler's avatar
Daniel Scheffler committed
112

Daniel Scheffler's avatar
Daniel Scheffler committed
113
114
115
116
    configuration = dict(
        db_host='localhost',
        CPUs=20
        )
Daniel Scheffler's avatar
Daniel Scheffler committed
117

Daniel Scheffler's avatar
Daniel Scheffler committed
118
119
    PC = ProcessController(jobID=123456, **configuration)
    PC.run_all_processors()
Daniel Scheffler's avatar
Daniel Scheffler committed
120

Daniel Scheffler's avatar
Daniel Scheffler committed
121
122
Possible configuration arguments can be found `here <https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/blob/master/gms_preprocessing/options/options_schema.py>`_.

123

Daniel Scheffler's avatar
Daniel Scheffler committed
124
125
126
127

Installation
------------

128
129
130
131
132
gms_preprocessing depends on some open source packages which are usually installed without problems by the automatic install
routine. However, for some projects, we strongly recommend resolving the dependency before the automatic installer
is run. This approach avoids problems with conflicting versions of the same software.
Using conda_, the recommended approach is:

Daniel Scheffler's avatar
Daniel Scheffler committed
133
.. code:: console
134

Daniel Scheffler's avatar
Daniel Scheffler committed
135
  # create virtual environment for gms_preprocessing, this is optional
136
  conda create -c conda-forge --name gms_preprocessing python=3
Daniel Scheffler's avatar
Daniel Scheffler committed
137
  source activate gms_preprocessing
138

139
140
  # install some dependencies that cause trouble when installed via pip
  conda install -c conda-forge numpy gdal scikit-image pyproj lxml geopandas ipython matplotlib basemap scikit-learn=0.23.2 shapely pyhdf python-fmask holoviews
141

142
143
  # install not pip-installable deps of arosics
  conda install -c conda-forge pyfftw pykrige
144

145
146
147
  # install not pip-installable deps of sicor
  conda install -c conda-forge glymur pygrib cachetools pyhdf h5py pytables llvmlite numba
  pip install ecmwf-api-client
Daniel Scheffler's avatar
Daniel Scheffler committed
148

149
150
151
152
  # install gms_preprocessing
  git clone https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing.git
  cd gms_preprocessing
  pip install .
Daniel Scheffler's avatar
Daniel Scheffler committed
153
154


Daniel Scheffler's avatar
Daniel Scheffler committed
155
156
To enable lock functionality (needed for CPU / memory / disk IO management), install redis-server_:

Daniel Scheffler's avatar
Daniel Scheffler committed
157
.. code-block:: console
Daniel Scheffler's avatar
Daniel Scheffler committed
158
159
160

    sudo apt-get install redis-server

Daniel Scheffler's avatar
Daniel Scheffler committed
161

Daniel Scheffler's avatar
Daniel Scheffler committed
162
163
License
-------
Daniel Scheffler's avatar
Daniel Scheffler committed
164

Daniel Scheffler's avatar
Daniel Scheffler committed
165
166
gms_preprocessing - Spatial and spectral homogenization of satellite remote sensing data.

167
Copyright 2020 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences, Potsdam, Germany
Daniel Scheffler's avatar
Daniel Scheffler committed
168

Daniel Scheffler's avatar
Daniel Scheffler committed
169
170
171
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
Daniel Scheffler's avatar
Daniel Scheffler committed
172
173
(at your option) any later version. This program is distributed in the hope
that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
Daniel Scheffler's avatar
Daniel Scheffler committed
174
warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
Daniel Scheffler's avatar
Daniel Scheffler committed
175
176
GNU General Public License for more details. You should have received a copy
of the GNU General Public License along with this program.
Daniel Scheffler's avatar
Daniel Scheffler committed
177
If not, see <http://www.gnu.org/licenses/>.
178
179


Daniel Scheffler's avatar
Daniel Scheffler committed
180
181
182
183
184
Contact
-------

.. line-block::

Daniel Scheffler's avatar
Daniel Scheffler committed
185
186
    Daniel Scheffler
    eMail: daniel.scheffler@gfz-potsdam.de
Daniel Scheffler's avatar
Daniel Scheffler committed
187

Daniel Scheffler's avatar
Daniel Scheffler committed
188
189
190
191
192
    Helmholtz Centre Potsdam GFZ German Research Centre for Geoscienes
    Section 1.4 Remote Sensing
    Telegrafenberg
    14473 Potsdam
    Germany
Daniel Scheffler's avatar
Daniel Scheffler committed
193
194


195
Credits
196
-------
197

198
199
200
201
The development of the gms_preprocessing package was funded by the German Federal Ministry of Education and Research
(BMBF, project grant code: 01 IS 14 010 A-C).

The package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.
202

203
Landsat-5/7/8 satellite data and SRTM/ASTER digital elevation models have been provided by the US Geological
Daniel Scheffler's avatar
Daniel Scheffler committed
204
205
Survey. Sentinel-2 data have been provided by ESA.

206
207
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
208
209
.. _coverage: https://geomultisens.gitext-pages.gfz-potsdam.de/gms_preprocessing/coverage/
.. _nosetests: https://geomultisens.gitext-pages.gfz-potsdam.de/gms_preprocessing/nosetests_reports/nosetests.html
210
.. _conda: https://conda.io/docs/
Daniel Scheffler's avatar
Daniel Scheffler committed
211
.. _redis-server: https://www.rosehosting.com/blog/how-to-install-configure-and-use-redis-on-ubuntu-16-04/
Daniel Scheffler's avatar
Daniel Scheffler committed
212
.. _gms-vis: https://gitext.gfz-potsdam.de/geomultisens/gms-vis