README.rst 8.56 KB
Newer Older
Daniel Scheffler's avatar
Daniel Scheffler committed
1
2
3
=========================================================================================
gms_preprocessing - Spatial and spectral homogenization of satellite remote sensing data.
=========================================================================================
4

Daniel Scheffler's avatar
Daniel Scheffler committed
5
6
7
The goal of the gms_preprocessing Python library is to provide a fully automatic
pre-precessing pipeline for spatial and spectral fusion (i.e., homogenization)
of multispectral satellite image data. Currently it offers compatibility to
Daniel Scheffler's avatar
Daniel Scheffler committed
8
Landsat-5, Landsat-7, Landsat-8, Sentinel-2A and Sentinel-2B.
9

Daniel Scheffler's avatar
Daniel Scheffler committed
10
* Free software: GNU General Public License v3 (`license details <https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/blob/master/LICENSE>`_)
11
* Documentation: http://geomultisens.gitext.gfz-potsdam.de/gms_preprocessing/doc/
Daniel Scheffler's avatar
Daniel Scheffler committed
12
* Code history: Release notes for the current and earlier versions of gms_preprocessing can be found `here <./HISTORY.rst>`_.
Daniel Scheffler's avatar
Daniel Scheffler committed
13
* OS compatibility: Linux
Daniel Scheffler's avatar
Daniel Scheffler committed
14
15
16
17
18


Status
------

19
20
.. .. image:: https://img.shields.io/travis/geomultisens/gms_preprocessing.svg
        :target: https://travis-ci.org/geomultisens/gms_preprocessing
21

22
23
.. .. image:: https://readthedocs.org/projects/gms_preprocessing/badge/?version=latest
        :target: https://gms_preprocessing.readthedocs.io/en/latest/?badge=latest
24
25
        :alt: Documentation Status

26
27
.. .. image:: https://pyup.io/repos/github/geomultisens/gms_preprocessing/shield.svg
     :target: https://pyup.io/repos/github/geomultisens/gms_preprocessing/
28
29
     :alt: Updates

Daniel Scheffler's avatar
Daniel Scheffler committed
30
.. image:: https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/badges/master/pipeline.svg
31
32
33
        :target: https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/commits/master
.. image:: https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/badges/master/coverage.svg
        :target: http://geomultisens.gitext.gfz-potsdam.de/gms_preprocessing/coverage/
Daniel Scheffler's avatar
Daniel Scheffler committed
34
.. .. image:: https://img.shields.io/pypi/v/gms_preprocessing.svg
Daniel Scheffler's avatar
Daniel Scheffler committed
35
        :target: https://pypi.python.org/pypi/gms_preprocessing
Daniel Scheffler's avatar
Daniel Scheffler committed
36
.. .. image:: https://img.shields.io/pypi/l/gms_preprocessing.svg
Daniel Scheffler's avatar
Daniel Scheffler committed
37
        :target: https://gitext.gfz-potsdam.de/danschef/gms_preprocessing/blob/master/LICENSE
Daniel Scheffler's avatar
Daniel Scheffler committed
38
.. .. image:: https://img.shields.io/pypi/pyversions/gms_preprocessing.svg
Daniel Scheffler's avatar
Daniel Scheffler committed
39
        :target: https://img.shields.io/pypi/pyversions/gms_preprocessing.svg
40

41
See also the latest coverage_ report and the nosetests_ HTML report.
42
43
44
45

Features
--------

Daniel Scheffler's avatar
Daniel Scheffler committed
46
47
48
49
50
Level-1 processing:
^^^^^^^^^^^^^^^^^^^

* data import and  metadata homogenization (compatibility: Landsat-5/7/8, Sentinel-2A/2B)
* equalization of acquisition- and illumination geometry
Daniel Scheffler's avatar
Daniel Scheffler committed
51
52
* atmospheric correction (using `SICOR <https://gitext.gfz-potsdam.de/EnMAP/sicor>`_)
* correction of geometric errors (using `AROSICS <https://gitext.gfz-potsdam.de/danschef/arosics>`_)
Daniel Scheffler's avatar
Daniel Scheffler committed
53
54
55
56
57

Level-2 processing:
^^^^^^^^^^^^^^^^^^^

* spatial homogenization
58
* spectral homogenization (using `SpecHomo <https://gitext.gfz-potsdam.de/geomultisens/spechomo>`_
Daniel Scheffler's avatar
Daniel Scheffler committed
59
60
61
62
63
64
65
66
67
68
69
* estimation of accuracy layers

=> application oriented analysis dataset


Getting started
---------------

Usage via WebApp
^^^^^^^^^^^^^^^^

Daniel Scheffler's avatar
Daniel Scheffler committed
70
71
72
73
The recommended way to use gms_preprocessing is to setup the WebApp (see the
gms-vis_ repository) providing a UI for GeoMultiSens. Using this UI, existing
satellite data can be explored, filtered and selected for processing. New data
homogenization jobs (using gms_preprocessing) can be defined and started. All
Daniel Scheffler's avatar
Daniel Scheffler committed
74
75
76
configuration parameters of gms_preprocessing are accessible in the UI.


Daniel Scheffler's avatar
Fix.    
Daniel Scheffler committed
77
.. image:: ./docs/images/webapp_screenshot_900x497.png
Daniel Scheffler's avatar
Daniel Scheffler committed
78
79
80
    :width: 900 px
    :height: 497 px
    :scale: 100 %
Daniel Scheffler's avatar
Daniel Scheffler committed
81
82
83
84
85


Usage via console interface
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Daniel Scheffler's avatar
Daniel Scheffler committed
86
Homogenization jobs can also be created and started using the command line
Daniel Scheffler's avatar
Daniel Scheffler committed
87
interface. Documentation can be found `here <http://geomultisens.gitext.gfz-potsdam.de/gms_preprocessing/doc/usage.html#gms-preprocessing-command-line-interface>`_.
Daniel Scheffler's avatar
Daniel Scheffler committed
88

Daniel Scheffler's avatar
Daniel Scheffler committed
89
90
91
Here is a small example:

.. code:: console
Daniel Scheffler's avatar
Daniel Scheffler committed
92

Daniel Scheffler's avatar
Daniel Scheffler committed
93
94
    # start the job with the ID 123456 and override default configuration with the given one.
    >>> run_gms.py jobid 123456 --json_config /path/to/my/config.json
Daniel Scheffler's avatar
Daniel Scheffler committed
95
96
97

There is a default configuration file, called `options_default.json <https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/blob/master/gms_preprocessing/options/options_default.json>`_.
This file contains the documentation for all the available configuration
Daniel Scheffler's avatar
Daniel Scheffler committed
98
99
100
101
102
103
parameters.


Usage via Python API
^^^^^^^^^^^^^^^^^^^^

Daniel Scheffler's avatar
Daniel Scheffler committed
104
There is also a Python API that allows to setup and start homogenization jobs
Daniel Scheffler's avatar
Daniel Scheffler committed
105
106
107
108
109
110
111
by a Python function call.

This is an example:

.. code:: python

    from gms_preprocessing import ProcessController
Daniel Scheffler's avatar
Daniel Scheffler committed
112

Daniel Scheffler's avatar
Daniel Scheffler committed
113
114
115
116
    configuration = dict(
        db_host='localhost',
        CPUs=20
        )
Daniel Scheffler's avatar
Daniel Scheffler committed
117

Daniel Scheffler's avatar
Daniel Scheffler committed
118
119
    PC = ProcessController(jobID=123456, **configuration)
    PC.run_all_processors()
Daniel Scheffler's avatar
Daniel Scheffler committed
120

Daniel Scheffler's avatar
Daniel Scheffler committed
121
122
Possible configuration arguments can be found `here <https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing/blob/master/gms_preprocessing/options/options_schema.py>`_.

123

Daniel Scheffler's avatar
Daniel Scheffler committed
124
125
126
127

Installation
------------

128
129
130
131
132
gms_preprocessing depends on some open source packages which are usually installed without problems by the automatic install
routine. However, for some projects, we strongly recommend resolving the dependency before the automatic installer
is run. This approach avoids problems with conflicting versions of the same software.
Using conda_, the recommended approach is:

Daniel Scheffler's avatar
Daniel Scheffler committed
133
.. code:: console
134

Daniel Scheffler's avatar
Daniel Scheffler committed
135
136
137
138
139
140
141
  # create virtual environment for gms_preprocessing, this is optional
  conda create -y -q --name gms_preprocessing python=3
  source activate gms_preprocessing
  conda install -y -q -c anaconda hdf5 pytables; \
  conda install -y -q -c conda-forge gdal pygrib h5py; \
  conda install -y -q -c numba llvmlite; \
  conda install -y -q -c conda-forge psycopg2 python-fmask"
142
143
144

Then install sicor:

Daniel Scheffler's avatar
Daniel Scheffler committed
145
.. code-block:: console
146

Niklas Bohn's avatar
Niklas Bohn committed
147
    git clone https://gitext.gfz-potsdam.de/EnMAP/sicor.git
148
149
150
    cd sicor; make install

Clone the following repository and install gms_preprocessing to your local python:
Daniel Scheffler's avatar
Daniel Scheffler committed
151

Daniel Scheffler's avatar
Daniel Scheffler committed
152
.. code-block:: console
Daniel Scheffler's avatar
Daniel Scheffler committed
153

Niklas Bohn's avatar
Niklas Bohn committed
154
    git clone https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing.git
155
    cd gms_preprocessing; make install
Daniel Scheffler's avatar
Daniel Scheffler committed
156

Daniel Scheffler's avatar
Daniel Scheffler committed
157
158
To enable lock functionality (needed for CPU / memory / disk IO management), install redis-server_:

Daniel Scheffler's avatar
Daniel Scheffler committed
159
.. code-block:: console
Daniel Scheffler's avatar
Daniel Scheffler committed
160
161
162

    sudo apt-get install redis-server

Daniel Scheffler's avatar
Daniel Scheffler committed
163
164
165
166

.. .. The pip installer does not work as long as the packages arosics, geoarray, py_tools_ds and sicor are not public.
.. .. * Use the pip-Installer:

167
168
.. ..     pip install git+https://gitext.gfz-potsdam.de/geomultisens/gms_preprocessing.git   # for HTTPS authentification
.. ..     pip install git+git@gitext.gfz-potsdam.de:geomultisens/gms_preprocessing.git   # for SSH-Key authentification
Daniel Scheffler's avatar
Daniel Scheffler committed
169
170


Daniel Scheffler's avatar
Daniel Scheffler committed
171
172
License
-------
Daniel Scheffler's avatar
Daniel Scheffler committed
173

Daniel Scheffler's avatar
Daniel Scheffler committed
174
175
gms_preprocessing - Spatial and spectral homogenization of satellite remote sensing data.

176
Copyright 2020 Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences, Potsdam, Germany
Daniel Scheffler's avatar
Daniel Scheffler committed
177

Daniel Scheffler's avatar
Daniel Scheffler committed
178
179
180
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
Daniel Scheffler's avatar
Daniel Scheffler committed
181
182
(at your option) any later version. This program is distributed in the hope
that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
Daniel Scheffler's avatar
Daniel Scheffler committed
183
warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
Daniel Scheffler's avatar
Daniel Scheffler committed
184
185
GNU General Public License for more details. You should have received a copy
of the GNU General Public License along with this program.
Daniel Scheffler's avatar
Daniel Scheffler committed
186
If not, see <http://www.gnu.org/licenses/>.
187
188


Daniel Scheffler's avatar
Daniel Scheffler committed
189
190
191
192
193
Contact
-------

.. line-block::

Daniel Scheffler's avatar
Daniel Scheffler committed
194
195
    Daniel Scheffler
    eMail: daniel.scheffler@gfz-potsdam.de
Daniel Scheffler's avatar
Daniel Scheffler committed
196

Daniel Scheffler's avatar
Daniel Scheffler committed
197
198
199
200
201
    Helmholtz Centre Potsdam GFZ German Research Centre for Geoscienes
    Section 1.4 Remote Sensing
    Telegrafenberg
    14473 Potsdam
    Germany
Daniel Scheffler's avatar
Daniel Scheffler committed
202
203


204
Credits
205
-------
206

207
208
209
210
The development of the gms_preprocessing package was funded by the German Federal Ministry of Education and Research
(BMBF, project grant code: 01 IS 14 010 A-C).

The package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.
211

212
Landsat-5/7/8 satellite data and SRTM/ASTER digital elevation models have been have been provided by the US Geological
Daniel Scheffler's avatar
Daniel Scheffler committed
213
214
Survey. Sentinel-2 data have been provided by ESA.

215
216
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
217
218
.. _coverage: http://geomultisens.gitext.gfz-potsdam.de/gms_preprocessing/coverage/
.. _nosetests: http://geomultisens.gitext.gfz-potsdam.de/gms_preprocessing/nosetests_reports/nosetests.html
219
.. _conda: https://conda.io/docs/
Daniel Scheffler's avatar
Daniel Scheffler committed
220
.. _redis-server: https://www.rosehosting.com/blog/how-to-install-configure-and-use-redis-on-ubuntu-16-04/
Daniel Scheffler's avatar
Daniel Scheffler committed
221
.. _gms-vis: https://gitext.gfz-potsdam.de/geomultisens/gms-vis