Commit ec52e2a8 authored by Daniel Scheffler's avatar Daniel Scheffler
Browse files

Merge branch 'master' into enhancement/improve_ecmwf_crawler

# Conflicts:
#	sicor/sicor_ac.py
parents 2def97e0 4c6abce9
......@@ -12,12 +12,10 @@ test_sicor_install:
stage: test
script:
- source /root/miniconda3/bin/activate
- conda create -y -q --name sicor_testinstall python=3
- source activate sicor_testinstall
- conda config --add channels conda-forge # Fix for gdal, libgdal installed from defaults channels causing ImportError: libpoppler.so.76: cannot open shared object file: No such file or directory
- conda config --set channel_priority strict
# run sicor installation
- conda install --yes -q sicor
# run sicor installation in a fresh environment
- mamba create -y -q -c conda-forge --name sicor_testinstall sicor
- conda activate sicor_testinstall
- mamba install -c conda-forge arosics=1.2.6 # temporarily needed in case noarch package of arosics is installed (currently not the latest release => causes ModuleNotFoundError for py_tools_ds version 0.16.7)
# test if its importable
- cd ..
- pwd
......@@ -31,14 +29,16 @@ test_sicor_enmap:
stage: test
script:
- source /root/miniconda3/bin/activate
- source activate sicor_env
- conda activate sicor_env
- export GDAL_DATA=/root/miniconda3/envs/sicor_env/share/gdal
- export PYTHONPATH=$PYTHONPATH:/root # /root <- directory needed later
- pip install pycodestyle --upgrade
- conda install -c conda-forge gdown
# copy testdata and download LUT
- cp -r /root/EnPT/enpt /root/miniconda3/envs/sicor_env/lib/python3.8/site-packages/enpt-0.16.3-py3.8.egg/
- cp -r /root/EnPT/tests /root/miniconda3/envs/sicor_env/lib/python3.8/site-packages/enpt-0.16.3-py3.8.egg/
- mamba install -c conda-forge gdown
# copy testdata
- cd /root/EnPT/enpt
- git pull
- cp -r /root/EnPT/enpt /root/miniconda3/envs/sicor_env/lib/python3.8/site-packages/enpt-0.17.2-py3.8.egg/
- cp -r /root/EnPT/tests /root/miniconda3/envs/sicor_env/lib/python3.8/site-packages/enpt-0.17.2-py3.8.egg/
- cd /builds/EnMAP/sicor
- make lint
- make nosetests_enmap # enmap tests are called here
- pip install sphinx_rtd_theme # Read-the-docs theme for SPHINX documentation
......@@ -60,16 +60,17 @@ test_sicor_s2:
only: # for first pipeline of new branch, changes are evaluated as true for all files and everything will be built
changes:
- sicor/AC/RtFo.py
- sicor/Tools/cB/*
- sicor/ECMWF/*
- sicor/sicor_ac.py
- tests/test_sicor_s2.py
script:
- source /root/miniconda3/bin/activate
- source activate sicor_env
- conda activate sicor_env
- export GDAL_DATA=/root/miniconda3/envs/sicor_env/share/gdal
- export PYTHONPATH=$PYTHONPATH:/root # /root <- directory needed later
- pip install pycodestyle --upgrade
- conda install -c conda-forge pygrib cachetools gdown
- mamba install -c conda-forge pygrib cachetools gdown scikit-learn
- make lint
- make nosetests_s2 # sentinel-2 tests are called here
- pip install sphinx_rtd_theme # Read-the-docs theme for SPHINX documentation
......@@ -130,7 +131,7 @@ deploy_pypi:
- test_sicor_enmap
script:
- source /root/miniconda3/bin/activate sicor_env
- conda install -c conda-forge pygrib
- mamba install -c conda-forge pygrib
- pip install -U twine
- python setup.py sdist
- twine upload dist/*
......
......@@ -11,6 +11,7 @@ Contributors
------------
* Daniel Scheffler <daniel.scheffler@gfz-potsdam.de>
(main developer of the EnPT source code)
* Maximilian Brell <maximilian.brell@gfz-potsdam.de>
* René Preusker <rene.preusker@wew.fu-berlin.de>
* Hannes Diedrich
......
......@@ -39,7 +39,8 @@ Write Documentation
~~~~~~~~~~~~~~~~~~~
SICOR could always use more documentation, whether as part of the
official SICOR docs, or in docstrings.
official SICOR docs, in docstrings, or even on the web in blog posts,
articles, and such.
Submit Feedback
~~~~~~~~~~~~~~~
......@@ -50,7 +51,47 @@ If you are proposing a feature:
* Explain in detail how it would work.
* Keep the scope as narrow as possible, to make it easier to implement.
* Remember that this is a volunteer-driven project, and that contributions are welcome :)
Get Started!
------------
Ready to contribute? Here's how to set up `sicor` for local development.
1. Fork the `sicor` repo on GitLab.
2. Clone your fork locally::
$ git clone git@gitext.gfz-potsdam.de:your_name_here/sicor.git
3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up
your fork for local development::
$ mkvirtualenv sicor
$ cd sicor/
$ python setup.py develop
4. Create a branch for local development::
$ git checkout -b name-of-your-bugfix-or-feature
Now you can make your changes locally.
5. When you're done making changes, check that your changes pass the codestyle and the software tests, including
testing other Python versions with tox::
$ make lint
$ python -m unittest
$ tox
To get flake8 and tox, just pip install them into your virtualenv.
6. Commit your changes and push your branch to GitLab::
$ git add .
$ git commit -m "Your detailed description of your changes."
$ git push origin name-of-your-bugfix-or-feature
7. Submit a merge request through the GitLab website.
Pull Request Guidelines
-----------------------
......@@ -61,4 +102,14 @@ Before you submit a pull request, check that it meets these guidelines:
2. If the pull request adds functionality, the docs should be updated. Put
your new functionality into a function with a docstring, and add the
feature to the list in README.rst.
3. The pull request should work for Python 3.5+.
3. The merge request should work for Python 2.6, 2.7, 3.4, 3.5, 3.6 and 3.7. Check
https://gitext.gfz-potsdam.de/EnMAP/sicor/merge_requests
and make sure that the tests pass for all supported Python versions.
Tips
----
To run a subset of tests::
$ python -m unittest tests.test_sicor_enmap
......@@ -2,7 +2,6 @@
History
=======
0.x.x (coming soon)
--------------------
......@@ -13,15 +12,118 @@ Bugfixes:
*
0.16.4 (2021-06-18)
--------------------
New features:
*
Bugfixes:
* Updated setup.py by removing check for packages that do not install well with pip. This avoids incompatibilities with the latest gdal versions.
0.16.3 (2021-06-17)
--------------------
New features:
*
Bugfixes:
* Disabled multiprocessing for both the optimization and the empirical line extrapolation in case SICOR is running on Windows or macOS.
0.16.2 (2021-05-26)
--------------------
New features:
* Dimensionality reduction of LUT grid to increase interpolation speed.
* Updated final log message of SICOR AC for EnMAP.
* First guess water vapor retrieval is only applied to land pixels if land_only is set to true.
Bugfixes:
* Fixed bug in empirical line function which produced unrealistic peaks in water reflectance spectra.
* Removed infinite values from water vapor first guess map to ensure convergence of Eigenvalues when calculating information content.
* Removed numba jit from hyperspectral LUT interpolation to avoid potential numba related bugs.
* Data arrays from the EnMAP L1B object are now safely copied instead of remaining mutable. This prevents issues with later usages.
0.16.1 (2021-03-24)
--------------------
New features:
* 'make lint' now directly prints errors instead of only logging them to logfiles.
* Automatic retraining of S2 novelty detector in case pretrained scikit-learn random forest model is out of date.
Bugfixes:
* Pinned gdal to version<=3.1.2 to avoid import error.
* Fixed bug in empirical line function, which caused one single remaining unprocessed segmentation label.
* Replaced deprecated gdal imports to fix "DeprecationWarning: gdal.py was placed in a namespace, it is now available as osgeo.gdal".
* Updated cerberus schema for SicorValidator to avoid missing path warning in case of LUT file.
* Updated download link and file size of S2 novelty detector and unpinned scikit-learn version.
0.16.0 (2021-02-23)
--------------------
New features:
* Transformation of VNIR data cube to SWIR sensor geometry to enable accurate segmentation and first guess retrievals.
* Well-arranged separation between EnMAP-specific AC and generic AC.
* Added incorporation of uncertainties due to model unknowns.
* Extended options files with additional parameters:
* Prior mean and standard deviation of state vector parameters
* Standard deviations of model unknowns
* Inversion parameters
* Extended optional output of Optimal Estimation:
* Jacobian of solution state
* Convergence message
* Number of iterations
* Gain matrix
* Averaging kernel matrix
* Value of cost function
* Degrees of freedom
* Information content
* Retrieval noise
* Smoothing error
* Updated first guess retrievals.
Bugfixes:
* Updated keyword for excluding patterns from URL check.
* Fixed bug in LUT file assertion.
* Removed slow inversion method based on downhill simplex algorithm.
* Removed option to turn off ice retrieval.
0.15.6 (2021-02-05)
--------------------
New features:
* Two optional processing modes for EnMAP data: 'land only' and 'land + water' based on water mask.
Bugfixes:
* Fixed bug in LUT file assertion.
* Replaced pandas xlrd dependency by openpyxl.
0.15.5 (2021-01-21)
--------------------
New features:
* Improved handling of clear and cloudy fraction. Additional logger warnings and infos are now printed.
Bugfixes:
* Fixed Qhull error within water vapor retrieval, which occurred while processing extremely cloudy images.
0.15.4 (2021-01-13)
--------------------
New features:
>>>>>>> HISTORY.rst
* Improved consistency in the logging of ECMWF errors within ac_gms().
* Default values and units for multispectral AC are now printed to the logs.
Bugfixes:
* Updated URLs due to changes on the server side.
* Deprecated raise of assertion error in case the LUT file only represents an LFS pointer.
* Fixed "RuntimeWarning: overflow encountered in reduce" within ac_gms().
* Implemented CWV default value for AC of Landsat data in case no ECMWF data are available.
......@@ -103,6 +205,7 @@ Bugfixes:
0.14.3 (2020-09-02)
-------------------
New features:
* The package is now available on the Python Package Index.
* Added 'deploy_pypi' CI job.
......@@ -110,6 +213,7 @@ New features:
0.14.2 (2020-05-14)
-------------------
New features:
* Segmentation of input radiance data cubes to enhance processing speed.
* Empirical line solution for extrapolating reflectance spectra based on segment averages.
......@@ -117,6 +221,7 @@ New features:
0.14.1 (2019-02-18)
-------------------
New features:
* Optimal estimation for atmospheric and surface parameters.
* Calculation of retrieval uncertainties.
......@@ -124,6 +229,7 @@ New features:
0.14.0 (2019-02-11)
-------------------
New features:
* New EnMAP atmospheric correction.
* 3 phases of water retrieval for hyperspectral data.
......
......@@ -92,16 +92,19 @@ clean-test: ## Remove test and coverage artifacts.
rm -rf report.html
lint: ## Check style and pep8 conformity using multiple pep8 and style checkers. For now, flake8, pycodestyle, pylint and pydocstyle are included, but their results ignored.
-flake8 --exclude="*.ipynb,*.ipynb*" --max-line-length=120 sicor tests > ./tests/linting/flake8.log
-pycodestyle sicor --exclude="*.ipynb,*.ipynb*" --max-line-length=120 > ./tests/linting/pycodestyle.log
-pylint --rcfile=tests/linting/pylint.rc sicor > tests/linting/pylint.log
-pydocstyle sicor > ./tests/linting/pydocstyle.log
-flake8 --exclude="*.ipynb,*.ipynb*" --max-line-length=120 sicor tests > ./tests/linting/flake8.log || \
(cat ./tests/linting/flake8.log && exit 1)
-pycodestyle sicor --exclude="*.ipynb,*.ipynb*" --max-line-length=120 > ./tests/linting/pycodestyle.log || \
(cat ./tests/linting/pycodestyle.log && exit 1)
-pylint --rcfile=tests/linting/pylint.rc sicor > tests/linting/pylint.log || \
(cat ./tests/linting/pylint.log && exit 1)
-pydocstyle sicor > ./tests/linting/pydocstyle.log || \
(cat ./tests/linting/pydocstyle.log && exit 1)
urlcheck: ## check for dead URLs
# white-listed: certificate checks fail although URLs work
urlchecker check . \
--file-types .py,.rst,.md,.json \
--white-listed-patterns www.enmap.org,S2_PDI_Level-1C_Tile_Metadata.xsd,S2_PDI_Level-2A_Tile_Metadata.xsd
--exclude-patterns www.enmap.org,S2_PDI_Level-1C_Tile_Metadata.xsd,S2_PDI_Level-2A_Tile_Metadata.xsd
test_enmap: ## Run a single test quickly with the default Python and without coverage. This is useful for debugging errors and feel free to change the considered test case to your liking.
echo ########### ####### ${PYTHONPATH}
......
......@@ -24,20 +24,20 @@ Using Anaconda or Miniconda (recommended)
Using conda_ (latest version recommended), SICOR is installed as follows:
1. Create virtual environment for SICOR (optional but recommended):
Create virtual environment for SICOR (optional but recommended), and install SICOR itself:
.. code-block:: bash
$ conda create -c conda-forge --name sicor python=3
$ conda activate sicor
$ conda create -c conda-forge --name sicor_env sicor
$ conda activate sicor_env
2. Then install SICOR itself:
Alternatively, you can of course install SICOR in an already existing environment by simply running:
.. code-block:: bash
$ conda install -c conda-forge sicor
This is the preferred method to install SICOR, as it will always install the most recent stable release and
conda_ is the preferred method to install SICOR, as it will always install the most recent stable release and
automatically resolve all the dependencies.
......@@ -181,7 +181,7 @@ Usage from python:
.. code-block:: python
from sicor.sicor_enmap import sicor_ac_enmap
enmap_l2a_vnir, enmap_l2a_swir, cwv_model, cwc_model, ice_model, toa_model, se, scem, srem = sicor_ac_enmap(data_l1b, options, logger)
enmap_l2a_vnir, enmap_l2a_swir, res = sicor_ac_enmap(data_l1b, options, logger)
From command line (currently, only applicable to multispectral case):
......
#!/usr/bin/env python
"""Command line program to run sicor for EnMAP Level-1B products."""
import argparse
import logging
from datetime import datetime
from enpt.io.reader import L1B_Reader
import pprint
import sys
import dill
from enpt.io.reader import L1B_Reader
from enpt.options.config import EnPTConfig, config_for_testing_dlr
from sicor import __version__
from sicor.options import get_options
from sicor.sicor_enmap_old import *
from sicor.sicor_enmap import *
def sicor_ac_enmap_parser():
"""Return argument parser for sicor EnMAP"""
parser = argparse.ArgumentParser()
parser.add_argument("-i", "--input", action="store", required=True, type=str,
"""Return argument parser for sicor EnMAP."""
parser = argparse.ArgumentParser(
prog='sicor_ac_EnMAP.py',
description='=' * 70 + '\n' + 'SICOR EnMAP console argument parser. ',
epilog="use '>>> sicor_ac_EnMAP.py -h' for detailed documentation and usage hints.")
parser.add_argument('--version', action='version', version=__version__)
parser.add_argument("-in", "--input", action="store", required=True, type=str,
help="Path to EnMAP Level-1B product (root dir).")
parser.add_argument("-s", "--settings", action="store", required=True, type=str,
help="Path to settings json file. ")
parser.add_argument("-o", "--output", action="store", required=True, type=str,
parser.add_argument("-set", "--settings", action="store", required=True, type=str,
help="Path to retrieval options json file. ")
parser.add_argument("-out", "--output", action="store", required=True, type=str,
help="Path to output directory (will be created if nonexistent).")
parser.add_argument("--snr_vnir", action="store", required=False, default=None)
parser.add_argument("--snr_swir", action="store", required=False, default=None)
parser.add_argument("-q", "--quiet", action="store_true", required=False, default=False)
parser.add_argument("-d", "--debug", action="store_true", required=False, default=False)
parser.add_argument("-un", "--unknowns", action="store", required=False, type=str,
help="True, if uncertainties due to unknown forward model parameters should be added to "
"S_epsilon; default: False")
return parser
if __name__ == "__main__":
args = sicor_ac_enmap_parser().parse_args()
logger = logging.getLogger("SICOR_EnMAP")
logger.setLevel(logging.DEBUG)
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)-10s %(levelname)-10s %(module)-10s - %(funcName)-10s: %(message)s',
datefmt="%H:%M:%S"
)
logger = logging.Logger("EnMAP")
fmt_suffix = None
formatter_ConsoleH = logging.Formatter('%(asctime)s' + (' [%s]' % fmt_suffix if fmt_suffix else '') +
': %(message)s', datefmt='%Y/%m/%d %H:%M:%S')
consoleHandler_out = logging.StreamHandler(stream=sys.stdout)
consoleHandler_out.setFormatter(formatter_ConsoleH)
logger.addHandler(consoleHandler_out)
logger.info("Sicor AC for EnMAP started.")
logger.info("Input = %s" % args.input)
logger.info("Output = %s" % args.output)
logger.info("SNR VNIR = %s" % str(args.snr_vnir))
logger.info("SNR SWIR = %s" % str(args.snr_swir))
logger.info("Settings = %s" % args.settings)
logger.info("Unknowns = %s" % args.unknowns)
if args.quiet is True:
logger.info("Disable progressbar due to --quiet parameter.")
def tqdm_quiet(*args):
"""Quiet version of tqdm status bar."""
return args[0]
tqdm = tqdm_quiet
import sicor.sicor_enmap_old as bf
bf.tqdm = tqdm_quiet
import sicor.AC.RtFo as bf
bf.tqdm = tqdm_quiet
options = get_options(args.settings, validation=False)
logger.info("Loading settings: \n" + pprint.pformat(options))
options = get_options(args.settings)
logger.info("Load settings: \n" + pprint.pformat(options))
# Read EnMAP Level-1B Product from file system
logger.info("Reading EnMAP Level-1B Product from file system...")
config = EnPTConfig(**config_for_testing_dlr)
RD = L1B_Reader(config=config)
L1_obj = RD.read_inputdata(args.input, compute_snr=False)
L1_obj.get_preprocessed_dem()
enmap_l1b = L1_obj
# Load EnMAP Level-1B Product from file system
logger.warning("Observation time missing in metadata") # TODO: Remove this
enmap_l2a_sens_geo, state, fits = sicor_ac_enmap(
enmap_l1b=L1B_Reader(logger=logger).read_inputdata(
root_dir=args.input,
observation_time=datetime(2015, 12, 7, 10),
lon_lat_smpl=options["EnMAP"]['lon_lat_smpl'],
snr_vnir=args.snr_vnir,
snr_swir=args.snr_swir),
options=options, logger=logger, debug=args.debug)
logger.info("Performing atmospheric correction...")
enmap_l2a_vnir, enmap_l2a_swir, res = sicor_ac_enmap(enmap_l1b, options, unknowns=args.unknowns, logger=logger)
out_fn = enmap_l2a_sens_geo.save(
outdir=args.output,
suffix="_surface_reflectance")
logger.info("Writing products to output path...")
with open(args.output, "wb") as fl:
dill.dump([enmap_l2a_vnir, enmap_l2a_swir, res], fl)
logger.info("Wrote product to: %s" % out_fn)
logger.info("EEooFF")
......@@ -41,7 +41,7 @@ def countdown(execute_every, print_update=timedelta(seconds=1), tsleep=1):
def sicor_ecmwf_parser():
"""Return parser for sicor_ecmwf.py script"""
"""Return parser for sicor_ecmwf.py script."""
default_products = [
"fc_T2M",
......
......@@ -2,50 +2,88 @@ Algorithm descriptions
======================
Forward operator
---------------------------------
Overview
********
.. automodule:: sicor.AC.RtFo_3_phases
:members:
:undoc-members:
:show-inheritance:
SICOR is subdivided into two sequential main algorithms: the retrieval of the three phases of water and the calculation
of the surface reflectance. The latter requires the output from the water retrieval to accurately model the level 2a
spectra. The input requires an EnMAP level 1b product generated by the EnPT EnMAP data reader and an options file in
json format. Via the options file, the user is able to set the path to the atmospheric Look-Up-Table (LUT), decide
whether the algorithm should retrieve vapor, liquid and ice or only vapor and liquid, and set the number of cpu's used
for processing. All other relevant parameters are retrieved from the EnMAP product metadata file during the EnPT
reader process.
LUT
-------------------------------
.. automodule:: sicor.Tools.EnMAP.LUT
:members:
:undoc-members:
:show-inheritance:
Conversion
--------------------------------------
.. automodule:: sicor.Tools.EnMAP.conversion
:members:
:undoc-members:
:show-inheritance:
Three Phases of Water Retrieval
*******************************
Metadata
------------------------------------
SICOR uses sufficient narrow bands in the NIR to identify and quantify water in different states. The path lengths of
water vapor, liquid water and ice are simultaneously estimated by applying a physically-based inversion of an
atmospheric radiative transfer model (RTM) which is linked to a well-parameterized surface reflectance model. The latter
incorporates the Beer-Lambert law to express the radiation absorption as a function of the path length of pure liquid
water and ice. While water vapor can be inferred from the RTM simulations, the surface reflectance model enables the
retrieval of the other two phases. The approach is based on the decoupling of the overlapping absorption lines of water
vapor, liquid water and ice. The lines of liquid water and ice are shifted towards longer wavelengths. This
displacement, in combination with the moderate absorption energies enables a spectroscopic separation of the three
phases.
.. automodule:: sicor.Tools.EnMAP.metadata
:members:
:undoc-members:
:show-inheritance:
Multiprocessing
-------------------------------------------
.. automodule:: sicor.Tools.EnMAP.multiprocessing
:members:
:undoc-members:
:show-inheritance:
Optimal Estimation (OE)
-----------------------------------------------
Forward Operator (FO)
*********************
.. automodule:: sicor.Tools.EnMAP.optimal_estimation
:members:
:undoc-members:
:show-inheritance:
Both the retrieval of the three phases of water and the modeling of the surface reflectance are based on the inversion
of a well-parameterized forward model (FO), which models the TOA radiance spectra. The water retrieval uses the FO to
estimate the values of vapor, liquid and ice by minimizing the difference between modeled and measured spectra.
The minimization uses a predefined cost function in an iterative optimization procedure. The modeling of the surface
reflectance applies the FO based on the retrieved combined atmospheric and surface state vector. Besides the path
lengths of the three water phases, the FO additionally requires state vector parameters such as observation geometry,
surface elevation and aerosol optical thickness (AOT). All these additional parameters are obtained from the metadata of
the input EnMAP dataset.
Atmospheric Model
*****************
Assuming clear sky and a plane-parallel atmosphere as well as a Lambertian surface, SICOR models the TOA radiance by a
simplified solution of the radiative transfer equation following the approach of Chandrasekhar (1960). To decrease the
computational burden and to increase the processing speed, the needed atmospheric components were previously calculated
for different atmospheric cases and stored in a multidimensional LUT.
Surface Reflectance Model
*************************
For the three phases of water retrieval, SICOR models the surface reflectance as a linear change in reflectance with
wavelength attenuated by the spectrally dependent absorption for liquid water and ice based on the Beer-Lambert law.
The wavelength dependent absorption coefficients of liquid water and ice are calculated by using the imaginary part of
the complex index of refraction k. To obtain k, SICOR uses the table of Kedenburg et al. (2012) for liquid water and
the values from Warren (1984) for ice.
Inversion
*********
SICOR iteratively adjusts water vapor, liquid water, and ice path lengths to match modeled and measured spectra within
the selected water absorption feature. For EnMAP datasets, the 1140 nm window is applied due to the VNIR and SWIR
detector overlapping around the 940 nm feature. The matching of the spectra is evaluated by a predefined cost function
and at each iteration step, the needed atmospheric parameters are obtained by a multidimensional interpolation within
the LUT. SICOR applies optimal estimation according to Rodgers (2000) to the iteration procedure.
Optimal Estimation
******************
Optimal estimation enables the possibility to incorporate an error vector in terms of measurement, forward model, and
a priori uncertainties.
Development
===========
Gitlab Continuous Integration (CI)
----------------------------------
Gitlab's feature to support basic continuous integration (CI_) is used to perform integration test on each push to
the gitlab repository. The setup is defined in the file `.gitlab-ci.yml`