Commit 3ac7fb66 authored by Daniel Scheffler's avatar Daniel Scheffler
Browse files

Merge branch 'feature/acwater_polymer' into 'master'

Added/revised parameters related to AC mode and ACwater.

See merge request !15
parents 102e6b34 20136389
Pipeline #24520 failed with stages
in 1 minute and 33 seconds
......@@ -2,6 +2,16 @@
History
=======
0.6.0 (2021-06-16)
------------------
* Added parameters related to three new AC modes in EnPT and ACwater.
* Revised descriptions and titles all over the GUI.
* Revised 'optional' flags.
* Improved connection of the QGIS feedback object to EnPT STDOUT and STDERR stream to fix missing log messages on Linux.
* Updated GUI screenshots and installation.rst.
0.5.0 (2021-06-04)
------------------
......
......@@ -29,9 +29,9 @@ See also the latest coverage_ report and the nosetests_ HTML report.
How the GUI looks like
----------------------
.. image:: ./docs/images/screenshot_enpt_enmapboxapp_874x1267.png
:width: 874 px
:height: 1267 px
.. image:: https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/enpt_enmapboxapp/raw/master/docs/images/screenshot_enpt_enmapboxapp_v0.6.0.png
:width: 1046 px
:height: 876 px
:scale: 100 %
......
......@@ -8,9 +8,9 @@ QGIS_ plugin to process imaging spectroscopy data and particularly the hyperspec
of QGIS_. It allows to parameterize EnPT_ and run the processing chain to generate EnMAP_ Level-2A from Level-1B data.
.. image:: ./images/screenshot_enpt_enmapboxapp_874x1267.png
:width: 874 px
:height: 1267 px
.. image:: ./images/screenshot_enpt_enmapboxapp_v0.6.0.png
:width: 1046 px
:height: 876 px
:scale: 100 %
.. _EnPT: https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT
......
enpt\_enmapboxapp package
=========================
Submodules
----------
enpt\_enmapboxapp.enpt\_enmapboxapp module
------------------------------------------
.. automodule:: enpt_enmapboxapp.enpt_enmapboxapp
:members:
:undoc-members:
:show-inheritance:
Module contents
---------------
.. automodule:: enpt_enmapboxapp
:members:
:undoc-members:
:show-inheritance:
EnPT EnMAP-Box app documentation
EnPT-EnMAP-Box-App documentation
================================
......
......@@ -4,25 +4,32 @@
Installation
============
Since enpt_enmapboxapp extends the functionality of the EnMAP-Box_, the installation of enpt_enmapboxapp requires
QGIS_ including the EnMAP-Box plugin. If QGIS_ or the EnMAP-Box_ is not yet installed on your system,
Since enpt_enmapboxapp (the EnPT GUI package) extends the functionality of the EnMAP-Box_, the installation of
enpt_enmapboxapp requires QGIS_ including the EnMAP-Box plugin. If QGIS_ or the EnMAP-Box_ is not yet installed on your system,
follow the installation instructions
`here <https://enmap-box.readthedocs.io/en/latest/usr_section/usr_installation.html>`__.
`here <https://enmap-box.readthedocs.io/en/latest/usr_section/usr_installation.html>`__. This should also automatically
install the enpt_enmapboxapp package into the QGIS_ Python environment.
The enpt_enmapboxapp package is then installed into the QGIS_ Python environment:
However, make sure you have the latest version of the GUI installed by running the following command on a command line
with the QGIS Python environment activated:
.. code-block:: console
pip install enpt_enmapboxapp
pip install --upgrade enpt_enmapboxapp
.. note::
On a Windows system, you have to activate the the QGIS environment first. To do so, run the OSGeo4W Shell (listed
under OSGeo4W in the Start Menu) as administrator and enter ``call py3_env.bat``.
To make the enpt_enmapboxapp GUI run together with EnPT_ (backend), EnPT_ has to be installed into a separate Anaconda
environment. Please refer to the EnPT_ installation instructions
`here <https://enmap.git-pages.gfz-potsdam.de/GFZ_Tools_EnMAP_BOX/EnPT/doc/installation.html>`__.
On a Windows system and in case your QGIS is NOT installed into a Conda Python environment but via the OSGeo4W
installer, you have to activate the the QGIS environment first. To do so, run the OSGeo4W Shell (listed under
OSGeo4W in the Start Menu) as administrator and enter :code:`call py3_env.bat`.
To make the enpt_enmapboxapp (GUI code) run together with EnPT_ (backend), the EnPT_ backend code has to be installed.
The easiest way (recommended) is to install QGIS, the EnMAP-Box and EnPT into a single Conda environment. Please
refer to the EnPT_ installation instructions
`here <https://enmap.git-pages.gfz-potsdam.de/GFZ_Tools_EnMAP_BOX/EnPT/doc/installation.html>`__ to do so. However,
if QGIS is already installed as a standalone-software which does not use a Conda Python environment in the background
(e.g., installed via the OSGeo4W installer on Windows), the EnPT backend code has to be installed into a separate
Conda environment that is called 'enpt' and then linked to the QGIS Python using the EnPT GUI.
.. _EnPT: https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT
.. _EnMAP-Box: https://www.enmap.org/data_tools/enmapbox/
......
API Reference
=============
.. toctree::
:maxdepth: 4
enpt_enmapboxapp
......@@ -3,10 +3,10 @@ Usage
=====
The enpt_enmapboxapp GUI can be found in
:menuselection:`QGIS 3.xx --> EnMAP-Box --> Processing Toolbox --> Pre-Processing --> EnMAP processing tool algorithm`
:menuselection:`QGIS 3.xx --> EnMAP-Box --> Processing Toolbox --> Pre-Processing --> EnPT - EnMAP Processing Tool`
.. image:: ./images/sceenshot_how_to_start_1342x1029.png
.. image:: ./images/screenshot_how_to_start.png
:scale: 75 %
Use the GUI to parameterize EnPT_ according to your input dataset and the desired processing settings.
......@@ -15,10 +15,11 @@ The input parameters are explained
.. hint::
Before you start the processing chain, make sure, that the Anaconda root directory is correctly set. It should
If your QGIS is not installed into a Conda Python environment but installed as a standalone software, make sure,
that the Anaconda root directory is correctly set before you start the processing chain. It should
point to the Anaconda installation containing the previously installed `enpt` Python environment.
The status of computation can watched in the log tab and the QGIS Python console.
The status of computation can be watched in the log tab and the QGIS Python console.
.. _EnPT: https://git.gfz-potsdam.de/EnMAP/GFZ_Tools_EnMAP_BOX/EnPT
......@@ -27,6 +27,7 @@
import os
from os.path import expanduser
import psutil
from pkgutil import get_loader
from datetime import date
from multiprocessing import cpu_count
from threading import Thread
......@@ -70,6 +71,11 @@ class _EnPTBaseAlgorithm(QgsProcessingAlgorithm):
P_enable_vnir_swir_coreg = 'enable_vnir_swir_coreg'
P_path_reference_image = 'path_reference_image'
P_enable_ac = 'enable_ac'
P_mode_ac = 'mode_ac'
P_polymer_root = 'polymer_root'
P_threads = 'threads'
P_blocksize = 'blocksize'
P_auto_download_ecmwf = 'auto_download_ecmwf'
P_scale_factor_boa_ref = 'scale_factor_boa_ref'
P_run_smile_P = 'run_smile_P'
P_run_deadpix_P = 'run_deadpix_P'
......@@ -97,11 +103,20 @@ class _EnPTBaseAlgorithm(QgsProcessingAlgorithm):
return 'EnPTAlgorithm'
def displayName(self):
return 'EnMAP processing tool algorithm (v%s)' % __version__
return f'EnPT - EnMAP Processing Tool (v{__version__})'
def createInstance(self, *args, **kwargs):
return type(self)()
@staticmethod
def _get_default_polymer_root():
try:
path_polymer = os.path.abspath(
os.path.join(os.path.dirname(get_loader("polymer").path), os.pardir))
except AttributeError:
path_polymer = ''
return path_polymer
@staticmethod
def _get_default_output_dir():
userhomedir = expanduser('~')
......@@ -148,8 +163,7 @@ class _EnPTBaseAlgorithm(QgsProcessingAlgorithm):
name=self.P_CPUs,
description='Number of CPU cores to be used for processing',
type=QgsProcessingParameterNumber.Integer,
defaultValue=cpu_count(), minValue=0, maxValue=cpu_count(),
optional=True),
defaultValue=cpu_count(), minValue=0, maxValue=cpu_count()),
advanced=True)
self.addParameter(
......@@ -178,8 +192,7 @@ class _EnPTBaseAlgorithm(QgsProcessingAlgorithm):
description='Average elevation in meters above sea level \n'
'(may be provided if no DEM is available and ignored if DEM is given)',
type=QgsProcessingParameterNumber.Integer,
defaultValue=0,
optional=True),
defaultValue=0),
advanced=True)
self.addParameter(
......@@ -252,7 +265,7 @@ class _EnPTBaseAlgorithm(QgsProcessingAlgorithm):
name=self.P_scale_factor_toa_ref,
description='Scale factor to be applied to TOA reflectance result',
type=QgsProcessingParameterNumber.Integer,
defaultValue=10000),
defaultValue=10000, minValue=0),
advanced=True)
self.addParameter(
......@@ -277,15 +290,58 @@ class _EnPTBaseAlgorithm(QgsProcessingAlgorithm):
self.addParameter(
QgsProcessingParameterBoolean(
name=self.P_enable_ac,
description='Enable atmospheric correction using SICOR algorithm',
description='Enable atmospheric correction',
defaultValue=True))
self.addParameter(QgsProcessingParameterNumber(
name=self.P_scale_factor_boa_ref,
description='Scale factor to be applied to BOA reflectance result',
type=QgsProcessingParameterNumber.Integer,
defaultValue=10000,
optional=True),
self.addParameter(
QgsProcessingParameterEnum(
name=self.P_mode_ac,
description="Atmospheric correction mode",
options=['land - SICOR is applied to land AND water',
'water - POLYMER is applied to water only; land is cleared ',
'combined - SICOR is applied to land and POLYMER to water'],
defaultValue=2))
self.addParameter(
QgsProcessingParameterFile(
name=self.P_polymer_root,
description='Polymer root directory (that contains the subdirectory for ancillary data)',
behavior=QgsProcessingParameterFile.Folder,
defaultValue=self._get_default_polymer_root(),
optional=True),
advanced=True)
self.addParameter(
QgsProcessingParameterNumber(
name=self.P_threads,
description='Number of threads for multiprocessing when running ACwater/Polymer \n'
"('0: no threads', '-1: automatic', '>0: number of threads')",
type=QgsProcessingParameterNumber.Integer,
defaultValue=-1, minValue=-1, maxValue=cpu_count()),
advanced=True)
self.addParameter(
QgsProcessingParameterNumber(
name=self.P_blocksize,
description='Block size for multiprocessing when running ACwater/Polymer',
type=QgsProcessingParameterNumber.Integer,
defaultValue=100, minValue=1),
advanced=True)
self.addParameter(
QgsProcessingParameterBoolean(
name=self.P_auto_download_ecmwf,
description='Automatically download ECMWF data for atmospheric correction '
'of water surfaces in ACwater/Polymer',
defaultValue=False),
advanced=True)
self.addParameter(
QgsProcessingParameterNumber(
name=self.P_scale_factor_boa_ref,
description='Scale factor to be applied to BOA reflectance result',
type=QgsProcessingParameterNumber.Integer,
defaultValue=10000, minValue=0),
advanced=True)
self.addParameter(
......@@ -377,10 +433,13 @@ class _EnPTBaseAlgorithm(QgsProcessingAlgorithm):
text = \
'<p>General information about this EnMAP box app can be found ' \
'<a href="https://enmap.git-pages.gfz-potsdam.de/GFZ_Tools_EnMAP_BOX/enpt_enmapboxapp/doc/">here</a>' \
'.</p>' \
'<p>Type <i>enpt -h</i> into a shell to get further information about individual ' \
'parameters.</p>'
'<a href="https://enmap.git-pages.gfz-potsdam.de/GFZ_Tools_EnMAP_BOX/enpt_enmapboxapp/doc/">here</a>. ' \
'For details, e.g., about all the algorithms implemented in EnPT, take a look at the ' \
'<a href="https://enmap.git-pages.gfz-potsdam.de/GFZ_Tools_EnMAP_BOX/EnPT/doc/index.html">EnPT backend ' \
'documentation</a>.</p>' \
'<p>Type <i>enpt -h</i> into a shell to get further information about individual parameters or check out ' \
'the <a href="https://enmap.git-pages.gfz-potsdam.de/GFZ_Tools_EnMAP_BOX/EnPT/doc/usage.html#' \
'command-line-utilities">documentation</a>.</p>'
return text
......@@ -412,6 +471,9 @@ class _EnPTBaseAlgorithm(QgsProcessingAlgorithm):
Thread(target=reader, args=[process.stdout, q]).start()
Thread(target=reader, args=[process.stderr, q]).start()
stdout_qname = None
stderr_qname = None
# for _ in range(2):
for source, line in iter(q.get, None):
if qgis_feedback.isCanceled():
......@@ -426,9 +488,18 @@ class _EnPTBaseAlgorithm(QgsProcessingAlgorithm):
linestr = line.decode('latin-1').rstrip()
# print("%s: %s" % (source, linestr))
if source.name == 3:
# source name seems to be platfor/environment specific, so grab it from dummy STDOUT/STDERR messages.
if linestr == 'Connecting to EnPT STDOUT stream.':
stdout_qname = source.name
continue
if linestr == 'Connecting to EnPT STDERR stream.':
stderr_qname = source.name
continue
if source.name == stdout_qname:
qgis_feedback.pushInfo(linestr)
if source.name == 4:
if source.name == stderr_qname:
qgis_feedback.reportError(linestr)
exitcode = process.poll()
......
......@@ -41,6 +41,7 @@ class EnPTAlgorithm(_EnPTBaseAlgorithm):
@staticmethod
def _prepare_enpt_environment() -> dict:
os.environ['PYTHONUNBUFFERED'] = '1'
os.environ['IS_ENPT_GUI_CALL'] = '1'
enpt_env = os.environ.copy()
enpt_env["PATH"] = ';'.join([i for i in enpt_env["PATH"].split(';') if 'OSGEO' not in i]) # actually not needed
......@@ -75,9 +76,10 @@ class EnPTAlgorithm(_EnPTBaseAlgorithm):
raise ImportError("enpt", "EnPT must be installed into the QGIS Python environment "
"when calling 'EnPTAlgorithm'.")
# replace Enum parameters with corresponding strings
# replace Enum parameters with corresponding strings (not needed in case of unittest)
for n, opts in [
('output_format', {0: 'GTiff', 1: 'ENVI'}),
('mode_ac', {0: 'land', 1: 'water', 2: 'combined'}),
('deadpix_P_algorithm', {0: 'spectral', 1: 'spatial'}),
('deadpix_P_interp_spectral', {0: 'linear', 1: 'bilinear', 2: 'cubic', 3: 'spline'}),
('deadpix_P_interp_spatial', {0: 'linear', 1: 'bilinear', 2: 'cubic', 3: 'spline'}),
......@@ -85,10 +87,8 @@ class EnPTAlgorithm(_EnPTBaseAlgorithm):
('vswir_overlap_algorithm', {0: 'order_by_wvl', 1: 'average', 2: 'vnir_only', 3: 'swir_only'}),
('target_projection_type', {0: 'UTM', 1: 'Geographic'}),
]:
parameters[n] = opts[parameters[n]]
feedback.pushInfo("The log messages of the EnMAP processing tool are written to the *.log file "
"in the specified output folder.")
if isinstance(parameters[n], int):
parameters[n] = opts[parameters[n]]
# remove all parameters not to be forwarded to the EnPT CLI
parameters = {k: v for k, v in parameters.items()
......
......@@ -118,6 +118,7 @@ class ExternalEnPTAlgorithm(_EnPTBaseAlgorithm):
@staticmethod
def _prepare_enpt_environment() -> dict:
os.environ['PYTHONUNBUFFERED'] = '1'
os.environ['IS_ENPT_GUI_CALL'] = '1'
enpt_env = os.environ.copy()
enpt_env["PATH"] = ';'.join([i for i in enpt_env["PATH"].split(';') if 'OSGEO' not in i]) # actually not needed
......@@ -170,9 +171,10 @@ class ExternalEnPTAlgorithm(_EnPTBaseAlgorithm):
parameters = {k: v for k, v in parameters.items()
if k not in ['anaconda_root']}
# replace Enum parameters with corresponding strings
# replace Enum parameters with corresponding strings (not needed in case of unittest)
for n, opts in [
('output_format', {0: 'GTiff', 1: 'ENVI'}),
('mode_ac', {0: 'land', 1: 'water', 2: 'combined'}),
('deadpix_P_algorithm', {0: 'spectral', 1: 'spatial'}),
('deadpix_P_interp_spectral', {0: 'linear', 1: 'bilinear', 2: 'cubic', 3: 'spline'}),
('deadpix_P_interp_spatial', {0: 'linear', 1: 'bilinear', 2: 'cubic', 3: 'spline'}),
......@@ -180,7 +182,8 @@ class ExternalEnPTAlgorithm(_EnPTBaseAlgorithm):
('vswir_overlap_algorithm', {0: 'order_by_wvl', 1: 'average', 2: 'vnir_only', 3: 'swir_only'}),
('target_projection_type', {0: 'UTM', 1: 'Geographic'}),
]:
parameters[n] = opts[parameters[n]]
if isinstance(parameters[n], int):
parameters[n] = opts[parameters[n]]
# print parameters and console call to log
# for key in sorted(parameters):
......
......@@ -22,6 +22,6 @@
# You should have received a copy of the GNU Lesser General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
__version__ = '0.5.0'
__versionalias__ = '20210604.01'
__version__ = '0.6.0'
__versionalias__ = '20210615.01'
__author__ = 'Daniel Scheffler'
......@@ -55,11 +55,10 @@ setup(
author="Daniel Scheffler",
author_email='danschef@gfz-potsdam.de',
classifiers=[
'Development Status :: 2 - Pre-Alpha',
'Intended Audience :: Developers',
'Development Status :: 4 - Beta',
'Intended Audience :: End Users/Desktop',
'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)',
'Natural Language :: English',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
......
......@@ -61,6 +61,7 @@ enpt_test_parameters = dict(
enable_vnir_swir_coreg=False,
json_config=None,
n_lines_to_append=NULL,
mode_ac='combined',
ortho_resampAlg='bilinear',
vswir_overlap_algorithm='vnir_only',
output_dir='TEMPORARY_OUTPUT',
......@@ -76,6 +77,7 @@ enpt_test_parameters = dict(
scale_factor_boa_ref=10000,
scale_factor_toa_ref=10000,
sicor_cache_dir=None,
target_projection_type='UTM',
working_dir=None)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment