Commit 1a7537a6 authored by Hannes Diedrich's avatar Hannes Diedrich
Browse files

Merge branch 'feature/merge_tifs' into 'master'

Feature/merge tifs

See merge request !6
parents ddd78468 38433daa
Pipeline #2603 canceled with stages
in 5 minutes and 46 seconds
......@@ -7,6 +7,9 @@ Status
[Coverage report](http://gts2.gitext.gfz-potsdam.de/gts2_client/coverage/)
## Release notes
* **2018-02-16:** Added installation information and installation script for miniconda and all needed packages
* **2018-01-18:** Added docker file and script for building an compatible image and running a container for the client
* **2018-01-17:** Added option for mosaicing/merging tifs and RGBs on client side
* **2018-01-10:** Added output of RGB images into jpg or png, nc-output still in progress
## Description
......@@ -16,8 +19,8 @@ time of interest and wanted band combination and saves them to geotiff (.tiff)
or alternatively as .json file or as netcdf (.nc) file.
## Requirements
1. Access to GTS2 API (username and password)
1. Python 3
1. Access to GTS2 API (username, password, port)
1. Python3
1. Python packages:
* numpy
* gdal
......@@ -33,21 +36,55 @@ Clone the repository with:
</code>
## Installation
1. Make sure your system meets all the Requirements (see above)
* For GFZ users: you can achieve that by using the gfz python.
For that please run: <code>module load pygfz</code>
2. Install the package by first going into the repository folder and then run:
<code> python gts2_client/setup.py install </code>
3. Create a credentials file in your home directory **credentials_gts2_client**
that contains the following structure:
### ... is easy (as long your python is compatible):
* Make sure your system meets all the requirements (see above)
* Install the package by running:
`python gts2_client/setup.py install`
### Install a compatible python:
The following instruction is only valid for Linux distributions but can be adapted on Windows machines.
Everything is based on bash shell.
#### EITHER: Use install script and follow instructions:
Run: `bash install_py_gts2_client.sh`
After providing the installation path the script will install miniconda
and all necessary packages as well as the gts2_client.
At the script will provide instructions which PATHs to add to your .bashrc.
#### OR: Per hand (expert modus):
* Download and install at least Miniconda with default settings:
```
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod 755 Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh -b
rm -f Miniconda3-latest-Linux-x86_64.sh
```
* Install necessary packages for gts2_client using **gts2_client_conda_install.yml**:
`conda env update -f gts2_client_conda_install.yml`
### Finally:
Create a credentials file in your home directory **credentials_gts2_client**
that contains the following structure:
```bash
{"user": "",
"password": ""}
"password": ""
"port": ""}
```
Please fill in your credentials (provided by GFZ Potsdam, please contact gts2@gfz-potsdam.de).
### If you want to use docker
**!! Attention: Expert mode !! root access to computer is needed.**
You only have to run **docker_gts2_client/build_run_gts2_client_docker.sh**
It creates an image that is based on a Centos:7, performs all needed installation steps and starts a container.
You end up with a shell where you can run `gts2_client`.
The building of the image can take some time (up to 30 Minutes), but once it is done,
**docker_gts2_client/build_run_gts2_client_docker.sh** skips the build and only starts the container.
## Usage
### As command line tool:
......@@ -56,7 +93,7 @@ gts2_client.py required_arguments [optional_arguments]
```
The list of arguments including their default values can be called from the command line with:
<code>gts2_client --help</code>
`gts2_client --help`
#### Arguments:
##### Required:
......@@ -108,8 +145,14 @@ The list of arguments including their default values can be called from the comm
* -q RGB_BANDS_SELECTION, --rgb_bands_selection RGB_BANDS_SELECTION
band selection for rgb production, choose from:
realistic, nice_looking, vegetation,
healthy_vegetation_urban, water, snow, agriculture
healthy_vegetation_urban, snow, agriculture
(default: realistic)
* -w MERGE_TIFS, --merge_tifs MERGE_TIFS
Merge tifs and RGBs if area in two or more MGRS tiles
per time step (True or False). (default: False)
* -x MERGE_TILE, --merge_tile MERGE_TILE
Choose MGRS tile into which the merge of files is
performed (e.g. 33UUV). (default: None)
### As python package
......@@ -122,3 +165,6 @@ result = gts2_client.client(out_mode="python", **kwargs)
## Limitations:
* Bands with spatial resolution 60m (Band 1) are not stacked.
* Requests with areas larger than 0.2°x0.2° will probably not be processed
if you also request a large time range this threshold can also be smaller
{
"user": "",
"password": ""
"password": "",
"port": 80
}
\ No newline at end of file
#!/usr/bin/env bash
context_dir="./context"
dockerfile="gts2_client.docker"
runner_os="centos"
runner_iname="gts2_client_runner"
runner_tag="${runner_os}:${runner_iname}"
container_name="gts2_client"
cred_file="$HOME/credentials_gts2_client"
out_data_folder="/tmp/gts2_client"
#Check if image exists
if [ $(sudo docker images | grep ${runner_iname} | wc -l) == 0 ]
then
# Copying credentials file into context dir, if exists
if [ -e ${cred_file} ]
then
echo "cp ${cred_file} ./context/"
cp ${cred_file} ./context/
else
echo "read"
read -p "Please enter GTS2 username: " username
read -p "Please enter GTS2 password: " password
read -p "Please enter GTS2 port (default 80): " port
cat > ${cred_file} <<EOL
{
"user": "${username}",
"password": "${password}",
"port": ${port}
}
EOL
fi
# build docker image
sudo docker build -f ${context_dir}/${dockerfile} -m 20G -t ${runner_tag} ${context_dir}
fi
#run container and start a shell where you can run the gts2_client
sudo docker rm -f ${container_name}
mkdir -p ${out_data_folder}
echo "Starting gts2_container, please write files to ${out_data_folder} (setting the -o option of client accordingly)"
sudo docker run -it --name ${container_name} -v ${out_data_folder}:${out_data_folder} ${runner_tag} \
bash -i -c "cd /home/gts2_client; git pull origin master; source ~/anaconda3/bin/activate; python setup.py install;
echo "";echo "";echo "";echo "";echo '#######';echo 'This is a shell were you can run the gts2_client :::';echo '#######'; bash"
\ No newline at end of file
FROM centos:7
RUN yum update -y && \
yum install -y wget vim bzip2 git
ENV anaconda_dl='Anaconda3-5.0.1-Linux-x86_64.sh'
RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/$anaconda_dl && \
bash ./$anaconda_dl -b && \
rm -f /root/$anaconda_dl"
RUN /bin/bash -i -c "source ~/anaconda3/bin/activate && \
conda install --yes -q -c conda-forge gdal 'icu=58.*' lxml pyqt && \
pip install netCDF4 && \
conda update -q --all"
RUN /bin/bash -i -c "cd /home/ && \
git clone https://gitext.gfz-potsdam.de/gts2/gts2_client.git"
COPY credentials_gts2_client /root/credentials_gts2_client
\ No newline at end of file
This diff is collapsed.
name: root
channels:
- conda-forge
dependencies:
- scipy
- requests
- scikit-image
- gdal
- ipython
- pip
- libssh2
- pip:
- netcdf4
#!/usr/bin/env bash
echo "###################################"
echo "This is the installation routine for installing Miniconda and all necessary packages for gts2_client."
echo "Depending on the load of your machine, this process can take up to an hour, so lean back and relax!"
echo "All files are stored into your installation path which you will provide in the following."
echo "Please make sure that you have sufficient free space in ths installation path (at least 3 GB)."
echo "###################################"
current_path=$(pwd)
read -p "Please enter path for the python installation: " inst_path
cd ${inst_path}
conda_vers="miniconda3"
url="https://repo.continuum.io/miniconda/"
conda_inst_name="Miniconda3-latest-Linux-x86_64.sh"
echo "#### Downloading ${conda_vers}.... ####"
wget ${url}${conda_inst_name}
chmod 755 ${conda_inst_name}
echo "#### Installing ${conda_vers}.... ####"
./${conda_inst_name} -p ${inst_path}/${conda_vers} -b
rm -f ${conda_inst_name}
echo "... Ready"
env_name=gts2_client
echo "Creating a conda environment named: ${env_name} ..."
export PATH=${inst_path}/${conda_vers}/bin/:$PATH
export PYTHONPATH=${inst_path}/${conda_vers}/lib/python3.6/site-packages/
echo "... Ready"
echo "Installing necessary packages ..."
conda env update -f ${current_path}/gts2_client_conda_install.yml
echo "... Ready"
echo "Installing gts2_client ..."
cd ${current_path}
python setup.py install
echo "... Ready"
echo "###############################"
echo "Congratulations! Miniconda and all necessary packages are installed"
echo " "
echo "###############################"
echo "In order to load your environment and run gts2_client, run the following commands in your bash shell:"
echo "###############################"
echo "### 1. ###"
echo "Set the paths right (you can also add the following lines to your $HOME/.bashrc [recommended]) :"
echo "--------"
echo "export PATH=${inst_path}/${conda_vers}/bin/:$PATH"
echo "export PYTHONPATH=${inst_path}/${conda_vers}/lib/python3.6/site-packages/:$PYTHONPATH"
echo "export LD_LIBRARY_PATH=${inst_path}/${conda_vers}/lib"
echo "--------"
echo "### 2. ###"
echo "Activate your conda environment:"
echo "source activate ${env_name}"
echo " "
echo "... Now you can use the gts2_client ..."
echo " "
from setuptools import setup, find_packages
from importlib import util
requirements = ["numpy", "scipy", "netCDF4", "requests", "scikit-image"]
requirements = ["numpy", "scipy", "netCDF4", "requests", "scikit-image", "gdal"]
other_requirements = ["gdal"]
test_requirements = requirements + ["coverage"]
......@@ -17,7 +17,7 @@ if not_installed != []:
', '.join(not_installed)))
setup(name='gts2_client',
version='0.4',
version='0.5',
packages=find_packages(exclude=['tests*']),
url='https://gitext.gfz-potsdam.de/gts2/gts2_client.git',
license='GNU LESSER GENERAL PUBLIC LICENSE - Version 3, 29 June 2007',
......@@ -28,4 +28,4 @@ setup(name='gts2_client',
install_requires=requirements,
test_suite='tests',
tests_require=test_requirements
)
\ No newline at end of file
)
......@@ -62,11 +62,44 @@ class TestGts2Client(unittest.TestCase):
minimum_fill=minimum_fill,
stack_resolution=stack_res)
list_of_files = glob(os.path.join(tmpdir, "*{mode}*.{form}".format(form=out_formats[ii],
mode=out_mode)))
list_of_files = glob(os.path.join(tmpdir, "*.{form}".format(form=out_formats[ii])))
if len(list_of_files) == 0:
raise FileNotFoundError("Could not find files matching: *{mode}*.{form}".format(
form=out_formats[ii], mode=out_mode))
raise FileNotFoundError("Could not find files matching: *.{form}".format(form=out_formats[ii]))
else:
print("Test OK.")
def test_gts2_client_merge(self):
"""
Test the merge tifs and rgbs if different tiles are retrieved.
:return:
"""
out_modes = ["single", "stack", "rgb", "stack"]
out_formats = ["tif", "tif", "jpg", "tif"]
levels = ["L2A", "L2A", "L2A", "L1C"]
merge_tile = ["32UQD", "32UQD", None, None]
for ii, out_mode in enumerate(out_modes):
with tempfile.TemporaryDirectory() as tmpdir:
gts2_client.client(outpath=tmpdir,
out_prefix="krh",
out_mode=out_mode,
geo_ll=(12.559433, 53.036066),
geo_ur=(12.737961, 53.238058),
sensor="S2A",
level=levels[ii],
version="0.12",
bands="B04_B03_B02",
max_cloudy="0.2",
suffix="",
start_date="20170501",
end_date="20170530",
minimum_fill="0.9",
merge_tifs=True,
merge_tile=merge_tile[ii])
list_of_files = glob(os.path.join(tmpdir, "*.{form}".format(form=out_formats[ii])))
if len(list_of_files) == 0:
raise FileNotFoundError("Could not find files matching: *.{form}".format(form=out_formats[ii]))
else:
print("Test OK.")
......@@ -162,6 +195,45 @@ class TestGts2Client(unittest.TestCase):
:return:
"""
try:
out_mode = "stack"
print("#### Testing '_' in prefix ")
gts2_client.client(
out_prefix="test_",
out_mode=out_mode,
geo_ll=geo_ll,
geo_ur=geo_ur,
bands="B05",
start_date=start_date,
end_date=end_date,
version=version,
level=level,
max_cloudy=max_cloudy,
minimum_fill=minimum_fill,
stack_resolution=stack_res,
quiet=False)
except ValueError:
print("Test OK.")
try:
out_mode = "stack"
print("#### Testing too large area")
gts2_client.client(
out_mode=out_mode,
geo_ll=geo_ll,
geo_ur=(12.737961+0.4, 53.238058+0.4),
bands="B05",
start_date=start_date,
end_date=end_date,
version=version,
level=level,
max_cloudy=max_cloudy,
minimum_fill=minimum_fill,
stack_resolution=stack_res,
quiet=False)
except ValueError:
print("Test OK.")
try:
out_mode = "python"
print("#### Testing stack_resolution=60")
......@@ -201,7 +273,7 @@ class TestGts2Client(unittest.TestCase):
print("Test OK.")
try:
out_mode = "dowsnotwork"
out_mode = "doesnotwork"
print("#### Testing wrong out_mode")
gts2_client.client(
out_mode=out_mode,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment