Commit 00065288 authored by Romulo Pereira Goncalves's avatar Romulo Pereira Goncalves

Merge branch 'dev' into 'master'

New release v1.1.0

See merge request !13
parents 7f66bf05 3d2297d6
Pipeline #13045 passed with stages
in 18 minutes and 27 seconds
......@@ -7,6 +7,7 @@ test_gts2_client:
script:
- source ~/anaconda3/bin/activate
- export GDAL_DATA=/home/gitlab-runner/anaconda3/share/gdal
- export PROJ_LIB=/root/anaconda3/share/proj
- make coverage
artifacts:
paths:
......@@ -22,4 +23,4 @@ pages:
artifacts:
paths:
- public
expire_in: 600 days
\ No newline at end of file
expire_in: 600 days
## Release notes
* **2020-09-29:** New scipy, add CHANGELOG.md, fix issues #24, #25, #29 and switched to AC version 0.14. Make this release 1.1.0.
* **2020-07-20:** Added timeout parameter to avoid freezing of client and changed versioning, making this version 1.0.0
* **2019-05-23:** Switched to AC version 0.13
* **2018-11-13:** Changed default port to 443
* **2018-02-20:** Improved output to netCDF file
* **2018-02-16:** Added installation information and installation script for miniconda and all needed packages
* **2018-01-18:** Added docker file and script for building an compatible image and running a container for the client
* **2018-01-17:** Added option for mosaicing/merging tifs and RGBs on client side
* **2018-01-10:** Added output of RGB images into jpg or png, nc-output still in progress
......@@ -8,16 +8,6 @@ Status
[Coverage report](http://gts2.gitext.gfz-potsdam.de/gts2_client/coverage/)
## Release notes
* **2020-07-20:** Added timeout parameter to avoid freezing of client and changed versioning, making this version 1.0.0
* **2019-05-23:** Switched to AC version 0.13
* **2018-11-13:** Changed default port to 443
* **2018-02-20:** Improved output to netCDF file
* **2018-02-16:** Added installation information and installation script for miniconda and all needed packages
* **2018-01-18:** Added docker file and script for building an compatible image and running a container for the client
* **2018-01-17:** Added option for mosaicing/merging tifs and RGBs on client side
* **2018-01-10:** Added output of RGB images into jpg or png, nc-output still in progress
## Description
This program package downloads Sentinel-2 data from the GTS2
(GFZ Time Series System for Sentinel-2) cloud for a area of interest,
......@@ -26,15 +16,8 @@ time of interest and wanted band combination and saves them to geotiff (.tiff)
## Requirements
1. Access to GTS2 API (username, password, port)
1. Python3
1. Python packages:
* numpy
* gdal
* scipy
* json
* netCDF4
* requests
* scikit-image
2. Python3
3. Python packages - check the requirements in [setup.py](./setup.py)
## Download package:
Clone the repository with:
......@@ -100,7 +83,7 @@ The building of the image can take some time (up to 30 Minutes), but once it is
### As command line tool:
```bash
gts2_client.py required_arguments [optional_arguments]
gts2_client required_arguments [optional_arguments]
```
The list of arguments including their default values can be called from the command line with:
......@@ -136,7 +119,7 @@ The list of arguments including their default values can be called from the comm
* -t LEVEL, --level LEVEL
processing level (e.g. L2A)
* -v VERSION, --version VERSION
version of atmospheric correction (e.g. 0.12)
version of atmospheric correction (e.g. 0.14 - recommended version)
* -c COREG, --coreg COREG
get coregistrated data with corrected pixel shifts (e.g. True)
[pixel shifts between time steps were eliminated by coregistrating
......
......@@ -4,7 +4,8 @@ context_dir="./context"
dockerfile="gts2_client.docker"
runner_os="centos"
runner_iname="gts2_client_runner"
runner_tag="${runner_os}:${runner_iname}"
runner_version="latest"
runner_tag="${runner_os}_${runner_iname}:${runner_version}"
container_name="gts2_client"
cred_file="$HOME/credentials_gts2_client"
out_data_folder="/tmp/gts2_client"
......@@ -43,4 +44,4 @@ mkdir -p ${out_data_folder}
echo "Starting gts2_container, please write files to ${out_data_folder} (setting the -o option of client accordingly)"
sudo docker run -it --name ${container_name} -v ${out_data_folder}:${out_data_folder} ${runner_tag} \
bash -i -c "cd /home/gts2_client; git pull origin master; source ~/anaconda3/bin/activate; python setup.py install;
echo "";echo "";echo "";echo "";echo '#######';echo 'This is a shell were you can run the gts2_client :::';echo '#######'; bash"
\ No newline at end of file
echo "";echo "";echo "";echo "";echo '#######';echo 'This is a shell were you can run the gts2_client :::';echo '#######'; bash"
FROM centos:7
RUN yum update -y && \
yum install -y wget vim bzip2 git
ENV anaconda_dl='Anaconda3-5.0.1-Linux-x86_64.sh'
yum install -y wget vim bzip2 gcc gcc-c++ make libgl1-mesa-glx mesa-libGL qt5-qtbase-gui git texlive
ENV anaconda_dl='Anaconda3-2019.07-Linux-x86_64.sh'
RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/$anaconda_dl && \
bash ./$anaconda_dl -b && \
rm -f /root/$anaconda_dl"
RUN /bin/bash -i -c "source ~/anaconda3/bin/activate && \
conda install --yes -q -c conda-forge gdal 'icu=58.*' lxml pyqt && \
pip install netCDF4 && \
conda update -q --all"
conda config --add channels conda-forge && \
conda install --yes -c conda-forge proj gdal=3.1.0 lxml pyqt netcdf4 coverage libiconv imageio"
RUN /bin/bash -i -c "cd /home/ && \
git clone https://gitext.gfz-potsdam.de/gts2/gts2_client.git"
COPY credentials_gts2_client /root/credentials_gts2_client
\ No newline at end of file
COPY credentials_gts2_client /root/credentials_gts2_client
......@@ -23,7 +23,7 @@ from os.path import join
from os.path import expanduser
from scipy.ndimage.interpolation import zoom
from skimage.exposure import rescale_intensity, adjust_gamma
from scipy.misc import imsave
from imageio import imwrite
from subprocess import Popen, PIPE
band_settings = {"realistic": ("B04", "B03", "B02"),
......@@ -361,7 +361,7 @@ def mk_rgb(basedir, outdir, rgb_comb=("B04", "B03", "B02"), rgb_gamma=(1.0, 1.0,
del data
del ds
imsave(fn_out, S2_rgb)
imwrite(fn_out, S2_rgb)
fnout_list.append(fn_out)
return fnout_list
......@@ -615,9 +615,9 @@ def json_to_netcdf(out_mode, api_result, outpath, out_prefix, geo_ll, geo_ur, st
for attr in api_result[key][tile][band][data_info].keys()]
if data_info == 'data':
band_arr = np.asarray(api_result[key][tile][band][data_info])
band_group.createDimension('x', band_arr.shape[2])
band_group.createDimension('x', band_arr.shape[0])
band_group.createDimension('y', band_arr.shape[1])
band_group.createDimension('t', band_arr.shape[0])
band_group.createDimension('t', band_arr.shape[2])
fill = api_result[key][tile][band]['fill_value']
data = band_group.createVariable('Data', 'i4', ('x', 'y', 't'), fill_value=fill)
data.units = "None"
......@@ -666,9 +666,9 @@ def json_to_netcdf(out_mode, api_result, outpath, out_prefix, geo_ll, geo_ur, st
api_result[key][tile][msk][data_info]['time'])
if data_info == 'data':
mask_arr = np.asarray(api_result[key][tile][msk][data_info])
mask_group.createDimension('x', mask_arr.shape[2])
mask_group.createDimension('x', mask_arr.shape[0])
mask_group.createDimension('y', mask_arr.shape[1])
mask_group.createDimension('t', mask_arr.shape[0])
mask_group.createDimension('t', mask_arr.shape[2])
fill = api_result[key][tile][msk]['fill_value']
data = mask_group.createVariable('Data', 'i4', ('x', 'y', 't'), fill_value=fill)
data.units = "None"
......
......@@ -5,11 +5,12 @@ dependencies:
- scipy
- requests
- scikit-image
- gdal
- proj
- gdal=3.1.0
- ipython
- pip
- libssh2
- pip:
- netcdf4
- netcdf4
- imageio
from setuptools import setup, find_packages
from importlib import util
requirements = ["numpy", "scipy", "netCDF4", "requests", "scikit-image", "gdal"]
requirements = ["proj", "numpy", "imageio", "scipy", "netCDF4", "requests", "scikit-image", "gdal"]
other_requirements = ["gdal"]
test_requirements = requirements + ["coverage"]
......@@ -17,14 +17,18 @@ if not_installed != []:
', '.join(not_installed)))
setup(name='gts2_client',
version='1.0.0',
version='1.1.0',
packages=find_packages(exclude=['tests*']),
url='https://gitext.gfz-potsdam.de/gts2/gts2_client.git',
license='GNU LESSER GENERAL PUBLIC LICENSE - Version 3, 29 June 2007',
author='Hannes Diedrich, Niklas Bohn, Andre Hollstein',
author_email='hannes.diedrich@gfz-potsdam.de',
description='Downloads Sentinel-2 data from GTS2 cloud',
scripts=['gts2_client/gts2_client.py'],
entry_points={
'console_scripts': [
'gts2_client = gts2_client.gts2_client:main'
],
},
install_requires=requirements,
test_suite='tests',
tests_require=test_requirements
......
......@@ -2,7 +2,8 @@
context_dir="./context"
dockerfile="build_runner_image.docker"
runner_tag="gts2_client_ci"
runner_version="latest"
runner_tag="gts2_client_ci:$runner_version"
gitlab_runner="gts2_client_gitlab_CI_runner"
echo "Preparing gts2 credentials file"
......@@ -37,5 +38,6 @@ sudo docker exec -it ${gitlab_runner} /bin/bash -c "export RUNNER_EXECUTOR=docke
--run-untagged=true \
--locked=true \
--tag-list gts2_client \
--docker-pull-policy='never' \
--description '${runner_name}' \
--docker-image '${runner_tag}:latest' "
--docker-image '${runner_tag}' "
FROM centos:7
RUN yum update -y && \
yum install -y wget vim bzip2 gcc gcc-c++ make libgl1-mesa-glx mesa-libGL qt5-qtbase-gui git texlive
RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/Anaconda3-4.3.1-Linux-x86_64.sh && \
bash ./Anaconda3-4.3.1-Linux-x86_64.sh -b && \
rm -f /root/Anaconda3-4.3.1-Linux-x86_64.sh"
ENV anaconda_dl='Anaconda3-2019.07-Linux-x86_64.sh'
RUN /bin/bash -i -c "wget https://repo.continuum.io/archive/$anaconda_dl && \
bash ./$anaconda_dl -b && \
rm -f /root/$anaconda_dl"
RUN /bin/bash -i -c "source ~/anaconda3/bin/activate && \
conda install --yes -c conda-forge gdal 'icu=58.*' lxml pyqt coverage && \
pip install netCDF4"
conda config --add channels conda-forge && \
conda install --yes -c conda-forge proj gdal=3.1.0 lxml pyqt netcdf4 coverage libiconv imageio"
RUN mkdir -p /home/tmp/
COPY credentials_gts2_client /root/
\ No newline at end of file
COPY credentials_gts2_client /root/
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment