Installation

1.) Install Java 8 (Ubuntu)

sudo add-apt-repository ppa:webupd8team/java
sudo apt update
sudo apt install oracle-java8-*

2.) Install Maven 3.x

sudo apt install maven

3.) Clone GMS-Meta

git clone --recursive https://gitext.gfz-potsdam.de/geomultisens/gms-meta.git

4.) Build Flink

cd pyflink
mvn clean install -DskipTests
cp -r flink-dist/target/flink-1.1-SNAPSHOT-bin/flink-1.1-SNAPSHOT /geoms/code/

shutdown flink

bin/stop-local.sh

TODO: host specific flink settings? (conf/flink-config.yml)

5.) install Anaconda

cd /opt/
wget wget https://repo.continuum.io/archive/Anaconda3-5.0.0-Linux-x86_64.sh
bash Anaconda3-5.0.0-Linux-x86_64.sh
source /opt/anaconda3/bin/activate

6.) install Python dependencies

conda install --yes pyqt numba  
conda install --yes -c conda-forge pyfftw=0.10.4  
conda install --yes -c ioam holoviews bokeh  
conda install --yes -c conda-forge numpy scikit-image scikit-learn matplotlib pyproj rasterio shapely basemap pykrige glymur pygrib pyproj cachetools pyhdf ephem python-fmask scipy  
conda install --yes -c conda-forge gdal
conda install --yes -c conda-forge 'icu=58.*' lxml  
pip install pandas geopandas dicttoxml jsmin cerberus pyprind pint iso8601 tqdm mpld3 sphinx-argparse dill pytz spectral>0.16 psycopg2 pyorbital pyinstrument geoalchemy2 sqlalchemy py_tools_ds>=0.9.1 geoarray>=0.6.12 arosics>=0.6.2 flake8 pycodestyle pylint pydocstyle nose nose2 nose-htmloutput coverage rednose six xattr

• install sicor (## move to gms-meta/install.sh)

sudo apt install curl
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash  
sudo apt-get install git-lfs  
git clone https://gitext.gfz-potsdam.de/EnMAP/sicor.git
cd sicor
make install  # this might fail if the machine has not enough RAM -> use 'pip --no-cache-dir install .' instead.


bash gms-meta/install.sh

7.) Setup the database

• simply execute the install_database.sh script from the gms-meta root folder, this will install all needed dependencies and import a lightweigt database dump

sudo ./install_database.sh

• after the database is up and running you might want to update the config table according to your environment

8.) Setup GMS Pipeline controller, downloader and WebUI

• install gms-aux containing all dependencies for gms-modules

cd gms-aux
mvn clean install -DskipTests

the generated jars will be found in the target folder of each submodule and copied to the local maven repo folder, e.g. ~/.m2/

• build gms-modules (make sure gms-aux is installed, see step above)

cd gms-modules
mvn clean package -DskipTests

• configure the controller via properties files gms-controller.properties and SceneDatabase.properties (have to be stored in the same folder as the executable gms-controller.jar, e.g. gms-modules/gms-controller/target/)

Templates for both properties files can be found in the resources folder of the gms-controller project (gms-modules/gms-controller/src/main/resources/)

A minimum gms-controller.properties for local execution of all modules looks as follows:

    # gms-controller default props file

    # global host definition used for all modules
    module.host = localhost
 
    # url (protocol, host and port) providing the flink REST API, default: http://localhost:8081
    flinkapi.url = http://localhost:8081

    # log level for this module, must be one of ("trace", "debug", "info", "warn", or "error"), default if omitted: info
    module.log.level = info 

The SceneDatabase.properties contains the connection details, like host, port, etc. for the database und might look like follows:

    # host the gms database is running, default: geoms.gfz-potsdam.de
    db.host = localhost
    # port the gms database is listening to, default: 5432
    db.port = 5432
    # name of the database, default: geomultisens
    db.name = geomultisens
    # database user, default: gmsdb
    db.user = gmsdb
    # database password for the above user, default: gmsdb
    db.pass = gmsdb
    # host the index mediator server is running on, default: value of db.host 
    index.host = localhost
    # port the index mediator server is listening to, default: 8654
    index.port = 8654

• start gms-controller

cd gms-modules/gms-controller/target
java -jar gms-controller.jar &

• start gms-downloader

cd gms-modules/gms-downloader/target
java -jar gms-downloader.jar &

• start gms-sceneprocessor

cd gms-modules/gms-sceneprocessor/target
java -jar gms-proc.jar &

8.) install and setup Jetty + deploy WebApp within Jetty

Deployment

Once the gms-modules or the gms-vis in particular is build and packaged by maven a gms.war web-archive is generated. This archive has to be deployed via a webserver supporting java servlet container like jetty (http://www.eclipse.org/jetty/).

Setup Jetty

Tested with jetty version (9.4.3), so any 9.4.x should do fine, but no guaranties for versions >9.4.

General howto: http://www.eclipse.org/jetty/documentation/9.4.x/quickstart-running-jetty.html

My step by step:

1. Download (http://www.eclipse.org/jetty/download.html) and extract jetty to an arbitrary folder (e.g. /opt/jetty/jetty-distribution-9.4.3/)
2. Create a softlink to this folder (e.g. /opt/jetty/current) pointing to the jetty-distribution folder
3. Create the jetty base folder (e.g. /opt/jetty/base)
4. Set the JETTY_HOME and JETTY_BASE environment variables in /etc/default/jetty to the linked folder and the base folder (in case the service refuses to start later on, it might help to also set the JETTY_START_LOG to e.g. /opt/jetty/base/logs/start.log)
5. init the jetty base folder (depending on your access rights you might need to prepend 'sudo') cd $JETTY_BASE java -jar $JETTY_HOME/start.jar --add-to-start=http,deploy,resources,ext,jmx,requestlog,console-capture,annotations,jsp,logging-slf4j,slf4j-simple-impl,websocket

Deploy gms-vis and start Jetty

1. deploy the gms-vis module by putting the generated gms-vis/target/gms.war into the $JETTY_BASE/webapps/ folder
2. deploy the gms-vis/src/main/resources/gms-vis.properties into the $JETTY_BASE/resources folder
3. deploy the gms-aux/gms-database/src/main/resources/SceneDatabase.properties into the $JETTY_BASE/resources folder
4. update the corresponding fields in the above mentioned property files
5. start jetty via the included startup script under $JETTY_HOME/bin/jetty.sh or via java -jar $JETTY_HOME/start.jar (for the latter make sure the environment variables are properly exported, defining them in /etc/default/jetty might not be sufficient!))

Updating Jetty

1. Simply download and extract the latest distribution package.
2. Shutdown any running jetty instance
3. Update the Softlink (e.g. /opt/jetty/current) point the new distribution folder
4. No need to touch any of the depolyed webapps under $JETTY_BASE
5. Start jetty via mentioned jar or system service

Web-Client

URL Parameters

parameter	description
job	enables the submission of download, processing and analysis jobs
job=jobid	attaches the web client to the job with the given jobid, "last" is also a valid value, attaching to the latest attached job of this client
allDatasets	shows all datasets available in the database, not only those that are declared "enabled" by the database

Script to setup jetty

• see https://gitext.gfz-potsdam.de/geomultisens/gms-meta/blob/master/ci/gms-tutorial/dockerfile

wget http://central.maven.org/maven2/org/eclipse/jetty/jetty-distribution/9.4.7.v20170914/jetty-distribution-9.4.7.v20170914.tar.gz
tar xvf jetty-distribution-9.4.7.v20170914.tar.gz

=> to be completed!

start jetty

opt/jetty/current/bin$ ./jetty.sh start