Installation
1.) Install Java 8 (Ubuntu)
sudo add-apt-repository ppa:webupd8team/java
sudo apt update
sudo apt install oracle-java8-*
2.) Install Maven 3.x
sudo apt install maven
3.) Clone GMS-Meta
git clone --recursive https://gitext.gfz-potsdam.de/geomultisens/gms-meta.git
4.) Build Flink
cd pyflink
mvn clean install -DskipTests
cp -r flink-dist/target/flink-1.1-SNAPSHOT-bin/flink-1.1-SNAPSHOT /geoms/code/
shutdown flink
bin/stop-local.sh
TODO: host specific flink settings? (conf/flink-config.yml)
5.) install Anaconda
cd /opt/
wget wget https://repo.continuum.io/archive/Anaconda3-5.0.0-Linux-x86_64.sh
bash Anaconda3-5.0.0-Linux-x86_64.sh
source /opt/anaconda3/bin/activate
6.) install Python dependencies
conda install --yes pyqt numba
conda install --yes -c conda-forge pyfftw=0.10.4
conda install --yes -c ioam holoviews bokeh
conda install --yes -c conda-forge numpy scikit-image scikit-learn matplotlib pyproj rasterio shapely basemap pykrige glymur pygrib pyproj cachetools pyhdf ephem python-fmask scipy
conda install --yes -c conda-forge gdal
conda install --yes -c conda-forge 'icu=58.*' lxml
pip install pandas geopandas dicttoxml jsmin cerberus pyprind pint iso8601 tqdm mpld3 sphinx-argparse dill pytz spectral>0.16 psycopg2 pyorbital pyinstrument geoalchemy2 sqlalchemy py_tools_ds>=0.9.1 geoarray>=0.6.12 arosics>=0.6.2 flake8 pycodestyle pylint pydocstyle nose nose2 nose-htmloutput coverage rednose six xattr
- install sicor (## move to gms-meta/install.sh)
sudo apt install curl
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfs
git clone https://gitext.gfz-potsdam.de/EnMAP/sicor.git
cd sicor
make install # this might fail if the machine has not enough RAM -> use 'pip --no-cache-dir install .' instead.
bash gms-meta/install.sh
7.) Setup the database
- simply execute the
install_database.sh
script from the gms-meta root folder, this will install all needed dependencies and import a lightweigt database dump
sudo ./install_database.sh
- after the database is up and running you might want to update the
config
table according to your environment
8.) Setup GMS Pipeline controller, downloader and WebUI
- install gms-aux containing all dependencies for
gms-modules
cd gms-aux
mvn clean install -DskipTests
the generated jars will be found in the target
folder of each submodule and copied to the local maven repo folder, e.g. ~/.m2/
- build gms-modules (make sure
gms-aux
is installed, see step above)
cd gms-modules
mvn clean package -DskipTests
- configure the controller via properties files
gms-controller.properties
andSceneDatabase.properties
(have to be stored in the same folder as the executablegms-controller.jar
, e.g.gms-modules/gms-controller/target/
)
Templates for both properties files can be found in the resources folder of the gms-controller project (gms-modules/gms-controller/src/main/resources/
)
A minimum gms-controller.properties
for local execution of all modules looks as follows:
# gms-controller default props file
# global host definition used for all modules
module.host = localhost
# url (protocol, host and port) providing the flink REST API, default: http://localhost:8081
flinkapi.url = http://localhost:8081
# log level for this module, must be one of ("trace", "debug", "info", "warn", or "error"), default if omitted: info
module.log.level = info
The SceneDatabase.properties
contains the connection details, like host, port, etc. for the database und might look like follows:
# host the gms database is running, default: geoms.gfz-potsdam.de
db.host = localhost
# port the gms database is listening to, default: 5432
db.port = 5432
# name of the database, default: geomultisens
db.name = geomultisens
# database user, default: gmsdb
db.user = gmsdb
# database password for the above user, default: gmsdb
db.pass = gmsdb
# host the index mediator server is running on, default: value of db.host
index.host = localhost
# port the index mediator server is listening to, default: 8654
index.port = 8654
- start gms-controller
cd gms-modules/gms-controller/target
java -jar gms-controller.jar &
- start gms-downloader
cd gms-modules/gms-downloader/target
java -jar gms-downloader.jar &
- start gms-sceneprocessor
cd gms-modules/gms-sceneprocessor/target
java -jar gms-proc.jar &
8.) install and setup Jetty + deploy WebApp within Jetty
Deployment
Once the gms-modules or the gms-vis in particular is build and packaged by maven a gms.war web-archive is generated. This archive has to be deployed via a webserver supporting java servlet container like jetty (http://www.eclipse.org/jetty/).
Setup Jetty
Tested with jetty version (9.4.3), so any 9.4.x should do fine, but no guaranties for versions >9.4.
General howto: http://www.eclipse.org/jetty/documentation/9.4.x/quickstart-running-jetty.html
My step by step:
- Download (http://www.eclipse.org/jetty/download.html) and extract jetty to an arbitrary folder (e.g.
/opt/jetty/jetty-distribution-9.4.3/
) - Create a softlink to this folder (e.g.
/opt/jetty/current
) pointing to the jetty-distribution folder - Create the jetty base folder (e.g.
/opt/jetty/base
) - Set the JETTY_HOME and JETTY_BASE environment variables in
/etc/default/jetty
to the linked folder and the base folder (in case the service refuses to start later on, it might help to also set the JETTY_START_LOG to e.g./opt/jetty/base/logs/start.log
) - init the jetty base folder (depending on your access rights you might need to prepend 'sudo')
cd $JETTY_BASE java -jar $JETTY_HOME/start.jar --add-to-start=http,deploy,resources,ext,jmx,requestlog,console-capture,annotations,jsp,logging-slf4j,slf4j-simple-impl,websocket
Deploy gms-vis and start Jetty
- deploy the gms-vis module by putting the generated
gms-vis/target/gms.war
into the$JETTY_BASE/webapps/
folder - deploy the
gms-vis/src/main/resources/gms-vis.properties
into the$JETTY_BASE/resources
folder - deploy the
gms-aux/gms-database/src/main/resources/SceneDatabase.properties
into the$JETTY_BASE/resources
folder - update the corresponding fields in the above mentioned property files
- start jetty via the included startup script under
$JETTY_HOME/bin/jetty.sh
or viajava -jar $JETTY_HOME/start.jar
(for the latter make sure the environment variables are properly exported, defining them in/etc/default/jetty
might not be sufficient!))
Updating Jetty
- Simply download and extract the latest distribution package.
- Shutdown any running jetty instance
- Update the Softlink (e.g.
/opt/jetty/current
) point the new distribution folder - No need to touch any of the depolyed webapps under $JETTY_BASE
- Start jetty via mentioned jar or system service
Web-Client
URL Parameters
parameter | description |
---|---|
job | enables the submission of download, processing and analysis jobs |
job=jobid | attaches the web client to the job with the given jobid, "last" is also a valid value, attaching to the latest attached job of this client |
allDatasets | shows all datasets available in the database, not only those that are declared "enabled" by the database |
Script to setup jetty
wget http://central.maven.org/maven2/org/eclipse/jetty/jetty-distribution/9.4.7.v20170914/jetty-distribution-9.4.7.v20170914.tar.gz
tar xvf jetty-distribution-9.4.7.v20170914.tar.gz
=> to be completed!
start jetty
opt/jetty/current/bin$ ./jetty.sh start