Commit 04ae3684 authored by Romulo Pereira Goncalves's avatar Romulo Pereira Goncalves
Browse files

Merge branch 'improve_readme' into 'documentation'

Improve the main README and re-structure the repository.

See merge request !9
parents 8c8f9579 2b859f43
Pipeline #23834 failed with stage
in 15 seconds
......@@ -23,8 +23,8 @@ pages: # this job must be called 'pages' to advise GitLab to upload content to
- mkdir -p public/images/
# Copy over the docs
- cp -r docs/_build/html/* public/doc/
- cp -r docs/images/* public/images//
- cp -r docs/*.html public/doc/
- cp -r docs/images/* public/images/
# Check if everything is working great
- ls -al public
......
......@@ -11,9 +11,9 @@ Description: Calculates samples and related classifiers for mapping gradual prob
License: GPL-3
Imports:
BH (<= 1.69.0-1),
sf (<= 0.8-1),
sp (<= 1.4-1),
rgdal (<= 1.4-8),
sf (<= 0.9-0),
sp (<= 1.4-4),
rgdal (<= 1.5-12),
raster,
geojsonio,
maptools,
......
......@@ -2,8 +2,8 @@
#'
#' Clips a raster object
#'
#' @param raster
#' @param shape
#' @param raster raster object
#' @param shape shape object
#'
#' @return a raster object
#' @export
......
......@@ -16,13 +16,13 @@ plot_results <- function(inPath, color = NULL) {
# Compares the number of .kmz to that of .tif. If .tif files are missing the
# user's attention is drawn.
if (length(list.files(
outPath,
inPath,
pattern = ".tif$",
all.files = FALSE,
include.dirs = TRUE,
no.. = TRUE
)) != length(list.files(
outPath,
inPath,
pattern = ".kmz$",
all.files = FALSE,
include.dirs = TRUE,
......
......@@ -7,7 +7,9 @@
clip(raster, shape)
}
\arguments{
\item{shape}{}
\item{raster}{raster object}
\item{shape}{shape object}
}
\value{
a raster object
......
.. figure:: GitDocs/Logo.png
.. figure:: docs/images/Logo.png
:target: https://github.com/carstennh/HabitatSampler/tree/master/demo
:align: center
......@@ -6,56 +6,65 @@
Procedure on Autonomous Sampling and Reductive Learning in Imagery
==================================================================================================
How to use
----------------
1. Stepwise Procedure
----------------------------------
* You need R to run the master script: **HabitatSampler.r**
* Within the master script a step by step procedure is executed, more info in **HabitatSampler.md**.
* All necessary information is available under the directory: `demo <https://github.com/carstennh/HabitatSampler/tree/master/demo>`__
--------------
1. R package
-----------------------
* You need R to install the **HaSa package** that includes all functions and demo data.
* The installation steps for R version **4.1.0** are defined in `Section 2.0 of the documentation <https://git.gfz-potsdam.de/habitat-sampler/HabitatSampler/-/blob/master/docs/HabitatSampler.md#2-hasa-installation>`_.
2. R package
--------------------
* You need R to install the **package HaSa** that includes all functions and test data
* devtools::install_github("carstennh/HabitatSampler", subdir="R-package", build_vignettes = TRUE)
* Sometimes there are problems, then do **1.** devtools:: install_version("velox", version = "0.2.0", repos = "https://cran.uni-muenster.de/")
* For Windows operating systems the `Rtools <https://cran.r-project.org/bin/windows/Rtools/>`__ are needed
* For Ubuntu systems the following system packages dependencies need to be installed:
.. code-block::
apt-get install -y libjq-dev protobuf-compiler libprotobuf-dev proj-bin gdal-bin libgdal-dev jq libv8-dev pandoc
* For Windows operating systems the `Rtools <https://cran.r-project.org/bin/windows/Rtools/>`_ are needed
* library(HaSa) and list datasets: data(package="HaSa") and functions: lsf.str("package:HaSa") or use library(help="HaSa")
* there are information available about programm execution and function behavior in Rmarkdown: `HabitatSampler_Usage <https://github.com/carstennh/HabitatSampler/tree/master/R-package/vignettes>`__
* Information about program execution and function behavior is available in Rmarkdown: `HabitatSampler_Usage <https://git.gfz-potsdam.de/habitat-sampler/HabitatSampler/-/blob/master/R-package/vignettes/Habitat_Sampler_Usage.Rmd>`_
2. Stepwise Procedure
----------------------------------
* The **demo** directory provides a step-wised procedure via an R script: **HabitatSampler.r**, but also via a Jupyter notebook **HabitatSampler.ipynb**.
* All necessary data and information is available under the directory: `demo <https://github.com/carstennh/HabitatSampler/tree/master/demo>`_
* For documentation please check the `docs <https://github.com/carstennh/HabitatSampler/tree/master/docs>`_ directory.
Input
----------------
* **Image File as Raster Layer Stack** (e.g. Satellite Time Series, RGB Drone, Orthophoto)
* **Reference File** (e.g. spectral-temporal profiles or point shape; one profile or point per category)
* **Class Names** (the categories that are defined to be delineated in imagery)
* **Reference File** (e.g. spectral-temporal profiles or point shape; one profile or point per class)
* **Class Names** (the classes that are defined to be delineated in imagery)
Output
----------------
* **Interactive Maps** of habitat type probailities
* **Interactive Maps** of class type probabilities
.. image:: docs/images/figure_1.jpg
:width: 700px
.. image:: GitDocs/figure_1.png
* **Classified Image** of chosen categories
* **Sample Distribution** of sampled categories
* **Spatial Statistics** of categories distribution
* the categories are refferred to as habitat types
* **Classified Image** of chosen class
* **Sample Distribution** of sampled class
* **Spatial Statistics** of class distribution
* the classes are referred to as class types
.. image:: GitDocs/figure_2.png
.. image:: docs/images/figure_2.jpg
:width: 450px
Key Features
----------------
* the algorithm provides a set of **reference samples** for each habitat type
* the algorithm provides an ensemble of calibrated **machine learning classifiers** for each habitat type
* the algorithm provides a map of **habitat type probabilities**
* the algorithm is optimzed for broad-scale **satellite image** time series (pixel size > 10m)
* the alogrthm can be applied on **variable image categories** in complex scenes
* the algorithm is tranferable to **variable input imagery**
* the algorithm provides a set of **reference samples** for each class type
* the algorithm provides an ensemble of calibrated **machine learning classifiers** for each class type
* the algorithm provides a map of **class type probabilities**
* the algorithm is optimized for broad-scale **satellite image** time series (pixel size > 10m)
* the algorithm can be applied on **variable image classes** in complex scenes
* the algorithm is transferable to **variable input imagery**
Citation
----------------
Neumann, C. (2020): Habitat sampler—A sampling algorithm for habitat type delineation in remote sensing imagery. - Diversity and Distributions, 26 (12), 1752-1766. `<https://doi.org/10.1111/ddi.13165>`__.
Neumann, C. (2020): Habitat sampler—A sampling algorithm for class type delineation in remote sensing imagery. - Diversity and Distributions, 26 (12), 1752-1766. `<https://doi.org/10.1111/ddi.13165>`__.
Credits
----------------
......@@ -63,9 +72,4 @@ Credits
HaSa was developed by Carsten Neumann (Helmholtz Centre Potsdam GFZ German Research Centre for Geosciences) within the context of the
`NaTec - KRH <http://www.heather-conservation-technology.com/>`__ project funded by the German Federal Ministry of Education and Research (BMBF) (grant number: 01 LC 1602A).
The test data represent pre-processed Copernicus Sentinel-2 satellite imagery (ESA 2018). Pre-processing was done using `GTS2 <https://www.gfz-potsdam.de/en/section/remote-sensing-and-geoinformatics/projects/closed-projects/gts2/>`__ and `AROSICS <https://github.com/GFZ/arosics>`__.
Community version
-----------------
HaSa will be further developed under a community version located at `GitLab's habitat-sampler group <https://git.gfz-potsdam.de/habitat-sampler/HabitatSampler>`__.
The test data represent pre-processed Copernicus Sentinel-2 satellite imagery (ESA 2018). Pre-processing was done using `GTS2 <https://gitext.gfz-potsdam.de/gts2>`__ and `AROSICS <https://gitext.gfz-potsdam.de/danschef/arosics>`__.
......@@ -8,11 +8,11 @@
]
},
{
"cell_type": "markdown",
"metadata": {},
 
 
"## **0** - Setup"
]
},
{
"cell_type": "markdown",
......@@ -20,14 +20,15 @@
"source": [
"### **0.0** - Pin the version of certain dependencies\n",
"\n",
"The `velox` library is not anymore developed. Its latest version does not compile with the latest version of `Boost`. To avoid warnings and issues with the migration of `rgdal` to `gdal 3 and proj 6` we decided to pin the versions of `sf`, `sp` and `rgdal`."
]
},
{
"cell_type": "code",
 
 
 
 
"outputs": [],
"source": [
"install_dependencies <- TRUE"
]
},
......@@ -52,11 +53,12 @@
"(as ‘lib’ is unspecified)\n",
"\n",
"Installing package into ‘/home/romulo/R/x86_64-pc-linux-gnu-library/4.1’\n",
"(as ‘lib’ is unspecified)\n",
"\n"
"\n"
]
}
],
"source": [
"if (install_dependencies == TRUE) {\n",
" install.packages(\"remotes\")\n",
" install.packages(\"https://cran.r-project.org/src/contrib/Archive/BH/BH_1.69.0-1.tar.gz\", repos=NULL, type=\"source\")\n",
......
......@@ -26,12 +26,14 @@
#####
##0.0##
install.packages("remotes")
install.packages("https://cran.r-project.org/src/contrib/Archive/BH/BH_1.69.0-1.tar.gz", repos=NULL, type="source")
install.packages("https://cran.r-project.org/src/contrib/Archive/sf/sf_0.8-1.tar.gz", repos=NULL, type="source")
install.packages("https://cran.r-project.org/src/contrib/Archive/sp/sp_1.4-1.tar.gz", repos=NULL, type="source")
install.packages("https://cran.r-project.org/src/contrib/Archive/rgdal/rgdal_1.4-8.tar.gz", repos=NULL, type="source")
install.packages("https://cran.r-project.org/src/contrib/Archive/sf/sf_0.9-0.tar.gz", repos=NULL, type="source")
install.packages("https://cran.r-project.org/src/contrib/Archive/sp/sp_1.4-4.tar.gz", repos=NULL, type="source")
install.packages("https://cran.r-project.org/src/contrib/Archive/rgdal/rgdal_1.5-12.tar.gz", repos=NULL, type="source")
##0.1##
library(remotes)
remotes::install_git(
"https://git.gfz-potsdam.de/habitat-sampler/HabitatSampler.git",
ref = "master",
......
......@@ -1579,7 +1579,7 @@
"mimetype": "text/x-r-source",
"name": "R",
"pygments_lexer": "r",
"version": "3.6.3"
"version": "4.1.0"
}
},
"nbformat": 4,
......
This diff is collapsed.
# # Marie little example
# ## Configuration
getwd()
inPath<-"./../R-package/R/"
dataPath <- "/home/romulo/gitlab/marie/data/HaSa/"
outPath <- "./Data/Results/" #has to be an absolute path!
# ## Load Libraries
install_dependencies <- FALSE
if (install_dependencies == TRUE) {
install.packages("glcm")
install.packages("dplyr")
}
options("rgdal_show_exportToProj4_warnings"="none")
from_sources <- TRUE
if (from_sources == TRUE) {
source(paste(inPath,"Class_Habitat.r",sep=""))
source(paste(inPath,"clip.r",sep=""))
source(paste(inPath,"inner_procedure.r",sep=""))
source(paste(inPath,"outer_procedure.r",sep=""))
source(paste(inPath,"model_opt.r",sep=""))
source(paste(inPath,"plot_results.r",sep=""))
source(paste(inPath,"plot_interactive.r",sep=""))
libraries <- c("sp", "sf", "rgdal","raster","maptools","spatialEco","randomForest","e1071","devtools","velox","rgeos","leaflet","htmlwidgets", "IRdisplay", "glcm")
} else {
libraries <- c("sp", "sf", "rgdal","raster","maptools","spatialEco","randomForest","e1071","devtools","velox","rgeos","leaflet","htmlwidgets", "IRdisplay", "glcm", "HaSa")
}
lapply(libraries, library, character.only = TRUE)
# ### Support functions for UTM
library(dplyr)
library(sp)
library(rgdal)
library(tibble)
find_UTM_zone <- function(longitude, latitude) {
# Special zones for Svalbard and Norway
if (latitude >= 72.0 && latitude < 84.0 )
if (longitude >= 0.0 && longitude < 9.0)
return(31);
if (longitude >= 9.0 && longitude < 21.0)
return(33)
if (longitude >= 21.0 && longitude < 33.0)
return(35)
if (longitude >= 33.0 && longitude < 42.0)
return(37)
(floor((longitude + 180) / 6) %% 60) + 1
}
find_UTM_hemisphere <- function(latitude) {
ifelse(latitude > 0, "north", "south")
}
# returns a DF containing the UTM values, the zone and the hemisphere
longlat_to_UTM <- function(long, lat, units = 'm') {
df <- data.frame(
id = seq_along(long),
x = long,
y = lat
)
sp::coordinates(df) <- c("x", "y")
hemisphere <- find_UTM_hemisphere(lat)
zone <- find_UTM_zone(long, lat)
sp::proj4string(df) <- sp::CRS("+init=epsg:4326")
CRSstring <- paste0(
"+proj=utm +zone=", zone,
" +ellps=WGS84",
" +", hemisphere,
" +units=", units)
if (dplyr::n_distinct(CRSstring) > 1L)
stop("multiple zone/hemisphere detected")
res <- sp::spTransform(df, sp::CRS(CRSstring[1L])) %>%
tibble::as_data_frame() %>%
dplyr::mutate(
zone = zone,
hemisphere = hemisphere
)
res
}
UTM_to_longlat <- function(utm_df, zone, hemisphere) {
CRSstring <- paste0("+proj=utm +zone=", zone, " +", hemisphere)
utmcoor <- sp::SpatialPoints(utm_df, proj4string = sp::CRS(CRSstring))
longlatcoor <- sp::spTransform(utmcoor, sp::CRS("+init=epsg:4326"))
tibble::as_data_frame(longlatcoor)
}
# ## Load data
# ### Load Timeseries stack
load_utm <- TRUE
if (load_utm == TRUE) {
# This file already was cropped and projected to UTM
a1 <- brick(paste(dataPath, "20200921_I_filt1_utm.tif", sep = ""))
} else {
a1 <- brick(paste(dataPath,"20200921_I_filt1.tif", sep = ""))
proj4string(a1) <- '+proj=longlat +zone=33 +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +no_defs'
a1 = a1[[-4]] #remove 4th channel, no variance (all =255)
a1 <- raster::projectRaster(a1, crs = "+proj=utm +zone=33 +datum=WGS84 +units=m +no_defs +ellps=WGS84 +towgs84=0,0,0", method = "ngb")
}
#outfile <- raster::writeRaster(a1, filename='20200921_I_filt1_utm.tif', format="GTiff", overwrite=TRUE,options=c("INTERLEAVE=BAND","COMPRESS=LZW"))
a1
# ### Load Reference data
cut <- readOGR(paste(dataPath,"test_site_I_hasa_klein.shp", sep = ""))
proj4string(cut) <- '+proj=longlat +zone=33 +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0'
cut <- sp::spTransform(cut, "+proj=utm +zone=33 +datum=WGS84 +units=m +no_defs +ellps=WGS84 +towgs84=0,0,0")
a1 <- clip(a1, cut)
#proj4string(a1) <- proj4string(cut)
# ### Texture
texture <- glcm(a1[[2]], window = c(5, 5), shift = c(1, 1), statistics = c("mean", "variance", "homogeneity", "contrast", "dissimilarity", "entropy", "second_moment", "correlation"), na_opt="center", na_val=NA,scale_factor=10000, asinteger=TRUE)
res <- texture$glcm_contrast
length(res[is.na(res)])
# ### RGB Transformation
brightCor<-function(x) {
if (any ( is.na(x) )) {
rep(NA, length(x))
} else {
flo1<-rgb2hsv(x[1],x[2],x[3],maxColorValue = 255)
back_rgb<-col2rgb(hsv(h =flo1[1], s = flo1[2], v = 0.8, alpha=1), alpha = FALSE)
#back_rgb<-col2rgb(test, alpha = FALSE)
#flower<-rgb(back_rgb[1,],back_rgb[2,],back_rgb[3,],maxColorValue = 255)
#return(flower)
}
}
a1_br_cor<-calc(a1, brightCor, forceapply=T)
# ### Indices
# #### Visible Atmospheric Resistant Index (VARI)
rgb = a1
rgb$vari = (rgb[[2]]-rgb[[1]])/(rgb[[2]]+rgb[[1]]-rgb[[3]])
# #### Triangular Greenness Index (TGI)
rgb$tgi<-rgb[[2]]-0.39*rgb[[1]]-0.61*rgb[[3]]
# #### Green Chromatic Value (gcv)
rgb$gcv<-rgb[[2]]/(rgb[[1]]+rgb[[2]]+rgb[[3]])
rgb$bcv<-rgb[[3]]/(rgb[[1]]+rgb[[2]]+rgb[[3]])
rgb$rcv<-rgb[[1]]/(rgb[[1]]+rgb[[2]]+rgb[[3]])
# #### Others
rgb$ngrdi<-(rgb[[2]]-rgb[[1]])/(rgb[[2]]+rgb[[1]])
rgb$mgrvi<-(rgb[[2]]^2-rgb[[1]]^2)/(rgb[[2]]^2+rgb[[1]]^2)
rgb$rgbvi<-rgb[[2]]^2 - (rgb[[1]]*rgb[[3]])/rgb[[2]]^2 + (rgb[[1]]*rgb[[3]])
rgb$gli<-(2*rgb[[2]]-rgb[[1]]-rgb[[3]]) /(2*rgb[[2]]+ rgb[[1]]+ rgb[[3]])
rgb$exg<-2*rgb[[2]]-rgb[[1]]-rgb[[3]]
rgb$ng<-rgb[[2]]/((rgb[[1]]^0.667)*(rgb[[3]]^(1 - 0.667)))
res <- a1$`layer.3`
length(res[is.na(res)])
res <- rgb$ng
length(res[is.na(res)])
# ### Stack all together
marie_version <- TRUE
if (marie_version == TRUE) {
#a3 = brick(list(a1, texture[[c(1,2,3,5,6,7)]], a1_br_cor)) #not correlated features
a3 <- brick(list(a1, texture[[c(1,2,3,5,6,7)]], rgb[[-c(1:3)]], a1_br_cor)) #not correlated features
# We do not need to do this anymore since the projections are all set when the data is loaded.
#proj4string(a3)<-proj4string(cut)
} else {
#a3 <- raster::stack(a1, texture[[c(1,2,3,5,6,7)]])
#a3 <- raster::stack(a3, rgb[[-c(1:3)]])
#a3 <- raster::stack(a3, a1_br_cor) #not correlated features
}
# ## Get spectra of ref_points
# ### 1.b.2
shp <- readOGR(paste(dataPath, "ref_shp.shp", sep = ""))
shp <- sp::spTransform(shp, "+proj=utm +zone=33 +datum=WGS84 +units=m +no_defs +ellps=WGS84 +towgs84=0,0,0")
shp = shp[c(1,6,7,10),] #reduce to most important classes
ref<-as.data.frame(extract(a3,shp))
# ### 1.c.2
r=1; g=2; b=3;
plotRGB(a3,r=r,g=g,b=b,stretch="lin", axes=T)
plot(shp,pch=21,bg="red",col="yellow",cex=1.9,lwd=2.5,add=T)
# ### 1.c.3
col<-colorRampPalette(c("lightgrey","orange","yellow","limegreen","forestgreen"))
# ### 1.c.4
classNames <- sub(pattern = "/", replacement = ".", x = shp$cover)
# # 2 Habitat Sampling
# ### 2.a.1
init.samples = 25
nb_models = 15 #200 #can be <200 for code testing (10-15)
nb_it = 10
buffer = 0.5 #in m -> should equal the size of 1-2 pixel?
mtry = 8 #has to be smaller, than the number of channels in order to get a random set
init.seed = "sample"
n_classes = 4
# ### Remove NaNs
mask <- a3[[20]]
for(f in seq(1:dim(a3)[3])) {
layer <- a3[[f]]
layer[is.na(mask)] <- NaN
a3[[f]] <- layer
}
# ### Remove Infinites
mask <- a3[[20]]
for(f in seq(1:dim(a3)[3])) {
layer <- a3[[f]]
layer[is.infinite(mask)] <- NaN
a3[[f]] <- layer
}
# ### Inpect NaNs and Infinite numbers
for(f in seq(1:dim(a3)[3])) {
layer <- a3[[f]]
print(sprintf("Number of NaNs is %d", length(layer[is.na(layer)])))
print(sprintf("Number of Infinites is %d", length(layer[is.infinite(layer)])))
}
multi_Class_Sampling(
in.raster=a3,
init.samples=init.samples,
sample_type="regular",
nb_models=nb_models,
nb_it=nb_it,
buffer=buffer,
reference=ref,
model="rf",
mtry=mtry,
last=F,
seed=3,
init.seed=init.seed,
outPath=outPath,
step=1,
#RGB = c(1,2,3),
classNames=classNames,
n_classes=n_classes,
multiTest=1,
overwrite = TRUE,
parallel_mode = FALSE,
max_num_cores = 4,
save_runs = FALSE,
plot_on_browser = FALSE)
# # 3. Plot results
plot_Results(inPath=outPath)
......@@ -7,7 +7,7 @@ IRkernel::installspec(user = FALSE)
# Install Hasa dependencies
install.packages("remotes")
install.packages("https://cran.r-project.org/src/contrib/Archive/BH/BH_1.69.0-1.tar.gz", repos=NULL, type="source")
install.packages("https://cran.r-project.org/src/contrib/Archive/sf/sf_0.8-1.tar.gz", repos=NULL, type="source", dependencies=TRUE)
install.packages("https://cran.r-project.org/src/contrib/Archive/sf/sf_0.9-0.tar.gz", repos=NULL, type="source")
install.packages("https://cran.r-project.org/src/contrib/Archive/sp/sp_1.4-4.tar.gz", repos=NULL, type="source", dependencies=TRUE)
install.packages("https://cran.r-project.org/src/contrib/Archive/rgdal/rgdal_1.5-12.tar.gz", repos=NULL, type="source")
......
......@@ -3,15 +3,27 @@ title: "An Introduction to Habitat Sampler"
author: "Carsten Neumann, Alison Beamish, Romulo Goncalves"
date: "01/06/2021"
output:
pdf_document: default
html_document:
df_print: paged
md_document:
variant: markdown_github
always_allow_html: true
pandoc_args: ["-output", "README.md"]
toc: true
toc_depth: 2
variant: gfm
pdf_document:
toc: true
toc_depth: 2
html_document:
theme: united
highlight: tango
toc: true
toc_depth: 2
toc_float:
collapsed: false
smooth_scroll: false
df_print: paged
always_allow_html: yes
header-includes:
- \usepackage{caption}
- \captionsetup[figure]{labelformat=empty}
- \usepackage{caption}
- \captionsetup[figure]{labelformat=empty}
---
```{r echo=FALSE}
......@@ -20,6 +32,7 @@ library(knitr)
knitr::opts_chunk$set(tidy.opts = list(width.cutoff = 75), tidy = TRUE, fig.pos = "!H", out.extra = "")
```
\newpage
# 1 Introduction
This manual introduces the Habitat Sampler (HaSa), an innovative tool that autonomously generates representative reference samples for predictive modelling of surface class probabilities. The tool can be applied to any image data that displays surface structures and dynamics of any kind at multiple spatial and temporal scales. HaSa was initially developed to classify habitat dynamics in semi-natural ecosystems but the procedure can theoretically be applied to any surface. The main innovation of the tool is that it reduces reliance on comprehensive in situ ground truth data or comprehensive training datasets which constrain accurate image classification particularly in complex scenes.
...