Commit 18dd5dfd authored by Maximilian Dolling's avatar Maximilian Dolling

Resolve "do the docs =("

parent 903bcfe1
# CI-Services
---
This Framework provides various services for the GitLab CI regarding to software quality.
The services are packed into a docker image.
You can run the image locally or by setting up a GitLab CI/CD pipeline.
## Currently implemented services
* **test**: runs no actual service, just gives information if everything seems fine
## Running the services
### via GitLab CI/CD
#### requirements
* repo hosted on [GFZ external Gitlab](gitext.gfz-potsdam.de)
### locally
#### requirements
* [docker](https://www.docker.com/)
```bash
docker login gitext.gfz-potsdam.de:5000
docker pull gitext.gfz-potsdam.de:5000/hifis/software-services/fair/ci-services/ci-services:latest
docker run -v [/absolute/path/to/repo]:/repo \
gitext.gfz-potsdam.de:5000/hifis/software-services/fair/ci-services/ci-services:latest \
[SERVICE]]
```
## Description
This software provides a docker image, which provides various services around software quality.
Its goal is to help researchers, developers and [RSEs](https://de-rse.org/en/) develop software with better quality.
Currently there are two ways to run the services.
Either via the [GitLab CI/CD](https://docs.gitlab.com/ee/ci/) (recommended) or a local run.
> **Example:** You can find a full report of all available services [here](meta/ci-services-report.md)
**Current available services:**
* Programming language detection
---
## Requirements
The software you want to run the services on **must** be a [git](https://git-scm.com/) repository!
**via GitLab CI/CD**
* available [GitLab Runner](https://docs.gitlab.com/runner/) with `ci-services` [tag](https://docs.gitlab.com/ee/ci/runners/#using-tags) and the [docker executor](https://docs.gitlab.com/runner/executors/docker.html)
**local usage**
* [docker](https://www.docker.com/) >= 19.03.1
---
## Usage
Running the services via Gitlab CI/CD is the recommended solution.
It keeps the report up to date, every time the project is changed.
You can read more about how it is set up [here](doc/doc_usage_ci.md).
Running it on you local machine is easier to set up.
Therefore it must be triggered manually before every release.
You can read more about how it is set up [here](doc/doc_usage_local.md).
---
## Further planned features
* license check
* license header check
* license generation
* comment check
* language detection
* check for nececary files
* check for necessary files
* generate necessary files
* check if tests exist
* credential check
* DOI request
* curate meta data
* generate report
* test for code hosting
# tmp
**SSH_KNOWN_HOSTS_GITLAB**
# gitext.gfz-potsdam.de:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
gitext.gfz-potsdam.de ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsiq5txhABcG9oePf47SCd+w/tfSs6NWKaMRd2BfBi9DGORQX4wwV4CYvSJbMfJvQO5SMGwhnEAXe0bWihC2V8X7lzcePrpjfP+uH2lmipcFbV9g3iQpM7Fusr96IV65v/qQ6HE4+KkHJLR1vEKj1AOaVgDww7CHhVAHCkvqcSwwtfOjBbqyfQ2Su7O6UNzs1ecQcIXnQNO8ebzimv3c8mKdo9j8i5eiWVqzRivmi/R3XIIc2T695mTLyxUplqfyvZpDL90F9VEjr/3HM9/3dASbonaTH8SoMRCxOy8Fpz0Va1MdpSor0uoemQjbu8aZzGB0+tsE5eWjuFaZib8QDF
# gitext.gfz-potsdam.de:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
gitext.gfz-potsdam.de ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFiDZULQkGfyG/dFvTtoE3PoANi1PccdjuYfdRId1LaqDEE/sDdwJoy/Ate0DbOoTrjfb22eawUkjVVEmvInXh4=
# gitext.gfz-potsdam.de:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
gitext.gfz-potsdam.de ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGRXeIQaIUcyF17/3TTZR2ILHoKqYpAD6/E3s8YTaq8G
---
**create deploy key**
## Help
ssh-keygen -t rsa -b 4096 -C "ci-services"
public key > settings/repository/deploy keys (write access must be allowed)
private key > settings/CI/vars
\ No newline at end of file
TODO =)
\ No newline at end of file
# CI-Services
---
## CI/CD usage
The following manual describes how to use the ci-services with GitLab CI/CD.
It is highly advisable to know how to use git and GitLab when using this method.
> **Note:** It is possible to hook the ci-services to GitHub Actions and other code hosting platforms.
---
## Decide on way to retrieve the report
There are two different ways to get the generated report by the `ci-services`.
**Artifact**
When you opt for this option, the report is available via the GitLab web-interface.
You can simply download.
More information available (here)[https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html]
This option is easier to set up.
**Commit** (recommended)
> **Note:** this option is **required** by certain (future) services.
---
### 1) Project member permissions
To apply certain changes described below, you need at least `Maintainer` [permissions](https://docs.gitlab.com/ee/user/permissions.html).
### 2) Host the project on any GitLab
How to do that, is described [here](https://docs.gitlab.com/ee/gitlab-basics/create-project.html)
### 3) Enable CI/CD
In the GitLab web-interface from the page of your repository navigate to `Settings > General > Visibility, project features, permissions`.
Enable `Pipelines` and set them to `Only Project Members`.
### 4) Add GitLab Runner
To have an available Runner is **required** for the ci-services to function.
This will not be covered here.
Talk to your local admin get one up and running.
**Runner requirements**
* tag: `ci-services`
* executor: docker
> **WARNING:** Be aware that the runtime for the services can go up to several hours, depending on the project size!
Further information are available [here](https://docs.gitlab.com/runner/install/).
---
### 5.1) Retrieve report by Artifact
#### 5.1.1) Set up .gitlab-ci.yml
This yaml block is set up for project without prior GitLab Ci/CD setup.
If you already have `.gitlab-ci.yml` in your project, there should be someone withing your project who can help you adapt.
Copy the content of the following code block to a file named `.gitlab-ci.yml` in the base directory of your project.
Commit and push your changes.
```yaml
stages:
- ci-services
ci-services:
stage: ci-services
image:
name: "gitext.gfz-potsdam.de:5000/hifis/software-services/fair/ci-services/ci-services:latest"
entrypoint: [""]
tags:
- ci-services
script:
- mkdir -p /repo
- cp -r "$CI_PROJECT_DIR/." /repo
- docker-entrypoint.sh -a
- cp -rf /repo/meta "$CI_PROJECT_DIR"
only:
refs:
- master
artifacts:
paths:
- meta/ci-services-report.md
expire_in: 30 days
```
---
### 5.2) Retrieve report by commit
This manual focuses on [UNIX](https://en.wikipedia.org/wiki/Unix) systems like Linux or MacOS.
> **Note:** for Windows the commands can slightly differ, but is fully compatible.
#### 5.2.1) Set up deploy key
First we need to generate a key pair via the terminal.
It will function as [deploy key](https://docs.gitlab.com/ee/ssh/#deploy-keys).
More information on generating a key pair can be found [here](https://docs.gitlab.com/ee/ssh/#generating-a-new-ssh-key-pair).
```console
ssh-keygen -t rsa -b 4096 -C "[PROJECT_NAME]_ci-services" -f [OUTPUTFILE] -q -N ""
```
In the GitLab web-interface from the page of your repository navigate to `Settings > CI/CD > Deploy Keys`.
Paste the content of `[OUTPUTFILE].pub` to the field `key` and give it any unique name (like *PROJECT_NAME*_deploy-key)
Check the 'Write access allowed' checkbox.
Save the changes.
> **Note:** it is recommended to either store `[OUTPUTFILE].pub` securely or delete it.
#### 5.2.2) Set up Variables
We need to tell our CI/CD the identity of the GitLab server we are using.
To do that, we request it from the server via a terminal command.
It returns two lines with the public identity of the GitLab server.
```console
ssh-keyscan -t rsa [GITLAB_DOMAIN]]
# example
ssh-keyscan -t rsa gitext.gfz-potsdam.de
# gitext.gfz-potsdam.de:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
gitext.gfz-potsdam.de ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsiq5txhABcG9oePf47SCd+w/tfSs6NWKaMRd2BfBi9DGORQX4wwV4CYvSJbMfJvQO5SMGwhnEAXe0bWihC2V8X7lzcePrpjfP+uH2lmipcFbV9g3iQpM7Fusr96IV65v/qQ6HE4+KkHJLR1vEKj1AOaVgDww7CHhVAHCkvqcSwwtfOjBbqyfQ2Su7O6UNzs1ecQcIXnQNO8ebzimv3c8mKdo9j8i5eiWVqzRivmi/R3XIIc2T695mTLyxUplqfyvZpDL90F9VEjr/3HM9/3dASbonaTH8SoMRCxOy8Fpz0Va1MdpSor0uoemQjbu8aZzGB0+tsE5eWjuFaZib8QDF
```
Now open the GitLab web-interface and navigate to `Settings > CI/CD > Variables` from from the page of your repository.
We need to fill in two variables here.
Click on 'Add Variable'.
As key type in `GITLAB_DEPLOY_KEY`.
As value paste the content of `[OUTPUTFILE]` (was generated in 5.2.1).
Click on 'Add Variable'.
For the second one:
As key type in `SSH_KNOWN_HOSTS_GITLAB`.
As value paste both output lines of the `ssh-keyscan` command (see above).
Click on 'Add Variable'.
#### 5.2.3) Set up .gitlab-ci.yml
This yaml block is set up for project without prior GitLab Ci/CD setup.
If you already have `.gitlab-ci.yml` in your project, there should be someone withing your project who can help you adapt.
Copy the content of the following code block to a file named `.gitlab-ci.yml` in the base directory of your project.
Commit and push your changes.
```yaml
stages:
- ci-services
ci-services:
stage: ci-services
image:
name: "gitext.gfz-potsdam.de:5000/hifis/software-services/fair/ci-services/ci-services:latest"
entrypoint: [""]
tags:
- ci-services
before_script:
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS_GITLAB" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- mkdir -p /repo
- cp -r "$CI_PROJECT_DIR/." /repo
- docker-entrypoint.sh -a
- cp -rf /repo/meta "$CI_PROJECT_DIR"
- eval $(ssh-agent -s)
- echo "$GITLAB_DEPLOY_KEY" | tr -d '\r' | ssh-add -
- git config user.name "GitLab CI:${CI_PROJECT_NAMESPACE}/${CI_PROJECT_NAME}"
- git config user.email "[PROJECT_NAME]_ci-services"
- git add -f meta/ci-services-report.md
- git commit -m "[AUTOMATIC] generated meta/ci-services-report.md by gitlab ci"
- git push "git@${CI_SERVER_HOST}:${CI_PROJECT_NAMESPACE}/${CI_PROJECT_NAME}.git" "HEAD:${CI_COMMIT_REF_NAME}"
only:
refs:
- master
```
**Important**
In the yaml file above fill in the line `- git config user.email "[PROJECT_NAME]_ci-services"`
The content of user.email must be identical with what was entered in 5.2.1)
> **Note:** it is recommended to either store `[OUTPUTFILE]` securely or delete it.
### 6) Examples
An example of both variants applied can be found [here](.gitlab-ci.yml)
# CI-Services
---
## Local usage
The following manual describes how to use the ci-services on a local machine.
This manual focuses on [UNIX](https://en.wikipedia.org/wiki/Unix) systems like Linux or MacOS.
> **Note:** for Windows the commands can slightly differ, but is fully compatible.
---
### 3) Assure project is a git repository
To run the services on your local machine the first thing to do is to assure, that your software is within a git repository.
If you know, it is a git repository, you can skip this part.
Open a Terminal and change to the project directory. And then check for git.
```console
cd [PATH_TO_PROJECT]
git status
```
If you see `fatal: not a git repository` you don't have a git repository.
Please make sure it becomes one. More information on doing that is available [here](https://swcarpentry.github.io/git-novice/).
### 4) Execute ci-services
Start the docker service:
```console
systemctl start docker.service
```
The general syntax looks like the following:
```console
docker run -v [/ABSOLUTE/PATH/TO/REPO]:/repo gitext.gfz-potsdam.de:5000/hifis/software-services/fair/ci-services/ci-services:latest [SERVICE]
```
To get a list of available services use:
```console
docker run -v [/ABSOLUTE/PATH/TO/REPO]:/repo gitext.gfz-potsdam.de:5000/hifis/software-services/fair/ci-services/ci-services:latest --help
```
### 5) Examples
Run all services for the project within the directory `~/Projects/volcano_detection`.
```console
docker run -v ~/Projects/volcano_detection:/repo gitext.gfz-potsdam.de:5000/hifis/software-services/fair/ci-services/ci-services:latest -a
```
Get the version of the ci-services. As project directory anything can be used. (using root directory here)
```console
docker run -v /:/repo gitext.gfz-potsdam.de:5000/hifis/software-services/fair/ci-services/ci-services:latest --version
```
......@@ -11,7 +11,7 @@
**Last Commit:** 168465c558f94e0d1b92c51060b1d956eeb85c91
**Report Time:** 22/03/2020 14:25:09
**Report Time:** 22/03/2020 14:27:53
---
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment