...
 
Commits (15)
......@@ -9,7 +9,7 @@ stages:
- ci-services
variables:
VERSION: 0.4.1
VERSION: 0.4.2
DOCKER_DST_IMAGE_NAME: ci-services
build-deploy:dep:
......
......@@ -43,6 +43,9 @@ The software you want to run the services on **must** be a [git](https://git-scm
## Usage
Some of the services require a config file.
To see which, take a look at the [service description](doc/doc_service_description.md)
Running the services via Gitlab CI/CD is the recommended solution.
It keeps the report up to date, every time the project is changed.
You can read more about how it is set up [here](doc/doc_usage_ci.md).
......
......@@ -37,7 +37,7 @@ RUN apk upgrade
# copy entrypoint script to executables
COPY ./build/docker/dev/docker-entrypoint.sh /usr/local/bin/
COPY ./build/docker/dev/entrypoint-manual.txt /usr/local/bin/
RUN echo '0.4.1' > /version.txt
RUN echo '0.4.2' > /version.txt
# copy service scripts to root dir
COPY ./src/services /services
......
......@@ -28,21 +28,28 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Version first, because it does not require any other files or settings.
if [ "$1" = "--version" ]; then
cat /version.txt
exit 0
fi
# checks if a repository was bound to the docker image
if [ ! -d /repo ]; then
echo
echo "Repository to run services on not bound! Showing help page..."
cat /usr/local/bin/entrypoint-manual.txt
echo
echo "Repository to run services on not bound! Showing help page..."
exit 1
fi
# checks if a .ci-service.yml exists
if [ ! -f /repo/.ci-services.yml ]; then
cat /usr/local/bin/entrypoint-manual.txt
echo
echo "Repository does not contain a config for the ci-services."
echo "Visit https://gitext.gfz-potsdam.de/software/services/fair/ci-services/-/blob/master/doc/doc_service_description.md#general"
echo "for more information"
cat /usr/local/bin/entrypoint-manual.txt
exit 1
fi
......@@ -128,13 +135,8 @@ if [ "$1" = "--test" ]; then
exit 0
fi
if [ "$1" = "--version" ]; then
cat /version.txt
exit 0
fi
# if no suitable argument was found, shows help page
cat /usr/local/bin/entrypoint-manual.txt
echo
echo "Unknown command line argument! Showing help page..."
cat /usr/local/bin/entrypoint-manual.txt
exit 1
<!--
SPDX-FileCopyrightText: 2020 Helmholtz Centre Potsdam - GFZ German Research Centre for Geosciences Potsdam, Germany (https://www.gfz-potsdam.de/)
SPDX-License-Identifier: CC0-1.0
-->
# CI-Services
---
### general
Some services require settings.
These settings will be done in a `.ci-services.yaml` file in the root directory of the project.
An example for a [.ci-services.yaml](.ci-services.yml) can be found in this project.
A minimal config is located [here](test/tetfiles/.min_ci-services.yml) and an empty commented file below:
```yaml
# exclude patterns for license check
# list of regular expressions
# every file and directory will be checked if it matched any of the expressions.
list-used-licenses:
exclude:
-
# matching of imported name to name in pkg manager
# sometimes the imported name does not match the pkg name
# they are sorted by programming language
rename:
lib:
language:
import name: 'pkg manager name'
# mark local imports to exclude from license check
# they are sorted by programming language
local_imports:
language:
- 'library name'
# license information used for the reuse tool
apply-license:
description: 'description for license header'
copyright: 'copyright holder for dep5 file'
time: 'time of copyright'
maintainer:
name: 'name of maintainer for dep5 file'
mail: 'mail of maintainer for dep5 file'
licenses:
data:
spdx-identifier: 'spdx-identifier'
copyright: 'copyright holder'
software:
spdx-identifier: 'spdx-identifier'
copyright: 'copyright holder'
pictures:
spdx-identifier: 'spdx-identifier'
copyright: 'copyright holder'
insignificant:
spdx-identifier: 'spdx-identifier'
copyright: 'copyright holder'
```
For more information which services need a config, see the [service description](doc/doc_service_description.md)
\ No newline at end of file
......@@ -20,53 +20,12 @@ SPDX-License-Identifier: CC0-1.0
All implemented services will generate a a yaml file in `meta/[SERVICE_NAME]` containing the results.
Furthermore all settings for the services will be done in a `.ci-services.yaml` file in the root directory of the project.
An example for a [.ci-services.yaml](.ci-services.yml) can be found in this project and an empty commented file below:
```yaml
# exclude patterns for license check
# list of regular expressions
# every file and directory will be checked if it matched any of the expressions.
exclude:
-
# matching of imported name to name in pkg manager
# sometimes the imported name does not match the pkg name
# they are sorted by programming language
rename:
lib:
language:
import name: 'pkg manager name'
# mark local imports to exclude from license check
# they are sorted by programming language
local_imports:
language:
- 'library name'
# license information used for the reuse tool
legal:
description: 'description for license header'
copyright: 'copyright holder for dep5 file'
time: 'time of copyright'
maintainer:
name: 'name of maintainer for dep5 file'
mail: 'mail of maintainer for dep5 file'
licenses:
data:
spdx-identifier: 'spdx-identifier'
copyright: 'copyright holder'
software:
spdx-identifier: 'spdx-identifier'
copyright: 'copyright holder'
pictures:
spdx-identifier: 'spdx-identifier'
copyright: 'copyright holder'
insignificant:
spdx-identifier: 'spdx-identifier'
copyright: 'copyright holder'
```
---
### apply license
> **Note:** config required
This service implements a wrapper for the [reuse tool](https://reuse.software/) by the fsfe.
The main concept of the service is to have dedicated licenses for all files.
This leads to the possibility of multiple licenses for a project.
......@@ -85,7 +44,8 @@ Afterwards it generates a license header or a .license file for every file in th
### generate report
The report service will read and merge all the sub-reports.
It will then generate acense_check markdown version of the assembled report.
If there is no subreport, the report will not be generated.
It will then generate a markdown version of the assembled report.
The source code can be found [here](../src/services/generate-report).
---
......@@ -111,6 +71,8 @@ The source code can be found [here](../src/services/list-used-languages).
### list used licenses
> **Note:** config required
The list authors service can be run with the `--list-used-licenses` flag when running it either local or via CI/CD.
The service loads the output from `list-used-languages`.
It takes every files from the repository and assigns them to a detected language.
......
......@@ -9,11 +9,11 @@
**Software Location:** [gitext](https://gitext.gfz-potsdam.de/software/services/fair/ci-services)
**Last Commit:** a4fb0656339aeb1b4df35b3f0f24013c9978740c
**Last Commit:** 7d4726a32cd76769a2144b092d737eff16f2e236
**Report Time:** 26/05/2020 07:57:23
**Report Time:** 27/05/2020 10:46:59
**Report Version:** 0.4.1
**Report Version:** 0.4.2
---
......@@ -36,9 +36,9 @@
|---|---|
|Dockerfile|6617|
|HTML|1834|
|Python|61126|
|Python|63535|
|Ruby|2990|
|Shell|21191|
|Shell|21273|
### list used licenses
......@@ -66,8 +66,8 @@
* Unused licenses:
* Used licenses: CC-BY-4.0, CC0-1.0, GPL-3.0-or-later
* Read errors: 0
* Files with copyright information: 69 / 69
* Files with license information: 69 / 69
* Files with copyright information: 71 / 71
* Files with license information: 71 / 71
Congratulations! Your project is compliant with version 3.0 of the REUSE Specification :-)
......
......@@ -67,9 +67,11 @@ report_time = datetime.now().strftime("%d/%m/%Y %H:%M:%S")
# consolidate subreports
report_dict = {'report version': report_version}
subreport_found = False
for root, dirs, files in os.walk(working_dir, topdown=False):
for file in files:
if file.endswith('.yml'):
subreport_found = True
subreport_path = os.path.join(root, file)
with open(subreport_path) as subreport_yml:
subreport_dict = yaml.load(subreport_yml, Loader=yaml.FullLoader)
......@@ -78,6 +80,11 @@ for root, dirs, files in os.walk(working_dir, topdown=False):
if cleanup:
shutil.rmtree(Path(subreport_path).parent)
# do not try to generate report if there is nothing to put into it.
if not subreport_found:
print('No subreport found. please run at least one other service to generate a report.')
sys.exit(1)
# changes to report to match test output, if test is running
# delete timestamp, last commit hash and set language values to 0 due to frequent changes
if test:
......
......@@ -53,7 +53,7 @@ def get_python_libs(file_path=''):
while line:
if line[0] != '#' and line != '' and line[0:2] != '-r' and line != '\n':
stripped_down_line = line.strip().split('#')[0].replace(' ', '')
if stripped_down_line[0:4] == 'http':
if ('/' or ':') in stripped_down_line:
used_libs_without_stdlib_list.extend([{stripped_down_line: 'unknown'}])
elif '/' in stripped_down_line:
used_libs_without_stdlib_list.extend([{stripped_down_line: 'unknown'}])
......@@ -79,10 +79,22 @@ def get_python_libs(file_path=''):
line = file.readline()
used_libs_list = []
for line in lines_with_libs_list:
lib_list = []
if line[0:7] == 'import ':
used_libs_list = used_libs_list + line[7:].replace(' ', '').split(",")
lib_list = line[7:].replace(' ', '').split(",")
if line[0:5] == 'from ':
used_libs_list.append(line.split(' ')[1])
lib_list = [line.split(' ')[1]]
tmp_lib_list =[]
for lib in lib_list:
if lib is not None:
if lib[0] == '.':
tmp_lib = lib[1:]
tmp_lib_list.append('.' + tmp_lib.split('.')[0])
else:
tmp_lib = lib
tmp_lib_list.append(tmp_lib.split('.')[0])
used_libs_list += tmp_lib_list
used_libs_list = list(dict.fromkeys(used_libs_list))
......
......@@ -59,7 +59,7 @@ def get_python_libraries_license(libraries_dict):
license_dict['licenses'][lib] = 'Python-2.0'
continue
# handle unknown location
if lib[0:4] == 'http' or '.' in lib or '/' in lib:
if '.' in lib or '/' in lib or ':' in lib:
license_dict['unknowns'][lib] = 'unknown location'
continue
# handle unknown version
......
......@@ -272,7 +272,8 @@ report = {'name': project_name,
'report version': report_version,
'list used licenses': {'description': 'Generates a list of dependencies and their license.',
'result': {'license list': cumulated_licenses_dict,
'unknowns': unknowns}}}
'unknowns': unknowns},
'debug used libs': used_libs_dict}}
# create report
......
......@@ -36,7 +36,7 @@ tail -n +7 test/test_report.md > test/tmp_test_report.md
IS_CHK_SUM=$(sha512sum meta/ci-services-report.md | awk '{ print $1 }')
PASS_CHK_SUM=$(sha512sum test/tmp_test_report.md | awk '{ print $1 }')
rm test/tmp_test_report.md
#rm test/tmp_test_report.md
if [ "$IS_CHK_SUM" != "$PASS_CHK_SUM" ]; then
echo "Generated report does not equal the test template"
......
......@@ -19,7 +19,7 @@ SPDX-License-Identifier: CC0-1.0
**Report Time:** 0
**Report Version:** 0.4.1
**Report Version:** 0.4.2
---
......@@ -56,8 +56,8 @@ SPDX-License-Identifier: CC0-1.0
|---|---|---|
|Apache 2.0|1|requests|
|GPL-3.0-or-later AND Apache-2.0 AND CC0-1.0 AND CC-BY-SA-4.0|1|reuse|
|MIT|2|PyYAML <br> stdlib-list|
|Python-2.0|10|datetime <br> importlib.util <br> json <br> os <br> pathlib <br> re <br> shutil <br> sys <br> time <br> unittest|
|MIT|3|PyYAML <br> httpretty <br> stdlib-list|
|Python-2.0|10|datetime <br> importlib <br> json <br> os <br> pathlib <br> re <br> shutil <br> sys <br> time <br> unittest|
|mock|1|stdlib|
#### Unknown objects
......@@ -78,6 +78,8 @@ SPDX-License-Identifier: CC0-1.0
* Python / coverage / unsupported version modifier (!=3.5)
* Python / django / unknown version
* Python / docopt / unsupported version modifier (>=0.5.0)
* Python / green / unknown version
......@@ -105,8 +107,8 @@ SPDX-License-Identifier: CC0-1.0
* Unused licenses:
* Used licenses: CC-BY-4.0, CC0-1.0, GPL-3.0-or-later
* Read errors: 0
* Files with copyright information: 68 / 68
* Files with license information: 68 / 68
* Files with copyright information: 70 / 70
* Files with license information: 70 / 70
Congratulations! Your project is compliant with version 3.0 of the REUSE Specification :-)
......
# ci-services
# This Software provides services to check and improve the source code quality of software-projects
#
# SPDX-FileCopyrightText: 2020 Helmholtz Centre Potsdam - GFZ German Research Centre for Geosciences Potsdam, Germany (https://www.gfz-potsdam.de/)
#
# SPDX-License-Identifier: GPL-3.0-or-later
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version, complemented with
# the following provision:
# For the scientific transparency and verification of results obtained
# and communicated to the public after using a modified version of the
# work, You (as the recipient of the source code and author of this
# modified version, used to produce the published results in scientific
# communications) commit to make this modified source code available in
# a repository that is easily and freely accessible for a duration of
# five years after the communication of the obtained results.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
from django.contrib.auth.mixins import UserPassesTestMixin
from django.shortcuts import redirect
from django.urls import reverse
import httpretty
......@@ -24,4 +24,5 @@ http://wxpython.org/Phoenix/snapshot-builds/wxPython_Phoenix-3.0.3.dev1820+49a88
# Same as 1st section, just here to show that you can put things in any order.
rejected
green
httpretty == 1.0.2
#
\ No newline at end of file