Commit 5a5cc1a3 authored by Maximilian Dolling's avatar Maximilian Dolling
Browse files

added some config validation

parent be05777a
......@@ -2,24 +2,25 @@
#
# SPDX-License-Identifier: CC0-1.0
# exclude patterns for license check
exclude:
- 'doc/.*' # documentation
- 'meta/.*' # meta data
- 'test/.*' # test files
- '.*dep5_template$' # plain text template file, not source code
# matching of imported name to name in pkg manager
rename:
lib:
list-used-licenses:
# exclude patterns for license check
exclude:
- 'doc/.*' # documentation
- 'meta/.*' # meta data
- 'test/.*' # test files
- '.*dep5_template$' # plain text template file, not source code
# matching of imported name to name in pkg manager
rename:
lib:
Python:
yaml: 'PyYAML'
stdlib_list: 'stdlib-list'
# mark local imports to exclude from license check
local_imports:
Python:
yaml: 'PyYAML'
stdlib_list: 'stdlib-list'
# mark local imports to exclude from license check
local_imports:
Python:
- 'functions'
# license used for fore files in this repository
legal:
- 'functions'
apply-license:
description: 'This Software provides services to check and improve the source code quality of software-projects'
copyright: 'Helmholtz Centre Potsdam - GFZ German Research Centre for Geosciences Potsdam, Germany (https://www.gfz-potsdam.de/)'
time: '2020'
......
......@@ -9,7 +9,7 @@ stages:
- ci-services
variables:
VERSION: 0.4.0
VERSION: 0.4.1
DOCKER_DST_IMAGE_NAME: ci-services
build-deploy:dep:
......
......@@ -37,7 +37,7 @@ RUN apk upgrade
# copy entrypoint script to executables
COPY ./build/docker/dev/docker-entrypoint.sh /usr/local/bin/
COPY ./build/docker/dev/entrypoint-manual.txt /usr/local/bin/
RUN echo '0.4.0' > /version.txt
RUN echo '0.4.1' > /version.txt
# copy service scripts to root dir
COPY ./src/services /services
......
......@@ -36,15 +36,31 @@ if [ ! -d /repo ]; then
exit 1
fi
# checks if a .ci-service.yml exists
if [ ! -f /repo/.ci-services.yml ]; then
echo
echo "Repository does not contain a config for the ci-services."
echo "Visit https://gitext.gfz-potsdam.de/software/services/fair/ci-services/-/blob/master/doc/doc_service_description.md#general"
echo "for more information"
cat /usr/local/bin/entrypoint-manual.txt
exit 1
fi
# checks for command line argument to start respective service
if [ "$1" = "--all" ]; then
echo "starting 'list-authors' service..."
/services/list-authors/list-authors.sh
echo "starting 'list-used-languages' service..."
/services/list-used-languages/list-used-languages.sh
echo "starting 'list-used-licenses' service..."
/services/list-used-licenses/list-used-licenses.sh
echo "starting 'apply-license' service..."
/services/apply-license/apply-license.sh
echo "starting 'reuse-lint' service..."
/services/reuse-lint/reuse-lint.sh
echo "starting 'generate-report' service..."
/services/generate-report/generate-report.sh
chmod -R 777 /repo
......@@ -93,13 +109,19 @@ if [ "$1" = "--reuse-lint" ]; then
fi
if [ "$1" = "--test" ]; then
echo "starting 'list-authors' service..."
/services/list-authors/list-authors.sh --test
echo "starting 'list-used-languages' service..."
/services/list-used-languages/list-used-languages.sh --test
echo "starting 'list-used-licenses' service..."
/services/list-used-licenses/list-used-licenses.sh --test
echo "starting 'apply-license' service..."
/services/apply-license/apply-license.sh --test
echo "starting 'reuse-lint' service..."
/services/reuse-lint/reuse-lint.sh --test
echo "starting 'generate-report' service..."
/services/generate-report/generate-report.sh --test
chmod -R 777 /repo
......
......@@ -3,7 +3,9 @@ Usage: docker run -v [PATH/TO/REPO]:/repo [SERVICE]
Runs various services for software quality detection.
Needs a directory to run services on bound to '/repo'
Needs a directory to run services on bound to '/repo'.
The directory needs to be a git repository and needs to contain a config yaml.
Visit https://gitext.gfz-potsdam.de/software/services/fair/ci-services for more help.
Available services:
--all runs all services (reporting is last)
......
......@@ -7,13 +7,13 @@
**Software Name:** ci-services
**Software Location:** [gitext](https://gitext.gfz-potsdam.de/software/services/fair/ci-services)
**Software Location:** local
**Last Commit:** f725c80fd5d03418ff674dd21542b3dce406b246
**Last Commit:** 0
**Report Time:** 25/05/2020 15:53:26
**Report Time:** 0
**Report Version:** 0.4.0
**Report Version:** 0.4.1
---
......@@ -34,11 +34,11 @@
|language|bytes of code|
|---|---|
|Dockerfile|6617|
|HTML|1834|
|Python|61173|
|Ruby|2990|
|Shell|19337|
|Dockerfile|0|
|HTML|0|
|Python|0|
|Ruby|0|
|Shell|0|
### list used licenses
......@@ -51,9 +51,42 @@
|Apache 2.0|1|requests|
|GPL-3.0-or-later AND Apache-2.0 AND CC0-1.0 AND CC-BY-SA-4.0|1|reuse|
|MIT|2|PyYAML <br> stdlib-list|
|Python-2.0|8|datetime <br> json <br> os <br> pathlib <br> re <br> shutil <br> sys <br> time|
|Python-2.0|10|datetime <br> importlib.util <br> json <br> os <br> pathlib <br> re <br> shutil <br> sys <br> time <br> unittest|
|mock|1|stdlib|
#### Unknown objects
**unknown language**
* src/services/apply-license/data/templates/dep5_template
* test/linter/pylintrc
* test/testfiles/unknown_language/test.testlanguage
**unknown license**
* Python / ./downloads/numpy-1.9.2-cp34-none-win32.whl / unknown location
* Python / Mopidy-Dirble / unsupported version modifier (~=1.1)
* Python / beautifulsoup4 / unknown version
* Python / coverage / unsupported version modifier (!=3.5)
* Python / docopt / unsupported version modifier (>=0.5.0)
* Python / green / unknown version
* Python / http://wxpython.org/Phoenix/snapshot-builds/wxPython_Phoenix-3.0.3.dev1820+49a8884-cp34-none-win_amd64.whl / unknown location
* Python / keyring / unsupported version modifier (>=4.1.1)
* Python / nose / unknown version
* Python / nose-cov / unknown version
* Python / rejected / unknown version
### reuse linter
> **Description: Prints the result of the [reuse](https://reuse.software/spec/) linter**
......
......@@ -49,7 +49,12 @@ if "--test" in sys.argv:
test = True
with open('/repo/.ci-services.yml') as settings_yaml:
settings = yaml.load(settings_yaml, Loader=yaml.FullLoader)
tmp = yaml.load(settings_yaml, Loader=yaml.FullLoader)
if 'apply-license' not in tmp:
print("no settings for this service in 'ci-services.yml - exiting service'")
sys.exit(1)
else:
settings = tmp['apply-license']
settings_yaml.close()
with open('/services/apply-license/data/apply-license_patterns.yml') as apply_license_patterns_yaml:
......@@ -68,8 +73,8 @@ with open('/version.txt', 'r') as file:
report_version = file.read().replace('\n', '')
# download license texts
for item in settings['legal']['licenses']:
os.system('reuse download {} &> /dev/null'.format(settings['legal']['licenses'][item]['spdx-identifier']))
for item in settings['licenses']:
os.system('reuse download {} &> /dev/null'.format(settings['licenses'][item]['spdx-identifier']))
# create main license file
......@@ -83,11 +88,11 @@ print(__file__)
# format template
license_year = time.strftime("%Y")
licensetext = licensetext.replace('{1}', license_year)
licensetext = licensetext.replace('{2}', settings['legal']['copyright'])
licensetext = licensetext.replace('{3}', settings['legal']['licenses']['data']['spdx-identifier'])
licensetext = licensetext.replace('{4}', settings['legal']['licenses']['pictures']['spdx-identifier'])
licensetext = licensetext.replace('{5}', settings['legal']['licenses']['software']['spdx-identifier'])
licensetext = licensetext.replace('{6}', settings['legal']['licenses']['insignificant']['spdx-identifier'])
licensetext = licensetext.replace('{2}', settings['copyright'])
licensetext = licensetext.replace('{3}', settings['licenses']['data']['spdx-identifier'])
licensetext = licensetext.replace('{4}', settings['licenses']['pictures']['spdx-identifier'])
licensetext = licensetext.replace('{5}', settings['licenses']['software']['spdx-identifier'])
licensetext = licensetext.replace('{6}', settings['licenses']['insignificant']['spdx-identifier'])
# write licensefile
with open(repo_dir + '/LICENSE.md', 'w') as licensefile:
......@@ -103,8 +108,8 @@ with open('/services/apply-license/data/templates/dep5_template', 'r') as dep5_t
# format template
dep5text = dep5text.replace('{1}', project_name)
dep5text = dep5text.replace('{2}', settings['legal']['maintainer']['name'])
dep5text = dep5text.replace('{3}', settings['legal']['maintainer']['mail'])
dep5text = dep5text.replace('{2}', settings['maintainer']['name'])
dep5text = dep5text.replace('{3}', settings['maintainer']['mail'])
dep5text = dep5text.replace('{4}', project_location)
# write licensefile
......@@ -131,7 +136,7 @@ for root, dirs, files in os.walk('/services/apply-license/data/templates/header'
template.close()
# modify template
template_text = template_text.replace('{1}', project_name)
template_text = template_text.replace('{2}', settings['legal']['description'])
template_text = template_text.replace('{2}', settings['description'])
# write template
with open(repo_dir + '/.reuse/templates/{}'.format(file), 'w') as template:
template.write(template_text)
......@@ -159,8 +164,8 @@ for root, dirs, files in os.walk(repo_dir, topdown=False):
type_found = False
command = 'reuse addheader --year="{1}" --copyright="{2}" --license="{3}" --template={4}{5} {6}&>/dev/null'
command = command.replace('{1}', settings['legal']['time'])
command = command.replace('{2}', settings['legal']['copyright'])
command = command.replace('{1}', settings['time'])
command = command.replace('{2}', settings['copyright'])
command = command.replace('{6}', filepath)
for file_type_pattern in apply_license_patterns['comment in file']:
......@@ -178,11 +183,11 @@ for root, dirs, files in os.walk(repo_dir, topdown=False):
p = re.compile(file_type_pattern, re.IGNORECASE)
if p.match(file):
command = command.replace('{3}',
settings['legal']['licenses'][file_type]['spdx-identifier'])
settings['licenses'][file_type]['spdx-identifier'])
if os.path.isfile(repo_dir + '/.reuse/templates/{}.jinja2'.format(
settings['legal']['licenses'][file_type]['spdx-identifier'])):
settings['licenses'][file_type]['spdx-identifier'])):
command = command.replace('{4}',
settings['legal']['licenses'][file_type]['spdx-identifier'])
settings['licenses'][file_type]['spdx-identifier'])
else:
command = command.replace('{4}', 'default')
os.system(command)
......@@ -191,10 +196,10 @@ for root, dirs, files in os.walk(repo_dir, topdown=False):
if type_found:
break
else:
command = command.replace('{3}', settings['legal']['licenses']['insignificant']['spdx-identifier'])
command = command.replace('{3}', settings['licenses']['insignificant']['spdx-identifier'])
if os.path.isfile(repo_dir + '/.reuse/templates/{}.jinja2'.format(
settings['legal']['licenses']['insignificant']['spdx-identifier'])):
command = command.replace('{4}', settings['legal']['licenses']['insignificant']['spdx-identifier'])
settings['licenses']['insignificant']['spdx-identifier'])):
command = command.replace('{4}', settings['licenses']['insignificant']['spdx-identifier'])
else:
command = command.replace('{4}', 'default')
os.system(command)
......@@ -46,10 +46,6 @@ test = False
if "--test" in sys.argv:
test = True
with open('/repo/.ci-services.yml') as settings_yaml:
settings = yaml.load(settings_yaml, Loader=yaml.FullLoader)
settings_yaml.close()
# collect report data
project_name = os.popen('basename $(git remote get-url origin)').read()[:-5]
last_commit_ref = os.popen('git rev-parse HEAD').read()[0:40]
......
......@@ -51,7 +51,12 @@ if "--test" in sys.argv:
test = True
with open('/repo/.ci-services.yml') as settings_yaml:
settings = yaml.load(settings_yaml, Loader=yaml.FullLoader)
tmp = yaml.load(settings_yaml, Loader=yaml.FullLoader)
if 'list-used-licenses' not in tmp:
print("no settings for this service in 'ci-services.yml - exiting service'")
sys.exit(1)
else:
settings = tmp['list-used-licenses']
settings_yaml.close()
# collect report data
......@@ -200,7 +205,7 @@ for language in used_libs_dict:
if lib_key in settings['rename']['lib'][language]:
lib_key = settings['rename']['lib'][language][lib_key]
# ignore local imports
if language in settings['local_imports']:
if settings['local_imports'] is not None and language in settings['local_imports']:
if lib_key in settings['local_imports'][language]:
continue
# check duplicates
......
......@@ -46,10 +46,6 @@ test = False
if "--test" in sys.argv:
test = True
with open('/repo/.ci-services.yml') as settings_yaml:
settings = yaml.load(settings_yaml, Loader=yaml.FullLoader)
settings_yaml.close()
# collect report data
project_name = os.popen('basename $(git remote get-url origin)').read()[:-5]
last_commit_ref = os.popen('git rev-parse HEAD').read()[0:40]
......
......@@ -19,7 +19,7 @@ SPDX-License-Identifier: CC0-1.0
**Report Time:** 0
**Report Version:** 0.4.0
**Report Version:** 0.4.1
---
......@@ -105,8 +105,8 @@ SPDX-License-Identifier: CC0-1.0
* Unused licenses:
* Used licenses: CC-BY-4.0, CC0-1.0, GPL-3.0-or-later
* Read errors: 0
* Files with copyright information: 67 / 67
* Files with license information: 67 / 67
* Files with copyright information: 68 / 68
* Files with license information: 68 / 68
Congratulations! Your project is compliant with version 3.0 of the REUSE Specification :-)
......
# SPDX-FileCopyrightText: 2020 Helmholtz Centre Potsdam - GFZ German Research Centre for Geosciences Potsdam, Germany (https://www.gfz-potsdam.de/)
#
# SPDX-License-Identifier: CC0-1.0
list-used-licenses:
# exclude patterns for license check
exclude:
-
# matching of imported name to name in pkg manager
rename:
lib:
# mark local imports to exclude from license check
local_imports:
apply-license:
description: 'software description'
copyright: 'copyright holder'
time: 'year'
maintainer:
name: 'maintainer name'
mail: 'maintainer mail'
licenses:
data:
spdx-identifier: 'CC0-1.0'
copyright: 'copyright holder'
software:
spdx-identifier: 'CC0-1.0'
copyright: 'copyright holder'
pictures:
spdx-identifier: 'CC0-1.0'
copyright: 'copyright holder'
insignificant:
spdx-identifier: 'CC0-1.0'
copyright: 'copyright holder'
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment