Metadata Submitter Backend¶
Note
Requirements:
- Python 3.8+
- MongoDB
Environment Setup¶
The application requires some environmental arguments in order to run properly, these are illustrated in the table below.
ENV | Default | Description | Mandatory |
MONGO_HOST |
localhost:27017 |
MongoDB server hostname, with port specified if needed. | Yes |
MONGO_AUTHDB |
- |
MongoDB authentication database. | Yes |
MONGO_DATABASE |
default |
MongoDB default database, will be used as authentication database if
MONGO_AUTHDB is not set. |
No |
MONGO_USERNAME |
admin |
Admin username for MongoDB. | Yes |
MONGO_PASSWORD |
admin |
Admin password for MongoDB. | Yes |
MONGO_SSL |
- |
Set to True to enable MongoDB TLS connection url. | No |
MONGO_SSL_CA |
- |
Path to CA file, required if MONGO_SSL enabled. |
No |
MONGO_SSL_CLIENT_CERT_KEY |
- |
Path to contains client’s TLS/SSL X.509 combined cert and key,
required if MONGO_SSL enabled. |
No |
AAI_CLIENT_SECRET |
public` |
OIDC client secret. | Yes |
AAI_CLIENT_ID |
secret |
OIDC client ID. | Yes |
BASE_URL |
http://localhost:5430 |
base URL of the metadata submitter. | Yes |
AUTH_METHOD |
code |
OIDC Authentication method to use. | No |
OIDC_URL |
- |
OIDC URL base URL, MUST resolve to configuration endpoint when appended with /.well-known/openid-configuration | Yes |
OIDC_SCOPE |
openid profile email |
Claims to request from AAI | No |
REDIRECT_URL |
- |
Required only for testing with front-end on localhost or change to
http://frontend:3000 if started using docker-compose (see Build and Deployment). |
No |
LOG_LEVEL |
INFO |
Set logging level, uppercase. | No |
SERVE_KEY |
- |
Keyfile used for TLS. | No |
SERVE_CERT |
- |
Certificate used for TLS. | No |
SERVE_CA |
- |
CA file used for TLS. | No |
SERVE_SLLVERSION |
- |
Version used for TLS, see the gunicorn documentation for ssl_version for more information. | No |
SERVE_CIPHERS |
- |
Ciphers used for TLS, see the gunicorn documentation for ciphers for more information. | No |
SERVE_CERTREQS |
- |
Client certificate requirement used for TLS, see the gunicorn documentation for cert_reqs for more information. | No |
Note
If just MONGO_DATABASE
is specified it will authenticate the user against it.
If just MONGO_AUTHDB
is specified it will authenticate the user against it.
If both MONGO_DATABASE
and MONGO_AUTHDB
are specified, the client will attempt to authenticate the specified user to the MONGO_AUTHDB database.
If both MONGO_DATABASE
and MONGO_AUTHDB
are unspecified, the client will attempt to authenticate the specified user to the admin database.
Install and run¶
For installing metadata-submitter
backend do the following:
$ git clone https://github.com/CSCfi/metadata-submitter
$ pip install .
Hint
Before running the application have MongoDB running.
MongoDB Server expects to find MongoDB instance running, specified with following environmental variables:
MONGO_INITDB_ROOT_USERNAME
(username for admin user to mongodb instance)MONGO_INITDB_ROOT_PASSWORD
(password for admin user to mongodb instance)MONGO_HOST
(host and port for MongoDB instance, e.g. localhost:27017)
To run the backend from command line set the environment variables required and use:
$ metadata_submitter
Hint
For a setup that requires also frontend follow the instructions in Build and Deployment.
Authentication¶
The Authentication follows the OIDC Specification.
We follow the steps of the OpenID Connect protocol.
- The RP (Client) sends a request to the OpenID Provider (OP),
for this we require
AAI_CLIENT_SECRET
,AAI_CLIENT_ID
,OIDC_URL
and a callback url constructed fromBASE_URL
. - The OP authenticates the End-User and obtains authorization.
- The OP responds with an ID Token and usually an Access Token, which are validated with configuration provided by
OIDC_URL
. - The RP can send a request with the Access Token to the UserInfo Endpoint.
- The UserInfo Endpoint returns Claims about the End-User, use claims
sub
,CSCUserName
orremoteUserIdentifier
to identify the user and start a session.
Information related to the OpenID Provider (OP) that needs to be configured is displayed in the table below.
Most of the information can be retrieved from OIDC Provider metadata
endpoint https://<provider_url>/.well-known/openid-configuration
.
ENV | Default | Description | Mandatory |
AAI_CLIENT_SECRET |
public` |
OIDC client secret. | Yes |
AAI_CLIENT_ID |
secret |
OIDC client ID. | Yes |
BASE_URL |
http://localhost:5430 |
base URL of the metadata submitter. | Yes |
AUTH_METHOD |
code |
OIDC Authentication method to use. | No |
OIDC_URL |
- |
OIDC URL base URL, MUST resolve to configuration endpoint when appended with /.well-known/openid-configuration | Yes |
OIDC_SCOPE |
openid profile email |
Claims to request from AAI | No |
REST API¶
View metadata submitter API in swagger editor.
The REST API is structured as follows:
- Submission Endpoints used in submitting data, mostly
POST
endpoints; - Query Endpoints used for data retrieval (
submissions
,objects
,users
) uses HTTPGET
; - Management Endpoints used for handling data updates and deletion, makes use of HTTP
PUT
,PATCH
andDELETE
.
The REST API is protected and can be accessed in two ways:
- Performing a successful login at frontend, which creates a session
- Using a
Bearer
token, which is issued byOIDC_URL
Important
A logged in user can only perform operations on the data it has associated.
The information for the current user can be retrieved at /v1/users/current
(the user ID is current
), and
any additional operations on other users are rejected.
Metadata Backend Modules¶
Backend for submitting and validating XML Files containing ENA metadata.
metadata_backend.api |
API endpoints and other api-related classes. |
metadata_backend.conf |
App configurations. |
metadata_backend.database |
Database services, initialisation and other tools. |
metadata_backend.helpers |
Helper tools, such as app configurations, logging and data validators. |
metadata_backend.server |
Metadata Backend API¶
API endpoints and other api-related classes.
metadata_backend.api.auth |
Handle Access for request and OIDC workflow. |
metadata_backend.api.handlers |
API handlers. |
metadata_backend.api.health |
Handle health check endpoint. |
metadata_backend.api.middlewares |
Middleware methods for server. |
metadata_backend.api.operators |
Operators for handling database-related operations. |
Database Operations¶
Database services, initialisation and other tools.
metadata_backend.database.db_service |
Services that handle database connections. |
Utility Functions¶
Helper tools, such as app configurations, logging and data validators.
metadata_backend.helpers.logger |
Logging formatting and functions for debugging. |
metadata_backend.helpers.parser |
Tool to parse XML and CSV files to JSON. |
metadata_backend.helpers.schema_loader |
Utility class to find XSD Schema that can be used to test XML files. |
metadata_backend.helpers.validator |
Utility classes for validating XML or JSON files. |