CanarieAPI documentation¶

CanarieAPI: Self describing REST service for Canarie registry¶

Provides documentation details, status monitoring and statistics reporting of services and platforms within the registered configuration.

Docker image available on DockerHub at pavics/canarieapi.

Refer to installation and usage to run the application locally.

User’s guide¶

Usage¶

Create a custom canarieapi/configuration.py file similar to canarieapi/default_configuration.py and define the environment variable CANARIE_API_CONFIG_FN pointing to it in order to use your desired configurations.

Run the CanarieAPI using gunicorn:

cd <CanarieAPI-root>/canarieapi
../bin/gunicorn -b 0.0.0.0:2000 --workers 1 --log-level=DEBUG --timeout 30 -k gevent wsgi

Run the monitoring and/or the log parsing task as cron jobs.

To run the log parsing job, add the following to a crontab file:

* * * * * python3 -c 'from canarieapi import logparser; logparser.cron_job()' 2>&1

The log parsing job requires that a nginx reverse proxy service is running and the DATABASE["access_log"] variable is set in the canarieapi/configuration.py script.

To run the the monitoring job, add the following to a crontab file:

* * * * * python3 -c 'from canarieapi import monitoring; monitoring.cron_job()' 2>&1

Package information¶

Installation¶

At the command line:

conda activate canarieapi     # or any other means of environment activation of your choice
make install

Contributing¶

Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

You can contribute in many ways:

Types of Contributions¶

Report Bugs¶

Report bugs at francois-xavier.derue@crim.ca.

If you are reporting a bug, please include:

Your operating system name and version.
Any details about your local setup that might be helpful in troubleshooting.
Detailed steps to reproduce the bug.

Write Documentation¶

Magpie could always use more documentation, whether as part of the official Magpie docs, in docstrings, or even on the web in blog posts, articles, and such.

Submit Feedback¶

The best way to send feedback is to file an issue at francois-xavier.derue@crim.ca.

If you are proposing a feature:

Explain in detail how it would work.
Keep the scope as narrow as possible, to make it easier to implement.
Remember that this is a volunteer-driven project, and that contributions are welcome :)

Get Started!¶

Ready to contribute? Here’s how to set up CanarieAPI for local development.

Clone the CanarieAPI repo from GitHub.

git clone https://github.com/Ouranosinc/CanarieAPI
Install your local copy and use a virtualenv. Assuming you have virtualenv installed, this is how you set up your fork for local development:
```
$ cd CanarieAPI/
$ virtualenv -p python 3.5 env
$ source env/bin/activate.csh
$ python setup.py develop
```
Now you can make your changes locally.
When you’re done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox:
```
$ flake8 CanarieAPI tests
$ python setup.py test
$ tox
```
To get flake8 and tox, just pip install them into your virtualenv.

Commit your changes and push your branch to GitHub:

$ git commit -m "Your detailed description of your changes."

Submit a pull request to the author.

Pull Request Guidelines¶

Before you submit a pull request, check that it meets these guidelines:

The pull request should include tests.
If the pull request adds functionality, the docs should be updated. Put your new functionality into a function with a docstring, and add the feature to the list in README.rst.
The tests should work for the specifid version of Python for this project.

Credits¶

Development Lead¶

David Byrns <david.byrns@crim.ca>

Contributors¶

Francis Charette-Migneault <francis.charette-migneault@crim.ca>

API Reference¶

This page contains auto-generated API reference documentation [1].

`canarieapi`¶

Submodules¶

`canarieapi.meta`¶

General meta information on the canarieapi package.

Module Contents¶

canarieapi.__meta__.__version__ = '1.0.0'[source]¶

canarieapi.__meta__.__author__ = 'David Byrns, Francis Charette-Migneault'[source]¶

canarieapi.__meta__.__email__ = 'francis.charette-migneault@crim.ca'[source]¶

canarieapi.__meta__.__title__ = 'Canarie API'[source]¶

canarieapi.__meta__.__description__ = 'CanarieAPI is a REST service for self-describing Canarie services and platforms'[source]¶

`canarieapi.api`¶

API definition.

This module defines the generic REST API for platforms and services as defined by the CANARIE API specification.

Module Contents¶

Functions¶

`handle_exceptions`(→ flask.typing.ResponseReturnValue)	Generate error response for raised exceptions.
`home`(→ flask.typing.ResponseReturnValue)
`manual_test`(→ flask.typing.ResponseReturnValue)
`extern_html_error_handler`(...)	Handle HTML errors from an external response.
`information`(→ flask.typing.ResponseReturnValue)	Info route required by CANARIE.
`collect_monitoring_statuses`(→ MonitorInfo)	Obtain all monitoring statuses for the requested service or platform.
`collect_cron_access_stats`(→ CronAccessStats)	Obtain access statuses of a service or platform from cron monitoring and logging jobs.
`collect_cron_last_status`(→ CronLastStatus)	Obtain the last time cron job have run (help to diagnose cron problem).
`stats`(→ flask.typing.ResponseReturnValue)	Stats route required by CANARIE.
`status`(→ flask.typing.ResponseReturnValue)	Extra route to know service status.
`simple_requests_handler`(→ flask.typing.ResponseReturnValue)	#Handle simple requests required by CANARIE.
`close_connection`(→ None)	Disconnect database.

Attributes¶

`CronAccessStats`
`CronLastStatus`
`MonitorStatus`
`MonitorInfo`
`START_UTC_TIME`
`CANARIE_API_TYPE`
`CANARIE_API_VALID_REQUESTS`
`_required`
`HANDLED_HTML_ERRORS`
`HANDLED_HTML_ERRORS_STR`

canarieapi.api.CronAccessStats[source]¶

canarieapi.api.CronLastStatus[source]¶

canarieapi.api.MonitorStatus[source]¶

canarieapi.api.MonitorInfo[source]¶

canarieapi.api.START_UTC_TIME[source]¶

canarieapi.api.CANARIE_API_TYPE = ['service', 'platform'][source]¶

canarieapi.api.CANARIE_API_VALID_REQUESTS[source]¶

canarieapi.api._required[source]¶

canarieapi.api.HANDLED_HTML_ERRORS = [400, 404, 405, 500, 503][source]¶

canarieapi.api.HANDLED_HTML_ERRORS_STR[source]¶

canarieapi.api.handle_exceptions(exception_instance: Exception) → flask.typing.ResponseReturnValue[source]¶

Generate error response for raised exceptions.

Parameters:: exception_instance – Exception instance.

canarieapi.api.home() → flask.typing.ResponseReturnValue[source]¶

canarieapi.api.manual_test() → flask.typing.ResponseReturnValue[source]¶

canarieapi.api.extern_html_error_handler(status_code_str: str) → flask.typing.ResponseReturnValue[source]¶

Handle HTML errors from an external response.

Handle errors that occur externally provided that Apache is configured so that it uses this route for handling errors.

For this add this line for each handled HTML errors in the Apache configuration:

ErrorDocument 400 <Rest root>/400

canarieapi.api.information(route_name: str, api_type: canarieapi.utility_rest.APIType) → flask.typing.ResponseReturnValue[source]¶: Info route required by CANARIE.

canarieapi.api.collect_monitoring_statuses(route_name: str, *, database: sqlite3.Connection | None = None) → MonitorInfo[source]¶: Obtain all monitoring statuses for the requested service or platform.

canarieapi.api.collect_cron_access_stats(route_name: str, *, database: sqlite3.Connection | None = None) → CronAccessStats[source]¶: Obtain access statuses of a service or platform from cron monitoring and logging jobs.

canarieapi.api.collect_cron_last_status(*, database: sqlite3.Connection | None = None) → CronLastStatus[source]¶: Obtain the last time cron job have run (help to diagnose cron problem).

canarieapi.api.stats(route_name: str, api_type: canarieapi.utility_rest.APIType) → flask.typing.ResponseReturnValue[source]¶: Stats route required by CANARIE.

canarieapi.api.status(route_name: str, api_type: canarieapi.utility_rest.APIType) → flask.typing.ResponseReturnValue[source]¶: Extra route to know service status.

canarieapi.api.simple_requests_handler(route_name: str, api_type: canarieapi.utility_rest.APIType, api_request: str = 'home') → flask.typing.ResponseReturnValue[source]¶: #Handle simple requests required by CANARIE.

canarieapi.api.close_connection(_: Exception) → None[source]¶

Disconnect database.

Parameters:: _ – Exception handled elsewhere, nothing to do with it

`canarieapi.app_object`¶

This module serves the purpose of centralizing the state object of Flask in a single place.

Module Contents¶

canarieapi.app_object.APP[source]¶

canarieapi.app_object.formatter[source]¶

canarieapi.app_object.ch[source]¶

`canarieapi.default_configuration`¶

Default configuration values for Canarie API.

To have the programs in the package override the values with the values found in this file, you need to set the environment variable named “CANARIE_API_CONFIG_FN” to the path of your own copy before launching the program.

Module Contents¶

canarieapi.default_configuration.MY_SERVER_NAME = 'http://localhost:2000'[source]¶

canarieapi.default_configuration.SERVER_MAIN_TITLE = 'Canarie API'[source]¶

canarieapi.default_configuration.PARSE_LOGS = True[source]¶

canarieapi.default_configuration.DATABASE[source]¶

canarieapi.default_configuration.SERVICES[source]¶

canarieapi.default_configuration.PLATFORMS[source]¶

`canarieapi.logparser`¶

Module Contents¶

Functions¶

`parse_log`(→ RouteStatistics)
`update_db`(→ None)
`cron_job`(→ None)

Attributes¶

RouteStatistics

canarieapi.logparser.RouteStatistics[source]¶

canarieapi.logparser.parse_log(filename: str, database: sqlite3.Connection | None = None) → RouteStatistics[source]¶

canarieapi.logparser.update_db(route_stats: RouteStatistics, database: sqlite3.Connection | None = None) → None[source]¶

canarieapi.logparser.cron_job() → None[source]¶

`canarieapi.monitoring`¶

Module Contents¶

Functions¶

`monitor`(→ None)
`check_service`(→ Tuple[canarieapi.status.Status, str])
`cron_job`(→ None)

Attributes¶

`Number`
`RequestConfig`
`ResponseConfig`

canarieapi.monitoring.Number[source]¶

canarieapi.monitoring.RequestConfig[source]¶

canarieapi.monitoring.ResponseConfig[source]¶

canarieapi.monitoring.monitor(*, update_db: bool = True, database: sqlite3.Connection | None = None) → None[source]¶

canarieapi.monitoring.check_service(request: RequestConfig, response: ResponseConfig) → Tuple[canarieapi.status.Status, str][source]¶

canarieapi.monitoring.cron_job() → None[source]¶

`canarieapi.reverse_proxied`¶

Middleware for reverse proxy.

This module implements a middleware that makes the Flask application work seamlessly behind a reverse proxy.

Module Contents¶

Classes¶

ReverseProxied

Class which implements a middleware so Flask can be used behind a reverse proxy.

class canarieapi.reverse_proxied.ReverseProxied(app: Any)[source]¶

Bases: object

Class which implements a middleware so Flask can be used behind a reverse proxy.

__call__(environ: Dict[str, str], start_response: Callable) → flask.Response[source]¶

`canarieapi.schema`¶

Module Contents¶

Functions¶

validate_config_schema(→ None)

Attributes¶

CONFIGURATION_SCHEMA

canarieapi.schema.CONFIGURATION_SCHEMA[source]¶

canarieapi.schema.validate_config_schema(update_db: bool, run_jobs: bool = True) → None[source]¶

`canarieapi.status`¶

Module Contents¶

Classes¶

Status

class canarieapi.status.Status[source]¶

ok = 'ok'[source]¶

bad = 'bad'[source]¶

down = 'down'[source]¶

static pretty_msg(status: Status) → str[source]¶

`canarieapi.utility_rest`¶

REST API utilities.

This module is a collection of utility functions used mainly by the rest_route module and which are placed here to keep the rest_route module as clean as possible.

Module Contents¶

Classes¶

`DatabaseRetryFunction`	Base class for protocol classes.
`AnyIntConverter`	Matches one of the items provided.

Functions¶

`request_wants_json`(→ bool)	Check if the request type is of type JSON considering both the `Accept` header and `f` query parameter.
`set_html_as_default_response`(→ None)	Set the default response Media-Type, with fallback to JSON.
`get_config`(→ JSON)	Return the config for the particular service/platform associated with the given route name.
`validate_route`(→ None)	Check if the route name is a value amongst known services/platforms in the configuration.
`get_api_title`(→ str)	Get the API title to be shown in rendered html page.
`get_canarie_api_response`(...)	Provide a valid response for the CANARIE API request based on the service route.
`make_error_response`(→ Tuple[Union[flask.Response, ...)	Make an error response based on the request type and given information.
`get_db`(→ sqlite3.Connection)	Get a connection to an existing database.
`init_db`(→ None)	Initialize a database from a schema.
`retry_db_error_after_init`(→ DatabaseRetryFunction)	Decorator that will retry a failing operation if an error related to database initialization occurred.

Attributes¶

`APIType`
`_JSON`
`JSON`
`ReturnType`

canarieapi.utility_rest.APIType[source]¶

canarieapi.utility_rest._JSON: typing_extensions.TypeAlias = 'JSON'[source]¶

canarieapi.utility_rest.JSON[source]¶

canarieapi.utility_rest.ReturnType[source]¶

canarieapi.utility_rest.request_wants_json() → bool[source]¶

Check if the request type is of type JSON considering both the Accept header and f query parameter.

The default Media-Type */* will be interpreted as JSON. Omitting a preferred type entirely will also default to JSON.

canarieapi.utility_rest.set_html_as_default_response() → None[source]¶

Set the default response Media-Type, with fallback to JSON.

By default, if the accepted mimetypes contains /, JSON format will be used. By calling this function, the / mimetype will be changed explicitly into text/html so that it becomes the mimetype used by default. This is useful for automatically rendering HTML by web browsers that do not provide explicitly the desired mimetype.

canarieapi.utility_rest.get_config(route_name: str, api_type: APIType) → JSON[source]¶

Return the config for the particular service/platform associated with the given route name.

Parameters:

route_name – Route name of the service/platform coming from the URL e.g. : [‘pavics’, ‘node’, ‘bias’, etc.]
api_type – Api type of the route which must be one of platform or service

Raises:

Exception if the route is unknown

canarieapi.utility_rest.validate_route(route_name: str, api_type: APIType) → None[source]¶

Check if the route name is a value amongst known services/platforms in the configuration.

Parameters:

route_name – Route name of the service/platform coming from the URL e.g. : [‘pavics’, ‘node’, ‘bias’, etc.]
api_type – Api type of the route which must be one of platform or service

Raises:

Exception if the route is unknown

canarieapi.utility_rest.get_api_title(route_name: str, api_type: APIType) → str[source]¶

Get the API title to be shown in rendered html page.

Parameters:

route_name – Route name of the service/platform coming from the URL e.g. : [‘pavics’, ‘node’, ‘bias’, etc.]
api_type – Api type of the route which must be one of platform or service

Returns:

An API title

canarieapi.utility_rest.get_canarie_api_response(route_name: str, api_type: APIType, api_request: str) → flask.typing.ResponseReturnValue[source]¶

Provide a valid response for the CANARIE API request based on the service route.

Parameters:

route_name – Route name of the service/platform coming from the URL e.g. : [‘pavics’, ‘node’, ‘bias’, etc.]
api_type – Api type of the route which must be one of platform or service
api_request – The request specified in the URL

Returns:

A valid HTML response

canarieapi.utility_rest.make_error_response(http_status: int | None = None, http_status_response: str | None = None) → Tuple[flask.Response | str, int][source]¶

Make an error response based on the request type and given information.

Parameters:

http_status – HTTP status
http_status_response – Standard message associated with a status code. Obtained via http.client.responses if not provided.

canarieapi.utility_rest.get_db(allow_cache: bool = True, connect: bool = True) → sqlite3.Connection[source]¶

Get a connection to an existing database.

If the database does not exist, create a connection to local sqlite3 file. If the local sqlite3 file doesn’t exist, initialize it using a schema.

Stores the established connection in the application’s global context to reuse it whenever required.

canarieapi.utility_rest.init_db(database: sqlite3.Connection) → None[source]¶: Initialize a database from a schema.

class canarieapi.utility_rest.DatabaseRetryFunction[source]¶

Bases: typing_extensions.Protocol

Base class for protocol classes.

Protocol classes are defined as:

class Proto(Protocol):
    def meth(self) -> int:
        ...

Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing), for example:

class C:
    def meth(self) -> int:
        return 0

def func(x: Proto) -> int:
    return x.meth()

func(C())  # Passes static type check

See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as:

class GenProto(Protocol[T]):
    def meth(self) -> T:
        ...

__call__(*args: Any, database: sqlite3.Connection | None = None, **kwargs: Any) → ReturnType[source]¶

canarieapi.utility_rest.retry_db_error_after_init(func: DatabaseRetryFunction) → DatabaseRetryFunction[source]¶: Decorator that will retry a failing operation if an error related to database initialization occurred.

class canarieapi.utility_rest.AnyIntConverter(mapping: werkzeug.routing.Map, *items: int | str)[source]¶

Bases: werkzeug.routing.BaseConverter

Matches one of the items provided.

Items must be integer and comma separated with a space to avoid confusion with floating point value in the parser.

For example:

1, 2, 3

And not:

1,2,3

Since it would parse as float 1,2 and 3.

CanarieAPI documentation¶

CanarieAPI: Self describing REST service for Canarie registry¶

User’s guide¶

Usage¶

Package information¶

Installation¶

Contributing¶

Types of Contributions¶

Report Bugs¶

Write Documentation¶

Submit Feedback¶

Get Started!¶

Pull Request Guidelines¶

Credits¶

Development Lead¶

Contributors¶

API Reference¶

Submodules¶

Module Contents¶

Module Contents¶

Functions¶

Attributes¶

Module Contents¶

Module Contents¶

Module Contents¶

Functions¶

Attributes¶

Module Contents¶

Functions¶

Attributes¶

Module Contents¶

Classes¶

Module Contents¶

Functions¶

Attributes¶

Module Contents¶

Classes¶

Module Contents¶

Classes¶

Functions¶

Attributes¶

Indices and tables¶