Files
pygeoapi/pygeoapi/plugin.py
T
Richard Law 60202129ec implement processing manager (#365)
* add process manager

* process manager updating

* control connectivity from inside manager

* missing method on execute_process function signature

* adds tinydb depenedency for requirements-provider

* adds some processes tests

* process manager under server config, not a process

* additional html rendering for jobs and processes

* fixes typo in docstring

* complete html templating for jobs and processes

* considers timezone when formatting processing start/end datetimes

* expanded locales; locale and tz environment variables

* makes environment TZ a global from pygeoapi/__init__.py

* don't redefine built-in format

* adds placeholder values in process form if given as examples

* placeholder in form handles null example

* changes to process progress update

* updates and changes to processing web UI

* adds python3-distutils dep, and env changes for TZs and locales

* adds tinydb dependency for processes

* Adds libpq-dev and related deps for processing - should eventually be reconciled

* minor margin change

* fix bug in postJob.js that prevented results from displaying

* move process elements around to economise on space

* adds second, complex process

* post data may be bytes

* make external process example

* adds redis process manager (may be moved to third-party plugin eventually)

* reintroduce default async processing; refactoring managers; improving sync/async processing ui

* throw pygeoapi ManagerExecuteError, not redis exception

* adds test for async-execute

* restores original test host

* separates async execute handler function

* adds support for file uploads as processing inputs

* adds endpoint for file download as process output

* naming consistency

* Fix job submission form in case server url is relative (#549)

`new URL()` fails if the first argument isn't a complete url

* Implement job deletion in UI, api and tinydb

* Also allow deletion from job list

The UI could be fancier by reloading the page or just removing the
deleted job right away, but I don't want to start building that now.

* Delete output file when deleting a job

* typo in test case

* move delete button in job view

* remove Dockerfile changes (cf #470)

* removes some docs/source*.rst absent in master

* fixes whitespace

* fixes pip installs

* removes more locale-specific content, cf 70

* fixes duplicate app following merge

* removes redis as default plugin

* removes ENV_TZ, cf #407

* fix typo

* fixes syntax error

* process execution test cleanup

* Fix format detection in execute_process (#589)

Before, html was always selected.

* remove db files in code directories, fix flake8, specific functionality

* remove interactive capabilities in lieu of Swagger

* Add basic openapi documentation for job detail endpoint (#590)

It's somewhat superficial still, e.g. jobId is missing as
proper parameter in the UI.

* Show elapsed time for currently running jobs (#591)

* flake8, jobs workflow

* update docs, DB path in test config

* more code cleanup

* manager workflow cleanup

* add default dummy manager

* update docs

* fix routes

* fixed per comments

* fix per comments

* fix per comments

* fix travis

* simply describe process logic

* simplify describe process logic

Co-authored-by: Tom Kralidis <tomkralidis@gmail.com>
Co-authored-by: Jorge de Jesus <jorge.jesus@gmail.com>
Co-authored-by: totycro <bernhard.mallinger@eox.at>
Co-authored-by: Richard Law <richard.law@cartoncloud.com.au>
2020-12-22 06:53:31 -05:00

109 lines
3.7 KiB
Python

# =================================================================
#
# Authors: Tom Kralidis <tomkralidis@gmail.com>
#
# Copyright (c) 2020 Tom Kralidis
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation
# files (the "Software"), to deal in the Software without
# restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following
# conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
#
# =================================================================
"""Plugin loader"""
import importlib
import logging
LOGGER = logging.getLogger(__name__)
#: Loads provider plugins to be used by pygeoapi,\
#: formatters and processes available
PLUGINS = {
'provider': {
'CSV': 'pygeoapi.provider.csv_.CSVProvider',
'Elasticsearch': 'pygeoapi.provider.elasticsearch_.ElasticsearchProvider', # noqa
'GeoJSON': 'pygeoapi.provider.geojson.GeoJSONProvider',
'OGR': 'pygeoapi.provider.ogr.OGRProvider',
'PostgreSQL': 'pygeoapi.provider.postgresql.PostgreSQLProvider',
'SQLiteGPKG': 'pygeoapi.provider.sqlite.SQLiteGPKGProvider',
'MongoDB': 'pygeoapi.provider.mongo.MongoProvider',
'FileSystem': 'pygeoapi.provider.filesystem.FileSystemProvider',
'rasterio': 'pygeoapi.provider.rasterio_.RasterioProvider',
'xarray': 'pygeoapi.provider.xarray_.XarrayProvider',
'MVT': 'pygeoapi.provider.mvt.MVTProvider'
},
'formatter': {
'CSV': 'pygeoapi.formatter.csv_.CSVFormatter'
},
'process': {
'HelloWorld': 'pygeoapi.process.hello_world.HelloWorldProcessor'
},
'process_manager': {
'Dummy': 'pygeoapi.process.manager.dummy.DummyManager',
'TinyDB': 'pygeoapi.process.manager.tinydb_.TinyDBManager'
}
}
def load_plugin(plugin_type, plugin_def):
"""
loads plugin by name
:param plugin_type: type of plugin (provider, formatter)
:param plugin_def: plugin definition
:returns: plugin object
"""
name = plugin_def['name']
if plugin_type not in PLUGINS.keys():
msg = 'Plugin type {} not found'.format(plugin_type)
LOGGER.exception(msg)
raise InvalidPluginError(msg)
plugin_list = PLUGINS[plugin_type]
LOGGER.debug('Plugins: {}'.format(plugin_list))
if '.' not in name and name not in plugin_list.keys():
msg = 'Plugin {} not found'.format(name)
LOGGER.exception(msg)
raise InvalidPluginError(msg)
if '.' in name: # dotted path
packagename, classname = name.rsplit('.', 1)
else: # core formatter
packagename, classname = plugin_list[name].rsplit('.', 1)
LOGGER.debug('package name: {}'.format(packagename))
LOGGER.debug('class name: {}'.format(classname))
module = importlib.import_module(packagename)
class_ = getattr(module, classname)
plugin = class_(plugin_def)
return plugin
class InvalidPluginError(Exception):
"""Invalid plugin"""
pass