Add cql-json support for ES (#723)
Fix starlette event loop Fix starlette event loop Fix starlette event loop Fix starlette event loop Fix provider regression Make method public Make method public Move function to the helpers utility Add the CQL lifecycle for development Add CQL docs Fix flake8 Isolate import for starlette codepath
This commit is contained in:
Francesco Bartoli
committed by
GitHub
GitHub
parent
0f38c764d6
commit
bb4cd0bf69
@@ -50,6 +50,7 @@ jobs:
|
||||
- name: Install requirements 📦
|
||||
run: |
|
||||
pip3 install -r requirements.txt
|
||||
pip3 install -r requirements-starlette.txt
|
||||
pip3 install -r requirements-dev.txt
|
||||
pip3 install -r requirements-provider.txt
|
||||
python3 setup.py install
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
.. _cql:
|
||||
|
||||
CQL support
|
||||
===========
|
||||
|
||||
Limitations
|
||||
-----------
|
||||
|
||||
The support to CQL is limited to `Simple CQL filter <https://portal.ogc.org/files/96288#cql-core>`_ and thus it allows to query with the
|
||||
following predicates:
|
||||
|
||||
- comparison predicates
|
||||
- spatial predicates
|
||||
- temporal predicates
|
||||
|
||||
Formats
|
||||
-------
|
||||
|
||||
At the moment pygeoapi supports only the CQL dialect with the JSON encoding `CQL-JSON <https://portal.ogc.org/files/96288#simple-cql-JSON>`_.
|
||||
|
||||
Providers
|
||||
---------
|
||||
|
||||
As of now the available providers supported for CQL filtering are limited to only :ref:`Elasticsearch <Elasticsearch>`.
|
||||
|
||||
Queries
|
||||
^^^^^^^
|
||||
|
||||
The following type of queries are supported right now:
|
||||
|
||||
- ``between`` predicate query
|
||||
- Logical ``and`` query with ``between`` and ``eq`` expression
|
||||
- Spatial query with ``bbox``
|
||||
|
||||
Examples
|
||||
^^^^^^^^
|
||||
|
||||
A ``between`` example for a specific property through an HTTP POST request:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl --location --request POST 'http://localhost:5000/collections/nhsl_hazard_threat_all_indicators_s_bc/items?f=json&limit=50&filter-lang=cql-json' \
|
||||
--header 'Content-Type: application/query-cql-json' \
|
||||
--data-raw '{
|
||||
"between": {
|
||||
"value": { "property": "properties.MHn_Intensity" },
|
||||
"lower": 0.59,
|
||||
"upper": 0.60
|
||||
}
|
||||
}'
|
||||
@@ -15,17 +15,17 @@ pygeoapi core feature providers are listed below, along with a matrix of support
|
||||
parameters.
|
||||
|
||||
.. csv-table::
|
||||
:header: Provider, properties (filters), resulttype, bbox, datetime, sortby, properties (display)
|
||||
:header: Provider, properties (filters), resulttype, bbox, datetime, sortby, properties (display), CQL
|
||||
:align: left
|
||||
|
||||
CSV,❌,results/hits,❌,❌,❌,✅
|
||||
Elasticsearch,✅,results/hits,✅,✅,✅,✅
|
||||
GeoJSON,❌,results/hits,❌,❌,❌,❌
|
||||
MongoDB,✅,results,✅,✅,✅,❌
|
||||
OGR,✅,results/hits,✅,❌,❌,❌
|
||||
PostgreSQL,✅,results/hits,✅,❌,❌,❌
|
||||
SQLiteGPKG,✅,results/hits,✅,❌,❌,❌
|
||||
SensorThingsAPI,✅,results/hits,✅,✅,✅,✅
|
||||
CSV,❌,results/hits,❌,❌,❌,✅,❌
|
||||
Elasticsearch,✅,results/hits,✅,✅,✅,✅,✅
|
||||
GeoJSON,❌,results/hits,❌,❌,❌,❌,❌
|
||||
MongoDB,✅,results,✅,✅,✅,❌,❌
|
||||
OGR,✅,results/hits,✅,❌,❌,❌,❌
|
||||
PostgreSQL,✅,results/hits,✅,❌,❌,❌,❌
|
||||
SQLiteGPKG,✅,results/hits,✅,❌,❌,❌,❌
|
||||
SensorThingsAPI,✅,results/hits,✅,✅,✅,✅,❌
|
||||
|
||||
|
||||
Below are specific connection examples based on supported providers.
|
||||
@@ -65,6 +65,7 @@ To publish a GeoJSON file, the file must be a valid GeoJSON FeatureCollection.
|
||||
data: tests/data/file.json
|
||||
id_field: id
|
||||
|
||||
.. _Elasticsearch:
|
||||
|
||||
Elasticsearch
|
||||
^^^^^^^^^^^^^
|
||||
@@ -87,6 +88,11 @@ To publish an Elasticsearch index, the following are required in your index:
|
||||
id_field: geonameid
|
||||
time_field: datetimefield
|
||||
|
||||
This provider has the support for the CQL queries as indicated in the table above.
|
||||
|
||||
.. seealso::
|
||||
:ref:`cql` for more details on how to use the Common Query Language to filter the collection with specific queries.
|
||||
|
||||
OGR
|
||||
^^^
|
||||
|
||||
|
||||
@@ -23,6 +23,39 @@ To run all tests, simply run ``pytest`` in the repository. To run a specific te
|
||||
run ``pytest tests/test_api.py``, for example.
|
||||
|
||||
|
||||
CQL extension lifecycle
|
||||
-----------------------
|
||||
|
||||
Limitations
|
||||
^^^^^^^^^^^
|
||||
|
||||
This workflow is valid only for the `CQL-JSON` format.
|
||||
|
||||
Schema
|
||||
^^^^^^
|
||||
|
||||
The Common Query Language (CQL) is the part 3 of the standard OGC API - Features. This extension has its specification available at
|
||||
`OGC API - Features - Part 3: Filtering and the Common Query Language (CQL) <https://portal.ogc.org/files/96288>`_ and the schema exists in development at
|
||||
`cql.json <https://github.com/opengeospatial/ogcapi-features/blob/master/extensions/cql/standard/schema/cql.json>`_.
|
||||
|
||||
Model generation
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
pygeoapi uses a class-based python model interface to translate the schema into python objects defined by `pydantic <https://pydantic-docs.helpmanual.io/>`_ models.
|
||||
The model is generated with the pre-processing of the schema through the utility ``datamodel-codegen``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Generate from local downloaded json schema file
|
||||
datamodel-codegen --input ~/Download/cql-schema.json --input-file-type jsonschema --output ./pygeoapi/models/cql_update.py --class-name CQLModel
|
||||
|
||||
How to merge
|
||||
^^^^^^^^^^^^
|
||||
|
||||
Once the new pydantic models have been generated then the content of the python file ``cql_update.py`` can be used to replace the old classes within the ``cql.py`` file.
|
||||
Update everything above the function ``get_next_node`` and then verify if the tests for the CQL are still passing, for example ``test_post_cql_json_between_query``
|
||||
in ``tests/test_elasticsearch__provider.py``.
|
||||
|
||||
Working with Spatialite on OSX
|
||||
------------------------------
|
||||
|
||||
|
||||
@@ -29,6 +29,7 @@ pygeoapi |release| documentation
|
||||
data-publishing/index
|
||||
plugins
|
||||
html-templating
|
||||
cql
|
||||
language
|
||||
development
|
||||
ogc-compliance
|
||||
|
||||
+286
-17
@@ -66,7 +66,7 @@ from pygeoapi.provider.base import (
|
||||
from pygeoapi.provider.tile import (ProviderTileNotFoundError,
|
||||
ProviderTileQueryError,
|
||||
ProviderTilesetIdNotFoundError)
|
||||
|
||||
from pygeoapi.models.cql import CQLModel
|
||||
from pygeoapi.util import (dategetter, DATETIME_FORMAT,
|
||||
filter_dict_by_key_value, get_provider_by_type,
|
||||
get_provider_default, get_typed_value, JobStatus,
|
||||
@@ -219,7 +219,10 @@ class APIRequest:
|
||||
self._args = self._get_params(request)
|
||||
|
||||
# Get path info
|
||||
self._path_info = request.headers.environ['PATH_INFO'].strip('/')
|
||||
if hasattr(request, 'scope'):
|
||||
self._path_info = request.scope['path'].strip('/')
|
||||
elif hasattr(request.headers, 'environ'):
|
||||
self._path_info = request.headers.environ['PATH_INFO'].strip('/')
|
||||
|
||||
# Extract locale from params or headers
|
||||
self._raw_locale, self._locale = self._get_locale(request.headers,
|
||||
@@ -228,6 +231,9 @@ class APIRequest:
|
||||
# Determine format
|
||||
self._format = self._get_format(request.headers)
|
||||
|
||||
# Get received headers
|
||||
self._headers = self.get_request_headers(request.headers)
|
||||
|
||||
@classmethod
|
||||
def with_data(cls, request, supported_locales) -> 'APIRequest':
|
||||
"""
|
||||
@@ -250,11 +256,17 @@ class APIRequest:
|
||||
# Set data from Flask request
|
||||
api_req._data = request.data
|
||||
elif hasattr(request, 'body'):
|
||||
# Set data from Starlette request after async coroutine completion
|
||||
# TODO: this now blocks, but once Flask v2 with async support
|
||||
# has been implemented, with_data() can become async too
|
||||
loop = asyncio.get_event_loop()
|
||||
api_req._data = loop.run_until_complete(request.body())
|
||||
try:
|
||||
import nest_asyncio
|
||||
nest_asyncio.apply()
|
||||
# Set data from Starlette request after async
|
||||
# coroutine completion
|
||||
# TODO: this now blocks, but once Flask v2 with async support
|
||||
# has been implemented, with_data() can become async too
|
||||
loop = asyncio.get_event_loop()
|
||||
api_req._data = loop.run_until_complete(request.body())
|
||||
except ModuleNotFoundError:
|
||||
LOGGER.error("Module nest-asyncio not found")
|
||||
return api_req
|
||||
|
||||
@staticmethod
|
||||
@@ -399,6 +411,17 @@ class APIRequest:
|
||||
|
||||
return self._format
|
||||
|
||||
@property
|
||||
def headers(self) -> dict:
|
||||
"""
|
||||
Returns the dictionary of the headers from
|
||||
the request.
|
||||
|
||||
:returns: Request headers dictionary
|
||||
"""
|
||||
|
||||
return self._headers
|
||||
|
||||
def get_linkrel(self, format_: str) -> str:
|
||||
"""
|
||||
Returns the hyperlink relationship (rel) attribute value for
|
||||
@@ -477,6 +500,19 @@ class APIRequest:
|
||||
headers['Content-Type'] = FORMAT_TYPES[self._format]
|
||||
return headers
|
||||
|
||||
def get_request_headers(self, headers) -> dict:
|
||||
"""
|
||||
Obtains and returns a dictionary with Request object headers.
|
||||
|
||||
This method adds the headers of the original request and
|
||||
makes them available to the API object.
|
||||
|
||||
:returns: A header dict
|
||||
"""
|
||||
|
||||
headers_ = {item[0]: item[1] for item in headers.items()}
|
||||
return headers_
|
||||
|
||||
|
||||
class API:
|
||||
"""API object"""
|
||||
@@ -1104,13 +1140,12 @@ class API:
|
||||
@pre_process
|
||||
def get_collection_items(
|
||||
self, request: Union[APIRequest, Any],
|
||||
dataset, pathinfo=None) -> Tuple[dict, int, str]:
|
||||
dataset) -> Tuple[dict, int, str]:
|
||||
"""
|
||||
Queries collection
|
||||
|
||||
:param request: A request object
|
||||
:param dataset: dataset name
|
||||
:param pathinfo: path location
|
||||
|
||||
:returns: tuple of headers, status code, content
|
||||
"""
|
||||
@@ -1393,14 +1428,9 @@ class API:
|
||||
|
||||
if request.format == F_HTML: # render
|
||||
# For constructing proper URIs to items
|
||||
if pathinfo:
|
||||
path_info = '/'.join([
|
||||
self.config['server']['url'].rstrip('/'),
|
||||
pathinfo.strip('/')])
|
||||
else:
|
||||
path_info = '/'.join([
|
||||
self.config['server']['url'].rstrip('/'),
|
||||
request.path_info])
|
||||
path_info = '/'.join([
|
||||
self.config['server']['url'].rstrip('/'),
|
||||
request.path_info])
|
||||
|
||||
content['items_path'] = path_info
|
||||
content['dataset_path'] = '/'.join(path_info.split('/')[:-1])
|
||||
@@ -1451,6 +1481,245 @@ class API:
|
||||
|
||||
return headers, 200, to_json(content, self.pretty_print)
|
||||
|
||||
@pre_process
|
||||
def post_collection_items(
|
||||
self, request: Union[APIRequest, Any],
|
||||
dataset) -> Tuple[dict, int, str]:
|
||||
"""
|
||||
Queries collection or filter an item
|
||||
|
||||
:param request: A request object
|
||||
:param dataset: dataset name
|
||||
|
||||
:returns: tuple of headers, status code, content
|
||||
"""
|
||||
|
||||
request_headers = request.headers
|
||||
|
||||
if not request.is_valid(PLUGINS['formatter'].keys()):
|
||||
return self.get_format_exception(request)
|
||||
|
||||
# Set Content-Language to system locale until provider locale
|
||||
# has been determined
|
||||
headers = request.get_response_headers(SYSTEM_LOCALE)
|
||||
|
||||
properties = []
|
||||
reserved_fieldnames = ['bbox', 'f', 'limit', 'startindex',
|
||||
'resulttype', 'datetime', 'sortby',
|
||||
'properties', 'skipGeometry', 'q',
|
||||
'filter-lang']
|
||||
|
||||
collections = filter_dict_by_key_value(self.config['resources'],
|
||||
'type', 'collection')
|
||||
|
||||
if dataset not in collections.keys():
|
||||
msg = 'Invalid collection'
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'InvalidParameterValue', msg)
|
||||
|
||||
LOGGER.debug('Processing query parameters')
|
||||
|
||||
LOGGER.debug('Processing startindex parameter')
|
||||
try:
|
||||
startindex = int(request.params.get('startindex'))
|
||||
if startindex < 0:
|
||||
msg = 'startindex value should be positive or zero'
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'InvalidParameterValue', msg)
|
||||
except TypeError as err:
|
||||
LOGGER.warning(err)
|
||||
startindex = 0
|
||||
except ValueError:
|
||||
msg = 'startindex value should be an integer'
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'InvalidParameterValue', msg)
|
||||
|
||||
LOGGER.debug('Processing limit parameter')
|
||||
try:
|
||||
limit = int(request.params.get('limit'))
|
||||
# TODO: We should do more validation, against the min and max
|
||||
# allowed by the server configuration
|
||||
if limit <= 0:
|
||||
msg = 'limit value should be strictly positive'
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'InvalidParameterValue', msg)
|
||||
except TypeError as err:
|
||||
LOGGER.warning(err)
|
||||
limit = int(self.config['server']['limit'])
|
||||
except ValueError:
|
||||
msg = 'limit value should be an integer'
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'InvalidParameterValue', msg)
|
||||
|
||||
resulttype = request.params.get('resulttype') or 'results'
|
||||
|
||||
LOGGER.debug('Processing bbox parameter')
|
||||
|
||||
bbox = request.params.get('bbox')
|
||||
|
||||
if bbox is None:
|
||||
bbox = []
|
||||
else:
|
||||
try:
|
||||
bbox = validate_bbox(bbox)
|
||||
except ValueError as err:
|
||||
msg = str(err)
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'InvalidParameterValue', msg)
|
||||
|
||||
LOGGER.debug('Processing datetime parameter')
|
||||
datetime_ = request.params.get('datetime')
|
||||
try:
|
||||
datetime_ = validate_datetime(collections[dataset]['extents'],
|
||||
datetime_)
|
||||
except ValueError as err:
|
||||
msg = str(err)
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'InvalidParameterValue', msg)
|
||||
|
||||
LOGGER.debug('processing q parameter')
|
||||
val = request.params.get('q')
|
||||
|
||||
q = None
|
||||
if val is not None:
|
||||
q = val
|
||||
|
||||
LOGGER.debug('Loading provider')
|
||||
|
||||
try:
|
||||
p = load_plugin('provider', get_provider_by_type(
|
||||
collections[dataset]['providers'], 'feature'))
|
||||
except ProviderTypeError:
|
||||
try:
|
||||
p = load_plugin('provider', get_provider_by_type(
|
||||
collections[dataset]['providers'], 'record'))
|
||||
except ProviderTypeError:
|
||||
msg = 'Invalid provider type'
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'NoApplicableCode', msg)
|
||||
except ProviderConnectionError:
|
||||
msg = 'connection error (check logs)'
|
||||
return self.get_exception(
|
||||
500, headers, request.format, 'NoApplicableCode', msg)
|
||||
except ProviderQueryError:
|
||||
msg = 'query error (check logs)'
|
||||
return self.get_exception(
|
||||
500, headers, request.format, 'NoApplicableCode', msg)
|
||||
|
||||
LOGGER.debug('processing property parameters')
|
||||
for k, v in request.params.items():
|
||||
if k not in reserved_fieldnames and k not in p.fields.keys():
|
||||
msg = 'unknown query parameter: {}'.format(k)
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'InvalidParameterValue', msg)
|
||||
elif k not in reserved_fieldnames and k in p.fields.keys():
|
||||
LOGGER.debug('Add property filter {}={}'.format(k, v))
|
||||
properties.append((k, v))
|
||||
|
||||
LOGGER.debug('processing sort parameter')
|
||||
val = request.params.get('sortby')
|
||||
|
||||
if val is not None:
|
||||
sortby = []
|
||||
sorts = val.split(',')
|
||||
for s in sorts:
|
||||
prop = s
|
||||
order = '+'
|
||||
if s[0] in ['+', '-']:
|
||||
order = s[0]
|
||||
prop = s[1:]
|
||||
|
||||
if prop not in p.fields.keys():
|
||||
msg = 'bad sort property'
|
||||
return self.get_exception(
|
||||
400, headers, request.format,
|
||||
'InvalidParameterValue', msg)
|
||||
|
||||
sortby.append({'property': prop, 'order': order})
|
||||
else:
|
||||
sortby = []
|
||||
|
||||
LOGGER.debug('processing properties parameter')
|
||||
val = request.params.get('properties')
|
||||
|
||||
if val is not None:
|
||||
select_properties = val.split(',')
|
||||
properties_to_check = set(p.properties) | set(p.fields.keys())
|
||||
|
||||
if (len(list(set(select_properties) -
|
||||
set(properties_to_check))) > 0):
|
||||
msg = 'unknown properties specified'
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'InvalidParameterValue', msg)
|
||||
else:
|
||||
select_properties = []
|
||||
|
||||
LOGGER.debug('processing skipGeometry parameter')
|
||||
val = request.params.get('skipGeometry')
|
||||
if val is not None:
|
||||
skip_geometry = str2bool(val)
|
||||
else:
|
||||
skip_geometry = False
|
||||
|
||||
LOGGER.debug('Processing filter-lang parameter')
|
||||
filter_lang = request.params.get('filter-lang')
|
||||
if filter_lang == 'cql-json': # @TODO add check from the configuration
|
||||
val = filter_lang
|
||||
else:
|
||||
msg = 'Invalid filter language'
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'InvalidParameterValue', msg)
|
||||
|
||||
LOGGER.debug('Querying provider')
|
||||
LOGGER.debug('startindex: {}'.format(startindex))
|
||||
LOGGER.debug('limit: {}'.format(limit))
|
||||
LOGGER.debug('resulttype: {}'.format(resulttype))
|
||||
LOGGER.debug('sortby: {}'.format(sortby))
|
||||
LOGGER.debug('bbox: {}'.format(bbox))
|
||||
LOGGER.debug('datetime: {}'.format(datetime_))
|
||||
LOGGER.debug('properties: {}'.format(select_properties))
|
||||
LOGGER.debug('skipGeometry: {}'.format(skip_geometry))
|
||||
LOGGER.debug('q: {}'.format(q))
|
||||
LOGGER.debug('filter-lang: {}'.format(filter_lang))
|
||||
|
||||
LOGGER.debug('Processing headers')
|
||||
|
||||
LOGGER.debug('Processing request content-type header')
|
||||
if (request_headers.get(
|
||||
'Content-Type') or request_headers.get(
|
||||
'content-type')) != 'application/query-cql-json':
|
||||
msg = ('Invalid body content-type')
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'InvalidHeaderValue', msg)
|
||||
|
||||
LOGGER.debug('Processing body')
|
||||
|
||||
if not request.data:
|
||||
msg = 'missing request data'
|
||||
return self.get_exception(
|
||||
400, headers, request.format, 'MissingParameterValue', msg)
|
||||
|
||||
try:
|
||||
# Parse bytes data, if applicable
|
||||
data = request.data.decode()
|
||||
LOGGER.debug(data)
|
||||
# @TODO validation function
|
||||
filter_ = None
|
||||
if val:
|
||||
filter_ = CQLModel.parse_raw(data)
|
||||
content = p.query(startindex=startindex, limit=limit,
|
||||
resulttype=resulttype, bbox=bbox,
|
||||
datetime_=datetime_, properties=properties,
|
||||
sortby=sortby,
|
||||
select_properties=select_properties,
|
||||
skip_geometry=skip_geometry,
|
||||
q=q,
|
||||
filterq=filter_)
|
||||
except (UnicodeDecodeError, AttributeError):
|
||||
pass
|
||||
|
||||
return headers, 200, to_json(content, self.pretty_print)
|
||||
|
||||
@pre_process
|
||||
def get_collection_item(self, request: Union[APIRequest, Any],
|
||||
dataset, identifier) -> Tuple[dict, int, str]:
|
||||
|
||||
+11
-4
@@ -175,7 +175,7 @@ def collection_queryables(collection_id=None):
|
||||
return get_response(api_.get_collection_queryables(request, collection_id))
|
||||
|
||||
|
||||
@BLUEPRINT.route('/collections/<collection_id>/items')
|
||||
@BLUEPRINT.route('/collections/<collection_id>/items', methods=['GET', 'POST'])
|
||||
@BLUEPRINT.route('/collections/<collection_id>/items/<item_id>')
|
||||
def collection_items(collection_id, item_id=None):
|
||||
"""
|
||||
@@ -187,9 +187,16 @@ def collection_items(collection_id, item_id=None):
|
||||
:returns: HTTP response
|
||||
"""
|
||||
if item_id is None:
|
||||
return get_response(api_.get_collection_items(request, collection_id))
|
||||
return get_response(
|
||||
api_.get_collection_item(request, collection_id, item_id))
|
||||
if request.method == 'GET': # list items
|
||||
return get_response(
|
||||
api_.get_collection_items(request, collection_id))
|
||||
elif request.method == 'POST': # filter items
|
||||
return get_response(
|
||||
api_.post_collection_items(request, collection_id))
|
||||
|
||||
else:
|
||||
return get_response(
|
||||
api_.get_collection_item(request, collection_id, item_id))
|
||||
|
||||
|
||||
@BLUEPRINT.route('/collections/<collection_id>/coverage')
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
# =================================================================
|
||||
#
|
||||
# Authors: Francesco Bartoli <xbartolone@gmail.com>
|
||||
#
|
||||
# Copyright (c) 2021 Francesco Bartoli
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person
|
||||
# obtaining a copy of this software and associated documentation
|
||||
# files (the "Software"), to deal in the Software without
|
||||
# restriction, including without limitation the rights to use,
|
||||
# copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
# copies of the Software, and to permit persons to whom the
|
||||
# Software is furnished to do so, subject to the following
|
||||
# conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
#
|
||||
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
|
||||
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
||||
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
|
||||
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
|
||||
# OTHER DEALINGS IN THE SOFTWARE.
|
||||
#
|
||||
# =================================================================
|
||||
|
||||
"""Interface module containing the models for the openapi/json schema"""
|
||||
@@ -0,0 +1,527 @@
|
||||
# ****************************** -*-
|
||||
# flake8: noqa
|
||||
# generated by datamodel-codegen:
|
||||
# filename: cql-schema.json
|
||||
# timestamp: 2021-03-13T21:05:20+00:00
|
||||
# =================================================================
|
||||
#
|
||||
# Authors: Francesco Bartoli <xbartolone@gmail.com>
|
||||
#
|
||||
# Copyright (c) 2021 Francesco Bartoli
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person
|
||||
# obtaining a copy of this software and associated documentation
|
||||
# files (the "Software"), to deal in the Software without
|
||||
# restriction, including without limitation the rights to use,
|
||||
# copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
# copies of the Software, and to permit persons to whom the
|
||||
# Software is furnished to do so, subject to the following
|
||||
# conditions:
|
||||
#
|
||||
# The above copyright notice and this permission notice shall be
|
||||
# included in all copies or substantial portions of the Software.
|
||||
#
|
||||
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
|
||||
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
||||
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
|
||||
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
|
||||
# OTHER DEALINGS IN THE SOFTWARE.
|
||||
#
|
||||
# =================================================================
|
||||
|
||||
from datetime import date, datetime
|
||||
from enum import Enum
|
||||
from typing import Any, List, Optional, Union
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class CQLModel(BaseModel):
|
||||
__root__: 'Union[\n ComparisonPredicate,\n SpatialPredicate,\n TemporalPredicate,\n AndExpression\n ]'
|
||||
|
||||
|
||||
class AndExpression(BaseModel):
|
||||
and_: 'List[ComparisonPredicate]' = Field(..., alias='and')
|
||||
|
||||
|
||||
class NotExpression(BaseModel):
|
||||
not_: 'List[Any]' = Field(..., alias='not')
|
||||
|
||||
|
||||
class OrExpression(BaseModel):
|
||||
or_: 'List[Any]' = Field(..., alias='or')
|
||||
|
||||
|
||||
class PropertyRef(BaseModel):
|
||||
property: 'Optional[str]' = None
|
||||
|
||||
|
||||
class ScalarLiteral(BaseModel):
|
||||
__root__: 'Union[str, float, bool]'
|
||||
|
||||
|
||||
class Bbox(BaseModel):
|
||||
__root__: 'List[float]'
|
||||
|
||||
|
||||
class LineStringType(Enum):
|
||||
LineString = 'LineString'
|
||||
|
||||
|
||||
class LinestringCoordinate(BaseModel):
|
||||
__root__: 'List[Any]'
|
||||
|
||||
|
||||
class Linestring(BaseModel):
|
||||
type: 'LineStringType'
|
||||
coordinates: 'List[LinestringCoordinate]' = Field(...)
|
||||
bbox: 'Optional[List[float]]' = Field(None)
|
||||
|
||||
|
||||
class MultiLineStringType(Enum):
|
||||
MultiLineString = 'MultiLineString'
|
||||
|
||||
|
||||
class MultilineStringCoordinate(BaseModel):
|
||||
__root__: 'List[Any]'
|
||||
|
||||
|
||||
class Multilinestring(BaseModel):
|
||||
type: 'MultiLineStringType'
|
||||
coordinates: 'List[List[MultilineStringCoordinate]]'
|
||||
bbox: 'Optional[List[float]]' = Field(None)
|
||||
|
||||
|
||||
class MultiPointType(Enum):
|
||||
MultiPoint = 'MultiPoint'
|
||||
|
||||
|
||||
class Multipoint(BaseModel):
|
||||
type: 'MultiPointType'
|
||||
coordinates: 'List[List[float]]'
|
||||
bbox: 'Optional[List[float]]' = Field(None)
|
||||
|
||||
|
||||
class MultiPolygonType(Enum):
|
||||
MultiPolygon = 'MultiPolygon'
|
||||
|
||||
|
||||
class MultipolygonCoordinateItem(BaseModel):
|
||||
__root__: 'List[Any]'
|
||||
|
||||
|
||||
class Multipolygon(BaseModel):
|
||||
type: 'MultiPolygonType'
|
||||
coordinates: 'List[List[List[MultipolygonCoordinateItem]]]'
|
||||
bbox: 'Optional[List[float]]' = Field(None)
|
||||
|
||||
|
||||
class PointType(Enum):
|
||||
Point = 'Point'
|
||||
|
||||
|
||||
class Point(BaseModel):
|
||||
type: 'PointType'
|
||||
coordinates: 'List[float]' = Field(...)
|
||||
bbox: 'Optional[List[float]]' = Field(None)
|
||||
|
||||
|
||||
class PolygonType(Enum):
|
||||
Polygon = 'Polygon'
|
||||
|
||||
|
||||
class PolygonCoordinatesItem(BaseModel):
|
||||
__root__: 'List[Any]'
|
||||
|
||||
|
||||
class Polygon(BaseModel):
|
||||
type: 'PolygonType'
|
||||
coordinates: 'List[List[PolygonCoordinatesItem]]'
|
||||
bbox: 'Optional[List[float]]' = Field(None)
|
||||
|
||||
|
||||
class TimeString(BaseModel):
|
||||
__root__: 'Union[date, datetime]'
|
||||
|
||||
|
||||
class EnvelopeLiteral(BaseModel):
|
||||
bbox: 'Bbox'
|
||||
|
||||
|
||||
class GeometryLiteral(BaseModel):
|
||||
__root__: 'Union[\n Point, Linestring, Polygon, Multipoint, Multilinestring, Multipolygon\n ]'
|
||||
|
||||
|
||||
class TypedTimeString(BaseModel):
|
||||
datetime: 'TimeString'
|
||||
|
||||
|
||||
class PeriodString(BaseModel):
|
||||
__root__: 'List[Union[TimeString, str]]' = Field(...)
|
||||
|
||||
|
||||
class SpatialLiteral(BaseModel):
|
||||
__root__: 'Union[GeometryLiteral, EnvelopeLiteral]'
|
||||
|
||||
|
||||
class TemporalLiteral(BaseModel):
|
||||
__root__: 'Union[TimeString, PeriodString]'
|
||||
|
||||
|
||||
class TypedPeriodString(BaseModel):
|
||||
datetime: 'PeriodString'
|
||||
|
||||
|
||||
class TypedTemporalLiteral(BaseModel):
|
||||
__root__: 'Union[TypedTimeString, TypedPeriodString]'
|
||||
|
||||
|
||||
class ArrayPredicate(BaseModel):
|
||||
__root__: 'Union[\n AequalsExpression,\n AcontainsExpression,\n AcontainedByExpression,\n AoverlapsExpression,\n ]'
|
||||
|
||||
|
||||
class ComparisonPredicate(BaseModel):
|
||||
__root__: 'Union[\n BinaryComparisonPredicate,\n IsLikePredicate,\n IsBetweenPredicate,\n IsInListPredicate,\n IsNullPredicate,\n ]'
|
||||
|
||||
|
||||
class SpatialPredicate(BaseModel):
|
||||
__root__: 'Union[\n IntersectsExpression,\n EqualsExpression,\n DisjointExpression,\n TouchesExpression,\n WithinExpression,\n OverlapsExpression,\n CrossesExpression,\n ContainsExpression,\n ]'
|
||||
|
||||
|
||||
class TemporalPredicate(BaseModel):
|
||||
__root__: 'Union[\n BeforeExpression,\n AfterExpression,\n MeetsExpression,\n MetbyExpression,\n ToverlapsExpression,\n OverlappedbyExpression,\n BeginsExpression,\n BegunbyExpression,\n DuringExpression,\n TcontainsExpression,\n EndsExpression,\n EndedbyExpression,\n TequalsExpression,\n AnyinteractsExpression,\n ]'
|
||||
|
||||
|
||||
class AcontainedByExpression(BaseModel):
|
||||
acontainedBy: 'ArrayExpression'
|
||||
|
||||
|
||||
class AcontainsExpression(BaseModel):
|
||||
acontains: 'ArrayExpression'
|
||||
|
||||
|
||||
class AequalsExpression(BaseModel):
|
||||
aequals: 'ArrayExpression'
|
||||
|
||||
|
||||
class AfterExpression(BaseModel):
|
||||
after: 'TemporalOperands'
|
||||
|
||||
|
||||
class AnyinteractsExpression(BaseModel):
|
||||
anyinteracts: 'TemporalOperands'
|
||||
|
||||
|
||||
class AoverlapsExpression(BaseModel):
|
||||
aoverlaps: 'ArrayExpression'
|
||||
|
||||
|
||||
class BeforeExpression(BaseModel):
|
||||
before: 'TemporalOperands'
|
||||
|
||||
|
||||
class BeginsExpression(BaseModel):
|
||||
begins: 'TemporalOperands'
|
||||
|
||||
|
||||
class BegunbyExpression(BaseModel):
|
||||
begunby: 'TemporalOperands'
|
||||
|
||||
|
||||
class BinaryComparisonPredicate(BaseModel):
|
||||
__root__: 'Union[\n EqExpression, LtExpression, GtExpression, LteExpression, GteExpression\n ]'
|
||||
|
||||
|
||||
class ContainsExpression(BaseModel):
|
||||
contains: 'SpatialOperands'
|
||||
|
||||
|
||||
class CrossesExpression(BaseModel):
|
||||
crosses: 'SpatialOperands'
|
||||
|
||||
|
||||
class DisjointExpression(BaseModel):
|
||||
disjoint: 'SpatialOperands'
|
||||
|
||||
|
||||
class DuringExpression(BaseModel):
|
||||
during: 'TemporalOperands'
|
||||
|
||||
|
||||
class EndedbyExpression(BaseModel):
|
||||
endedby: 'TemporalOperands'
|
||||
|
||||
|
||||
class EndsExpression(BaseModel):
|
||||
ends: 'TemporalOperands'
|
||||
|
||||
|
||||
class EqualsExpression(BaseModel):
|
||||
equals: 'SpatialOperands'
|
||||
|
||||
|
||||
class IntersectsExpression(BaseModel):
|
||||
intersects: 'SpatialOperands'
|
||||
|
||||
|
||||
class Between(BaseModel):
|
||||
value: 'ValueExpression'
|
||||
lower: 'ScalarExpression' = Field(None)
|
||||
upper: 'ScalarExpression' = Field(None)
|
||||
|
||||
|
||||
class IsBetweenPredicate(BaseModel):
|
||||
between: 'Between'
|
||||
|
||||
|
||||
class In(BaseModel):
|
||||
value: 'ValueExpression'
|
||||
list: 'List[ValueExpression]'
|
||||
nocase: 'Optional[bool]' = True
|
||||
|
||||
|
||||
class IsInListPredicate(BaseModel):
|
||||
in_: 'In' = Field(..., alias='in')
|
||||
|
||||
|
||||
class IsLikePredicate(BaseModel):
|
||||
like: 'ScalarOperands'
|
||||
wildcard: 'Optional[str]' = '%'
|
||||
singleChar: 'Optional[str]' = '.'
|
||||
escapeChar: 'Optional[str]' = '\\'
|
||||
nocase: 'Optional[bool]' = True
|
||||
|
||||
|
||||
class IsNullPredicate(BaseModel):
|
||||
isNull: 'ScalarExpression'
|
||||
|
||||
|
||||
class MeetsExpression(BaseModel):
|
||||
meets: 'TemporalOperands'
|
||||
|
||||
|
||||
class MetbyExpression(BaseModel):
|
||||
metby: 'TemporalOperands'
|
||||
|
||||
|
||||
class OverlappedbyExpression(BaseModel):
|
||||
overlappedby: 'TemporalOperands'
|
||||
|
||||
|
||||
class OverlapsExpression(BaseModel):
|
||||
overlaps: 'SpatialOperands'
|
||||
|
||||
|
||||
class TcontainsExpression(BaseModel):
|
||||
tcontains: 'TemporalOperands'
|
||||
|
||||
|
||||
class TequalsExpression(BaseModel):
|
||||
tequals: 'TemporalOperands'
|
||||
|
||||
|
||||
class TouchesExpression(BaseModel):
|
||||
touches: 'SpatialOperands'
|
||||
|
||||
|
||||
class ToverlapsExpression(BaseModel):
|
||||
toverlaps: 'TemporalOperands'
|
||||
|
||||
|
||||
class WithinExpression(BaseModel):
|
||||
within: 'SpatialOperands'
|
||||
|
||||
|
||||
class ArrayExpression(BaseModel):
|
||||
__root__: 'List[Union[PropertyRef, FunctionRef, ArrayLiteral]]' = Field(
|
||||
... # , max_items=2, min_items=2
|
||||
)
|
||||
|
||||
|
||||
class EqExpression(BaseModel):
|
||||
eq: 'ScalarOperands'
|
||||
|
||||
|
||||
class GtExpression(BaseModel):
|
||||
gt: 'ScalarOperands'
|
||||
|
||||
|
||||
class GteExpression(BaseModel):
|
||||
gte: 'ScalarOperands'
|
||||
|
||||
|
||||
class LtExpression(BaseModel):
|
||||
lt: 'ScalarOperands'
|
||||
|
||||
|
||||
class LteExpression(BaseModel):
|
||||
lte: 'ScalarOperands'
|
||||
|
||||
|
||||
class ScalarExpression(BaseModel):
|
||||
__root__: 'Union[ScalarLiteral, PropertyRef,\n FunctionRef, ArithmeticExpression]'
|
||||
|
||||
|
||||
class ScalarOperands(BaseModel):
|
||||
__root__: 'List[ScalarExpression]' = Field(...)
|
||||
|
||||
|
||||
class SpatialOperands(BaseModel):
|
||||
__root__: 'List[GeomExpression]' = Field(...)
|
||||
|
||||
|
||||
class TemporalOperands(BaseModel):
|
||||
__root__: 'List[TemporalExpression]' = Field(...)
|
||||
# , max_items=2, min_items=2)
|
||||
|
||||
|
||||
class ValueExpression(BaseModel):
|
||||
__root__: 'Union[ScalarExpression, SpatialLiteral, TypedTemporalLiteral]'
|
||||
|
||||
|
||||
class ArithmeticExpression(BaseModel):
|
||||
__root__: 'Union[AddExpression, SubExpression, MulExpression, DivExpression]'
|
||||
|
||||
|
||||
class ArrayLiteral(BaseModel):
|
||||
__root__: 'List[\n Union[\n ScalarLiteral,\n SpatialLiteral,\n TypedTemporalLiteral,\n PropertyRef,\n FunctionRef,\n ArithmeticExpression,\n ArrayLiteral,\n ]\n ]'
|
||||
|
||||
|
||||
class FunctionRef(BaseModel):
|
||||
function: 'Function'
|
||||
|
||||
|
||||
class GeomExpression(BaseModel):
|
||||
__root__: 'Union[SpatialLiteral, PropertyRef, FunctionRef]'
|
||||
|
||||
|
||||
class TemporalExpression(BaseModel):
|
||||
__root__: 'Union[TemporalLiteral, PropertyRef, FunctionRef]'
|
||||
|
||||
|
||||
class AddExpression(BaseModel):
|
||||
_: 'ArithmeticOperands' = Field(..., alias='+')
|
||||
|
||||
|
||||
class DivExpression(BaseModel):
|
||||
_: 'Optional[ArithmeticOperands]' = Field(None, alias='/')
|
||||
|
||||
|
||||
class Function(BaseModel):
|
||||
name: 'str'
|
||||
arguments: 'Optional[\n List[\n Union[\n ScalarLiteral,\n SpatialLiteral,\n TypedTemporalLiteral,\n PropertyRef,\n FunctionRef,\n ArithmeticExpression,\n ArrayLiteral,\n ]\n ]\n ]' = None
|
||||
|
||||
|
||||
class MulExpression(BaseModel):
|
||||
_: 'ArithmeticOperands' = Field(..., alias='*')
|
||||
|
||||
|
||||
class SubExpression(BaseModel):
|
||||
_: 'ArithmeticOperands' = Field(..., alias='-')
|
||||
|
||||
|
||||
class ArithmeticOperands(BaseModel):
|
||||
__root__: 'List[\n Union[ArithmeticExpression, PropertyRef, FunctionRef, float]\n ]' = Field(...)
|
||||
|
||||
|
||||
CQLModel.update_forward_refs()
|
||||
AndExpression.update_forward_refs()
|
||||
ArrayPredicate.update_forward_refs()
|
||||
ComparisonPredicate.update_forward_refs()
|
||||
SpatialPredicate.update_forward_refs()
|
||||
TemporalPredicate.update_forward_refs()
|
||||
AcontainedByExpression.update_forward_refs()
|
||||
AcontainsExpression.update_forward_refs()
|
||||
AequalsExpression.update_forward_refs()
|
||||
AfterExpression.update_forward_refs()
|
||||
AnyinteractsExpression.update_forward_refs()
|
||||
AoverlapsExpression.update_forward_refs()
|
||||
BeforeExpression.update_forward_refs()
|
||||
BeginsExpression.update_forward_refs()
|
||||
BegunbyExpression.update_forward_refs()
|
||||
BinaryComparisonPredicate.update_forward_refs()
|
||||
ContainsExpression.update_forward_refs()
|
||||
CrossesExpression.update_forward_refs()
|
||||
DisjointExpression.update_forward_refs()
|
||||
DuringExpression.update_forward_refs()
|
||||
EndedbyExpression.update_forward_refs()
|
||||
EndsExpression.update_forward_refs()
|
||||
EqualsExpression.update_forward_refs()
|
||||
IntersectsExpression.update_forward_refs()
|
||||
Between.update_forward_refs()
|
||||
In.update_forward_refs()
|
||||
IsBetweenPredicate.update_forward_refs()
|
||||
IsLikePredicate.update_forward_refs()
|
||||
IsNullPredicate.update_forward_refs()
|
||||
ValueExpression.update_forward_refs()
|
||||
MeetsExpression.update_forward_refs()
|
||||
MetbyExpression.update_forward_refs()
|
||||
OverlappedbyExpression.update_forward_refs()
|
||||
OverlapsExpression.update_forward_refs()
|
||||
TcontainsExpression.update_forward_refs()
|
||||
TequalsExpression.update_forward_refs()
|
||||
TouchesExpression.update_forward_refs()
|
||||
ToverlapsExpression.update_forward_refs()
|
||||
WithinExpression.update_forward_refs()
|
||||
ArrayExpression.update_forward_refs()
|
||||
EqExpression.update_forward_refs()
|
||||
GtExpression.update_forward_refs()
|
||||
GteExpression.update_forward_refs()
|
||||
LtExpression.update_forward_refs()
|
||||
LteExpression.update_forward_refs()
|
||||
ScalarExpression.update_forward_refs()
|
||||
ScalarOperands.update_forward_refs()
|
||||
SpatialOperands.update_forward_refs()
|
||||
TemporalOperands.update_forward_refs()
|
||||
ArithmeticExpression.update_forward_refs()
|
||||
ArrayLiteral.update_forward_refs()
|
||||
ScalarLiteral.update_forward_refs()
|
||||
PropertyRef.update_forward_refs()
|
||||
FunctionRef.update_forward_refs()
|
||||
AddExpression.update_forward_refs()
|
||||
DivExpression.update_forward_refs()
|
||||
MulExpression.update_forward_refs()
|
||||
SubExpression.update_forward_refs()
|
||||
|
||||
|
||||
def get_next_node(obj):
|
||||
logical_op = None
|
||||
if obj.__repr_name__() == 'AndExpression':
|
||||
next_node = obj.and_
|
||||
logical_op = 'and'
|
||||
elif obj.__repr_name__() == 'OrExpression':
|
||||
next_node = obj.or_
|
||||
logical_op = 'or'
|
||||
elif obj.__repr_name__() == 'NotExpression':
|
||||
next_node = obj.not_
|
||||
logical_op = 'not'
|
||||
elif obj.__repr_name__() == 'ComparisonPredicate':
|
||||
next_node = obj.__root__
|
||||
elif obj.__repr_name__() == 'SpatialPredicate':
|
||||
next_node = obj.__root__
|
||||
elif obj.__repr_name__() == 'TemporalPredicate':
|
||||
next_node = obj.__root__
|
||||
elif obj.__repr_name__() == 'IsBetweenPredicate':
|
||||
next_node = obj.between
|
||||
elif obj.__repr_name__() == 'Between':
|
||||
next_node = obj.value
|
||||
elif obj.__repr_name__() == 'ValueExpression':
|
||||
next_node = obj.__root__ or obj.lower or obj.upper
|
||||
elif obj.__repr_name__() == 'ScalarExpression':
|
||||
next_node = obj.__root__
|
||||
elif obj.__repr_name__() == 'ScalarLiteral':
|
||||
next_node = obj.__root__
|
||||
elif obj.__repr_name__() == 'PropertyRef':
|
||||
next_node = obj.property
|
||||
elif obj.__repr_name__() == 'BinaryComparisonPredicate':
|
||||
next_node = obj.__root__
|
||||
elif obj.__repr_name__() == 'EqExpression':
|
||||
next_node = obj.eq
|
||||
logical_op = 'eq'
|
||||
else:
|
||||
raise ValueError("Object not valid")
|
||||
|
||||
return (logical_op, next_node)
|
||||
@@ -3,6 +3,7 @@
|
||||
# Authors: Tom Kralidis <tomkralidis@gmail.com>
|
||||
#
|
||||
# Copyright (c) 2021 Tom Kralidis
|
||||
# Copyright (c) 2021 Francesco Bartoli
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person
|
||||
# obtaining a copy of this software and associated documentation
|
||||
@@ -27,6 +28,7 @@
|
||||
#
|
||||
# =================================================================
|
||||
|
||||
from typing import Dict
|
||||
from collections import OrderedDict
|
||||
import json
|
||||
import logging
|
||||
@@ -34,10 +36,14 @@ from urllib.parse import urlparse
|
||||
|
||||
from elasticsearch import Elasticsearch, exceptions, helpers
|
||||
from elasticsearch.client.indices import IndicesClient
|
||||
from elasticsearch_dsl import Search, Q
|
||||
|
||||
from pygeoapi.provider.base import (BaseProvider, ProviderConnectionError,
|
||||
ProviderQueryError,
|
||||
ProviderItemNotFoundError)
|
||||
from pygeoapi.models.cql import CQLModel, get_next_node
|
||||
from pygeoapi.util import get_envelope
|
||||
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
@@ -144,7 +150,8 @@ class ElasticsearchProvider(BaseProvider):
|
||||
|
||||
def query(self, startindex=0, limit=10, resulttype='results',
|
||||
bbox=[], datetime_=None, properties=[], sortby=[],
|
||||
select_properties=[], skip_geometry=False, q=None, **kwargs):
|
||||
select_properties=[], skip_geometry=False, q=None,
|
||||
filterq=None, **kwargs):
|
||||
"""
|
||||
query Elasticsearch index
|
||||
|
||||
@@ -158,6 +165,7 @@ class ElasticsearchProvider(BaseProvider):
|
||||
:param select_properties: list of property names
|
||||
:param skip_geometry: bool of whether to skip geometry (default False)
|
||||
:param q: full-text search term(s)
|
||||
:param filterq: filter object
|
||||
|
||||
:returns: dict of 0..n GeoJSON features
|
||||
"""
|
||||
@@ -291,6 +299,9 @@ class ElasticsearchProvider(BaseProvider):
|
||||
query['_source'] = {'excludes': ['geometry']}
|
||||
try:
|
||||
LOGGER.debug('querying Elasticsearch')
|
||||
if filterq:
|
||||
LOGGER.debug('adding cql object: {}'.format(filterq.json()))
|
||||
query = update_query(input_query=query, cql=filterq)
|
||||
LOGGER.debug(json.dumps(query, indent=4))
|
||||
|
||||
LOGGER.debug('Setting ES paging zero-based')
|
||||
@@ -495,3 +506,187 @@ class ElasticsearchCatalogueProvider(ElasticsearchProvider):
|
||||
|
||||
def __repr__(self):
|
||||
return '<ElasticsearchCatalogueProvider> {}'.format(self.data)
|
||||
|
||||
|
||||
class ESQueryBuilder:
|
||||
def __init__(self):
|
||||
self._operation = None
|
||||
self.must_value = {}
|
||||
self.should_value = {}
|
||||
self.mustnot_value = {}
|
||||
self.filter_value = {}
|
||||
|
||||
def must(self, must_value):
|
||||
self.must_value = must_value
|
||||
return self
|
||||
|
||||
def should(self, should_value):
|
||||
self.should_value = should_value
|
||||
return self
|
||||
|
||||
def must_not(self, mustnot_value):
|
||||
self.mustnot_value = mustnot_value
|
||||
return self
|
||||
|
||||
def filter(self, filter_value):
|
||||
self.filter_value = filter_value
|
||||
return self
|
||||
|
||||
@property
|
||||
def operation(self):
|
||||
return self._operation
|
||||
|
||||
@operation.setter
|
||||
def operation(self, value):
|
||||
self._operation = value
|
||||
|
||||
def build(self):
|
||||
if self.must_value:
|
||||
must_clause = self.must_value or {}
|
||||
if self.should_value:
|
||||
should_clause = self.should_value or {}
|
||||
if self.mustnot_value:
|
||||
mustnot_clause = self.mustnot_value or {}
|
||||
if self.filter_value:
|
||||
filter_clause = self.filter_value or {}
|
||||
else:
|
||||
filter_clause = {}
|
||||
|
||||
# to figure out how to deal with logical operations
|
||||
# return match_clause & range_clause
|
||||
clauses = must_clause or should_clause or mustnot_clause
|
||||
filters = filter_clause
|
||||
if self.operation == 'and':
|
||||
res = Q(
|
||||
'bool',
|
||||
must=[clause for clause in clauses],
|
||||
filter=[filter for filter in filters])
|
||||
elif self.operation == 'or':
|
||||
res = Q(
|
||||
'bool',
|
||||
should=[clause for clause in clauses],
|
||||
filter=[filter for filter in filters])
|
||||
elif self.operation == 'not':
|
||||
res = Q(
|
||||
'bool',
|
||||
must_not=[clause for clause in clauses],
|
||||
filter=[filter for filter in filters])
|
||||
else:
|
||||
if filters:
|
||||
res = Q(
|
||||
'bool',
|
||||
must=[clauses],
|
||||
filter=[filters])
|
||||
else:
|
||||
res = Q(
|
||||
'bool',
|
||||
must=[clauses])
|
||||
|
||||
return res
|
||||
|
||||
|
||||
def _build_query(q, cql):
|
||||
|
||||
# this would be handled by the AST with the traverse of CQL model
|
||||
op, node = get_next_node(cql.__root__)
|
||||
q.operation = op
|
||||
if isinstance(node, list):
|
||||
query_list = []
|
||||
for elem in node:
|
||||
op, next_node = get_next_node(elem)
|
||||
if not getattr(next_node, 'between', 0) == 0:
|
||||
property = next_node.between.value.__root__.__root__.property
|
||||
lower = next_node.between.lower.__root__.__root__
|
||||
upper = next_node.between.upper.__root__.__root__
|
||||
query_list.append(Q(
|
||||
{
|
||||
'range':
|
||||
{
|
||||
f'{property}': {
|
||||
'gte': lower, 'lte': upper
|
||||
}
|
||||
}
|
||||
}
|
||||
))
|
||||
if not getattr(next_node, '__root__', 0) == 0:
|
||||
scalars = tuple(next_node.__root__.eq.__root__)
|
||||
property = scalars[0].__root__.property
|
||||
value = scalars[1].__root__.__root__
|
||||
query_list.append(Q(
|
||||
{'match': {f'{property}': f'{value}'}}
|
||||
))
|
||||
q.must(query_list)
|
||||
elif not getattr(node, 'between', 0) == 0:
|
||||
property = node.between.value.__root__.__root__.property
|
||||
lower = None
|
||||
if not getattr(node.between.lower,
|
||||
'__root__', 0) == 0:
|
||||
lower = node.between.lower.__root__.__root__
|
||||
upper = None
|
||||
if not getattr(node.between.upper,
|
||||
'__root__', 0) == 0:
|
||||
upper = node.between.upper.__root__.__root__
|
||||
query = Q(
|
||||
{
|
||||
'range':
|
||||
{
|
||||
f'{property}': {
|
||||
'gte': lower, 'lte': upper
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
q.must(query)
|
||||
elif not getattr(node, '__root__', 0) == 0:
|
||||
next_op, next_node = get_next_node(node)
|
||||
if not getattr(next_node, 'eq', 0) == 0:
|
||||
scalars = tuple(next_node.eq.__root__)
|
||||
property = scalars[0].__root__.property
|
||||
value = scalars[1].__root__.__root__
|
||||
query = Q(
|
||||
{'match': {f'{property}': f'{value}'}}
|
||||
)
|
||||
q.must(query)
|
||||
elif not getattr(node, 'intersects', 0) == 0:
|
||||
property = node.intersects.__root__[0].__root__.property
|
||||
if property == 'geometry':
|
||||
geom_type = node.intersects.__root__[
|
||||
1].__root__.__root__.__root__.type
|
||||
if geom_type.value == 'Polygon':
|
||||
coordinates = node.intersects.__root__[
|
||||
1].__root__.__root__.__root__.coordinates
|
||||
coords_list = [
|
||||
poly_coords.__root__ for poly_coords in coordinates[0]
|
||||
]
|
||||
filter_ = Q(
|
||||
{
|
||||
'geo_shape': {
|
||||
'geometry': {
|
||||
'shape': {
|
||||
'type': 'envelope',
|
||||
'coordinates': get_envelope(
|
||||
coords_list)
|
||||
},
|
||||
'relation': 'intersects'
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
query_all = Q(
|
||||
{'match_all': {}}
|
||||
)
|
||||
q.must(query_all)
|
||||
q.filter(filter_)
|
||||
return q.build()
|
||||
|
||||
|
||||
def update_query(input_query: Dict, cql: CQLModel):
|
||||
s = Search.from_dict(input_query)
|
||||
query = ESQueryBuilder()
|
||||
output_query = _build_query(query, cql)
|
||||
s = s.query(output_query)
|
||||
|
||||
LOGGER.debug('Enhanced query: {}'.format(
|
||||
json.dumps(s.to_dict())
|
||||
))
|
||||
return s.to_dict()
|
||||
|
||||
@@ -220,8 +220,8 @@ async def get_collection_items_tiles(request: Request, name=None,
|
||||
request, name, tileMatrixSetId, tile_matrix, tileRow, tileCol))
|
||||
|
||||
|
||||
@app.route('/collections/{collection_id}/items')
|
||||
@app.route('/collections/{collection_id}/items/')
|
||||
@app.route('/collections/{collection_id}/items', methods=['GET', 'POST'])
|
||||
@app.route('/collections/{collection_id}/items/', methods=['GET', 'POST'])
|
||||
@app.route('/collections/{collection_id}/items/{item_id}')
|
||||
@app.route('/collections/{collection_id}/items/{item_id}/')
|
||||
async def collection_items(request: Request, collection_id=None, item_id=None):
|
||||
@@ -239,8 +239,14 @@ async def collection_items(request: Request, collection_id=None, item_id=None):
|
||||
if 'item_id' in request.path_params:
|
||||
item_id = request.path_params['item_id']
|
||||
if item_id is None:
|
||||
return get_response(api_.get_collection_items(
|
||||
request, collection_id, pathinfo=request.scope['path']))
|
||||
if request.method == 'GET': # list items
|
||||
return get_response(
|
||||
api_.get_collection_items(
|
||||
request, collection_id))
|
||||
elif request.method == 'POST': # filter items
|
||||
return get_response(
|
||||
api_.post_collection_items(
|
||||
request, collection_id))
|
||||
else:
|
||||
return get_response(api_.get_collection_item(
|
||||
request, collection_id, item_id))
|
||||
|
||||
@@ -30,6 +30,7 @@
|
||||
"""Generic util functions used in the code"""
|
||||
|
||||
import base64
|
||||
from typing import List
|
||||
from datetime import date, datetime, time
|
||||
from decimal import Decimal
|
||||
from enum import Enum
|
||||
@@ -41,6 +42,7 @@ import os
|
||||
import re
|
||||
from urllib.request import urlopen
|
||||
from urllib.parse import urlparse
|
||||
from shapely.geometry import Polygon
|
||||
|
||||
import dateutil.parser
|
||||
# from babel.support import Translations
|
||||
@@ -511,3 +513,20 @@ def url_join(*parts):
|
||||
"""
|
||||
|
||||
return '/'.join([p.strip().strip('/') for p in parts])
|
||||
|
||||
|
||||
def get_envelope(coords_list: List[List[float]]):
|
||||
"""
|
||||
helper function to get the envelope for a given coordinates
|
||||
list through the Shapely API.
|
||||
|
||||
:param coords_list: list of coordinates
|
||||
|
||||
:returns: list of the envelope's coordinates
|
||||
"""
|
||||
|
||||
coords = [tuple(item) for item in coords_list]
|
||||
polygon = Polygon(coords)
|
||||
bounds = polygon.bounds
|
||||
return [[bounds[0], bounds[3]],
|
||||
[bounds[2], bounds[1]]]
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
# Flask-based CORS setup
|
||||
flask_cors
|
||||
|
||||
# Generate pydantic models from json schema
|
||||
datamodel-code-generator
|
||||
|
||||
# testing
|
||||
pytest
|
||||
pytest-cov
|
||||
|
||||
@@ -10,3 +10,4 @@ pymongo==3.10.1
|
||||
scipy
|
||||
xarray
|
||||
zarr
|
||||
elasticsearch-dsl
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
aiofiles
|
||||
starlette
|
||||
uvicorn
|
||||
nest-asyncio
|
||||
|
||||
@@ -10,3 +10,5 @@ rasterio
|
||||
shapely
|
||||
tinydb
|
||||
unicodecsv
|
||||
future-annotations
|
||||
pydantic
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -3,6 +3,7 @@
|
||||
# Authors: Tom Kralidis <tomkralidis@gmail.com>
|
||||
#
|
||||
# Copyright (c) 2020 Tom Kralidis
|
||||
# Copyright (c) 2021 Francesco Bartoli
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person
|
||||
# obtaining a copy of this software and associated documentation
|
||||
@@ -31,6 +32,7 @@ import pytest
|
||||
|
||||
from pygeoapi.provider.base import ProviderItemNotFoundError
|
||||
from pygeoapi.provider.elasticsearch_ import ElasticsearchProvider
|
||||
from pygeoapi.models.cql import CQLModel
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
@@ -43,6 +45,105 @@ def config():
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def config_cql():
|
||||
return {
|
||||
'name': 'Elasticsearch',
|
||||
'type': 'feature',
|
||||
'data': 'http://localhost:9200/nhsl_hazard_threat_all_indicators_s_bc', # noqa
|
||||
'id_field': 'Sauid'
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def between():
|
||||
between_ = {
|
||||
"between": {
|
||||
"value": {"property": "properties.pop_max"},
|
||||
"lower": 10000,
|
||||
"upper": 100000
|
||||
}
|
||||
}
|
||||
return CQLModel.parse_obj(between_)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def between_upper():
|
||||
between_ = {
|
||||
"between": {
|
||||
"value": {"property": "properties.pop_max"},
|
||||
"upper": 100000
|
||||
}
|
||||
}
|
||||
return CQLModel.parse_obj(between_)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def between_lower():
|
||||
between_ = {
|
||||
"between": {
|
||||
"value": {"property": "properties.pop_max"},
|
||||
"lower": 10000
|
||||
}
|
||||
}
|
||||
return CQLModel.parse_obj(between_)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def eq():
|
||||
eq_ = {
|
||||
"eq": [
|
||||
{"property": "properties.featurecla"},
|
||||
"Admin-0 capital"
|
||||
]
|
||||
}
|
||||
return CQLModel.parse_obj(eq_)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def _and(eq, between):
|
||||
and_ = {
|
||||
"and": [
|
||||
{
|
||||
"between": {
|
||||
"value": {
|
||||
"property": "properties.pop_max"
|
||||
},
|
||||
"lower": 100000,
|
||||
"upper": 1000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"eq": [
|
||||
{"property": "properties.featurecla"},
|
||||
"Admin-0 capital"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
return CQLModel.parse_obj(and_)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def intersects():
|
||||
intersects = {"intersects": [
|
||||
{"property": "geometry"},
|
||||
{
|
||||
"type": "Polygon",
|
||||
"coordinates": [
|
||||
[
|
||||
[10.497565, 41.520355],
|
||||
[10.497565, 43.308645],
|
||||
[15.111823, 43.308645],
|
||||
[15.111823, 41.520355],
|
||||
[10.497565, 41.520355]
|
||||
]
|
||||
]
|
||||
}
|
||||
]}
|
||||
return CQLModel.parse_obj(intersects)
|
||||
|
||||
|
||||
def test_query(config):
|
||||
p = ElasticsearchProvider(config)
|
||||
|
||||
@@ -121,3 +222,72 @@ def test_get_not_existing_item_raise_exception(config):
|
||||
p = ElasticsearchProvider(config)
|
||||
with pytest.raises(ProviderItemNotFoundError):
|
||||
p.get('404')
|
||||
|
||||
|
||||
def test_post_cql_json_between_query(config, between):
|
||||
"""Testing cql json query for a between object"""
|
||||
p = ElasticsearchProvider(config)
|
||||
|
||||
results = p.query(limit=100, filterq=between)
|
||||
|
||||
assert len(results['features']) == 23
|
||||
assert results['numberMatched'] == 23
|
||||
assert results['numberReturned'] == 23
|
||||
|
||||
for item in results['features']:
|
||||
assert 10000 <= item["properties"]["pop_max"] <= 100000
|
||||
|
||||
|
||||
def test_post_cql_json_between_lte_query(config, between_upper):
|
||||
"""Testing cql json query for a between object"""
|
||||
p = ElasticsearchProvider(config)
|
||||
|
||||
results = p.query(limit=100, filterq=between_upper)
|
||||
|
||||
assert len(results['features']) == 28
|
||||
assert results['numberMatched'] == 28
|
||||
assert results['numberReturned'] == 28
|
||||
|
||||
for item in results['features']:
|
||||
assert item["properties"]["pop_max"] <= 100000
|
||||
|
||||
|
||||
def test_post_cql_json_between_gte_query(config, between_lower):
|
||||
"""Testing cql json query for a between object"""
|
||||
p = ElasticsearchProvider(config)
|
||||
|
||||
results = p.query(limit=500, filterq=between_lower)
|
||||
|
||||
assert len(results['features']) == 237
|
||||
assert results['numberMatched'] == 237
|
||||
assert results['numberReturned'] == 237
|
||||
|
||||
for item in results['features']:
|
||||
assert 10000 <= item["properties"]["pop_max"]
|
||||
|
||||
|
||||
def test_post_cql_json_eq_query(config, eq):
|
||||
"""Testing cql json query for an eq object"""
|
||||
p = ElasticsearchProvider(config)
|
||||
|
||||
results = p.query(limit=500, filterq=eq)
|
||||
|
||||
assert len(results['features']) == 235
|
||||
|
||||
|
||||
def test_post_cql_json_and_query(config, _and):
|
||||
"""Testing cql json query for an and object"""
|
||||
p = ElasticsearchProvider(config)
|
||||
|
||||
results = p.query(limit=1000, filterq=_and)
|
||||
|
||||
assert len(results['features']) == 77
|
||||
|
||||
|
||||
def test_post_cql_json_intersects_query(config, intersects):
|
||||
"""Testing cql json query for an intersects object"""
|
||||
p = ElasticsearchProvider(config)
|
||||
|
||||
results = p.query(limit=100, filterq=intersects)
|
||||
|
||||
assert len(results['features']) == 2
|
||||
|
||||
Reference in New Issue
Block a user