huanld/pygeoapi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add json-ld templating for feature collection item (#868)

Browse Source 
* Render template if specified in config and provide example config

Update linked_data.py

Merge branch 'geopython:master' into jsonld

Rended from json-ld output

Render jinja2 json-ld template from json-ld output instead of json output

Merge branch 'geopython:master' into jsonld

Add documentation

- Add documentation
- Add test to workflow

Update pygeoapi-test-config.yml

Update test_api.py

Update api.py

Update linked_data.py

Move template declaration in configuration

Update docs

Update configuration.rst

Update configuration.rst

* Updates per requested changes

* Fix spelling

* Fix json-ld template pathing

* Remove  root path for the templating

* Move json-ld template from api.py

- Move single item json-ld templating to inside geojson2jsonld
- Reformat json-ld configuration for context and item_template to children of json-ld block
- Update docs and example configurations

* Fix ref

* Use FileSystemLoader to control template search path

search for templates is in order of `template_paths` list

* s/json-ld/linked-data/ig

rename json-ld to more generic name
This commit is contained in:
Benjamin Webb
2023-02-15 21:12:19 -05:00
committed by GitHub
parent 25e60c4a8b
commit e1af5e1ca5
15 changed files with 200 additions and 153 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker/default.config.yml
+6 -5
View File
@@ -97,11 +97,12 @@ resources:
keywords:
- observations
- monitoring
context:
- datetime: https://schema.org/DateTime
- vocab: https://example.com/vocab#
stn_id: "vocab:stn_id"
value: "vocab:value"
linked-data:
context:
- datetime: https://schema.org/DateTime
- vocab: https://example.com/vocab#
stn_id: "vocab:stn_id"
value: "vocab:value"
links:
- type: text/csv
rel: canonical
docker/examples/sensorthings/brgm.sta.pygeoapi.config.yml
+19 -16
View File
@@ -103,11 +103,12 @@ resources:
- Things
- SensorThings
- BRGM
context:
- sosa: "http://www.w3.org/ns/sosa/"
ssn: "http://www.w3.org/ns/ssn/"
Datastreams: sosa:ObservationCollection
name: schema:name
linked-data:
context:
- sosa: "http://www.w3.org/ns/sosa/"
ssn: "http://www.w3.org/ns/ssn/"
Datastreams: sosa:ObservationCollection
name: schema:name
links:
- type: application/html
rel: canonical
@@ -140,12 +141,13 @@ resources:
- Datastreams
- SensorThings
- BRGM
context:
- sosa: http://www.w3.org/ns/sosa/
ssn: http://www.w3.org/ns/ssn/
Observations: sosa:hasMember
Thing: sosa:hasFeatureOfInterest
name: schema:name
linked-data:
context:
- sosa: http://www.w3.org/ns/sosa/
ssn: http://www.w3.org/ns/ssn/
Observations: sosa:hasMember
Thing: sosa:hasFeatureOfInterest
name: schema:name
links:
- type: application/html
rel: canonical
@@ -181,11 +183,12 @@ resources:
- Observations
- SensorThings
- BRGM
context:
- sosa: http://www.w3.org/ns/sosa/
ssn: http://www.w3.org/ns/ssn/
Datastream: sosa:isMemberOf
name: schema:name
linked-data:
context:
- sosa: http://www.w3.org/ns/sosa/
ssn: http://www.w3.org/ns/ssn/
Datastream: sosa:isMemberOf
name: schema:name
links:
- type: application/html
rel: canonical
docker/examples/sensorthings/iow.sta.pygeoapi.config.yml
+19 -16
View File
@@ -97,11 +97,12 @@ resources:
- Things
- SensorThings
- IoW
context:
- sosa: "http://www.w3.org/ns/sosa/"
ssn: "http://www.w3.org/ns/ssn/"
Datastreams: sosa:ObservationCollection
name: schema:name
linked-data:
context:
- sosa: "http://www.w3.org/ns/sosa/"
ssn: "http://www.w3.org/ns/ssn/"
Datastreams: sosa:ObservationCollection
name: schema:name
links:
- type: application/html
rel: canonical
@@ -134,12 +135,13 @@ resources:
- Datastreams
- SensorThings
- IoW
context:
- sosa: http://www.w3.org/ns/sosa/
ssn: http://www.w3.org/ns/ssn/
Observations: sosa:hasMember
Thing: sosa:hasFeatureOfInterest
name: schema:name
linked-data:
context:
- sosa: http://www.w3.org/ns/sosa/
ssn: http://www.w3.org/ns/ssn/
Observations: sosa:hasMember
Thing: sosa:hasFeatureOfInterest
name: schema:name
links:
- type: application/html
rel: canonical
@@ -176,11 +178,12 @@ resources:
- Observations
- SensorThings
- IoW
context:
- sosa: http://www.w3.org/ns/sosa/
ssn: http://www.w3.org/ns/ssn/
Datastream: sosa:isMemberOf
name: schema:name
linked-data:
context:
- sosa: http://www.w3.org/ns/sosa/
ssn: http://www.w3.org/ns/ssn/
Datastream: sosa:isMemberOf
name: schema:name
links:
- type: application/html
rel: canonical
docker/examples/sensorthings/sta.pygeoapi.config.yml
+19 -16
View File
@@ -97,11 +97,12 @@ resources:
keywords:
- Things
- SensorThings
context:
- sosa: "http://www.w3.org/ns/sosa/"
ssn: "http://www.w3.org/ns/ssn/"
Datastreams: sosa:ObservationCollection
name: schema:name
linked-data:
context:
- sosa: "http://www.w3.org/ns/sosa/"
ssn: "http://www.w3.org/ns/ssn/"
Datastreams: sosa:ObservationCollection
name: schema:name
links:
- type: application/html
rel: canonical
@@ -132,12 +133,13 @@ resources:
keywords:
- Datastreams
- SensorThings
context:
- sosa: http://www.w3.org/ns/sosa/
ssn: http://www.w3.org/ns/ssn/
Observations: sosa:hasMember
Thing: sosa:hasFeatureOfInterest
name: schema:name
linked-data:
context:
- sosa: http://www.w3.org/ns/sosa/
ssn: http://www.w3.org/ns/ssn/
Observations: sosa:hasMember
Thing: sosa:hasFeatureOfInterest
name: schema:name
links:
- type: application/html
rel: canonical
@@ -172,11 +174,12 @@ resources:
keywords:
- Observations
- SensorThings
context:
- sosa: http://www.w3.org/ns/sosa/
ssn: http://www.w3.org/ns/ssn/
Datastream: sosa:isMemberOf
name: schema:name
linked-data:
context:
- sosa: http://www.w3.org/ns/sosa/
ssn: http://www.w3.org/ns/ssn/
Datastream: sosa:isMemberOf
name: schema:name
links:
- type: application/html
rel: canonical
docker/examples/skin/skin.config.yml
+6 -5
View File
@@ -108,11 +108,12 @@ resources:
keywords:
- observations
- monitoring
context:
- datetime: https://schema.org/DateTime
- vocab: https://example.com/vocab#
stn_id: "vocab:stn_id"
value: "vocab:value"
linked-data:
context:
- datetime: https://schema.org/DateTime
- vocab: https://example.com/vocab#
stn_id: "vocab:stn_id"
value: "vocab:value"
links:
- type: text/csv
rel: canonical
docs/source/configuration.rst
+40 -16
View File
@@ -47,7 +47,7 @@ The ``server`` section provides directives on binding and high level tuning.
limit: 10 # server limit on number of items to return
templates: # optional configuration to specify a different set of templates for HTML pages. Recommend using absolute paths. Omit this to use the default provided templates
path: /path/to/jinja2/templates/folder # path to templates folder containing the jinja2 template HTML files
path: /path/to/jinja2/templates/folder # path to templates folder containing the Jinja2 template HTML files
static: /path/to/static/folder # path to static folder containing css, js, images and other static files referenced by the template
map: # leaflet map setup for HTML pages
@@ -128,7 +128,7 @@ The ``resource.type`` property is required. Allowed types are:
- ``process``
- ``stac-collection``
The ``providers`` block is a list of 1..n providers with which to operate the data on. Each
The ``providers`` block is a list of 1..n providers with which to operate the data on. Each
provider requires a ``type`` property. Allowed types are:
- ``feature``
@@ -150,11 +150,13 @@ default.
keywords: # list of related keywords
- observations
- monitoring
context: # linked data configuration (see Linked Data section)
- datetime: https://schema.org/DateTime
- vocab: https://example.com/vocab#
stn_id: "vocab:stn_id"
value: "vocab:value"
linked-data: # linked data configuration (see Linked Data section)
item_template: tests/data/base.jsonld
context:
- datetime: https://schema.org/DateTime
- vocab: https://example.com/vocab#
stn_id: "vocab:stn_id"
value: "vocab:value"
links: # list of 1..n related links
- type: text/csv # MIME type
rel: canonical # link relations per https://www.iana.org/assignments/link-relations/link-relations.xhtml
@@ -351,7 +353,7 @@ For collections, at the level of item, the default JSON-LD representation adds:
- An ``@id`` for the item, which is the URL for that item. If uri_field is specified,
it is used, otherwise the URL is to its HTML representation in pygeoapi.
- Separate GeoSPARQL/WKT and `schema.org/geo` versions of the geometry. `schema.org/geo`
- Separate GeoSPARQL/WKT and `schema.org/geo` versions of the geometry. `schema.org/geo`
only supports point, line, and polygon geometries. Multipart lines are merged into a single line.
The rest of the multipart geometries are transformed reduced and into a polygon via unary union
or convex hull transform.
@@ -377,7 +379,8 @@ The default pygeoapi configuration includes an example for the ``obs`` sample da
.. code-block:: yaml
context:
linked-data:
context:
- datetime: https://schema.org/DateTime
- vocab: https://example.com/vocab#
stn_id: "vocab:stn_id"
@@ -389,7 +392,8 @@ one with terms defined by schema.org:
.. code-block:: yaml
context:
linked-data:
context:
- schema: https://schema.org/
stn_id: schema:identifer
datetime:
@@ -411,32 +415,52 @@ by the dataset provider, not pygeoapi.
An example of a data provider that includes relationships between items is the SensorThings API provider.
SensorThings API, by default, has relationships between entities within its data model.
Setting the ``intralink`` field of the SensorThings provider to ``true`` sets pygeoapi
to represent the relationship between configured entities as intra-pygeoapi links or URIs.
This relationship can further be maintained in the JSON-LD structured data using the appropiate
Setting the ``intralink`` field of the SensorThings provider to ``true`` sets pygeoapi
to represent the relationship between configured entities as intra-pygeoapi links or URIs.
This relationship can further be maintained in the JSON-LD structured data using the appropiate
``@context`` with the sosa/ssn ontology. For example:
.. code-block:: yaml
Things:
context:
linked-data:
context:
- sosa: "http://www.w3.org/ns/sosa/"
ssn: "http://www.w3.org/ns/ssn/"
Datastreams: sosa:ObservationCollection
Datastreams:
context:
linked-data:
context:
- sosa: "http://www.w3.org/ns/sosa/"
ssn: "http://www.w3.org/ns/ssn/"
Observations: sosa:hasMember
Thing: sosa:hasFeatureOfInterest
Observations:
context:
linked-data:
context:
- sosa: "http://www.w3.org/ns/sosa/"
ssn: "http://www.w3.org/ns/ssn/"
Datastream: sosa:isMemberOf
Sometimes, the JSON-LD desired for an individual feature in a collection is more complicated than can be achieved by
aliasing properties using a context. In thise case, it is possible to specify a Jinja2 template. When ``item_template``
is defined for a feature collection, the json-ld prepared by pygeoapi will be used to render the Jinja2 template
specified by the path. The path specified can be absolute or relative to pygeoapi's template folder. For even more
deployment flexibility, the path can be specified with string interpolation of environment variables.
.. code-block:: yaml
linked-data:
item_template: tests/data/base.jsonld
context:
- datetime: https://schema.org/DateTime
.. note::
The template ``tests/data/base.jsonld`` renders the unmodified JSON-LD. For more information on the capacities
of Jinja2 templates, see :ref:`html-templating`.
Summary
-------
examples/django/sample_project/pygeoapi-config.yml
+6 -5
View File
@@ -108,11 +108,12 @@ resources:
keywords:
- observations
- monitoring
context:
- datetime: https://schema.org/DateTime
- vocab: https://example.com/vocab#
stn_id: "vocab:stn_id"
value: "vocab:value"
linked-data:
context:
- datetime: https://schema.org/DateTime
- vocab: https://example.com/vocab#
stn_id: "vocab:stn_id"
value: "vocab:value"
links:
- type: text/csv
rel: canonical
pygeoapi-config.yml
+6 -5
View File
@@ -108,11 +108,12 @@ resources:
keywords:
- observations
- monitoring
context:
- datetime: https://schema.org/DateTime
- vocab: https://example.com/vocab#
stn_id: "vocab:stn_id"
value: "vocab:value"
linked-data:
context:
- datetime: https://schema.org/DateTime
- vocab: https://example.com/vocab#
stn_id: "vocab:stn_id"
value: "vocab:value"
links:
- type: text/csv
rel: canonical
pygeoapi/linked_data.py
+23 -10
View File
@@ -31,10 +31,11 @@
Returns content as linked data representations
"""
import json
import logging
from typing import Callable
from pygeoapi.util import is_url
from pygeoapi.util import is_url, render_j2_template
from pygeoapi import l10n
from shapely.geometry import shape
from shapely.ops import unary_union
@@ -188,17 +189,21 @@ def geojson2jsonld(config: dict, data: dict, dataset: str,
:returns: string of rendered JSON (GeoJSON-LD)
"""
context = config['resources'][dataset].get('context', []).copy()
LOGGER.debug('Fetching context and template from resource configuration')
jsonld = config['resources'][dataset].get('linked-data', {})
context = jsonld.get('context', []).copy()
template = jsonld.get('item_template', None)
defaultVocabulary = {
'schema': 'https://schema.org/',
id_field: '@id',
'type': '@type'
}
if identifier:
# Single jsonld
defaultVocabulary.update({
'geosparql': 'http://www.opengis.net/ont/geosparql#'
'gsp': 'http://www.opengis.net/ont/geosparql#'
})
# Expand properties block
@@ -207,7 +212,7 @@ def geojson2jsonld(config: dict, data: dict, dataset: str,
# Include multiple geometry encodings
data['type'] = 'schema:Place'
jsonldify_geometry(data)
data[id_field] = identifier
data['@id'] = identifier
else:
# Collection of jsonld
@@ -223,10 +228,10 @@ def geojson2jsonld(config: dict, data: dict, dataset: str,
identifier = feature.get(id_field,
feature['properties'].get(id_field, ''))
if not is_url(str(identifier)):
identifier = f"config['server']['url']/collections/{dataset}/items/{feature['id']}" # noqa
identifier = f"{config['server']['url']}/collections/{dataset}/items/{feature['id']}" # noqa
data['features'][i] = {
id_field: identifier,
'@id': identifier,
'type': 'schema:Place'
}
@@ -240,7 +245,15 @@ def geojson2jsonld(config: dict, data: dict, dataset: str,
**data
}
return ldjsonData
if None in (template, identifier):
return ldjsonData
else:
# Render jsonld template for single item with template configured
LOGGER.debug(f'Rendering JSON-LD template: {template}')
content = render_j2_template(
config, template, ldjsonData)
ldjsonData = json.loads(content)
return ldjsonData
def jsonldify_geometry(feature: dict) -> None:
@@ -260,9 +273,9 @@ def jsonldify_geometry(feature: dict) -> None:
feature['geometry'] = feature.pop('geometry')
# Geosparql geometry
feature['geosparql:hasGeometry'] = {
feature['gsp:hasGeometry'] = {
'@type': f'http://www.opengis.net/ont/sf#{geom.geom_type}',
'geosparql:asWKT': {
'gsp:asWKT': {
'@type': 'http://www.opengis.net/ont/geosparql#wktLiteral',
'@value': f'{geom.wkt}'
}
pygeoapi/schemas/config/pygeoapi-config-0.x.yml
+18 -11
View File
@@ -55,7 +55,7 @@ properties:
properties:
path:
type: string
description: path to templates folder containing the jinja2 template HTML files
description: path to templates folder containing the Jinja2 template HTML files
static:
type: string
description: path to static folder containing css, js, images and other static files referenced by the template
@@ -264,16 +264,23 @@ properties:
keywords:
$ref: '#/definitions/i18n_array'
description: list of keywords about the service
context:
type: array
description: linked data configuration
items:
type: object
patternProperties:
"^.*$":
anyOf:
- type: string
- type: object
linked-data:
type: object
description: linked data configuration
properties:
item_template:
type: string
description: path to JSON-LD Jinja2 template
context:
type: array
description: additional JSON-LD context
items:
type: object
patternProperties:
"^.*$":
anyOf:
- type: string
- type: object
links:
type: array
description: list of related links
pygeoapi/util.py
+10 -25
View File
@@ -49,7 +49,6 @@ from shapely.geometry import Polygon
import dateutil.parser
from jinja2 import Environment, FileSystemLoader, select_autoescape
from babel.support import Translations
from jinja2.exceptions import TemplateNotFound
import yaml
from pygeoapi import __version__
@@ -328,22 +327,19 @@ def render_j2_template(config: dict, template: Path,
:returns: string of rendered template
"""
custom_templates = False
template_paths = {TEMPLATES, '.'}
try:
templates_path = config['server']['templates']['path']
env = Environment(loader=FileSystemLoader(templates_path),
extensions=['jinja2.ext.i18n',
'jinja2.ext.autoescape'],
autoescape=select_autoescape(['html', 'xml']))
custom_templates = True
LOGGER.debug(f'using custom templates: {templates_path}')
templates = config['server']['templates']['path']
template_paths.add(templates)
LOGGER.debug(f'using custom templates: {templates}')
except (KeyError, TypeError):
env = Environment(loader=FileSystemLoader(TEMPLATES),
extensions=['jinja2.ext.i18n',
'jinja2.ext.autoescape'],
autoescape=select_autoescape(['html', 'xml']))
LOGGER.debug(f'using default templates: {TEMPLATES}')
env = Environment(loader=FileSystemLoader(template_paths),
extensions=['jinja2.ext.i18n',
'jinja2.ext.autoescape'],
autoescape=select_autoescape(['html', 'xml']))
env.filters['to_json'] = to_json
env.filters['format_datetime'] = format_datetime
env.filters['format_duration'] = format_duration
@@ -362,18 +358,7 @@ def render_j2_template(config: dict, template: Path,
translations = Translations.load('locale', [locale_])
env.install_gettext_translations(translations)
try:
template = env.get_template(template)
except TemplateNotFound as err:
if custom_templates:
LOGGER.debug(err)
LOGGER.debug('Custom template not found; using default')
env = Environment(loader=FileSystemLoader(TEMPLATES),
extensions=['jinja2.ext.i18n'])
env.install_gettext_translations(translations)
template = env.get_template(template)
else:
raise
template = env.get_template(template)
return template.render(config=l10n.translate_struct(config, locale_, True),
data=data, locale=locale_, version=__version__)
tests/data/base.jsonld
+1
View File
@@ -0,0 +1 @@
{{ data | to_json | safe }}
tests/pygeoapi-test-config-hidden-resources.yml
+12 -11
View File
@@ -121,17 +121,18 @@ resources:
title: data
href: https://raw.githubusercontent.com/mapserver/mapserver/branch-7-0/msautotest/wxs/data/obs.csv
hreflang: en-US
context:
- schema: https://schema.org/
stn_id:
"@id": schema:identifier
"@type": schema:Text
datetime:
"@type": schema:DateTime
"@id": schema:observationDate
value:
"@type": schema:Number
"@id": schema:QuantitativeValue
linked-data:
context:
- schema: https://schema.org/
stn_id:
"@id": schema:identifier
"@type": schema:Text
datetime:
"@type": schema:DateTime
"@id": schema:observationDate
value:
"@type": schema:Number
"@id": schema:QuantitativeValue
extents:
spatial:
bbox: [-180,-90,180,90]
tests/pygeoapi-test-config.yml
+14 -11
View File
@@ -120,17 +120,18 @@ resources:
title: data
href: https://raw.githubusercontent.com/mapserver/mapserver/branch-7-0/msautotest/wxs/data/obs.csv
hreflang: en-US
context:
- schema: https://schema.org/
stn_id:
"@id": schema:identifier
"@type": schema:Text
datetime:
"@type": schema:DateTime
"@id": schema:observationDate
value:
"@type": schema:Number
"@id": schema:QuantitativeValue
linked-data:
context:
- schema: https://schema.org/
stn_id:
"@id": schema:identifier
"@type": schema:Text
datetime:
"@type": schema:DateTime
"@id": schema:observationDate
value:
"@type": schema:Number
"@id": schema:QuantitativeValue
extents:
spatial:
bbox: [-180,-90,180,90]
@@ -285,6 +286,8 @@ resources:
title: data source
href: https://en.wikipedia.org/wiki/GeoJSON
hreflang: en-US
linked-data:
item_template: tests/data/base.jsonld
extents:
spatial:
bbox: [-180,-90,180,90]
tests/test_api.py
+1 -1
View File
@@ -1109,7 +1109,7 @@ def test_get_collection_item_json_ld(config, api_):
feature = json.loads(response)
assert '@context' in feature
assert all((f in feature['@context'][0] for
f in ('schema', 'type', 'geosparql')))
f in ('schema', 'type', 'gsp')))
assert len(feature['@context']) == 1
assert 'schema' in feature['@context'][0]
assert feature['@context'][0]['schema'] == 'https://schema.org/'