* Implemented Connection Pooling
First part of collection pooling, improve by making pool size and other params configurable in YAML file of pygeoapi
* updated session pooling with more robust creation of dsn
* Further changes for releasing connections
* further little tweaks
* Added Thread Lock and building of DSN
* Session Pooling now only happens when ENV VAR ORACLE_POOL_MIN and ORACLE_POOL_MAX are set. Else usual behaviour is restored and single connections are opened and closed. Added a Lock for avoiding race conditions and threat safety.
* Flake8 changes
* More Flake8
* Added integer type to env variables
* Added Tests for Session Pooling
* Flake 8 changes for tests
* More Flake8 fun
* Fixed typo for env vars
* Fixed typo for env var and added noqa for variables
* More Typos
* More Flake8
* Added documentation to the oracle section for session pooling
* Review and Feedback with @totycro
* Update oracle.py
Added Author
* Added feedback from review
---------
Co-authored-by: root <root@els01entw.ama.at>
Co-authored-by: xlanger <moritz.langer@ama.gv.at>
* Exclude None from `get_processor` return type annotation
An exception is raised in case of error, so it can't ever return None
* Add support for OGC API Processes Subscriber
The subscription URLs are passed to the manager, which
then has to call them appropriately.
By default, managers have the attribute `supports_subscribing`
set to `False` in order to not break the API for these. The
subscriptions are only passed to if this is set to `True`
* Add ogc api callback class to conformance
https://docs.ogc.org/is/18-062r2/18-062r2.html#toc67
* Make successUri mandatory in subscriber
It's mandatory in the standard.
Thx @ricardogsilva !
* Use snake case in python for fields which are camel case in the api
Thx @ricardogsilva !
* Add subscriber to method docstring
* Provide default value for subscriber for managers not supporting it
Thanks @ricardogsilva !
* Factor out notification call into methods
This increases reusability by other managers
Thanks @ricardogsilva !
* Add an example call for a process subscriber
* Change test urls to valid urls
* Third party imports in own block
* Implement extra_properties in oracle Provider
These can be used to configure additional database-computed fields
in the config file which are returned on `get` and `query` calls
* Allow mandating properties which are not part of the output
Previously, properties which were not requested for the output
were not part of `fields`, which means that they were not passed
in to the provider as filter properties for e.g. `query()`.
This commit adds them there and introduces a new variable
`filtered_fields`, which is used for limiting the output of queries.
There is also some minor refactoring, but the existing and also the
newly written tests should avoid regressions.
* Restore previous behavior for default arguments
* - Remove the need to define a tiling scheme in the configuration, for providers that support only one scheme
- Support only WebMercatorQuad for mvt-elastic tiles
* - updated documentation for MVT-elastic provider
* - remove the need of a tiling scheme option in MVT-tippecanoe provider
* - updated tiles documentation with support to tiling-schemes
* - fixed regression bug with coordinate order on vector tile preview
* - add working url for wmts-facade code snippet
* - fixed flake8 example
* - refactored mvt classes to support all implemented metadata formats, regardless of the provider
* - fixed formatting issues
* Implementing basic tile metadata methods
* Fixing yml models
* Adding additional format
* Fixing schema set on load
* Removing unused field from documentation
* Change method name to generic vendor
* Keeping extra metadata info for tippecanoe provider
* Fix flake validations error
---------
Co-authored-by: doublebyte <doublebyte@hushmail.com>
* - Refactor MVTProvider, to support subclassing
- Added provider for MVT Tippecanoe
* - fixed test configuration for the MVT tippecanoe provider
* - added MVT elastic provider
* - added documentation for mvt providers
- fixed formatting
* - removed tiles publishing from elastic provider example
- updated gitignore to ignore data folder from elastic example
* - added docker example for MVT-elastic backend provider
* - Added docker example for tippecanoe MVT provider
* - updated README of the docker examples section, to catch up with the provided examples.
* - renamed mvt-elastic and mvt-tippecanoe docker example folders.
* - renamed elastic and tippecanoe folders to lower case
* - fixed formatting issue
* - s/Mabox/Mapbox/
* - put plugin provider names in single line
* - renamed MVT provider base class to BaseMVTProvider.
* - added MVT base class
* - reviwed uppercase on .gitignore
- add folder to .gitignore
* - removed new line and brackets
* - removed redundant line with base class
---------
Co-authored-by: doublebyte1 <info@doublebyte.net>
* Added support for table synonyms
* Added new parameters to query and manipulator call
* Changed error types
* Mount volumes to oracle container
* workflow part 2
* workflow part 3
* workflow part 4
* Changed file permissions to 777
* Deleted folder
* Recreated folder
* Changed to official Oracle Docker-Image
* Added Chown user
* back to gvenzl/oracle-xe:latest
* Tried docker-entrypoint-startdb.d
* Added addnab/docker-run-action@v3
* Added port and deamon mode
* next try
* added job.container.network
* next try
* + docker ps
* next try
* using docker run
* next try
* next try
* Changed len of array to 11
* Use sdo_util.from_geojsonfor create and update
* Flake8 changes
* Fixed error with views
* Added crs_transform_spec support
* Without default_crs
* Updated documentation for Oracle Provider
* changes for flake8
* Added crs_transform_spec support to get function
* review changes
* Added configurable SDO operator
* read from s3 bucket
* fix formatting for build workflow
* add s3fs to provider requirements
* storage options for fsspec
* Revert "storage options for fsspec"
This reverts commit dcfaceedd78d0f0e0dc3e38610958671c27b06a6.
* Revert "storage options for fsspec"
This reverts commit dcfaceedd78d0f0e0dc3e38610958671c27b06a6.
* delete extraneous code for cloud storage read
* update documentation
* add ci testing
* update test suite config file
* update number of collections
* update test query for s3
* ci testing updates
* read from s3 bucket
* fix formatting for build workflow
* add s3fs to provider requirements
* storage options for fsspec
* Revert "storage options for fsspec"
This reverts commit dcfaceedd78d0f0e0dc3e38610958671c27b06a6.
* Revert "storage options for fsspec"
This reverts commit dcfaceedd78d0f0e0dc3e38610958671c27b06a6.
* delete extraneous code for cloud storage read
* update documentation
* add ci testing
* update test suite config file
* update number of collections
* update test query for s3
* ci testing updates
* documentation updates
* config updates
* flake8
* modify sfs3 version
installing requirements threw an error when specifying 2023.6.0 (only identifies up through 2023.1.0). Had added the sfs3 requirement to deal with issues thrown when the 2023.9.0 release came out.
* updated coverages documentation
* - look for metadata.json, only when we are using the tilejson format; in other cases, this file is missing and that is going to make pygeoapi fail
* - updated configuration examples in documentation, to use "default" as metadata for vector tiles ("raw" does not work)
- added metadata format configuration for the pg_tileserv provider example
* Added enum for requested process execution modes
* Added data dict as expected base process.execute param
* Modified base process manager to select execution mode
* Pass requested execution mode to process manager
* Refactored dummy process manager to deal with request execution mode
* Use num value in response headers
* fix bug with non-existent process response headers
* fix flake8 errors
* fix python version errors
* fix python version errors
* fix python version errors
* refactor failing test
* refactor failing test
* fix line length
* fix using tuple in type annotations
* update docs and add missing param docstring
* swapped double quotes for single quotes for consistency
* OGC API - Features Part 2 (groundwork+CRS-BBOX) from PR #1155 - contributes to issue #1128
* #1128 provide conformance class for OAPIF Part 2 in /conformance page
* #1128 bitten by flake8...
* #1128 configurability CRS Feature Providers with syntax, defaults and tests
* #1128 configurability CRS Feature Providers refine for default values
* #1128 display supported CRSs in HTML Collection template
* #1128 config, mmetadata and tests for storageCRS and storageCrsCoordinateEpoch
* #1128 WIP for bbox-crs parameter support
* #1128 utility function and tests for default/mandatory supprted CRS list
* #1128 default supported CRS adaptation to OAPIF Part 2 standard
* #1128 grr flake8 whitespace
* #1128 start adding full API tests OGR for bbox-crs and crs parms
* #1128 fix flake8
* #1128 fix flake8 - install GDAL in workflow main for OGR tests
* #1128 fix flake8 - install GDAL in workflow main for OGR tests - need pip package?
* #1128 fix flake8 - install GDAL in workflow main for OGR tests - using libgdal-dev gdal-bin
* #1128 fix SensorThings test for main.yml Workflow
* #1128 fix SensorThings test for main.yml Workflow nr 2
* #1128 make all OGR tests working again
* #1128 make all OGR tests working again - flake8
* #1128 make all OGR tests working again - GeoSolutions WFS bbox
* #1128#1155 add documentation for OGC OAPIF Part 2 CRS CRS BBOX support
* #1128#1155 refine documentation for OGC OAPIF Part 2 CRS CRS BBOX support
* #1128#1155 refine documentation to align with #1149
* #1128#1155 rework from review OAS and pygeoapi config schema
* #1128#1155 minor: compile Re for CRS URI only once as global var
* #1128 merge in changes from PR #1173 - fix missing import
* WIP Ogcapi features part 2 - Support for crs query parameter (#1149)
* feat(ogcapi_features_crs): start implementing crs support from ogcapi features part2
* Pass input and output CRSs WKT instead of crs transformation object
* fix longs lines and blank lines
* fix typo
* fix import for type annotation not supported by python version
* fix variable visibility in local scope
* fix tabs/spaces indentations
* Add support for the crs parameter to OGRProvider
* make flake8 happy
* Make crs transformation mechanism more consistent between PostgreSQL and OGR providers
* test(util): add two test functions in util.py
New functions: test_get_crs_from_uri and test_get_transform_from_crs
* fix too long lines...
* Update get_crs_from_uri and corresponding test function
* fix(get_crs_from_uri): make the error more explicit in if wrong crs uri format
* flake8 again...
* Keep support for source_srs/target_srs in config for OGRProvider
* revert changes made to pygeoapi-config-0.x.yml, overlap with PR 1155
* test: add test data and update test config file
* Extract 'crs' and 'storage_crs' and provider level instead of collection level
* feat(crs): new decorator to support coordinates transformation of feature collections
* feat(crs): 'crs' query parameter for CSVProvider
* test(crs): add tests for 'crs' query parameter
* test: update number of collections in test_describe_collections
* test: update number of collections in test_filter_dict_by_key_value
* fix(crs_transform): change the crs transformation decorator
Change the logic of the decorator so that it works for both functions that
return FeatureCollections and for functions tha return single Features.
* test: add tests for get_collection_item end-point with 'crs' parameter
* fix(test_get_collection_item_crs): id as path parameter, not query parameter
* test: unpack coordinates to create point geometry
* feat(crs): add suuport for crs query parameter for all providers of type 'feature'
* docs(crs): add documentation to illustrate use of 'crs' query parameters
* docs(crs): more data access examples
* fix typo and add new line
* refactor: specify None as default value for crs_transform_out parameter in _sqlalchemy_to_feature method
* changes for PR 1149, test_api and style formatting
* CRS84 as default crs also for test_get_collection_items_crs
* test(crs): test coordinates transformation implementation of PostgreSQLProvider
* test(crs): move tests to test_postgresql_provider
* fix test function calls
* change test to ensure returned features are the same
* add json format to request object
* test(crs): test coordinates transformation implementation of OGRProvider
* refactor(crs): make more compact get_collection_item and get_collection_items
Define two new static methods in API class, to create crs_transform_wkt and
setting content-crs header. These methods can be re-used in both
get_collection_item and get_collection_items methods and removes code
duplication.
---------
Co-authored-by: Just van den Broecke <just@justobjects.nl>
* #1178 fix flake8 error
* #1178 use EPSG:28992 i.s.o. 32631 - fix unit test OGR Shapefile
* #1174 use CRS-compliant Axis ordering for crs support
* #1174 fix and honour CRS 4258disable native CRS Transform in OGR Provider - Axis ordering not honoured...
* #1174 remove ADR tests rom test_util.py
* #1174 enable native CRS transform again in OGR Provider
* #1174 enable native CRS transform again in OGR Provider - fix config
* #1174 remove support for source/target_srs in OGRProvider - enforce transforms always based on storageCRS
* #1174 fix tests Postgresql Provider for Transforms
* #1174 fix tests Postgresql Provider for Transforms
* #1174 add tests for OGR Transformation and Axis Order
* #1174 Suppress potential axis-swapping in OGR ExportToJSON
* #1174 minor fix test - unassign spatialref before setgeom infeat
* #1174 minor fix test - unassign spatialref before setgeom infeat - flake8
* #1174 solve CI WFS test failures with GDAL HTTP config options
* #1174 bbox and bbox-crs defs local in openapi.py for CITE validators
* #1174 merge master - #1152#1203 etc
* #1174 small doc changes
* #1174 move GeomObject typedef to beginning of util.py
* #1174 added debug logging in transform Decorator func
---------
Co-authored-by: Mathieu Tachon <92298764+MTachon@users.noreply.github.com>
* MongoDB Jobmanager
* Add MongoDB-Jobmanager
Example configuration included
* Requested Fixes
Fixes Issues mentioned in latest Review
* Flake8 fixes
Code is fixed on the basis of a flake8 report
* Update ogcapi-processes.rst
* Update ogcapi-processes.rst
* Flake8 fixes
Fixes of flake8 errors
* feat(OGRProvider): allow to use 'geom_field' in the config for OGR provider
* docs: show an example of using OGR for publishing data from a PostGIS database
* fix: does not check for empty geometry in this PR
* docs: add back ESRIJSON source_type
* - Added support to read from a generic url template as vector tile backend.
* - Add support for not rendering the tileset metadata
* - added support for z/y/x vector tile layers
* - fixed formatting issues
* - refactored code to use get_layer function, everywhere we need to parse username
- added a couple of debug statements, to make sure we are getting the layer name
correctly
- added an error for url templates that follow a schema, which is not supported yet
* - Added documentation for tiles provider, to show how to read a generic url
* - fixed formatting issues
* - added example of ES vector tiles in the docker-config of elasticsearch
Co-authored-by: doublebyte1 <info@doublebyte.net>
* Add pygeofilter, unpin psycopg2
* Add pygeofilter SQLAlchemy query demo
* Add note about primary key
* Add todos to script
* Make script generic
* Add (commented out) PostgreSQL provider hotosm_bdi_waterways to config
* Update to use local test database
* Return result as dictionary
* Pass GEOM_FIELD as variable
* Add offset and limt, fix two names
* Implement order by
* Use function for cql query
* Add cql query block to query and test
* Link up to db engine
* Convert WKB geometry to GeoJSON
* Add shapely as explicit dependency
* Add tests for CQL queries
* Add test using CROSSES with LINESTRING
* Uncomment Docker PostGIS test layer
* First pass at wiring up API
The CQL is read from the request but the parser fails to convert to
AST.
* Turn logging to DEBUG
Example query
http://localhost:5000/collections/hot_osm_waterways/items?cql=%27osm_id%20BETWEEN%2080800000%20AND%2080900000%20AND%20name%20IS%20NULL%27
* Create dedicated pygeoapi-config.yml for testing PostgreSQL
* Add test for PostgreSQL CQL
* Return 400 for bad CQL
* Tidy up old files
* Bring API error type tests together
* Only reflect the table of interest
* Add pygeofilter to requirements
* Remove local setup_env.sh from .gitignore
* Remove lark exception handling
* Remove lark codes from tests, docstring change
* Split cql test into good and error
* Rename cql parameter to filter
* Add optional filter-lang parameter with test
* Initial changes
* Rename cql text parser
* Use CQL-JSON POST for PostgreSQL query
* Refactor to get data early and handle errors
* Add error tests for CQL-JSON PostgreSQL
* Add tests for prev/next and provider instantiation
* As filter-lang is set the block in not needed
* Clarify test fixtures
* Use filterq only in all CQL calls
* Make engine and table_model attributes of provider
* Handle different instantiation failures
* Use SQLAlchemy for get_fields()
* Minor doc edits
* Use SQLAlchemy for get()
* Simplify _sqlalchemy_to_feature
* Store database parameters in dedicated function
* Strip out non-SQLAlchemy code
* Add draft query() based on CQL filters
* Add property filtering to query()
* Implement select_properties and skip_geometry
* Implement bbox filter
* Implement properties subset defined in config
* Clean up materialized view test
We only need to test the fields that are present to be sure that the
view has been accessed, so other checks have been removed.
* Flake8 fixes and tidy up
* Add combined test for CQL with bbox and properties
* Create sessions directly where required
* Add Engine store to allow connection reuse
* Replace cql_ast with filterq
* Tidy up tests by using fixture for api
* Add authors
* Use Session() as context manager
A session requires a connection to the database, as provided by the
SQLAlchemy engine. When the session goes out of scope, e.g. at exit
from the function when it was used, it should be garbage collected and
the connection freed. However, during load testing it was found that
this wasn't happening. This commit uses a context manager pattern to
start the session, which ensures that the session is closed at the end
of the `with` block.
* Return fields type as {'type': ...}
* Cache the table_model with the engine
Generating the table_model by "reflecting" the database is expensive but
the resulting model doesn't change. If we cache the table_model with
the engine we save on a round-trip to the database and speed up response
time.
* Use separate stores for Engine and table_models
Using a combined store based on both engine and table_model meant that
a pygeoapi serving multiple tables on the same database would maintain
a connection pool for each table. This could eventually saturate the
number of available connections. This commit separates the store into
two parts.
* Update documentation with example
* Manage or post items based on POST content type
* ghcr build action
* only ghcr job
* run criteria edit
* Update README.md
* job = main
* remove success requirememt
* add tag
* tags: bgs-cql
* Update main.yml
* Update README.md
* remove binary
* update
* Update actions to latest versions
* Dockerfile refactor
* Dockerfile refactor
* restore dockerhub
* fix pygeoif version
* # 3.6 test & dockerhub ref
* Pin pygeoif version to less than 1
* Replace psycopg-binary with psycopg
* Split Docker build into sections
* Pin click version >7,<=8
* Add further cql examples
* Update workflows to match GeoPython repo
* update flake python version
* Unquote PostgreSQL column names
* Update container workflow uses versions
* flake8 use python version 3.7
* Use pre ping to get db connection
* Test that properties are set in provider
* Amendments as requested by @tomkralidis
* Correction
* Fix reviewed changes in api code
Co-authored-by: Colin Blackburn <colb@bgs.ac.uk>
Co-authored-by: KoalaGeo <eddlewis85@gmail.com>
Co-authored-by: Edd <edlew@bgs.ac.uk>
Co-authored-by: Colin Blackburn <ximenesuk@users.noreply.github.com>
Benjamin Webb