* Implement extra_properties in oracle Provider
These can be used to configure additional database-computed fields
in the config file which are returned on `get` and `query` calls
* Allow mandating properties which are not part of the output
Previously, properties which were not requested for the output
were not part of `fields`, which means that they were not passed
in to the provider as filter properties for e.g. `query()`.
This commit adds them there and introduces a new variable
`filtered_fields`, which is used for limiting the output of queries.
There is also some minor refactoring, but the existing and also the
newly written tests should avoid regressions.
* Restore previous behavior for default arguments
* EDR queries without instance id works
Django URL's without instance IDs called a method requiring instance
IDs, resulting in a crash.
Instance ID defaults to None, which is also standard for the provider
that ultimately will be called.
Regression test added
* Use fixtures in django server tests
---------
Co-authored-by: Peter Garnæs <pga@dmi.dk>
* Allow either URL or URN CRS URIs
* Implemented transformation of geometries in CQL filter
* Fixed flake8 issue
* Removed commented out code
* Implemented support for the filter-crs query parameter
* Remove unneeded print() calls
* - refactored mvt classes to support all implemented metadata formats, regardless of the provider
* - fixed formatting issues
* Implementing basic tile metadata methods
* Fixing yml models
* Adding additional format
* Fixing schema set on load
* Removing unused field from documentation
* - added support to TileMatrixSets endpoint
- added TileMatrixSet definitions for WorldCRS84Quad and WebMercatorQuad
* - added tiling-schemes link in the json representation of the landing page
* - added html pages for tilematrixset endpoints
* - advertise json and html representations of the tiling schemes in the landing page
* - Use api definition of Well-known TileMatrixSets in the tiling-schemes element of the tiles metadata page
* - added tiling-scheme url on tiles metadata page, for es and tippecanoe providers
* - fixed flak8 formatting errors
* - updated number of links on the landing page, on the api test
* - Manage tile matrix set id dinamically, on tilematrix set flask endpoint
* - renamed functions to lower case
* - renamed tilematrix set functions on flask
* - Use TtileMatrixSetId parameter in tilematrixset api function
* - added support to TileMatrixSet endpoints on starlette
* - added test for tileMatrixSets api endpoint
* - added test for the tilematrixset endpoint
* - added routes for django
---------
Co-authored-by: Antonio Cerciello <antonio.cerciello@gmail.com>
* Pass OPENAPI_DOCUMENT to API also in django mode
Fixes#1509
* Add basic test for django
This only verifies that one page loads, which however
covers most of the django setup.
* Install django requirements in CI
* Added .pre-commit-config.yaml file
* Initial pass of pre-commit with its default rules and flake8
* Revert "Initial pass of pre-commit with its default rules and flake8"
This reverts commit 86d39e964e8122580d675f4639ae2d15b85c02e2.
* Made pre-commit only act on code files
* Added sql to the pre-commit file pattern
* Initial pass of pre-commit
* - refactored mvt classes to support all implemented metadata formats, regardless of the provider
* - fixed formatting issues
* Implementing basic tile metadata methods
* Fixing yml models
* Adding additional format
* Fixing schema set on load
* Removing unused field from documentation
* Change method name to generic vendor
* Keeping extra metadata info for tippecanoe provider
* Fix flake validations error
---------
Co-authored-by: doublebyte <doublebyte@hushmail.com>
This api method did not have any tests until now.
This also contains a small fix: If no content type was requested,
then by default, the header specified the content type
`application/json`, whereas the content was actually html. Note that
this is rarely a problem because browsers do request html explicitly.
* Add Admin API
- Create `admin.py` to serve as Admin API Core
- Create `flask_admin.py` to create flask blueprint for admin API
- Consolidate configuration getter
- Add Pathlib serializing
- Add docker example
* Add integration tests
- Amend admin example to allow writing to configuration. If FS is read only admin API does not work. Returns a 500 and logs `OSError: [Errno 30] Read-only file system: '/pygeoapi/local.config.yml' `
* Preserve env variables in configuration
* Use common accessor functions
- Use common configuration accessor methods for Django and Starlette
* GET returns raw config file
Return configuration with environment variables preserved on GET requests
* Safeguard env variables for root cfg view
"bind": {
"host": "localhost",
"port": "6000"
}
->
"bind": {
"host": "${HOST}",
"port": "${PORT}"
}
* Simplify admin HTML imports
- Use jinja recursion to expand the configuration
- Remove vue from templates
* Create admin API documentation
* Use render_item_value in admin template
* Add Admin API
- Create `admin.py` to serve as Admin API Core
- Create `flask_admin.py` to create flask blueprint for admin API
- Consolidate configuration getter
- Add Pathlib serializing
- Add docker example
* Update GitHub Actions deployment
* Update admin entrypoint
Update admin entrypoint to align with upstream pygeoapi implementation
* Make requested changes
Co-Authored-By: Tom Kralidis <tomkralidis@gmail.com>
* Amend test url
Co-Authored-By: Tom Kralidis <tomkralidis@gmail.com>
* Fix Admin CI tests
* Add PUT and PATCH for root configuration
- Add put and patch for root configuration
- Add CI tests for PUT and PATCH of root
* Update OpenAPI document wording
* Update entrypoint.sh
Replace tabs with spaces
* Remove unused step
Error from rebasing. Admin API tests are moved to their own job.
* Use jsonpatch
- Use debian supported packaging
- Use custom merge function
* Move test data location
* Create Starlette and Django app
- Fold flask_admin.py into flask_app.py
Co-Authored-By: Tom Kralidis <tomkralidis@gmail.com>
* Make requirements-admin.txt
Move admin dependencies to requirements-admin.txt
* Delete guiblock.html
* Update test count for STAC
Update expected test count for addt'l admin test data
* Relegate config warning to config.py
* Move admin tests out of example
* Delete admin docker example
* Update admin-api.rst
* Update pygeoapi-config-0.x.yml
* Update configuration.rst
* Update config.py
* Update admin.py
* Update admin.py
---------
Co-authored-by: Tom Kralidis <tomkralidis@gmail.com>
* Use generic error handling in get_collection_item
* Use generic error handling in get_collection_items
* Allow specifying a user-facing message in ProviderGenericError
* ose generic error handling in more instances
* Use generic error handling in more instances
* Linting fixes
* - Refactor MVTProvider, to support subclassing
- Added provider for MVT Tippecanoe
* - fixed test configuration for the MVT tippecanoe provider
* - added MVT elastic provider
* - added documentation for mvt providers
- fixed formatting
* - removed tiles publishing from elastic provider example
- updated gitignore to ignore data folder from elastic example
* - added docker example for MVT-elastic backend provider
* - Added docker example for tippecanoe MVT provider
* - updated README of the docker examples section, to catch up with the provided examples.
* - renamed mvt-elastic and mvt-tippecanoe docker example folders.
* - renamed elastic and tippecanoe folders to lower case
* - fixed formatting issue
* - s/Mabox/Mapbox/
* - put plugin provider names in single line
* - renamed MVT provider base class to BaseMVTProvider.
* - added MVT base class
* - reviwed uppercase on .gitignore
- add folder to .gitignore
* - removed new line and brackets
* - removed redundant line with base class
---------
Co-authored-by: doublebyte1 <info@doublebyte.net>
* Added support for table synonyms
* Added new parameters to query and manipulator call
* Changed error types
* Mount volumes to oracle container
* workflow part 2
* workflow part 3
* workflow part 4
* Changed file permissions to 777
* Deleted folder
* Recreated folder
* Changed to official Oracle Docker-Image
* Added Chown user
* back to gvenzl/oracle-xe:latest
* Tried docker-entrypoint-startdb.d
* Added addnab/docker-run-action@v3
* Added port and deamon mode
* next try
* added job.container.network
* next try
* + docker ps
* next try
* using docker run
* next try
* next try
* Changed len of array to 11
* Use sdo_util.from_geojsonfor create and update
* Flake8 changes
* Fixed error with views
* Added crs_transform_spec support
* Without default_crs
* Updated documentation for Oracle Provider
* changes for flake8
* Added crs_transform_spec support to get function
* review changes
* Added configurable SDO operator
This now fixes the issue that the `Accept` header wasn't considered for
format selection.
This adds netcdf as a known mime type in order to be able to test this,
but also for actual clients to use.
Note that if we want to allow other mime types such as `application/x-grib2`,
we also have to add them to `FORMAT_TYPES`.
Fixes#1390
* Refactored get_collection_coverage tests
Using pytest's parametrize mark. This makes it easier to isolate test cases in order to fix the errors that are showing up when running under Python3.8
* Refactor tests from test_api.py that were failing under Python 3.8
* Fix flake8 issues
* Add Python 3.8 to the CI matrix
* Replaced pandas usage with numpy for datetime
* Reverted tests back to non-parametrized form
* Dropped Python 3.7 from main github CI workflow
---------
Co-authored-by: Ricardo Garcia Silva <ricardo@kartoza.com>
* resolving case when lat dimensions are decreasing
CF conventions state only that a "coordinate variable must have values that are strictly monotonic (increasing or decreasing)." In the case of PRISM, the lat coordinates are decreasing. pygeoapi failed due to the lat dimension being empty with existing implementation; this update addresses the issue.
* remove blank space
* resolving case when lat dimensions are decreasing
CF conventions state only that a "coordinate variable must have values that are strictly monotonic (increasing or decreasing)." In the case of PRISM, the lat coordinates are decreasing. pygeoapi failed due to the lat dimension being empty with existing implementation; this update addresses the issue.
* remove blank space
* test decreasing latitude
The new test will now uncover if you can successfully serve data stored on S3 AND test if you can perform a cube query on data with monotonically decreasing latitude coordinates at the same time.
* flake8 py3 fixes
* read from s3 bucket
* fix formatting for build workflow
* add s3fs to provider requirements
* storage options for fsspec
* Revert "storage options for fsspec"
This reverts commit dcfaceedd78d0f0e0dc3e38610958671c27b06a6.
* Revert "storage options for fsspec"
This reverts commit dcfaceedd78d0f0e0dc3e38610958671c27b06a6.
* delete extraneous code for cloud storage read
* update documentation
* add ci testing
* update test suite config file
* update number of collections
* update test query for s3
* ci testing updates
* read from s3 bucket
* fix formatting for build workflow
* add s3fs to provider requirements
* storage options for fsspec
* Revert "storage options for fsspec"
This reverts commit dcfaceedd78d0f0e0dc3e38610958671c27b06a6.
* Revert "storage options for fsspec"
This reverts commit dcfaceedd78d0f0e0dc3e38610958671c27b06a6.
* delete extraneous code for cloud storage read
* update documentation
* add ci testing
* update test suite config file
* update number of collections
* update test query for s3
* ci testing updates
* documentation updates
* config updates
* flake8
* modify sfs3 version
installing requirements threw an error when specifying 2023.6.0 (only identifies up through 2023.1.0). Had added the sfs3 requirement to deal with issues thrown when the 2023.9.0 release came out.
* updated coverages documentation
* Fix for Django build, changes to documentation, and dev dependencies updated
* remove example configuration files
* Added readthedocs configuration file
* added section on building the documentation
* renamed docs/requirements-docs.txt to docs/requirements.txt
* renamed requirements file in readthedocs configuration
* Added documentation requirements installation to GitHub build workflow
* fixed pagination next page response
* fixed line too long
* Update get_collection_items test to accomodate last page fix
Note that this feature can't be tested with the CSV provider
because it uses the number of current items as numberMatched,
which doesn't seem correct but is an independent issue.
https://github.com/geopython/pygeoapi/blob/c7c07855e5cfa95b88d6d0d11f5c23111e38da95/pygeoapi/provider/csv_.py#L173
---------
Co-authored-by: Juan Pablo Duque <juan-pablo-duque@hotmail.com>
Co-authored-by: Juan Duque <juanpablo.duque@polimi.it>
* Process manager now knows about existing processes
* Move get_manager to base module
* Process manager now knows about existing processes
* Guard API from manager.get_job() errors
* Guard API from job-related errors
* fix flak8-related issue on tests
* moved process-related exceptions to pygeoapi.process.base
---------
Co-authored-by: Ricardo Garcia Silva <ricardo@kartoza.com>
* Process manager now knows about existing processes
* Move get_manager to base module
* manager is now responsible for instantiating a processor
* manager.execute_process now accepts process_id instead of processor instance
* Add license to test file
* fix email address typo
---------
Co-authored-by: Ricardo Garcia Silva <ricardo@kartoza.com>
* Process manager now knows about existing processes
* get processes from the process manager rather than the config
* Move get_manager to base module
---------
Co-authored-by: Ricardo Garcia Silva <ricardo@kartoza.com>
* Update tinydb manager test to current api
* Add tinydb parallel tests to CI
* Use filelock library to help make tinydb threadsafe
The tests for parallel access sometimes failed with the previous
implementation. I'm currently not 100% sure why, but it might have
to do with the fact that the file is already created in the TinyDB
constructor which was not protected by a lock.
Also not protecting the reads can read to invalid reads in the case
when a read and a write happen simultaneously and the read catches
partially written data.
This implementation allows for fewer concurrency but is quite simple,
so we should be able to be confident about its safety.
Also the TinyDB manager is not intended for high performance and its
IO operations are orders of magnitude faster than requests over the
network or process executions anyway.
* - Added conformance class for Mapbox Vector Tiles
* - updated number of conformance classes on tests
---------
Co-authored-by: doublebyte1 <info@doublebyte.net>
* update time handling for edr provider
* add logging for time parsing
* fix time handling errors
Convert times to np.datetime64 to allow greater than/less than comparison. If slicing over time, separate xarray sel into temporal and spatial component (cannot use method='nearest' for time slices).
* fix temporal query dictionary
* fix time error handling
* fix list handling for single timestep
* single timestep handling
* spatial-only query amendment
* code clean-up
* convert datetime
* move code to methods
for reuse between position and cube. #1239
* formatting fixes
#1239
* time query test
#1239
* clean-up
* move time querying tests to api testing
* Update test_api.py
* update time key
* modify test values
* updates for desired query behavior
#1239
* Update xarray_edr.py
* sanitize datetime_ parameter with Z
OpenAPI document provides RFC 3339 compliant examples (exclusively specify UTC as the timezone with a trailing Z; want to avoid confusion between allowed datetime query and the documentation. #1239
* requested changes from @webb-ben and @tomkralidis
#1247
* Added enum for requested process execution modes
* Added data dict as expected base process.execute param
* Modified base process manager to select execution mode
* Pass requested execution mode to process manager
* Refactored dummy process manager to deal with request execution mode
* Use num value in response headers
* fix bug with non-existent process response headers
* fix flake8 errors
* fix python version errors
* fix python version errors
* fix python version errors
* refactor failing test
* refactor failing test
* fix line length
* fix using tuple in type annotations
* update docs and add missing param docstring
* swapped double quotes for single quotes for consistency
* OGC API - Features Part 2 (groundwork+CRS-BBOX) from PR #1155 - contributes to issue #1128
* #1128 provide conformance class for OAPIF Part 2 in /conformance page
* #1128 bitten by flake8...
* #1128 configurability CRS Feature Providers with syntax, defaults and tests
* #1128 configurability CRS Feature Providers refine for default values
* #1128 display supported CRSs in HTML Collection template
* #1128 config, mmetadata and tests for storageCRS and storageCrsCoordinateEpoch
* #1128 WIP for bbox-crs parameter support
* #1128 utility function and tests for default/mandatory supprted CRS list
* #1128 default supported CRS adaptation to OAPIF Part 2 standard
* #1128 grr flake8 whitespace
* #1128 start adding full API tests OGR for bbox-crs and crs parms
* #1128 fix flake8
* #1128 fix flake8 - install GDAL in workflow main for OGR tests
* #1128 fix flake8 - install GDAL in workflow main for OGR tests - need pip package?
* #1128 fix flake8 - install GDAL in workflow main for OGR tests - using libgdal-dev gdal-bin
* #1128 fix SensorThings test for main.yml Workflow
* #1128 fix SensorThings test for main.yml Workflow nr 2
* #1128 make all OGR tests working again
* #1128 make all OGR tests working again - flake8
* #1128 make all OGR tests working again - GeoSolutions WFS bbox
* #1128#1155 add documentation for OGC OAPIF Part 2 CRS CRS BBOX support
* #1128#1155 refine documentation for OGC OAPIF Part 2 CRS CRS BBOX support
* #1128#1155 refine documentation to align with #1149
* #1128#1155 rework from review OAS and pygeoapi config schema
* #1128#1155 minor: compile Re for CRS URI only once as global var
* #1128 merge in changes from PR #1173 - fix missing import
* WIP Ogcapi features part 2 - Support for crs query parameter (#1149)
* feat(ogcapi_features_crs): start implementing crs support from ogcapi features part2
* Pass input and output CRSs WKT instead of crs transformation object
* fix longs lines and blank lines
* fix typo
* fix import for type annotation not supported by python version
* fix variable visibility in local scope
* fix tabs/spaces indentations
* Add support for the crs parameter to OGRProvider
* make flake8 happy
* Make crs transformation mechanism more consistent between PostgreSQL and OGR providers
* test(util): add two test functions in util.py
New functions: test_get_crs_from_uri and test_get_transform_from_crs
* fix too long lines...
* Update get_crs_from_uri and corresponding test function
* fix(get_crs_from_uri): make the error more explicit in if wrong crs uri format
* flake8 again...
* Keep support for source_srs/target_srs in config for OGRProvider
* revert changes made to pygeoapi-config-0.x.yml, overlap with PR 1155
* test: add test data and update test config file
* Extract 'crs' and 'storage_crs' and provider level instead of collection level
* feat(crs): new decorator to support coordinates transformation of feature collections
* feat(crs): 'crs' query parameter for CSVProvider
* test(crs): add tests for 'crs' query parameter
* test: update number of collections in test_describe_collections
* test: update number of collections in test_filter_dict_by_key_value
* fix(crs_transform): change the crs transformation decorator
Change the logic of the decorator so that it works for both functions that
return FeatureCollections and for functions tha return single Features.
* test: add tests for get_collection_item end-point with 'crs' parameter
* fix(test_get_collection_item_crs): id as path parameter, not query parameter
* test: unpack coordinates to create point geometry
* feat(crs): add suuport for crs query parameter for all providers of type 'feature'
* docs(crs): add documentation to illustrate use of 'crs' query parameters
* docs(crs): more data access examples
* fix typo and add new line
* refactor: specify None as default value for crs_transform_out parameter in _sqlalchemy_to_feature method
* changes for PR 1149, test_api and style formatting
* CRS84 as default crs also for test_get_collection_items_crs
* test(crs): test coordinates transformation implementation of PostgreSQLProvider
* test(crs): move tests to test_postgresql_provider
* fix test function calls
* change test to ensure returned features are the same
* add json format to request object
* test(crs): test coordinates transformation implementation of OGRProvider
* refactor(crs): make more compact get_collection_item and get_collection_items
Define two new static methods in API class, to create crs_transform_wkt and
setting content-crs header. These methods can be re-used in both
get_collection_item and get_collection_items methods and removes code
duplication.
---------
Co-authored-by: Just van den Broecke <just@justobjects.nl>
* #1178 fix flake8 error
* #1178 use EPSG:28992 i.s.o. 32631 - fix unit test OGR Shapefile
* #1174 use CRS-compliant Axis ordering for crs support
* #1174 fix and honour CRS 4258disable native CRS Transform in OGR Provider - Axis ordering not honoured...
* #1174 remove ADR tests rom test_util.py
* #1174 enable native CRS transform again in OGR Provider
* #1174 enable native CRS transform again in OGR Provider - fix config
* #1174 remove support for source/target_srs in OGRProvider - enforce transforms always based on storageCRS
* #1174 fix tests Postgresql Provider for Transforms
* #1174 fix tests Postgresql Provider for Transforms
* #1174 add tests for OGR Transformation and Axis Order
* #1174 Suppress potential axis-swapping in OGR ExportToJSON
* #1174 minor fix test - unassign spatialref before setgeom infeat
* #1174 minor fix test - unassign spatialref before setgeom infeat - flake8
* #1174 solve CI WFS test failures with GDAL HTTP config options
* #1174 bbox and bbox-crs defs local in openapi.py for CITE validators
* #1174 merge master - #1152#1203 etc
* #1174 small doc changes
* #1174 move GeomObject typedef to beginning of util.py
* #1174 added debug logging in transform Decorator func
---------
Co-authored-by: Mathieu Tachon <92298764+MTachon@users.noreply.github.com>
* API design rule support (#1134):
- Change config model
- Change Flask, Starlette, and Django apps for API rule adherence
- Add Flask and Starlette mock clients to test API rule adherence
- Add get_base_url() util function to replace all config['server']['url'] refs
- Ensure that any internal links have URL prefixes if needed
- Add tests and update docs
* Prevent fcntl import error (breaks tests when running locally on Windows)
* Prefer trailing slash in landing page URL when strict_slashes=True (#1134)
* fix(automap): fix naming conflicts for PostgreSQLProvider
Automaping classes and relationships from database schema throws an error if a
column name is the same as a relationship name (see
https://docs-sqlalchemy.readthedocs.io/ko/latest/orm/extensions/automap.html#handling-simple-naming-conflicts).
* renaming function name_for_scalar_relationship -> _name_for_scalar_relationship
* test: test that PostgreSQL provider can handle naming conflicts
Re-use data file from https://github.com/geopython/pygeoapi/pull/1185, and make
to new and empty tables that create naming conflicts when queried.
* Change test function post_collection_items -> get_collection_items
* fix path to table
Bernhard Mallinger