* Implemented Connection Pooling
First part of collection pooling, improve by making pool size and other params configurable in YAML file of pygeoapi
* updated session pooling with more robust creation of dsn
* Further changes for releasing connections
* further little tweaks
* Added Thread Lock and building of DSN
* Session Pooling now only happens when ENV VAR ORACLE_POOL_MIN and ORACLE_POOL_MAX are set. Else usual behaviour is restored and single connections are opened and closed. Added a Lock for avoiding race conditions and threat safety.
* Flake8 changes
* More Flake8
* Added integer type to env variables
* Added Tests for Session Pooling
* Flake 8 changes for tests
* More Flake8 fun
* Fixed typo for env vars
* Fixed typo for env var and added noqa for variables
* More Typos
* More Flake8
* Added documentation to the oracle section for session pooling
* Review and Feedback with @totycro
* Update oracle.py
Added Author
* Added feedback from review
---------
Co-authored-by: root <root@els01entw.ama.at>
Co-authored-by: xlanger <moritz.langer@ama.gv.at>
* Move api to subdirectory
* Move processes api to own file
* Adapt processes view methods
* Move openapi definition to processes api
* Use processes api in flask
* Linter
* Fix import issues
* Allow calling refactored views from starlette
* Allow calling refactored views from django
* Linter
* Move edr api to own file
* Adapt edr api to new style
* Fix typo in django views
* Move maps api to own file
* Adapt maps api to new style
* Move edr openapi to edr api file
* Move maps openapi to maps api file
* Move stac views to own file
* Refactor stac views to new file
* Move stac openapi to stac api file
* Move tiles api to own file
* Adapt tiles api to new style
* Also move tilematrixset to tiles api
* Adapt tilesetmatrix views to new style
NOTE: I had to remove one tilematrixsets test because
it tested that an invalid format would produce an error.
This now happens by default for all views, but the actual
code is outside of the endpoint function.
* update features, records, coverages
* update release version
* switch back to dev
* backport of #1313
* backport of #1313 fix
* backport of #1585
* Flask: sanitize OGC schema pathing (#1593)
* update release version
* switch back to dev
* backport of #1596
* Port test_gzip_csv test
Note that apply_gzip is now called by the web framework adapters,
so to test it in general, we have to call it in the test manually
* Add empty conformance class list to stac api
* Fix queryables call in starlette
* fix ref
* Unify request validity checking
The default case is handled by the web framework adapters. If custom
format handling is required, the check in the adapter must be skipped.
* Fix imports in django views
* backport #1598
* Remove test about format handling in endpoint
This is now handled outside of the endpoint function
* add docstring to base process manager (#1603)
* backport of #1601
* Port api ogr tests to new style
* Move processes tests to own file
* Run api tests from new dir in CI
* Move edr tests to own file
* Move maps tests to own file
* Move tiles tests to own file
* Actually hide hidden layers in openapi
* 1600 allow providing default value in config (#1604)
* move coverages tests to own file
* move itemtypes to own file, move core into init test
* fix OpenAPI output
* update tests
* add missing descriptions to OpenAPI admin responses
* update tests
* fix tests autodiscovery
* remove unused logging in tests
* address PR comments
* test with xarray 2024.2.0
* remove unneeded file
* safeguard xarray error
* unpin xarray
* fix OpenAPI generation
* fix schema endpoint in Flask and Starlette
* Safely serialize configuration JSON (#1605)
* Safely serialize configuration JSON
Co-Authored-By: Tom Kralidis <tomkralidis@gmail.com>
* Revert "Safely serialize configuration JSON"
This reverts commit 36feb067ee6f87e61955852dc48994f075806370.
* Add test for datetime with Admin API
* Safely serialize configuration JSON
---------
Co-authored-by: Tom Kralidis <tomkralidis@gmail.com>
* backport #1611
* Also fix schema endpoint for django
Fix is analogous to e72d4ba3a5ba3b8621ca839e7814429beeeb8f01
* address additional PR comments
---------
Co-authored-by: Tom Kralidis <tomkralidis@gmail.com>
Co-authored-by: Angelos Tzotsos <gcpp.kalxas@gmail.com>
Co-authored-by: Ricardo Garcia Silva <ricardo.garcia.silva@gmail.com>
Co-authored-by: Benjamin Webb <40066515+webb-ben@users.noreply.github.com>
* Exclude None from `get_processor` return type annotation
An exception is raised in case of error, so it can't ever return None
* Add support for OGC API Processes Subscriber
The subscription URLs are passed to the manager, which
then has to call them appropriately.
By default, managers have the attribute `supports_subscribing`
set to `False` in order to not break the API for these. The
subscriptions are only passed to if this is set to `True`
* Add ogc api callback class to conformance
https://docs.ogc.org/is/18-062r2/18-062r2.html#toc67
* Make successUri mandatory in subscriber
It's mandatory in the standard.
Thx @ricardogsilva !
* Use snake case in python for fields which are camel case in the api
Thx @ricardogsilva !
* Add subscriber to method docstring
* Provide default value for subscriber for managers not supporting it
Thanks @ricardogsilva !
* Factor out notification call into methods
This increases reusability by other managers
Thanks @ricardogsilva !
* Add an example call for a process subscriber
* Change test urls to valid urls
* Third party imports in own block
* Revert pydantic models to v1 version
Revert pydantic models to v1 version
Revert pydantic models to v1 version
Revert pydantic models to v1 version
* Add initial tests for models
Add initial tests for models
* Revert pydantic models to v1 version
Revert pydantic models to v1 version
Revert pydantic models to v1 version
Revert pydantic models to v1 version
* Add initial tests for models
Add initial tests for models
* Fix and replace methods from pydantic v2
* Add more tests for cql models
* Mock server in WFS provider tests
This commit mocks the layer in WFS tests and prepares return values
required to satisfy the tests. This reduces the test fidelity, but it
makes them stable and increases execution speed (literally 100x on my
machine, 66 sec to 0.6 sec).
Since the return value is mocked, it doesn't make sense to send
different parameters to the server, so some tests became obsolete.
The original tests are still there in another file, but are not supposed
to be run in CI.
* Mention wfs live tests in github CI
* Comment test file where all tests are skipped
pytest considers this to be a failure, but we still want the test file
to be mentioned here
* Implement extra_properties in oracle Provider
These can be used to configure additional database-computed fields
in the config file which are returned on `get` and `query` calls
* Allow mandating properties which are not part of the output
Previously, properties which were not requested for the output
were not part of `fields`, which means that they were not passed
in to the provider as filter properties for e.g. `query()`.
This commit adds them there and introduces a new variable
`filtered_fields`, which is used for limiting the output of queries.
There is also some minor refactoring, but the existing and also the
newly written tests should avoid regressions.
* Restore previous behavior for default arguments
* EDR queries without instance id works
Django URL's without instance IDs called a method requiring instance
IDs, resulting in a crash.
Instance ID defaults to None, which is also standard for the provider
that ultimately will be called.
Regression test added
* Use fixtures in django server tests
---------
Co-authored-by: Peter Garnæs <pga@dmi.dk>
* Allow either URL or URN CRS URIs
* Implemented transformation of geometries in CQL filter
* Fixed flake8 issue
* Removed commented out code
* Implemented support for the filter-crs query parameter
* Remove unneeded print() calls
* - refactored mvt classes to support all implemented metadata formats, regardless of the provider
* - fixed formatting issues
* Implementing basic tile metadata methods
* Fixing yml models
* Adding additional format
* Fixing schema set on load
* Removing unused field from documentation
* - added support to TileMatrixSets endpoint
- added TileMatrixSet definitions for WorldCRS84Quad and WebMercatorQuad
* - added tiling-schemes link in the json representation of the landing page
* - added html pages for tilematrixset endpoints
* - advertise json and html representations of the tiling schemes in the landing page
* - Use api definition of Well-known TileMatrixSets in the tiling-schemes element of the tiles metadata page
* - added tiling-scheme url on tiles metadata page, for es and tippecanoe providers
* - fixed flak8 formatting errors
* - updated number of links on the landing page, on the api test
* - Manage tile matrix set id dinamically, on tilematrix set flask endpoint
* - renamed functions to lower case
* - renamed tilematrix set functions on flask
* - Use TtileMatrixSetId parameter in tilematrixset api function
* - added support to TileMatrixSet endpoints on starlette
* - added test for tileMatrixSets api endpoint
* - added test for the tilematrixset endpoint
* - added routes for django
---------
Co-authored-by: Antonio Cerciello <antonio.cerciello@gmail.com>
* Pass OPENAPI_DOCUMENT to API also in django mode
Fixes#1509
* Add basic test for django
This only verifies that one page loads, which however
covers most of the django setup.
* Install django requirements in CI
* Added .pre-commit-config.yaml file
* Initial pass of pre-commit with its default rules and flake8
* Revert "Initial pass of pre-commit with its default rules and flake8"
This reverts commit 86d39e964e8122580d675f4639ae2d15b85c02e2.
* Made pre-commit only act on code files
* Added sql to the pre-commit file pattern
* Initial pass of pre-commit
* - refactored mvt classes to support all implemented metadata formats, regardless of the provider
* - fixed formatting issues
* Implementing basic tile metadata methods
* Fixing yml models
* Adding additional format
* Fixing schema set on load
* Removing unused field from documentation
* Change method name to generic vendor
* Keeping extra metadata info for tippecanoe provider
* Fix flake validations error
---------
Co-authored-by: doublebyte <doublebyte@hushmail.com>
This api method did not have any tests until now.
This also contains a small fix: If no content type was requested,
then by default, the header specified the content type
`application/json`, whereas the content was actually html. Note that
this is rarely a problem because browsers do request html explicitly.
* Add Admin API
- Create `admin.py` to serve as Admin API Core
- Create `flask_admin.py` to create flask blueprint for admin API
- Consolidate configuration getter
- Add Pathlib serializing
- Add docker example
* Add integration tests
- Amend admin example to allow writing to configuration. If FS is read only admin API does not work. Returns a 500 and logs `OSError: [Errno 30] Read-only file system: '/pygeoapi/local.config.yml' `
* Preserve env variables in configuration
* Use common accessor functions
- Use common configuration accessor methods for Django and Starlette
* GET returns raw config file
Return configuration with environment variables preserved on GET requests
* Safeguard env variables for root cfg view
"bind": {
"host": "localhost",
"port": "6000"
}
->
"bind": {
"host": "${HOST}",
"port": "${PORT}"
}
* Simplify admin HTML imports
- Use jinja recursion to expand the configuration
- Remove vue from templates
* Create admin API documentation
* Use render_item_value in admin template
* Add Admin API
- Create `admin.py` to serve as Admin API Core
- Create `flask_admin.py` to create flask blueprint for admin API
- Consolidate configuration getter
- Add Pathlib serializing
- Add docker example
* Update GitHub Actions deployment
* Update admin entrypoint
Update admin entrypoint to align with upstream pygeoapi implementation
* Make requested changes
Co-Authored-By: Tom Kralidis <tomkralidis@gmail.com>
* Amend test url
Co-Authored-By: Tom Kralidis <tomkralidis@gmail.com>
* Fix Admin CI tests
* Add PUT and PATCH for root configuration
- Add put and patch for root configuration
- Add CI tests for PUT and PATCH of root
* Update OpenAPI document wording
* Update entrypoint.sh
Replace tabs with spaces
* Remove unused step
Error from rebasing. Admin API tests are moved to their own job.
* Use jsonpatch
- Use debian supported packaging
- Use custom merge function
* Move test data location
* Create Starlette and Django app
- Fold flask_admin.py into flask_app.py
Co-Authored-By: Tom Kralidis <tomkralidis@gmail.com>
* Make requirements-admin.txt
Move admin dependencies to requirements-admin.txt
* Delete guiblock.html
* Update test count for STAC
Update expected test count for addt'l admin test data
* Relegate config warning to config.py
* Move admin tests out of example
* Delete admin docker example
* Update admin-api.rst
* Update pygeoapi-config-0.x.yml
* Update configuration.rst
* Update config.py
* Update admin.py
* Update admin.py
---------
Co-authored-by: Tom Kralidis <tomkralidis@gmail.com>
* Use generic error handling in get_collection_item
* Use generic error handling in get_collection_items
* Allow specifying a user-facing message in ProviderGenericError
* ose generic error handling in more instances
* Use generic error handling in more instances
* Linting fixes
* - Refactor MVTProvider, to support subclassing
- Added provider for MVT Tippecanoe
* - fixed test configuration for the MVT tippecanoe provider
* - added MVT elastic provider
* - added documentation for mvt providers
- fixed formatting
* - removed tiles publishing from elastic provider example
- updated gitignore to ignore data folder from elastic example
* - added docker example for MVT-elastic backend provider
* - Added docker example for tippecanoe MVT provider
* - updated README of the docker examples section, to catch up with the provided examples.
* - renamed mvt-elastic and mvt-tippecanoe docker example folders.
* - renamed elastic and tippecanoe folders to lower case
* - fixed formatting issue
* - s/Mabox/Mapbox/
* - put plugin provider names in single line
* - renamed MVT provider base class to BaseMVTProvider.
* - added MVT base class
* - reviwed uppercase on .gitignore
- add folder to .gitignore
* - removed new line and brackets
* - removed redundant line with base class
---------
Co-authored-by: doublebyte1 <info@doublebyte.net>
* Added support for table synonyms
* Added new parameters to query and manipulator call
* Changed error types
* Mount volumes to oracle container
* workflow part 2
* workflow part 3
* workflow part 4
* Changed file permissions to 777
* Deleted folder
* Recreated folder
* Changed to official Oracle Docker-Image
* Added Chown user
* back to gvenzl/oracle-xe:latest
* Tried docker-entrypoint-startdb.d
* Added addnab/docker-run-action@v3
* Added port and deamon mode
* next try
* added job.container.network
* next try
* + docker ps
* next try
* using docker run
* next try
* next try
* Changed len of array to 11
* Use sdo_util.from_geojsonfor create and update
* Flake8 changes
* Fixed error with views
* Added crs_transform_spec support
* Without default_crs
* Updated documentation for Oracle Provider
* changes for flake8
* Added crs_transform_spec support to get function
* review changes
* Added configurable SDO operator
This now fixes the issue that the `Accept` header wasn't considered for
format selection.
This adds netcdf as a known mime type in order to be able to test this,
but also for actual clients to use.
Note that if we want to allow other mime types such as `application/x-grib2`,
we also have to add them to `FORMAT_TYPES`.
Fixes#1390
* Refactored get_collection_coverage tests
Using pytest's parametrize mark. This makes it easier to isolate test cases in order to fix the errors that are showing up when running under Python3.8
* Refactor tests from test_api.py that were failing under Python 3.8
* Fix flake8 issues
* Add Python 3.8 to the CI matrix
* Replaced pandas usage with numpy for datetime
* Reverted tests back to non-parametrized form
* Dropped Python 3.7 from main github CI workflow
---------
Co-authored-by: Ricardo Garcia Silva <ricardo@kartoza.com>
* resolving case when lat dimensions are decreasing
CF conventions state only that a "coordinate variable must have values that are strictly monotonic (increasing or decreasing)." In the case of PRISM, the lat coordinates are decreasing. pygeoapi failed due to the lat dimension being empty with existing implementation; this update addresses the issue.
* remove blank space
* resolving case when lat dimensions are decreasing
CF conventions state only that a "coordinate variable must have values that are strictly monotonic (increasing or decreasing)." In the case of PRISM, the lat coordinates are decreasing. pygeoapi failed due to the lat dimension being empty with existing implementation; this update addresses the issue.
* remove blank space
* test decreasing latitude
The new test will now uncover if you can successfully serve data stored on S3 AND test if you can perform a cube query on data with monotonically decreasing latitude coordinates at the same time.
* flake8 py3 fixes
* read from s3 bucket
* fix formatting for build workflow
* add s3fs to provider requirements
* storage options for fsspec
* Revert "storage options for fsspec"
This reverts commit dcfaceedd78d0f0e0dc3e38610958671c27b06a6.
* Revert "storage options for fsspec"
This reverts commit dcfaceedd78d0f0e0dc3e38610958671c27b06a6.
* delete extraneous code for cloud storage read
* update documentation
* add ci testing
* update test suite config file
* update number of collections
* update test query for s3
* ci testing updates
* read from s3 bucket
* fix formatting for build workflow
* add s3fs to provider requirements
* storage options for fsspec
* Revert "storage options for fsspec"
This reverts commit dcfaceedd78d0f0e0dc3e38610958671c27b06a6.
* Revert "storage options for fsspec"
This reverts commit dcfaceedd78d0f0e0dc3e38610958671c27b06a6.
* delete extraneous code for cloud storage read
* update documentation
* add ci testing
* update test suite config file
* update number of collections
* update test query for s3
* ci testing updates
* documentation updates
* config updates
* flake8
* modify sfs3 version
installing requirements threw an error when specifying 2023.6.0 (only identifies up through 2023.1.0). Had added the sfs3 requirement to deal with issues thrown when the 2023.9.0 release came out.
* updated coverages documentation
* Fix for Django build, changes to documentation, and dev dependencies updated
* remove example configuration files
* Added readthedocs configuration file
* added section on building the documentation
* renamed docs/requirements-docs.txt to docs/requirements.txt
* renamed requirements file in readthedocs configuration
* Added documentation requirements installation to GitHub build workflow
* fixed pagination next page response
* fixed line too long
* Update get_collection_items test to accomodate last page fix
Note that this feature can't be tested with the CSV provider
because it uses the number of current items as numberMatched,
which doesn't seem correct but is an independent issue.
https://github.com/geopython/pygeoapi/blob/c7c07855e5cfa95b88d6d0d11f5c23111e38da95/pygeoapi/provider/csv_.py#L173
---------
Co-authored-by: Juan Pablo Duque <juan-pablo-duque@hotmail.com>
Co-authored-by: Juan Duque <juanpablo.duque@polimi.it>
* Process manager now knows about existing processes
* Move get_manager to base module
* Process manager now knows about existing processes
* Guard API from manager.get_job() errors
* Guard API from job-related errors
* fix flak8-related issue on tests
* moved process-related exceptions to pygeoapi.process.base
---------
Co-authored-by: Ricardo Garcia Silva <ricardo@kartoza.com>
* Process manager now knows about existing processes
* Move get_manager to base module
* manager is now responsible for instantiating a processor
* manager.execute_process now accepts process_id instead of processor instance
* Add license to test file
* fix email address typo
---------
Co-authored-by: Ricardo Garcia Silva <ricardo@kartoza.com>
* Process manager now knows about existing processes
* get processes from the process manager rather than the config
* Move get_manager to base module
---------
Co-authored-by: Ricardo Garcia Silva <ricardo@kartoza.com>
Fredrik Bore