* Release Process Manager backed by Postgresql
* Update postgresdb_.py
* Changes as for Tom comment on 16.7
* Update postgresql.py
* Included further Tom suggestions
* Renamed Manager requirements file
* renamed table job_info -> jobs
* Update postgres_manager_full_structure.backup.sql
* Update postgresql.py
* Update requirements-manager.txt
Accepted @Tom requirement.
I found the issue about compiling psycopg2:
it was a bug in my configuration.
* Modified to adhere to CI
* Update postgresql.py
---------
Co-authored-by: FrancescoIngv <FrancescoIngv@users.noreply.github.com>
* Move api to subdirectory
* Move processes api to own file
* Adapt processes view methods
* Move openapi definition to processes api
* Use processes api in flask
* Linter
* Fix import issues
* Allow calling refactored views from starlette
* Allow calling refactored views from django
* Linter
* Move edr api to own file
* Adapt edr api to new style
* Fix typo in django views
* Move maps api to own file
* Adapt maps api to new style
* Move edr openapi to edr api file
* Move maps openapi to maps api file
* Move stac views to own file
* Refactor stac views to new file
* Move stac openapi to stac api file
* Move tiles api to own file
* Adapt tiles api to new style
* Also move tilematrixset to tiles api
* Adapt tilesetmatrix views to new style
NOTE: I had to remove one tilematrixsets test because
it tested that an invalid format would produce an error.
This now happens by default for all views, but the actual
code is outside of the endpoint function.
* update features, records, coverages
* update release version
* switch back to dev
* backport of #1313
* backport of #1313 fix
* backport of #1585
* Flask: sanitize OGC schema pathing (#1593)
* update release version
* switch back to dev
* backport of #1596
* Port test_gzip_csv test
Note that apply_gzip is now called by the web framework adapters,
so to test it in general, we have to call it in the test manually
* Add empty conformance class list to stac api
* Fix queryables call in starlette
* fix ref
* Unify request validity checking
The default case is handled by the web framework adapters. If custom
format handling is required, the check in the adapter must be skipped.
* Fix imports in django views
* backport #1598
* Remove test about format handling in endpoint
This is now handled outside of the endpoint function
* add docstring to base process manager (#1603)
* backport of #1601
* Port api ogr tests to new style
* Move processes tests to own file
* Run api tests from new dir in CI
* Move edr tests to own file
* Move maps tests to own file
* Move tiles tests to own file
* Actually hide hidden layers in openapi
* 1600 allow providing default value in config (#1604)
* move coverages tests to own file
* move itemtypes to own file, move core into init test
* fix OpenAPI output
* update tests
* add missing descriptions to OpenAPI admin responses
* update tests
* fix tests autodiscovery
* remove unused logging in tests
* address PR comments
* test with xarray 2024.2.0
* remove unneeded file
* safeguard xarray error
* unpin xarray
* fix OpenAPI generation
* fix schema endpoint in Flask and Starlette
* Safely serialize configuration JSON (#1605)
* Safely serialize configuration JSON
Co-Authored-By: Tom Kralidis <tomkralidis@gmail.com>
* Revert "Safely serialize configuration JSON"
This reverts commit 36feb067ee6f87e61955852dc48994f075806370.
* Add test for datetime with Admin API
* Safely serialize configuration JSON
---------
Co-authored-by: Tom Kralidis <tomkralidis@gmail.com>
* backport #1611
* Also fix schema endpoint for django
Fix is analogous to e72d4ba3a5ba3b8621ca839e7814429beeeb8f01
* address additional PR comments
---------
Co-authored-by: Tom Kralidis <tomkralidis@gmail.com>
Co-authored-by: Angelos Tzotsos <gcpp.kalxas@gmail.com>
Co-authored-by: Ricardo Garcia Silva <ricardo.garcia.silva@gmail.com>
Co-authored-by: Benjamin Webb <40066515+webb-ben@users.noreply.github.com>
* Added initial stale bot CI workflow
* Added initial stale bot CI workflow
* Added permissions and exempt label to stale action
* Tweaked stale bot workflow file
---------
Co-authored-by: Ricardo Garcia Silva <ricardo@kartoza.com>
* Mock server in WFS provider tests
This commit mocks the layer in WFS tests and prepares return values
required to satisfy the tests. This reduces the test fidelity, but it
makes them stable and increases execution speed (literally 100x on my
machine, 66 sec to 0.6 sec).
Since the return value is mocked, it doesn't make sense to send
different parameters to the server, so some tests became obsolete.
The original tests are still there in another file, but are not supposed
to be run in CI.
* Mention wfs live tests in github CI
* Comment test file where all tests are skipped
pytest considers this to be a failure, but we still want the test file
to be mentioned here
* Clear up space before running CI
With all the databases and dependencies more than the 19G is
needed (this is the amount which is currenlty available on runners).
There is a github action which meddles with the disc layout, which is
probably too much for our use case, but it gave me the idea that you can
delete files which are needed to support certain platforms such as
dotnet and android.
https://github.com/easimon/maximize-build-space
With only manually removing those files, around 30G are available which
should be enough for the foreseeable future
* Update main.yml
---------
Co-authored-by: Tom Kralidis <tomkralidis@gmail.com>
* Pass OPENAPI_DOCUMENT to API also in django mode
Fixes#1509
* Add basic test for django
This only verifies that one page loads, which however
covers most of the django setup.
* Install django requirements in CI
* Add step for checking docker image CVEs
* Add buildx command and path to Dockerfile
* Fix working directory
Fix working directory
Fix working directory
Fix working directory
* Use trivy action
Use trivy action
* Use github placeholder over custom variable
* Add a step in CI for CVEs scanning
* Move the check for vulnerabilities on its own workflow
* Clean the clone job and comment accepted risk
* Add a badge for vulnerabilities clearance in the readme
* Add Admin API
- Create `admin.py` to serve as Admin API Core
- Create `flask_admin.py` to create flask blueprint for admin API
- Consolidate configuration getter
- Add Pathlib serializing
- Add docker example
* Add integration tests
- Amend admin example to allow writing to configuration. If FS is read only admin API does not work. Returns a 500 and logs `OSError: [Errno 30] Read-only file system: '/pygeoapi/local.config.yml' `
* Preserve env variables in configuration
* Use common accessor functions
- Use common configuration accessor methods for Django and Starlette
* GET returns raw config file
Return configuration with environment variables preserved on GET requests
* Safeguard env variables for root cfg view
"bind": {
"host": "localhost",
"port": "6000"
}
->
"bind": {
"host": "${HOST}",
"port": "${PORT}"
}
* Simplify admin HTML imports
- Use jinja recursion to expand the configuration
- Remove vue from templates
* Create admin API documentation
* Use render_item_value in admin template
* Add Admin API
- Create `admin.py` to serve as Admin API Core
- Create `flask_admin.py` to create flask blueprint for admin API
- Consolidate configuration getter
- Add Pathlib serializing
- Add docker example
* Update GitHub Actions deployment
* Update admin entrypoint
Update admin entrypoint to align with upstream pygeoapi implementation
* Make requested changes
Co-Authored-By: Tom Kralidis <tomkralidis@gmail.com>
* Amend test url
Co-Authored-By: Tom Kralidis <tomkralidis@gmail.com>
* Fix Admin CI tests
* Add PUT and PATCH for root configuration
- Add put and patch for root configuration
- Add CI tests for PUT and PATCH of root
* Update OpenAPI document wording
* Update entrypoint.sh
Replace tabs with spaces
* Remove unused step
Error from rebasing. Admin API tests are moved to their own job.
* Use jsonpatch
- Use debian supported packaging
- Use custom merge function
* Move test data location
* Create Starlette and Django app
- Fold flask_admin.py into flask_app.py
Co-Authored-By: Tom Kralidis <tomkralidis@gmail.com>
* Make requirements-admin.txt
Move admin dependencies to requirements-admin.txt
* Delete guiblock.html
* Update test count for STAC
Update expected test count for addt'l admin test data
* Relegate config warning to config.py
* Move admin tests out of example
* Delete admin docker example
* Update admin-api.rst
* Update pygeoapi-config-0.x.yml
* Update configuration.rst
* Update config.py
* Update admin.py
* Update admin.py
---------
Co-authored-by: Tom Kralidis <tomkralidis@gmail.com>
* Added support for table synonyms
* Added new parameters to query and manipulator call
* Changed error types
* Mount volumes to oracle container
* workflow part 2
* workflow part 3
* workflow part 4
* Changed file permissions to 777
* Deleted folder
* Recreated folder
* Changed to official Oracle Docker-Image
* Added Chown user
* back to gvenzl/oracle-xe:latest
* Tried docker-entrypoint-startdb.d
* Added addnab/docker-run-action@v3
* Added port and deamon mode
* next try
* added job.container.network
* next try
* + docker ps
* next try
* using docker run
* next try
* next try
* Changed len of array to 11
* Use sdo_util.from_geojsonfor create and update
* Flake8 changes
* Fixed error with views
* Added crs_transform_spec support
* Without default_crs
* Updated documentation for Oracle Provider
* changes for flake8
* Added crs_transform_spec support to get function
* review changes
* Added configurable SDO operator
* Refactored get_collection_coverage tests
Using pytest's parametrize mark. This makes it easier to isolate test cases in order to fix the errors that are showing up when running under Python3.8
* Add Python 3.8 to the CI matrix
* Reverted tests back to non-parametrized form
* Dropped Python 3.7 from main github CI workflow
* replaced ubuntu 20.04 with ubuntu 22.04 in CI workflow files
---------
Co-authored-by: Ricardo Garcia Silva <ricardo@kartoza.com>
* Refactored get_collection_coverage tests
Using pytest's parametrize mark. This makes it easier to isolate test cases in order to fix the errors that are showing up when running under Python3.8
* Refactor tests from test_api.py that were failing under Python 3.8
* Fix flake8 issues
* Add Python 3.8 to the CI matrix
* Replaced pandas usage with numpy for datetime
* Reverted tests back to non-parametrized form
* Dropped Python 3.7 from main github CI workflow
---------
Co-authored-by: Ricardo Garcia Silva <ricardo@kartoza.com>
* Update tinydb manager test to current api
* Add tinydb parallel tests to CI
* Use filelock library to help make tinydb threadsafe
The tests for parallel access sometimes failed with the previous
implementation. I'm currently not 100% sure why, but it might have
to do with the fact that the file is already created in the TinyDB
constructor which was not protected by a lock.
Also not protecting the reads can read to invalid reads in the case
when a read and a write happen simultaneously and the read catches
partially written data.
This implementation allows for fewer concurrency but is quite simple,
so we should be able to be confident about its safety.
Also the TinyDB manager is not intended for high performance and its
IO operations are orders of magnitude faster than requests over the
network or process executions anyway.
* Fix for Django build, changes to documentation, and dev dependencies updated
* remove example configuration files
* Added readthedocs configuration file
* added section on building the documentation
* renamed docs/requirements-docs.txt to docs/requirements.txt
* renamed requirements file in readthedocs configuration
* Added documentation requirements installation to GitHub build workflow
* OGC API - Features Part 2 (groundwork+CRS-BBOX) from PR #1155 - contributes to issue #1128
* #1128 provide conformance class for OAPIF Part 2 in /conformance page
* #1128 bitten by flake8...
* #1128 configurability CRS Feature Providers with syntax, defaults and tests
* #1128 configurability CRS Feature Providers refine for default values
* #1128 display supported CRSs in HTML Collection template
* #1128 config, mmetadata and tests for storageCRS and storageCrsCoordinateEpoch
* #1128 WIP for bbox-crs parameter support
* #1128 utility function and tests for default/mandatory supprted CRS list
* #1128 default supported CRS adaptation to OAPIF Part 2 standard
* #1128 grr flake8 whitespace
* #1128 start adding full API tests OGR for bbox-crs and crs parms
* #1128 fix flake8
* #1128 fix flake8 - install GDAL in workflow main for OGR tests
* #1128 fix flake8 - install GDAL in workflow main for OGR tests - need pip package?
* #1128 fix flake8 - install GDAL in workflow main for OGR tests - using libgdal-dev gdal-bin
* #1128 fix SensorThings test for main.yml Workflow
* #1128 fix SensorThings test for main.yml Workflow nr 2
* #1128 make all OGR tests working again
* #1128 make all OGR tests working again - flake8
* #1128 make all OGR tests working again - GeoSolutions WFS bbox
* #1128#1155 add documentation for OGC OAPIF Part 2 CRS CRS BBOX support
* #1128#1155 refine documentation for OGC OAPIF Part 2 CRS CRS BBOX support
* #1128#1155 refine documentation to align with #1149
* #1128#1155 rework from review OAS and pygeoapi config schema
* #1128#1155 minor: compile Re for CRS URI only once as global var
* #1128 merge in changes from PR #1173 - fix missing import
* WIP Ogcapi features part 2 - Support for crs query parameter (#1149)
* feat(ogcapi_features_crs): start implementing crs support from ogcapi features part2
* Pass input and output CRSs WKT instead of crs transformation object
* fix longs lines and blank lines
* fix typo
* fix import for type annotation not supported by python version
* fix variable visibility in local scope
* fix tabs/spaces indentations
* Add support for the crs parameter to OGRProvider
* make flake8 happy
* Make crs transformation mechanism more consistent between PostgreSQL and OGR providers
* test(util): add two test functions in util.py
New functions: test_get_crs_from_uri and test_get_transform_from_crs
* fix too long lines...
* Update get_crs_from_uri and corresponding test function
* fix(get_crs_from_uri): make the error more explicit in if wrong crs uri format
* flake8 again...
* Keep support for source_srs/target_srs in config for OGRProvider
* revert changes made to pygeoapi-config-0.x.yml, overlap with PR 1155
* test: add test data and update test config file
* Extract 'crs' and 'storage_crs' and provider level instead of collection level
* feat(crs): new decorator to support coordinates transformation of feature collections
* feat(crs): 'crs' query parameter for CSVProvider
* test(crs): add tests for 'crs' query parameter
* test: update number of collections in test_describe_collections
* test: update number of collections in test_filter_dict_by_key_value
* fix(crs_transform): change the crs transformation decorator
Change the logic of the decorator so that it works for both functions that
return FeatureCollections and for functions tha return single Features.
* test: add tests for get_collection_item end-point with 'crs' parameter
* fix(test_get_collection_item_crs): id as path parameter, not query parameter
* test: unpack coordinates to create point geometry
* feat(crs): add suuport for crs query parameter for all providers of type 'feature'
* docs(crs): add documentation to illustrate use of 'crs' query parameters
* docs(crs): more data access examples
* fix typo and add new line
* refactor: specify None as default value for crs_transform_out parameter in _sqlalchemy_to_feature method
* changes for PR 1149, test_api and style formatting
* CRS84 as default crs also for test_get_collection_items_crs
* test(crs): test coordinates transformation implementation of PostgreSQLProvider
* test(crs): move tests to test_postgresql_provider
* fix test function calls
* change test to ensure returned features are the same
* add json format to request object
* test(crs): test coordinates transformation implementation of OGRProvider
* refactor(crs): make more compact get_collection_item and get_collection_items
Define two new static methods in API class, to create crs_transform_wkt and
setting content-crs header. These methods can be re-used in both
get_collection_item and get_collection_items methods and removes code
duplication.
---------
Co-authored-by: Just van den Broecke <just@justobjects.nl>
* #1178 fix flake8 error
* #1178 use EPSG:28992 i.s.o. 32631 - fix unit test OGR Shapefile
* #1174 use CRS-compliant Axis ordering for crs support
* #1174 fix and honour CRS 4258disable native CRS Transform in OGR Provider - Axis ordering not honoured...
* #1174 remove ADR tests rom test_util.py
* #1174 enable native CRS transform again in OGR Provider
* #1174 enable native CRS transform again in OGR Provider - fix config
* #1174 remove support for source/target_srs in OGRProvider - enforce transforms always based on storageCRS
* #1174 fix tests Postgresql Provider for Transforms
* #1174 fix tests Postgresql Provider for Transforms
* #1174 add tests for OGR Transformation and Axis Order
* #1174 Suppress potential axis-swapping in OGR ExportToJSON
* #1174 minor fix test - unassign spatialref before setgeom infeat
* #1174 minor fix test - unassign spatialref before setgeom infeat - flake8
* #1174 solve CI WFS test failures with GDAL HTTP config options
* #1174 bbox and bbox-crs defs local in openapi.py for CITE validators
* #1174 merge master - #1152#1203 etc
* #1174 small doc changes
* #1174 move GeomObject typedef to beginning of util.py
* #1174 added debug logging in transform Decorator func
---------
Co-authored-by: Mathieu Tachon <92298764+MTachon@users.noreply.github.com>
* fix(automap): fix naming conflicts for PostgreSQLProvider
Automaping classes and relationships from database schema throws an error if a
column name is the same as a relationship name (see
https://docs-sqlalchemy.readthedocs.io/ko/latest/orm/extensions/automap.html#handling-simple-naming-conflicts).
* renaming function name_for_scalar_relationship -> _name_for_scalar_relationship
* test: test that PostgreSQL provider can handle naming conflicts
Re-use data file from https://github.com/geopython/pygeoapi/pull/1185, and make
to new and empty tables that create naming conflicts when queried.
* Change test function post_collection_items -> get_collection_items
* fix path to table
* Add pygeofilter, unpin psycopg2
* Add pygeofilter SQLAlchemy query demo
* Add note about primary key
* Add todos to script
* Make script generic
* Add (commented out) PostgreSQL provider hotosm_bdi_waterways to config
* Update to use local test database
* Return result as dictionary
* Pass GEOM_FIELD as variable
* Add offset and limt, fix two names
* Implement order by
* Use function for cql query
* Add cql query block to query and test
* Link up to db engine
* Convert WKB geometry to GeoJSON
* Add shapely as explicit dependency
* Add tests for CQL queries
* Add test using CROSSES with LINESTRING
* Uncomment Docker PostGIS test layer
* First pass at wiring up API
The CQL is read from the request but the parser fails to convert to
AST.
* Turn logging to DEBUG
Example query
http://localhost:5000/collections/hot_osm_waterways/items?cql=%27osm_id%20BETWEEN%2080800000%20AND%2080900000%20AND%20name%20IS%20NULL%27
* Create dedicated pygeoapi-config.yml for testing PostgreSQL
* Add test for PostgreSQL CQL
* Return 400 for bad CQL
* Tidy up old files
* Bring API error type tests together
* Only reflect the table of interest
* Add pygeofilter to requirements
* Remove local setup_env.sh from .gitignore
* Remove lark exception handling
* Remove lark codes from tests, docstring change
* Split cql test into good and error
* Rename cql parameter to filter
* Add optional filter-lang parameter with test
* Initial changes
* Rename cql text parser
* Use CQL-JSON POST for PostgreSQL query
* Refactor to get data early and handle errors
* Add error tests for CQL-JSON PostgreSQL
* Add tests for prev/next and provider instantiation
* As filter-lang is set the block in not needed
* Clarify test fixtures
* Use filterq only in all CQL calls
* Make engine and table_model attributes of provider
* Handle different instantiation failures
* Use SQLAlchemy for get_fields()
* Minor doc edits
* Use SQLAlchemy for get()
* Simplify _sqlalchemy_to_feature
* Store database parameters in dedicated function
* Strip out non-SQLAlchemy code
* Add draft query() based on CQL filters
* Add property filtering to query()
* Implement select_properties and skip_geometry
* Implement bbox filter
* Implement properties subset defined in config
* Clean up materialized view test
We only need to test the fields that are present to be sure that the
view has been accessed, so other checks have been removed.
* Flake8 fixes and tidy up
* Add combined test for CQL with bbox and properties
* Create sessions directly where required
* Add Engine store to allow connection reuse
* Replace cql_ast with filterq
* Tidy up tests by using fixture for api
* Add authors
* Use Session() as context manager
A session requires a connection to the database, as provided by the
SQLAlchemy engine. When the session goes out of scope, e.g. at exit
from the function when it was used, it should be garbage collected and
the connection freed. However, during load testing it was found that
this wasn't happening. This commit uses a context manager pattern to
start the session, which ensures that the session is closed at the end
of the `with` block.
* Return fields type as {'type': ...}
* Cache the table_model with the engine
Generating the table_model by "reflecting" the database is expensive but
the resulting model doesn't change. If we cache the table_model with
the engine we save on a round-trip to the database and speed up response
time.
* Use separate stores for Engine and table_models
Using a combined store based on both engine and table_model meant that
a pygeoapi serving multiple tables on the same database would maintain
a connection pool for each table. This could eventually saturate the
number of available connections. This commit separates the store into
two parts.
* Update documentation with example
* Manage or post items based on POST content type
* ghcr build action
* only ghcr job
* run criteria edit
* Update README.md
* job = main
* remove success requirememt
* add tag
* tags: bgs-cql
* Update main.yml
* Update README.md
* remove binary
* update
* Update actions to latest versions
* Dockerfile refactor
* Dockerfile refactor
* restore dockerhub
* fix pygeoif version
* # 3.6 test & dockerhub ref
* Pin pygeoif version to less than 1
* Replace psycopg-binary with psycopg
* Split Docker build into sections
* Pin click version >7,<=8
* Add further cql examples
* Update workflows to match GeoPython repo
* update flake python version
* Unquote PostgreSQL column names
* Update container workflow uses versions
* flake8 use python version 3.7
* Use pre ping to get db connection
* Test that properties are set in provider
* Amendments as requested by @tomkralidis
* Correction
* Fix reviewed changes in api code
Co-authored-by: Colin Blackburn <colb@bgs.ac.uk>
Co-authored-by: KoalaGeo <eddlewis85@gmail.com>
Co-authored-by: Edd <edlew@bgs.ac.uk>
Co-authored-by: Colin Blackburn <ximenesuk@users.noreply.github.com>
* Removed branches requirement on build action
Build action will now run whenever Lint job is completed: All pushes, PR to Master branch & releases
* Put lint in same main.yml
Lint is still a requirement for the build job
* Docker build only runs on master branch
Tom Kralidis