* Add pygeofilter, unpin psycopg2
* Add pygeofilter SQLAlchemy query demo
* Add note about primary key
* Add todos to script
* Make script generic
* Add (commented out) PostgreSQL provider hotosm_bdi_waterways to config
* Update to use local test database
* Return result as dictionary
* Pass GEOM_FIELD as variable
* Add offset and limt, fix two names
* Implement order by
* Use function for cql query
* Add cql query block to query and test
* Link up to db engine
* Convert WKB geometry to GeoJSON
* Add shapely as explicit dependency
* Add tests for CQL queries
* Add test using CROSSES with LINESTRING
* Uncomment Docker PostGIS test layer
* First pass at wiring up API
The CQL is read from the request but the parser fails to convert to
AST.
* Turn logging to DEBUG
Example query
http://localhost:5000/collections/hot_osm_waterways/items?cql=%27osm_id%20BETWEEN%2080800000%20AND%2080900000%20AND%20name%20IS%20NULL%27
* Create dedicated pygeoapi-config.yml for testing PostgreSQL
* Add test for PostgreSQL CQL
* Return 400 for bad CQL
* Tidy up old files
* Bring API error type tests together
* Only reflect the table of interest
* Add pygeofilter to requirements
* Remove local setup_env.sh from .gitignore
* Remove lark exception handling
* Remove lark codes from tests, docstring change
* Split cql test into good and error
* Rename cql parameter to filter
* Add optional filter-lang parameter with test
* Initial changes
* Rename cql text parser
* Use CQL-JSON POST for PostgreSQL query
* Refactor to get data early and handle errors
* Add error tests for CQL-JSON PostgreSQL
* Add tests for prev/next and provider instantiation
* As filter-lang is set the block in not needed
* Clarify test fixtures
* Use filterq only in all CQL calls
* Make engine and table_model attributes of provider
* Handle different instantiation failures
* Use SQLAlchemy for get_fields()
* Minor doc edits
* Use SQLAlchemy for get()
* Simplify _sqlalchemy_to_feature
* Store database parameters in dedicated function
* Strip out non-SQLAlchemy code
* Add draft query() based on CQL filters
* Add property filtering to query()
* Implement select_properties and skip_geometry
* Implement bbox filter
* Implement properties subset defined in config
* Clean up materialized view test
We only need to test the fields that are present to be sure that the
view has been accessed, so other checks have been removed.
* Flake8 fixes and tidy up
* Add combined test for CQL with bbox and properties
* Create sessions directly where required
* Add Engine store to allow connection reuse
* Replace cql_ast with filterq
* Tidy up tests by using fixture for api
* Add authors
* Use Session() as context manager
A session requires a connection to the database, as provided by the
SQLAlchemy engine. When the session goes out of scope, e.g. at exit
from the function when it was used, it should be garbage collected and
the connection freed. However, during load testing it was found that
this wasn't happening. This commit uses a context manager pattern to
start the session, which ensures that the session is closed at the end
of the `with` block.
* Return fields type as {'type': ...}
* Cache the table_model with the engine
Generating the table_model by "reflecting" the database is expensive but
the resulting model doesn't change. If we cache the table_model with
the engine we save on a round-trip to the database and speed up response
time.
* Use separate stores for Engine and table_models
Using a combined store based on both engine and table_model meant that
a pygeoapi serving multiple tables on the same database would maintain
a connection pool for each table. This could eventually saturate the
number of available connections. This commit separates the store into
two parts.
* Update documentation with example
* Manage or post items based on POST content type
* ghcr build action
* only ghcr job
* run criteria edit
* Update README.md
* job = main
* remove success requirememt
* add tag
* tags: bgs-cql
* Update main.yml
* Update README.md
* remove binary
* update
* Update actions to latest versions
* Dockerfile refactor
* Dockerfile refactor
* restore dockerhub
* fix pygeoif version
* # 3.6 test & dockerhub ref
* Pin pygeoif version to less than 1
* Replace psycopg-binary with psycopg
* Split Docker build into sections
* Pin click version >7,<=8
* Add further cql examples
* Update workflows to match GeoPython repo
* update flake python version
* Unquote PostgreSQL column names
* Update container workflow uses versions
* flake8 use python version 3.7
* Use pre ping to get db connection
* Test that properties are set in provider
* Amendments as requested by @tomkralidis
* Correction
* Fix reviewed changes in api code
Co-authored-by: Colin Blackburn <colb@bgs.ac.uk>
Co-authored-by: KoalaGeo <eddlewis85@gmail.com>
Co-authored-by: Edd <edlew@bgs.ac.uk>
Co-authored-by: Colin Blackburn <ximenesuk@users.noreply.github.com>
* Start a wip alignment
* Adding CRS to tile_matrix_set_links
* Refactor json structure tiles endpoint
* Link section in tileset
* Unit test
Co-authored-by: Francesco Bartoli <francesco.bartoli@geobeyond.it>
* Add gzip to API.py
* pytest for gzip
* fix pytest
* Update test_api.py
* Add gzip to server block of configuration files
* Update api.py to include gzip opts from config
Update logic behind gzip compression such that compression will only happen when gzip in the Accept-Encoding request header and config.server.gzip is True. Use server charset for encoding/decoding and include charset in Content-Type when compressed.
* Update pytest to use gzip config options
Update pytest to test with (gzip in Accept-Encoding & server.gzip = False), (gzip in Accept-Encoding & server.gzip = True), (gzip not in Accept-Encoding & server.gzip = True)
* Better Content Negotiation (#7)
* Change content negotiation in _get_format()
* Add to pytest and force h to always be string
* Force F_JSON to default response
* Update test_api.py for utf-16 encoding
* better content negotiation pytest.
* Add comments to pytest
* Fix flake 8
* Improve docstring for gzip decorator function
* fix Coverage output and format exception MIME type
* Minor tweaks, added test to assert that a text/html header (without 'f' query param) returns JSON for get_collection_coverage()
Co-authored-by: Sander Schaminee <sander.schaminee@geocat.net>
* Created localization (l10n) module + tests. Added l10n support to API and plugins (wip).
* Big refactor:
* All routed API methods are now decorated by @pre_process (consistency) and no longer have a headers+format argument but a request argument (**kwargs also removed)
* The pre_process decorator turns an incoming Flask/Starlette request into a generic APIRequest instance
* The new APIRequest class extracts all relevant info (params, data, locale, etc.) from the request and exposes them as properties
* Removed a lot of boilerplate (i.e. format checking) and wrapped that into methods
* Updated server-specific API calls in each route method (pass entire request object, not headers and query params)
* Several improvements and fixes:
* Updated OpenAPI page with "l" query param
* Added example translations (metadata)
* Changed plugin signature: added explicit locale attribute (instead of **kwargs)
* Moved locale processing to get_plugin_locale() function in l10n module
* API should pass raw requested locale to plugins, locale should always be set
* Fixed API tests and added APIRequest tests
* Prepared utils.py for Jinja2 i18n extension
* Rebased on commit b40297a8 and fixed compatibility with #661 and #662
* Updated documentation for language support
* Rebased and fixed compatibility with PR #658:
* Fixed EDR provider signature (added locale)
* Fixed EDR API routes and query function (and improved parameter-name handling)
* Fixed EDR tests
* Translate entire config in render_j2_template for requested locale:
* Added new translate_dict function to l10n module (+ tests)
* Updated all render_j2_template calls with locale parameter
* Updated pygeoapi-test-config.yml with some language structs
* Minor improvements
* support both 'language' and 'languages' property in server config and provider definitions
* renamed and modified translate_dict() to more generic translate_struct() function (l10n module)
* remove Content-Language header from provider responses if provider has no language support and format is json(ld)
* updated tests
* Leave provider locale handling to API
* Moved code to determine locale from providers to API class (and remove for formatters and processes)
* Removed locale parameter from plugin __init__ signatures
* Removed locale parameter from load_plugin()
* Added **kwargs to provider implementations for get, query, get_metadata, get_coverage_domainset and get_coverage_rangetype method signatures
* Added language=<locale> to all API calls to provider get, query, get_metadata, get_coverage_domainset and get_coverage_rangetype methods
* Use 'lang' instead of 'l' as language query parameter
* Updated Open API
* Updated documentation
* Fixed tests
* Implemented requested PR changes:
* Added usage examples to the APIRequest docstring
* Removed language support from coverage functions
* Updated plugins.rst and language.rst to match new behavior
* Removed language struct from resource links in pygeoapi-config.yml
* Rebased on latest master (fixed test_api.py)
* Rebased and applied fixes:
* Data property in APIRequest now is an awaitable attribute (fixed for Starlette compatibility)
* Named references to 'l' parameter to 'lang'
* Final changes/improvements:
* Make sure that Content-Language is always set;
* Added more tests to ensure that the default language returned is the first configured language (if no language was requested by the user);
* Updated docs;
* Replaced re-occuring strings with constants in api.py;
* Fixed Flake8 checks.
* add missing async to starlette routes (#704)
Co-authored-by: Tom Kralidis <tomkralidis@gmail.com>
* add process manager
* process manager updating
* control connectivity from inside manager
* missing method on execute_process function signature
* adds tinydb depenedency for requirements-provider
* adds some processes tests
* process manager under server config, not a process
* additional html rendering for jobs and processes
* fixes typo in docstring
* complete html templating for jobs and processes
* considers timezone when formatting processing start/end datetimes
* expanded locales; locale and tz environment variables
* makes environment TZ a global from pygeoapi/__init__.py
* don't redefine built-in format
* adds placeholder values in process form if given as examples
* placeholder in form handles null example
* changes to process progress update
* updates and changes to processing web UI
* adds python3-distutils dep, and env changes for TZs and locales
* adds tinydb dependency for processes
* Adds libpq-dev and related deps for processing - should eventually be reconciled
* minor margin change
* fix bug in postJob.js that prevented results from displaying
* move process elements around to economise on space
* adds second, complex process
* post data may be bytes
* make external process example
* adds redis process manager (may be moved to third-party plugin eventually)
* reintroduce default async processing; refactoring managers; improving sync/async processing ui
* throw pygeoapi ManagerExecuteError, not redis exception
* adds test for async-execute
* restores original test host
* separates async execute handler function
* adds support for file uploads as processing inputs
* adds endpoint for file download as process output
* naming consistency
* Fix job submission form in case server url is relative (#549)
`new URL()` fails if the first argument isn't a complete url
* Implement job deletion in UI, api and tinydb
* Also allow deletion from job list
The UI could be fancier by reloading the page or just removing the
deleted job right away, but I don't want to start building that now.
* Delete output file when deleting a job
* typo in test case
* move delete button in job view
* remove Dockerfile changes (cf #470)
* removes some docs/source*.rst absent in master
* fixes whitespace
* fixes pip installs
* removes more locale-specific content, cf 70
* fixes duplicate app following merge
* removes redis as default plugin
* removes ENV_TZ, cf #407
* fix typo
* fixes syntax error
* process execution test cleanup
* Fix format detection in execute_process (#589)
Before, html was always selected.
* remove db files in code directories, fix flake8, specific functionality
* remove interactive capabilities in lieu of Swagger
* Add basic openapi documentation for job detail endpoint (#590)
It's somewhat superficial still, e.g. jobId is missing as
proper parameter in the UI.
* Show elapsed time for currently running jobs (#591)
* flake8, jobs workflow
* update docs, DB path in test config
* more code cleanup
* manager workflow cleanup
* add default dummy manager
* update docs
* fix routes
* fixed per comments
* fix per comments
* fix per comments
* fix travis
* simply describe process logic
* simplify describe process logic
Co-authored-by: Tom Kralidis <tomkralidis@gmail.com>
Co-authored-by: Jorge de Jesus <jorge.jesus@gmail.com>
Co-authored-by: totycro <bernhard.mallinger@eox.at>
Co-authored-by: Richard Law <richard.law@cartoncloud.com.au>
Tom Kralidis