644281359a
* Add pygeofilter, unpin psycopg2 * Add pygeofilter SQLAlchemy query demo * Add note about primary key * Add todos to script * Make script generic * Add (commented out) PostgreSQL provider hotosm_bdi_waterways to config * Update to use local test database * Return result as dictionary * Pass GEOM_FIELD as variable * Add offset and limt, fix two names * Implement order by * Use function for cql query * Add cql query block to query and test * Link up to db engine * Convert WKB geometry to GeoJSON * Add shapely as explicit dependency * Add tests for CQL queries * Add test using CROSSES with LINESTRING * Uncomment Docker PostGIS test layer * First pass at wiring up API The CQL is read from the request but the parser fails to convert to AST. * Turn logging to DEBUG Example query http://localhost:5000/collections/hot_osm_waterways/items?cql=%27osm_id%20BETWEEN%2080800000%20AND%2080900000%20AND%20name%20IS%20NULL%27 * Create dedicated pygeoapi-config.yml for testing PostgreSQL * Add test for PostgreSQL CQL * Return 400 for bad CQL * Tidy up old files * Bring API error type tests together * Only reflect the table of interest * Add pygeofilter to requirements * Remove local setup_env.sh from .gitignore * Remove lark exception handling * Remove lark codes from tests, docstring change * Split cql test into good and error * Rename cql parameter to filter * Add optional filter-lang parameter with test * Initial changes * Rename cql text parser * Use CQL-JSON POST for PostgreSQL query * Refactor to get data early and handle errors * Add error tests for CQL-JSON PostgreSQL * Add tests for prev/next and provider instantiation * As filter-lang is set the block in not needed * Clarify test fixtures * Use filterq only in all CQL calls * Make engine and table_model attributes of provider * Handle different instantiation failures * Use SQLAlchemy for get_fields() * Minor doc edits * Use SQLAlchemy for get() * Simplify _sqlalchemy_to_feature * Store database parameters in dedicated function * Strip out non-SQLAlchemy code * Add draft query() based on CQL filters * Add property filtering to query() * Implement select_properties and skip_geometry * Implement bbox filter * Implement properties subset defined in config * Clean up materialized view test We only need to test the fields that are present to be sure that the view has been accessed, so other checks have been removed. * Flake8 fixes and tidy up * Add combined test for CQL with bbox and properties * Create sessions directly where required * Add Engine store to allow connection reuse * Replace cql_ast with filterq * Tidy up tests by using fixture for api * Add authors * Use Session() as context manager A session requires a connection to the database, as provided by the SQLAlchemy engine. When the session goes out of scope, e.g. at exit from the function when it was used, it should be garbage collected and the connection freed. However, during load testing it was found that this wasn't happening. This commit uses a context manager pattern to start the session, which ensures that the session is closed at the end of the `with` block. * Return fields type as {'type': ...} * Cache the table_model with the engine Generating the table_model by "reflecting" the database is expensive but the resulting model doesn't change. If we cache the table_model with the engine we save on a round-trip to the database and speed up response time. * Use separate stores for Engine and table_models Using a combined store based on both engine and table_model meant that a pygeoapi serving multiple tables on the same database would maintain a connection pool for each table. This could eventually saturate the number of available connections. This commit separates the store into two parts. * Update documentation with example * Manage or post items based on POST content type * ghcr build action * only ghcr job * run criteria edit * Update README.md * job = main * remove success requirememt * add tag * tags: bgs-cql * Update main.yml * Update README.md * remove binary * update * Update actions to latest versions * Dockerfile refactor * Dockerfile refactor * restore dockerhub * fix pygeoif version * # 3.6 test & dockerhub ref * Pin pygeoif version to less than 1 * Replace psycopg-binary with psycopg * Split Docker build into sections * Pin click version >7,<=8 * Add further cql examples * Update workflows to match GeoPython repo * update flake python version * Unquote PostgreSQL column names * Update container workflow uses versions * flake8 use python version 3.7 * Use pre ping to get db connection * Test that properties are set in provider * Amendments as requested by @tomkralidis * Correction * Fix reviewed changes in api code Co-authored-by: Colin Blackburn <colb@bgs.ac.uk> Co-authored-by: KoalaGeo <eddlewis85@gmail.com> Co-authored-by: Edd <edlew@bgs.ac.uk> Co-authored-by: Colin Blackburn <ximenesuk@users.noreply.github.com>
132 lines
4.5 KiB
YAML
132 lines
4.5 KiB
YAML
name: Build
|
|
|
|
on:
|
|
push:
|
|
paths-ignore:
|
|
- '**.md'
|
|
pull_request:
|
|
branches:
|
|
- master
|
|
paths-ignore:
|
|
- '!**.md'
|
|
release:
|
|
types:
|
|
- released
|
|
|
|
jobs:
|
|
flake8_py3:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Setup Python
|
|
uses: actions/setup-python@v1
|
|
with:
|
|
python-version: 3.7
|
|
architecture: x64
|
|
- name: Checkout pygeoapi
|
|
uses: actions/checkout@master
|
|
- name: Install flake8
|
|
run: pip install flake8
|
|
- name: Run flake8
|
|
uses: suo/flake8-github-action@releases/v1
|
|
with:
|
|
checkName: 'flake8_py3' # NOTE: this needs to be the same as the job name
|
|
env:
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
main:
|
|
needs: [flake8_py3]
|
|
runs-on: ubuntu-20.04
|
|
strategy:
|
|
matrix:
|
|
include:
|
|
- python-version: 3.6
|
|
- python-version: 3.7
|
|
env:
|
|
PYGEOAPI_CONFIG: "$(pwd)/pygeoapi-config.yml"
|
|
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
- uses: actions/setup-python@v2
|
|
name: Setup Python ${{ matrix.python-version }}
|
|
with:
|
|
python-version: ${{ matrix.python-version }}
|
|
- name: Configure sysctl limits
|
|
run: |
|
|
sudo swapoff -a
|
|
sudo sysctl -w vm.swappiness=1
|
|
sudo sysctl -w fs.file-max=262144
|
|
sudo sysctl -w vm.max_map_count=262144
|
|
- name: Install and run PostgreSQL/PostGIS 📦
|
|
uses: huaxk/postgis-action@v1
|
|
with:
|
|
postgresql password: ${{ secrets.DatabasePassword || 'postgres' }}
|
|
postgresql db: 'test'
|
|
- name: Install and run Elasticsearch 📦
|
|
uses: getong/elasticsearch-action@v1.2
|
|
with:
|
|
elasticsearch version: '7.6.1'
|
|
host port: 9200
|
|
container port: 9200
|
|
host node port: 9300
|
|
node port: 9300
|
|
discovery type: 'single-node'
|
|
- name: Install and run MongoDB
|
|
uses: supercharge/mongodb-github-action@1.5.0
|
|
with:
|
|
mongodb-version: 4.4
|
|
- name: Install and run SensorThingsAPI
|
|
run: |
|
|
wget https://raw.githubusercontent.com/webb-ben/data/main/docker-compose.yaml
|
|
docker-compose up -d
|
|
- name: Install requirements 📦
|
|
run: |
|
|
pip3 install -r requirements.txt
|
|
pip3 install -r requirements-starlette.txt
|
|
pip3 install -r requirements-dev.txt
|
|
pip3 install -r requirements-provider.txt
|
|
python3 setup.py install
|
|
pip3 install --upgrade numpy
|
|
#pip3 install --upgrade rasterio==1.1.8
|
|
- name: setup test data ⚙️
|
|
run: |
|
|
python3 tests/load_es_data.py tests/data/ne_110m_populated_places_simple.geojson geonameid
|
|
python3 tests/load_es_data.py tests/cite/ogcapi-features/canada-hydat-daily-mean-02HC003.geojson IDENTIFIER
|
|
python3 tests/load_mongo_data.py tests/data/ne_110m_populated_places_simple.geojson
|
|
gunzip < tests/data/hotosm_bdi_waterways.sql.gz | psql postgresql://postgres:${{ secrets.DatabasePassword || 'postgres' }}@localhost:5432/test
|
|
python3 tests/load_sta_data.py tests/data/sta_iow_datastreams.json
|
|
- name: run unit tests ⚙️
|
|
env:
|
|
POSTGRESQL_PASSWORD: ${{ secrets.DatabasePassword || 'postgres' }}
|
|
run: |
|
|
pytest tests/test_api.py
|
|
pytest tests/test_config.py
|
|
pytest tests/test_csv__formatter.py
|
|
pytest tests/test_csv__provider.py
|
|
pytest tests/test_elasticsearch__provider.py
|
|
pytest tests/test_esri_provider.py
|
|
pytest tests/test_filesystem_provider.py
|
|
pytest tests/test_geojson_provider.py
|
|
pytest tests/test_mongo_provider.py
|
|
#pytest tests/test_ogr_csv_provider.py
|
|
#pytest tests/test_ogr_esrijson_provider.py
|
|
#pytest tests/test_ogr_gpkg_provider.py
|
|
#pytest tests/test_ogr_shapefile_provider.py
|
|
#pytest tests/test_ogr_sqlite_provider.py
|
|
#pytest tests/test_ogr_wfs_provider.py
|
|
pytest tests/test_openapi.py
|
|
pytest tests/test_postgresql_provider.py
|
|
pytest tests/test_rasterio_provider.py
|
|
pytest tests/test_sensorthings_provider.py
|
|
pytest tests/test_socrata_provider.py
|
|
#pytest tests/test_sqlite_geopackage_provider.py
|
|
pytest tests/test_tinydb_catalogue_provider.py
|
|
pytest tests/test_util.py
|
|
pytest tests/test_xarray_netcdf_provider.py
|
|
pytest tests/test_xarray_zarr_provider.py
|
|
- name: build docs 🏗️
|
|
run: cd docs && make html
|
|
- name: failed tests 🚩
|
|
if: ${{ failure() }}
|
|
run: |
|
|
pip3 list -v
|