huanld/pygeoapi
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

#114 refine Docker assets and the simple and ElasticSearch examples

Browse Source
This commit is contained in:
Just van den Broecke
2019-05-23 15:34:56 +02:00
parent 3f700ab500
commit c31e173eff
22 changed files with 208 additions and 279 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Dockerfile
+1 -1
View File
@@ -30,7 +30,7 @@ ARG ADD_PIP_PACKAGES=""
ENV TZ=${TIMEZONE} \
DEBIAN_FRONTEND="noninteractive" \
DEB_BUILD_DEPS="tzdata build-essential python3-setuptools python3-pip apt-utils git" \
DEB_PACKAGES="locales libgdal20 python3-gdal libsqlite3-mod-spatialite ${ADD_DEB_PACKAGES}" \
DEB_PACKAGES="locales libgdal20 python3-gdal libsqlite3-mod-spatialite curl ${ADD_DEB_PACKAGES}" \
PIP_PACKAGES="gunicorn==19.9.0 gevent==1.4.0 wheel==0.33.4 ${ADD_PIP_PACKAGES}"
ADD . /pygeoapi
README.md
+16 -56
View File
@@ -58,68 +58,28 @@ docker run -p 80:8080 swaggerapi/swagger-ui
## Docker
Docker folder contains 2 sub-folders:
Best/easiest way to run `pygeoapi` is to use Docker.
On DockerHub [pygeoapi Docker Images](https://cloud.docker.com/u/geopython/repository/docker/geopython/pygeoapi)
are available.
- Simple
- Compose
The version tagged `latest` is automatically built whenever code
in the `master` branch of this GitHub repo changes (autobuild).
This also cascades to updating the [pygeoapi demo service](https://demo.pygeoapi.io/pygeoapi).
First folder will create a simple docker image with only GeoJSON, CSV as SQLite providers. While the second folder contains a full docker composition to run pygeoapi with ES.
Please read the [docker/README](https://github.com/geopython/pygeoapi/blob/master/docker/README.md) for
details of the Docker implementation. To get started quickly
[several examples](https://github.com/geopython/pygeoapi/blob/master/docker/examples) will get you up and running.
For simple testing and demonstration is more convenient to use the simple image.
### Unit Testing
Docker images have the following settings:
- Alpine edge OS
- spatialite compilation 4.3.0a
- pygeoapi running on port **5000**
### Simple (image)
Simple sub folder contains a simple implementation of pygeoapi with out ES (only: GeoJSON, CSV and SQLite provider).
```
cd docker/simple
docker build -t pygeoapi:latest .
docker run -p5000:5000 -v /pygeoapi/tests/data pygeoapi:latest
```
## Docker (composition)
Docker folder contains a docker-composition necessary to build a minimal pygeoapi using the complete set of providers providers (ES needs to be run as a separated service). Composition is only for development and testing in local environment:
### ES
- official elasticsearch: **5.6.8** on **CentosOS 7**
- ports **9300** and **9200**
Elasticsearch requires the host system to have its virtual memory parameter (**max_map_count**) [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html)
```
sudo sysctl -w vm.max_map_count=262144
```
If the docker composition fails with the following error:
```
docker_elastic_search_1 exited with code 78
```
it is very likely that you forgot to setup the sysctl
### Building and Running composition:
To build and run the composition in localhost
```
cd docker/compose
sudo sysctl -w vm.max_map_count=262144
docker-compose build
docker-compose up
```
### Testing code
Unit tests are run using pytest on the top project folder:
Unit tests are run using `pytest` from the top project folder:
```
pytest tests
```
Environment variables are set on file `pytest.ini`
NB beware that some tests require Provider dependencies (libraries) to be available
and that the ElasticSearch and Postgres tests require their respective
backend servers running.
Environment variables are set in the file [pytest.ini](pytest.ini).
docker/README.md
+43 -14
View File
@@ -1,12 +1,15 @@
# pygeoapi Docker HOWTO
Docker Images `geopython/pygeoapi:latest` and versions are available from DockerHub.
Docker Images `geopython/pygeoapi:latest` and versions are
[available from DockerHub](https://cloud.docker.com/u/geopython/repository/docker/geopython/pygeoapi).
Each Docker Image contains a default configuration [default.config.yml](default.config.yml)
with the project's test data and WFS3 datasets.
You can override this default config via Docker Volume mapping.
See an [example for the geoapi demo server](https://github.com/geopython/demo.pygeoapi.io/tree/master/services/pygeoapi).
You can override this default config via Docker Volume mapping or by extending the Docker Image
and copying in your config.
See an [example for the geoapi demo server](https://github.com/geopython/demo.pygeoapi.io/tree/master/services/pygeoapi)
for the latter method.
Depending on your config you may need specific backends to be available.
@@ -50,25 +53,29 @@ version: "3"
services:
pygeoapi:
image: geopython/pygeoapi
env_file:
- pygeoapi.env
image: geopython/pygeoapi:latest
volumes:
- ./my.config.yml:/pygeoapi/local.config.yml
```
See how the demo server is setup at
Or you can create a `Dockerfile` extending the base Image and `COPY` in your config:
```
FROM geopython/pygeoapi:latest
COPY ./my.config.yml /pygeoapi/local.config.yml
```
See how the demo server is setup this way at
https://github.com/geopython/demo.pygeoapi.io/tree/master/services/pygeoapi
This also shows the `pygeoapi.env` file mainly needed when running from a sub-domain.
## Running - Running on your domain
## Running - Running on a sub-path
By default the `pygeoapi` Docker Image will run from the `root` domain `/`.
If you need to run from a sub-domain and have all internal URLs correct
By default the `pygeoapi` Docker Image will run from the `root` path `/`.
If you need to run from a sub-path and have all internal URLs correct
you need to set `SCRIPT_NAME` environment variable.
For example to run with `my.config.yml` on http://localhost:5000/mypygeoapi:
@@ -78,7 +85,29 @@ For example to run with `my.config.yml` on http://localhost:5000/mypygeoapi:
# browse to http://localhost:5000/mypygeoapi
```
See https://github.com/geopython/demo.pygeoapi.io/tree/master/services/pygeoapi
Or within a `docker-compose.yml` full example:
```
version: "3"
services:
pygeoapi:
image: geopython/pygeoapi:latest
volumes:
- ./my.config.yml:/pygeoapi/local.config.yml
ports:
- "5000:80"
environment:
- SCRIPT_NAME=/pygeoapi
```
See [pygeoapi demo service](https://github.com/geopython/demo.pygeoapi.io/tree/master/services/pygeoapi)
for an full example.
docker/compose/docker-compose.yml
-28
View File
@@ -1,28 +0,0 @@
version: '3.3'
services:
pygeoapi:
build: ./pygeoapi
ports:
- "5000:5000"
volumes:
- pygeoapi_data:/pygeoapi/tests/data
links:
- elastic_search
depends_on:
- elastic_search
elastic_search:
build: ./ES
ports:
- 9300:9300
- 9200:9200
volumes:
- elastic_search_data:/usr/share/elasticsearch/data
volumes:
pygeoapi_data: {}
elastic_search_data: {}
#NOTE: Host requires changes in configuration to run ES
#sudo sysctl -w vm.max_map_count=262144
docker/compose/pygeoapi/Dockerfile
-58
View File
@@ -1,58 +0,0 @@
FROM alpine:edge
LABEL maintainer="jorge.dejesus@geocat.net"
ENV GITHUB https://github.com/geopython/pygeoapi.git
ENV PYGEOAPI_CONFIG /pygeoapi/local.config.yml
ENV PYGEOAPI_OPENAPI /pygeoapi/openapi.yml
ARG FREEXL=freexl-1.0.5.tar.gz
ARG LIBSPATIALITE=libspatialite-4.3.0a.tar.gz
RUN echo "@edge-testing http://nl.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories
RUN apk update && apk add --no-cache \
git \
python3-dev \
libffi \
libffi-dev \
musl-dev \
gcc \
openssl-dev \
g++ \
make \
curl \
sqlite-dev \
libxml2-dev \
"proj4-dev@edge-testing" \
"geos-dev@edge-testing"
RUN wget "http://www.gaia-gis.it/gaia-sins/${FREEXL}" && \
tar zxvf "${FREEXL}" && cd "${FREEXL%.*.*}" $ && \
./configure && make -j 4 && make install
RUN ldconfig /usr/local/lib
RUN wget "http://www.gaia-gis.it/gaia-sins/${LIBSPATIALITE}" && \
tar zxvf "${LIBSPATIALITE}" && cd "${LIBSPATIALITE%.*.*}" && \
./configure && make -j 4 && make install
RUN ldconfig /usr/local/lib
#RUN git clone -b docker ${GITHUB}
RUN git clone $GITHUB
COPY wait-for-elasticsearch.sh /wait-for-elasticsearch.sh
RUN chmod +x /wait-for-elasticsearch.sh
COPY docker-entrypoint.sh /docker-entrypoint.sh
COPY run_pygeoapi.sh /run_pygeoapi.sh
WORKDIR /pygeoapi
RUN pip3 install -r requirements.txt
RUN pip3 install -r requirements-dev.txt
RUN pip3 install -e .
COPY docker.config.yml local.config.yml
ENTRYPOINT ["/docker-entrypoint.sh"]
#Run using
#docker run -p5000:5000 pygeoapi:latest
docker/compose/pygeoapi/docker-entrypoint.sh
-14
View File
@@ -1,14 +0,0 @@
#!/bin/sh
set +e
echo "Waiting for the ES to be up to generate openapi yml"
/wait-for-elasticsearch.sh http://elastic_search:9200 -- /run_pygeoapi.sh
if [ $? -eq 1 ]; then
echo $?
"Wait for elastic search returned an error, carry on with out ES"
/run_pygeoapi.sh
fi
exec $@
docker/compose/pygeoapi/run_pygeoapi.sh
-17
View File
@@ -1,17 +0,0 @@
#!/bin/sh
set +e
echo "Trying to generate openapi.yml"
cd /pygeoapi
pygeoapi generate-openapi-document -c local.config.yml > openapi.yml
if [ $? -ne 0 ] ; then
echo "openapi.yml couldnt be generate ERROR, but carry on"
else
echo "openapi.yml generated continue to pygeoapi"
fi
pygeoapi serve
docker/examples/README.md
+9
View File
@@ -0,0 +1,9 @@
# Running pygeoapi with Docker - Examples
This folder contains the sub-folders:
- simple
- elastic
The [simple](simple) example will run pygeoapi with Docker with your local config.
The [elastic](elastic) example demonstrates a docker compose configuration to run pygeoapi with local ElasticSearch backend.
docker/compose/ES/Dockerfile → docker/examples/elastic/ES/Dockerfile
+10 -9
View File
@@ -1,28 +1,29 @@
FROM docker.elastic.co/elasticsearch/elasticsearch:5.6.8
LABEL maintainer="jorge.dejesus@geocat.net"
LABEL maintainer="jorge.dejesus@geocat.net justb4@gmail.com"
ARG DATA_FOLDER=/usr/share/elasticsearch/data
USER root
COPY docker-entrypoint.sh /docker-entrypoint.sh
COPY docker-entrypoint.sh /docker-entrypoint.sh
COPY add_data.sh /add_data.sh
RUN wget https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -O bin/wait-for-it.sh
RUN wget https://raw.githubusercontent.com/geopython/pygeoapi/master/tests/data/ne_110m_populated_places_simple.geojson -O /ne_110m_populated_places_simple.geojson
RUN chmod +x bin/wait-for-it.sh
RUN chmod +x bin/wait-for-it.sh
RUN wget https://raw.githubusercontent.com/geopython/pygeoapi/master/tests/data/ne_110m_populated_places_simple.geojson -O ${DATA_FOLDER}/ne_110m_populated_places_simple.geojson
RUN wget https://raw.githubusercontent.com/geopython/pygeoapi/master/tests/load_es_data.py -O /load_es_data.py
RUN echo "xpack.security.enabled: false" >> config/elasticsearch.yml
RUN echo "http.host: 0.0.0.0" >> config/elasticsearch.yml
RUN yum -y install python-setuptools \
&& easy_install elasticsearch \
RUN yum --enablerepo=extras -y install epel-release \
&& yum install -y python-pip \
&& yum install -y python-setuptools \
&& pip install --upgrade pip && pip install elasticsearch \
&& yum clean packages
RUN wget https://raw.githubusercontent.com/geopython/pygeoapi/master/tests/load_es_data.py -O /load_es_data.py
USER elasticsearch
CMD ["/usr/share/elasticsearch/bin/elasticsearch"]
ENTRYPOINT ["/docker-entrypoint.sh"]
docker/compose/ES/add_data.sh → docker/examples/elastic/ES/add_data.sh
View File
docker/compose/ES/docker-entrypoint.sh → docker/examples/elastic/ES/docker-entrypoint.sh
View File
docker/examples/elastic/README.md
+36
View File
@@ -0,0 +1,36 @@
# pygeoapi with ElasticSearch (ES)
These folders contain a Docker Compose configuration necessary to setup a minimal
`pygeoapi` server that uses a local ES backend service.
This config is only for local development and testing.
## ElasticSearch
- official ElasticSearch: **5.6.8** on **CentosOS 7**
- ports **9300** and **9200**
ES requires the host system to have its virtual memory
parameter (**max_map_count**) [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html)
set as follows:
```
sudo sysctl -w vm.max_map_count=262144
```
If the docker composition fails with the following error:
```
docker_elastic_search_1 exited with code 78
```
it is very likely that you forgot to setup the `sysctl`.
## Building and Running
To build and run the [Docker compose file](docker-compose.yml) in localhost:
```
sudo sysctl -w vm.max_map_count=262144
docker-compose build
docker-compose up
```
docker/examples/elastic/docker-compose.yml
+41
View File
@@ -0,0 +1,41 @@
version: '3.3'
services:
pygeoapi:
image: geopython/pygeoapi:latest
container_name: pygeoapi_es
entrypoint:
- /es-entrypoint.sh
ports:
- 5000:80
volumes:
- ./pygeoapi/docker.config.yml:/pygeoapi/local.config.yml
- ./pygeoapi/es-entrypoint.sh:/es-entrypoint.sh
- ./pygeoapi/wait-for-elasticsearch.sh:/wait-for-elasticsearch.sh
links:
- elastic_search
depends_on:
- elastic_search
elastic_search:
build: ./ES
container_name: elastic
ports:
- 9300:9300
- 9200:9200
volumes:
- elastic_search_data:/usr/share/elasticsearch/data
volumes:
elastic_search_data: {}
#NOTE: Host requires changes in configuration to run ES
#sudo sysctl -w vm.max_map_count=262144
docker/compose/pygeoapi/docker.config.yml → docker/examples/elastic/pygeoapi/docker.config.yml
+5
View File
@@ -160,3 +160,8 @@ datasets:
data: tests/data/ne_110m_admin_0_countries.sqlite
id_field: ogc_fid
table: ne_110m_admin_0_countries
processes:
hello-world:
processor:
name: HelloWorld
docker/examples/elastic/pygeoapi/es-entrypoint.sh
Executable
+7
View File
@@ -0,0 +1,7 @@
#!/bin/sh
set +e
echo "Waiting for ElasticSearch container..."
# First wait for ES to be up and then execute the original pygeoapi entrypoint.
/wait-for-elasticsearch.sh http://elastic_search:9200 /entrypoint.sh || echo "ES failed: $?, exit" && exit 1
docker/compose/pygeoapi/wait-for-elasticsearch.sh → docker/examples/elastic/pygeoapi/wait-for-elasticsearch.sh
View File
docker/examples/simple/README.md
+18
View File
@@ -0,0 +1,18 @@
# Simple Example
This is the simplest example to run `pygeoapi` with a [local config](my.config.yml)
using Docker.
## Using Docker directly
Execute [./run_pygeoapi.sh](run_pygeoapi.sh). This will pull the `pygeoapi` Image from
DockerHub and start the `pygeoapi` server. With your browser got to http://localhost:5000.
## Using Docker Compose
Run the [docker-compose.yml](docker-compose.yml) as follows:
```
docker-compose up [-d]
```
docker/examples/simple/docker-compose.yml
+9
View File
@@ -0,0 +1,9 @@
version: "3"
services:
pygeoapi:
image: geopython/pygeoapi:latest
volumes:
- ./my.config.yml:/pygeoapi/local.config.yml
docker/simple/docker.config.yml → docker/examples/simple/my.config.yml
+7 -2
View File
@@ -1,8 +1,8 @@
server:
bind:
host: 0.0.0.0
port: 5000
url: http://localhost:5000/
port: 80
url: http://localhost:5000
mimetype: application/json; charset=UTF-8
encoding: utf-8
language: en-US
@@ -131,3 +131,8 @@ datasets:
data: tests/data/ne_110m_admin_0_countries.sqlite
id_field: ogc_fid
table: ne_110m_admin_0_countries
processes:
hello-world:
processor:
name: HelloWorld
docker/examples/simple/run_pygeoapi.sh
Executable
+6
View File
@@ -0,0 +1,6 @@
#!/bin/sh
echo "Running latest Docker version of pygeoapi with local config"
echo "Go with your browser to http://localhost:5000 to view"
docker run -p 5000:80 -v $(pwd)/my.config.yml:/pygeoapi/local.config.yml -it geopython/pygeoapi:latest
docker/simple/Dockerfile
-63
View File
@@ -1,63 +0,0 @@
FROM alpine:3.8
LABEL maintainer="jorge.dejesus@geocat.net"
ENV GITHUB https://github.com/geopython/pygeoapi.git
ENV PYGEOAPI_CONFIG /pygeoapi/local.config.yml
ENV PYGEOAPI_OPENAPI /pygeoapi/openapi.yml
ARG FREEXL=freexl-1.0.5.tar.gz
ARG LIBSPATIALITE=libspatialite-4.3.0a.tar.gz
RUN echo "@edge-testing http://nl.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories
RUN apk update && apk add --no-cache \
git \
python3-dev \
libffi \
libffi-dev \
musl-dev \
gcc \
openssl-dev \
g++ \
make \
curl \
sqlite-dev \
libxml2-dev \
"proj4-dev@edge-testing" \
"geos-dev@edge-testing"
RUN wget "http://www.gaia-gis.it/gaia-sins/${FREEXL}" && \
tar zxvf "${FREEXL}" && cd "${FREEXL%.*.*}" $ && \
./configure && make -j 4 && make install
RUN ldconfig /usr/local/lib
RUN wget "http://www.gaia-gis.it/gaia-sins/${LIBSPATIALITE}" && \
tar zxvf "${LIBSPATIALITE}" && cd "${LIBSPATIALITE%.*.*}" && \
./configure && make -j 4 && make install
RUN ldconfig /usr/local/lib
#RUN git clone -b docker ${GITHUB}
RUN git clone $GITHUB
#remove elatic search
RUN rm /pygeoapi/pygeoapi/provider/elasticsearch_.py
RUN sed -i '/pygeoapi.provider.elasticsearch/d' /pygeoapi/pygeoapi/provider/__init__.py
WORKDIR /pygeoapi
RUN pip3 install -r requirements.txt
RUN pip3 install -r requirements-dev.txt
RUN pip3 install -e .
COPY docker.config.yml local.config.yml
COPY run_pygeoapi.sh /run_pygeoapi.sh
ENTRYPOINT ["/run_pygeoapi.sh"]
#Run using:
#docker run -p5000:5000 pygeoapi:latest
#Run using unamed volume:
#docker run -p5000:5000 -v /pygeoapi/tests/data pygeoapi:latest
#Run using named volume:
#docker volume create pygeoapi_volume
#docker run -p5000:5000 -v pygeoapi_volume:/pygeoapi/tests/data pygeoapi:latest
docker/simple/run_pygeoapi.sh
-17
View File
@@ -1,17 +0,0 @@
#!/bin/sh
set +e
echo "Trying to generate openapi.yml"
cd /pygeoapi
pygeoapi generate-openapi-document -c local.config.yml > openapi.yml
if [ $? -ne 0 ] ; then
echo "openapi.yml couldnt be generate ERROR, but carry on"
else
echo "openapi.yml generated continue to pygeoapi"
fi
pygeoapi serve