Gergő Jedlicska
d817e6dd0a
* feat(server): add server authz pipeline rework first sketch * feat(server authz): add new server authz middleware poc implementation * test(server authz): add unittests for the new server authz workflow * feat(wip rework of fileuploads vs blob storage): add basim impl of separate blob storage service * feat(fileimport service): refactored file import service to utilize the new asssetstorage service * refactor(server errors): refactor server errors to use the shared module definitions Now all the errors inherit from BaseError * refactor(fileimport service): cleanup after refactor * feat(frontend fileimports): use the new blob storage for downloading the original file * refactor(server fileimports): clean up the remnants of S3 storage from file imports * refactor(server authz): centralize generic authz pipeline configs * refactor(server blob storage): refactor / rename everything to use the `blob-storage` name * ci(circleci): add s3 objectstorage environment variables * ci(circleci): fix missing env variables * ci(circleci): add minio test container * ci(circleci): fix minio app startup * ci(circleci): enable circleci remote docker * ci(circleci): fix minio startup * ci(cirleci): detach and wait properly for minio to start * ci(circleci): revert to additional minio img config, it only fails when the container is stopped ?! * ci(circleci): disable file uploads * fix(fileimports): update with blob storage refactor leftovers * feat(server blob storage): add blob storage graphql api * refactor(server errors): merge new errors to shared module * fix(server comments rte): fix import for RTE error * chore(fileimports): remove node-fetch from dependency * chore(server): remove body parser dependency * fix(server blob storage): fix gql api * fix(frontend): fix fileupload item not loading the new upload status, cause of premature event fire * feat(server blob storage): fix file size limit and allow for public streams * Update packages/server/modules/blobstorage/graph/schemas/blobstorage.graphql Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com> * chore(blobstorage): fix PR review issues * fix(server): fix import bugs * chore(docker): ignore python venv data for docker build context * feat(knex): update knex configuration with min max connections and application_name * feat(helm chart): configure postgres max connections for server in the helm chart Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
PreviewService
This component generates object previews for Speckle Objects.
It reads preview tasks from the DB and uses Puppeteer and an internal Viewer to generate previews, which are currently stored in the DB.
This is an overview of this service:
Run locally
With an updated viewer installed in the current directory, you should first build the frontend-part of the preview service: The simple webpage with the viewer that will be accessed with Puppeteer to generate the preview:
yarn build
This should be rerun whenever you make changes to the viewer (if you make local viewer changes, don't forget to build the viewer module before running this)
After the viewer web page is up to date, run the preview service with:
yarn dev
This will use the default dev DB connection of postgres://speckle:speckle@localhost/speckle. You can pass the environment variable PG_CONNECTION_STRING to change this to a different DB.
Deployment notes
When deploying the PreviewService, it's important to pay attention to the memory requirements: Speckle streams can be arbitrarily large and the preview service must load the entire stream in order to generate the preview image.
You must limit the PreviewService container memory to a value that you want to allocate for preview generation. If a stream requires more memory then the limit, the preview for that stream will fail, but the entire system remains stable.
To limit the container memory when running with docker run, you can use the -m flag.
With docker-compose, you must use a docker-compose file at version 2 (not 3) and use the mem_limit option.