* feat(fileimport-service): add next gen file importer
* feat(fileimports): integrate server and fileimporter
* chore(dui3): remove leftover artifacts
* fix(server): test typing fixes
* fix(fileimports): test and pr comment fixes
* feat(fileimports: moare test fixes
* fix(fileimports): tests and yarn dedupe
* feat(webhook-service): learn to speak multi region
* feat(fileimport-service): talk multi region to me
* feat(fileuploads, blobs): multi region
* feat(fileimport-service): multi region fixes
* feat(branchesAndCommits): multi region resolvers
* fix(fileimports): no need for ts ignore
* fix(fileimports): fix pr comments
* Improves error logging
- use pino error logger correctly by passing in error as first argument
* monitor deployment: Filter logging at INFO level and above
* Use structured logging to create parameters for monitoring results
* Add structured logging to obj fileimport service
* Fileimport service, fix and improve logging
- use child logger with additional context where possible
- select appropriate logging level
- fix duplicated context in log statement
* REST endpoints, add context to structured logging and remove same context from message
* Webhook service provides context to bound logger to properly use structured logging
- Pass bound logger containing context to `makeNetworkRequest`
- do not log url, as it may contain a secret (like Discord's webhook urls), instead log the webhook Id
- log error message when network call fails
* upload: make better use of structured logging when recording data
* pino-pretty when in dev or test mode
- pino-pretty configured to send to stderr
* LOG_PRETTY env var
* Silence structured logging during testing
- can not rely on determining the port number by reading from stdout/stderr
- instead we determine which port is free, then create our server on that port
- we then poll that port until the server is ready before commencing tests
* Allow puppeteer to install chromium
* Do not need to install chromium separately
* chore(node): upgrades to node 18
Node 16 was out of support (but not security upgrades), so bumping to next stable version.
https://github.com/specklesystems/speckle-server/issues/1187
* Update server liveness and readiness probes for node 18
* Bump web-ifc to 0.0.36
* Apply `--no-experimental-fetch` flag to fileimport-service to prevent issues in web-ifc (via emscripten) with node 18
* Revert "Revert structured logging 2 (#1240)"
This reverts commit 78ecaeffcb.
* Logging should not be bundled into core shared directory
* making sure observability stuff isnt bundled into frontend
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
* Revert "'@' shortcut must come after it is configured in bootstrap (#1239)"
This reverts commit 967329473f.
* Revert "Structured logging (attempt 2) (#1234)"
This reverts commit 444d2ca7dd.
* Revert "Revert "feat(structured logging): implements structured logging for backend (#1217)" (#1227)"
This reverts commit 63e6581162.
* Use pino-http instead of express pino logger
* Use correct reference to knex and do not instantiate HttpLogger prematurely
* Adds missing dependency for pino to webhook-service
* Do not instantiate middleware when passed to express
* Refactor to move logging into shared
* Copy shared packages into dockerfiles
* Build shared workspace in docker build-stage for fileimport & webhook
* each log line is a json object
* structured logging allows logs to be ingested by machines and the logs to be indexed and queried addresses #1105
* structured logging allows arbitrary properties to be appended to each log line, and ingestion of logs to remain robust
* Structured logging provided by `pino` library
* Add `express-pino-logger` dependency
* Remove `debug`, `morgan`, and `morgan-debug` and replace with structured logging
* `console.log` & `console.error` replaced with structured logging in backend
* Remove `DEBUG` environment variable and replace with `LOG_LEVEL`
- Note that there is a test which reads from a logged line on `stdout`. This is not robust, it would be better to use the childProcess.pid to look up the port number.
* Log errors at points we explicitly send error to Sentry
* Amend indentation of a couple of log messages to align indentation with others
Fileimport service retreives blobs via the server storage API, and not directly from s3. Fileimport
service no longer requires information or credentials about s3.
* feat(server): add server authz pipeline rework first sketch
* feat(server authz): add new server authz middleware poc implementation
* test(server authz): add unittests for the new server authz workflow
* feat(wip rework of fileuploads vs blob storage): add basim impl of separate blob storage service
* feat(fileimport service): refactored file import service to utilize the new asssetstorage service
* refactor(server errors): refactor server errors to use the shared module definitions
Now all the errors inherit from BaseError
* refactor(fileimport service): cleanup after refactor
* feat(frontend fileimports): use the new blob storage for downloading the original file
* refactor(server fileimports): clean up the remnants of S3 storage from file imports
* refactor(server authz): centralize generic authz pipeline configs
* refactor(server blob storage): refactor / rename everything to use the `blob-storage` name
* ci(circleci): add s3 objectstorage environment variables
* ci(circleci): fix missing env variables
* ci(circleci): add minio test container
* ci(circleci): fix minio app startup
* ci(circleci): enable circleci remote docker
* ci(circleci): fix minio startup
* ci(cirleci): detach and wait properly for minio to start
* ci(circleci): revert to additional minio img config, it only fails when the container is stopped ?!
* ci(circleci): disable file uploads
* fix(fileimports): update with blob storage refactor leftovers
* feat(server blob storage): add blob storage graphql api
* refactor(server errors): merge new errors to shared module
* fix(server comments rte): fix import for RTE error
* chore(fileimports): remove node-fetch from dependency
* chore(server): remove body parser dependency
* fix(server blob storage): fix gql api
* fix(frontend): fix fileupload item not loading the new upload status, cause of premature event fire
* feat(server blob storage): fix file size limit and allow for public streams
* Update packages/server/modules/blobstorage/graph/schemas/blobstorage.graphql
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
* chore(blobstorage): fix PR review issues
* fix(server): fix import bugs
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
* yarn first go
* fix frontend build cache loader
* yarn workspaces built server Docker
* build(yarn): add workspaces plugin config
* chore(package defs): clean package*.json -s
* chore(gitignore): ignore yarn error log
* build(yarn): update yarn lock
* build(preview-service webpack): add extra resolved path to preview service webpack config
because of yarn package hoisting, there are no package level node_modules folder anymore.
* build(docker): update dockerignore with yarn specific configs
* build(docker): update Dockerfiles for yarn workspaces utilization
* ci(circleci): update server test job to yarn
* ci(circle): disable cache restore
* ci(circleci): trying the node orb yarn-run
* ci(circleci): yarn-run again
* ci(circleci): disable node orb
* ci(circleci): change base node image for tests
* ci(circleci): add yarn cache
* ci(circleci): remove node install step
* ci(circleci): add server specific cache archives
* ci(circleci): test build and publish
* ci(circleci): change npm auth method to suit yarn
* ci(circleci): trying new builder image
* ci(circleci): another base image, maybe this works
* ci(circleci): force a specific docker engine version
* ci(circleci): add yarn version plugin and its changes
* ci(circleci): cleanup and remove temp branch config
* chore(package defs): moving from npm run to yarn
* explicitly specifying webpack4 as a frontend dep
* chore(package defs): replace npm with yarn everywhere
* docs(root readme): update with some yarn specific docs
* chore(root workspace): update dev scripts and package lock
* ci(circleci): enable package publish step with yarn
Co-authored-by: Fabians <fabis94@live.com>
Gergő Jedlicska