* feat(preview-generator): add new preview generator webapp
* wip(preview-service): reworking the preview service backend
* feat(previews): logging
* feat(preview-service): streamline payloads
* fix(preview-service): do not log the full payload
* feat(preview-service): build new preview service
* feat(preview-service): add separate response queue
* feat(previews): integrate preview queues with the server
* feat(previews): use module alias
* chore(previews): remove old preview service code
* feat(previews): log stuff on job statuses
* fix(previews): add missing deps and scripts
* fix(previews): package deps fix
* fix(server): moar typing fixes
* Metrics related to jobs: total count, request failures, response errors & durations
* duration should include unit.
- histogram metric should be summary
- error responses include duration in seconds
- attempt to remove metric before adding it (prevent errors with duplicate metrics)
* fix(server, frontend): some ts fixes
* fixes
* fix(frontend): remove unneeded ts-expect-error
* chore(preview-service): eslint
* TS fix
* feat(previews): more smoal fixes
* fix(preview-service): alias loading
* feat(helm): updates for new preview service queue setup
* feat(preview-service): launch new browser for each job
* feat(preview-service): add timeout, fix liveliness
* fix(helm): add access to new secret in service accounts
* tidy metrics into a separate file
* Remove broken preview service acceptance test
* fix broken import
* Add metrics to test
* feat(preview-service): handle preview service shutdown properly
* fix(previews): merge bork
---------
Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
- should run on changes to files in directory
- Remove pnpm
- Allow postgres connection string to be configured for acceptance test
- Different postgres connection string if running inside preview container
- Run the preview-service image as a github action service
- separate the jobs into a build and a test job
- do not run the preview-service via the acceptance test, instead run it via github actions
- Add correct permission to job
- Add logging to the test, to understand progress
- Allow database name to be passed in to acceptance test
- Only delete the database if the test helper owns (created) it
- Upload image to s3 bucket
* Pinned the viewer version for the preview-service to the last viewer version before the API 2.0 merge as a temporary solution until the preview service is updated to use API 2.0
* updated lock
* Improves error logging
- use pino error logger correctly by passing in error as first argument
* monitor deployment: Filter logging at INFO level and above
* Use structured logging to create parameters for monitoring results
* Add structured logging to obj fileimport service
* Fileimport service, fix and improve logging
- use child logger with additional context where possible
- select appropriate logging level
- fix duplicated context in log statement
* REST endpoints, add context to structured logging and remove same context from message
* Webhook service provides context to bound logger to properly use structured logging
- Pass bound logger containing context to `makeNetworkRequest`
- do not log url, as it may contain a secret (like Discord's webhook urls), instead log the webhook Id
- log error message when network call fails
* upload: make better use of structured logging when recording data
* pino-pretty when in dev or test mode
- pino-pretty configured to send to stderr
* LOG_PRETTY env var
* Silence structured logging during testing
- can not rely on determining the port number by reading from stdout/stderr
- instead we determine which port is free, then create our server on that port
- we then poll that port until the server is ready before commencing tests
* Allow puppeteer to install chromium
* Do not need to install chromium separately
* chore(node): upgrades to node 18
Node 16 was out of support (but not security upgrades), so bumping to next stable version.
https://github.com/specklesystems/speckle-server/issues/1187
* Update server liveness and readiness probes for node 18
* Bump web-ifc to 0.0.36
* Apply `--no-experimental-fetch` flag to fileimport-service to prevent issues in web-ifc (via emscripten) with node 18
* Revert "Revert structured logging 2 (#1240)"
This reverts commit 78ecaeffcb.
* Logging should not be bundled into core shared directory
* making sure observability stuff isnt bundled into frontend
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
* Revert "'@' shortcut must come after it is configured in bootstrap (#1239)"
This reverts commit 967329473f.
* Revert "Structured logging (attempt 2) (#1234)"
This reverts commit 444d2ca7dd.
* Revert "Revert "feat(structured logging): implements structured logging for backend (#1217)" (#1227)"
This reverts commit 63e6581162.
* Use pino-http instead of express pino logger
* Use correct reference to knex and do not instantiate HttpLogger prematurely
* Adds missing dependency for pino to webhook-service
* Do not instantiate middleware when passed to express
* Refactor to move logging into shared
* Copy shared packages into dockerfiles
* Build shared workspace in docker build-stage for fileimport & webhook
* each log line is a json object
* structured logging allows logs to be ingested by machines and the logs to be indexed and queried addresses #1105
* structured logging allows arbitrary properties to be appended to each log line, and ingestion of logs to remain robust
* Structured logging provided by `pino` library
* Add `express-pino-logger` dependency
* Remove `debug`, `morgan`, and `morgan-debug` and replace with structured logging
* `console.log` & `console.error` replaced with structured logging in backend
* Remove `DEBUG` environment variable and replace with `LOG_LEVEL`
- Note that there is a test which reads from a logged line on `stdout`. This is not robust, it would be better to use the childProcess.pid to look up the port number.
* Log errors at points we explicitly send error to Sentry
* Amend indentation of a couple of log messages to align indentation with others
Daniel Gak Anagrov