* feat(backgroundjobs): add new background jobs module for file imports
queueing
* fix(fileuploads): a merge gone wrong
* feat(backgroundjobs): rename rhino queue env var
* test(backgroundjob): use deep equal claude
* fix(fileuploads): sync PR review
* feat(ifc_importer): initial importer app implementation with a sleeping
worker
* chore(pre-commit): remove black as a formatter, its now handled by ruff
* fix(ifc-importer): better handling of max job attempt
* feat(eslint): ignore package from eslint
* fix(helm test): pass in env vars at runtime not build time
- we configure the test at runtime, not buildtime
- we were never setting these build args at build time, so they were obsolete
* remove unnecessary build arg from server Dockerfile
* chore(Dockerfile): pin images by digest this ensures consistent builds with images pinned by SHA
* introduces a GitHub Action will create a PR to update the images when there is a new version, and will pin to that newer version SHA.
* Bump debian to 12 'bookworm'
* Bump pypi libraries
* except frontend-1, as only debian 11 is currently supported by openresty
* Bump debian packages
* Improves error logging
- use pino error logger correctly by passing in error as first argument
* monitor deployment: Filter logging at INFO level and above
* Use structured logging to create parameters for monitoring results
* Add structured logging to obj fileimport service
* Fileimport service, fix and improve logging
- use child logger with additional context where possible
- select appropriate logging level
- fix duplicated context in log statement
* REST endpoints, add context to structured logging and remove same context from message
* Webhook service provides context to bound logger to properly use structured logging
- Pass bound logger containing context to `makeNetworkRequest`
- do not log url, as it may contain a secret (like Discord's webhook urls), instead log the webhook Id
- log error message when network call fails
* upload: make better use of structured logging when recording data
* pino-pretty when in dev or test mode
- pino-pretty configured to send to stderr
* LOG_PRETTY env var
* Silence structured logging during testing
- can not rely on determining the port number by reading from stdout/stderr
- instead we determine which port is free, then create our server on that port
- we then poll that port until the server is ready before commencing tests
* Allow puppeteer to install chromium
* Do not need to install chromium separately
* Moves speckle-server, webhook-service, fileimport-service, monitoring-deployment, and test-deployment images to Distroless.
Partially addresses https://github.com/specklesystems/speckle-server/issues/883
* preview-service uses similar image for building and production stages
* explicitly include chromium-common dependency to prevent error in preview service
* Bump chromium packages due to package versions not being found
* Handle machine-id in distroless
- distroless has no shell, so node-machine-id will result in an error
- this commit introduces error handling and defaults to a uuid v4 in the case of an error
* Update binary location for readiness and liveness checks to match the binary location in Distroless
* Allow node binary path to be set as environment variable in fileimport service
* feat(helm chart): deployes Cilium Network Policies when configured
Cilium Network Policies provide more features over regular Kubernetes Network Policies, but Cilium
is not available everywhere. When selected by an operator, Cilium Network Policies will be deployed
instead of Kubernetes Network Policies.
Fixes https://github.com/specklesystems/speckle-server/issues/913
* Cilium Network Policy for fileimport service.
* tested only for external host.
* Still to test internal pod and external IP.
* Cilium network policy for file import service restricts DNS
* allows egress to service instead of endpoint
* file import service uses service url of speckle-server
* helper functions for server and dns
* DRY the prometheus selector
* CiliumNetworkPolicy for frontend
* CiliumNetworkPolicy for monitoring service
* CiliumNetworkPolicy for preview service
* CiliumNetworkPolicy for test
* CiliumNetworkPolicy for webhook_service
* CiliumNetworkPolicy for Server
* Test should egress to domain, not internally
* Test should be in tests directory to match Helm convention for tests
* Test should explicitly deny ingress from everywhere
* Server needs to egress to canonical domain (i.e. itself)
- DNS and egress for canonical domain added to Server
- As Test also egresses via canonical domain to access Server, we do not require the intra-cluster ingress to the server from the test pod
- Explicitly deny all egress from frontend
* WIP update to schema.json
* Breaking Change: inCluster network policies supported for cilium
* Breaking change: kubernetes network policy podSelector and namespaceSelector are now at a different level
* Updates schema.json
* add notes to remove egress once bug is fixed
- perf(server, webhook-service): production images are based on distroless
Reduces image file size by >40% for images which can use distroless base image. As
well as improving boot-up & restart time (via smaller download & load size), Distroless reduces the
attack surface area by removing almost all binaries & packages (e.g. shell, chown) that are not
necessary to run node.
- ensures distroless node images run tini
- removes fonts-dejavu-core and fontconfig from speckle-server
- Remove man and doc files if they exist
- args hoisted to top of Dockerfile and consolidated
- env vars consolidated to prevent additional layers
address https://github.com/specklesystems/speckle-server/issues/883
* Adds hadolint as pre-commit step
* Addresses all hadolint comments
* Use noninteractive apt-get frontend and clean after install
* dockerfile RUN statements are consolidated to prevent additional layers
* installed packages have pinned versions
* build(circleci): use speckle pre-commit runner with built-in hadolint
* Integrate pre-commit with husky bash script for git pre-commit hooks
* catches errors in bash
* if pre-commit is installed, it is run
* if optional additional binaries are installed, further pre-commit steps are run
* Update README with revised developer instructions
* Adds a pre-commit yarn script
Gergő Jedlicska