* feat(server): add speckle automate as a configurable default app
* feat(server): add default automate url, and helm values
* fix default app tests reporting the old number
* feat: register flag passed to fe
* feat: mixpanel tracking for all sign ups
* feat: utm first touch & last touch tracking
* feat(helm): Allows Environment Variable for MP to be configured
- default is enabled
- renames environment variable to ENABLE_MP
* feat(helm network policy): allowlist analytics.speckle.systems
---------
Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
- feat(logging): log all http requests and responses
- the auto logger does not log the body, to ensure sensitive payloads are not logged. Unfortunately this means that error messages are not logged either, so need to be manually logged.
- fix(logging): 400 errors should be info not error severity logging
- chore(server): adds endpoint parameter to some log messages
- and any other relevant parameters we have to hand (streamId, userId etc..)
- logging severity is info for `400` status codes, error for `500` status codes or equivalent.
- errors should be logged to stdout in a structured format
- currently still throws errors which are ultimately unhandled, this matches existing behaviour
- Consolidate redis creation and error handling in a shared module
* remove unused 'redis' module, in favour of 'ioredis'
* fix(logging): user login attempt without invite to invite-only server is info severity, not error severity
* fix(logging): a prompt for user to verify should be logged as information, not error severity
* fix(logging): user input errors should be logged as info severity
* Tidy the code to use a constant and a specific error type
* Attempting to remove the last admin user is invalid user input and should not be logged as error severity
* Missing passwords are user input errors
- logs are now warn severity, as they are not necessarily system errors and may be user errors or incorrectly configured application errors.
- improved the error messages, as we should not have multiple errors with the same message as this makes debugging difficult.
* feat(server default apps): add powerbi as a default app on the servers
* feat(server powerbi app): remove stream write permission from the powerbi app
* fix(defaultApps): make sure id and secret are the proper length
* feat(defaultApps): throw if something goes wrong during init
* tests(server default apps): update the default app count on the server
* feat(server default apps): add powerbi as a default app on the servers
* feat(server powerbi app): remove stream write permission from the powerbi app
* Revert "Revert structured logging 2 (#1240)"
This reverts commit 78ecaeffcb.
* Logging should not be bundled into core shared directory
* making sure observability stuff isnt bundled into frontend
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
* Revert "'@' shortcut must come after it is configured in bootstrap (#1239)"
This reverts commit 967329473f.
* Revert "Structured logging (attempt 2) (#1234)"
This reverts commit 444d2ca7dd.
* Revert "Revert "feat(structured logging): implements structured logging for backend (#1217)" (#1227)"
This reverts commit 63e6581162.
* Use pino-http instead of express pino logger
* Use correct reference to knex and do not instantiate HttpLogger prematurely
* Adds missing dependency for pino to webhook-service
* Do not instantiate middleware when passed to express
* Refactor to move logging into shared
* Copy shared packages into dockerfiles
* Build shared workspace in docker build-stage for fileimport & webhook
* Demonstration of bug to test when middleware added
- Adding middleware, even no-op, causes test to fail
* Make middleware async, but introduce delay.
Revert test back to original.
* Revert tests
* Add a 1ms sleep to the test to reduce likelihood of flakiness
* Rate limiting on all express endpoints using middleware
* Adds all configuration for existing rate limited endpoints
* It is helpful to add the package to yarn first
* Implements respectsLimits using Redis rate limiter
* Fix for test `Should rate-limit user creation`
- if rate limit error, post to `/auth/local/register` will return a 429 status code
* All rate limiting provided by new ratelimiter.ts
* Consolidate typescript interfaces
* Amend signature of function to require source to be passed in, and not try to guess it from the request
* Rename respectsLimits to isWithinRateLimits
* Throw within catch of Promise
* Replace rejectsRequestWithRatelimitStatusIfNeeded throughout code
* Sending rate limit response should deal with other types of error
- Sentry notified of the error
* Express middleware rate limits by a 3 second burst or a daily rate
- Provide action when generating 429 response
* Prevent DOS of Redis
* Add 'Retry-After' for all cases when responding with 429 status code
- default of 1 day, but dynamic based on available information
* Generate rate limiters once, on init
- Improved and consistent handling of exit from functions
- fixed environment variable names
* WIP Refactor rate limiting setup
Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com>
* WIP: fixed references, now runs but tests fail
* Use getSourceFromRequest where possible
* WIP: unit tests for rate limiter
* Unit tests for ratelimiter
* feat(IFC): WIP IFC parser improvements
* Revert "feat(IFC): WIP IFC parser improvements"
This reverts commit 093089a2c4.
* refactor authz, rate limiting middleware to global
Co-authored-by: Kristaps Fabians Geikins <fabis94@users.noreply.github.com>
Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com>
* invites tests fix
* fix(server ratelimiter): export public interfaces
* Unit test for rate limiter use in memory rate limiter
- in memory rate limiter is configured with zero limit by default
* Fixed#1219 (#1221)
* WIP: improve auth test for rate limiting user creation
* ci(circleci config): publishing was broken when main branch was tagged (i.e. for releases) (#1224)
* Gitignore CPU profiles
* All tests are now passing locally
* Fixed an issue in the frontend which was causing the views not to work. Fixed an issue with object selection camera animation where the dolly lerp factor was much too high for smooth animation (#1225)
* feat(structured logging): implements structured logging for backend (#1217)
* each log line is a json object
* structured logging allows logs to be ingested by machines and the logs to be indexed and queried addresses #1105
* structured logging allows arbitrary properties to be appended to each log line, and ingestion of logs to remain robust
* Structured logging provided by `pino` library
* Add `express-pino-logger` dependency
* Remove `debug`, `morgan`, and `morgan-debug` and replace with structured logging
* `console.log` & `console.error` replaced with structured logging in backend
* Remove `DEBUG` environment variable and replace with `LOG_LEVEL`
- Note that there is a test which reads from a logged line on `stdout`. This is not robust, it would be better to use the childProcess.pid to look up the port number.
* Log errors at points we explicitly send error to Sentry
* Amend indentation of a couple of log messages to align indentation with others
* Revert "feat(structured logging): implements structured logging for backend (#1217)" (#1227)
This reverts commit 84cb74e8b3.
* Move error to core/errors
- augmented typescript types moved to type-augmentations
* Added a missing wait in the screenshot generation loop (#1228)
* refactor(server rest api): remove duplicate rate limit requests
* feat(server rate limits): increase rate limits for the upload endpoints
* chore(server rate limits): final cleanup
Co-authored-by: Gergő Jedlicska <gergo@jedlicska.com>
Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com>
Co-authored-by: Dimitrie Stefanescu <didimitrie@gmail.com>
Co-authored-by: Kristaps Fabians Geikins <fabis94@users.noreply.github.com>
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
Co-authored-by: Alexandru Popovici <alexandrupopoviciioan@gmail.com>
* each log line is a json object
* structured logging allows logs to be ingested by machines and the logs to be indexed and queried addresses #1105
* structured logging allows arbitrary properties to be appended to each log line, and ingestion of logs to remain robust
* Structured logging provided by `pino` library
* Add `express-pino-logger` dependency
* Remove `debug`, `morgan`, and `morgan-debug` and replace with structured logging
* `console.log` & `console.error` replaced with structured logging in backend
* Remove `DEBUG` environment variable and replace with `LOG_LEVEL`
- Note that there is a test which reads from a logged line on `stdout`. This is not robust, it would be better to use the childProcess.pid to look up the port number.
* Log errors at points we explicitly send error to Sentry
* Amend indentation of a couple of log messages to align indentation with others
* fix: some extra param validation for some API endpoints
* fix(server): potentially leaking internal error details
* fix: secure session cookie for ssl servers
* fix(server): fixing tests
* Update to new specklepy (#1173)
* Publish images for all branches but limit tagging
* only tag 'latest' and '2' when 'SHOULD_PUBLISH' variable is 'true'
* Publishing helm chart should check for `SHOULD_PUBLISH`
* Move blocking step to publish-helm chart, and allow images to be published
* Pin python requirements and bump to latest versions
* Fix EOL whitespace
* use valid version for psycopg2-binary (the clue is in the 2!)
* fix(fileimports): add exception printing to file imports
* fix(fileimports): bump specklepy version
move to a specklepy version that contains a fix for send without writing to disk
Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
* Fixes liveness and readiness checks to prevent CSRF error message (#1169)
- provides content-type header
- check that status code is 200
* Fixes broken helm template by adding quotation marks around liveness probe command (#1171)
* fix(server activities): make sure the stream events are properly dispatched
* feat(server webhooks): add scheduled orphaned webhook cleanup
* test(server webhooks): add test to webhook cleanup service
* feat(server webhooks): drop foreign key reference for webhooks schema to streams
* refactor(server req context): refactor req context to have the ip attribute for all requests
* feat(server objects rest api): add ratelimits to objects rest api endpoints
* fix(server rest api): properly handle returning 419
Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
* feat(server): add server authz pipeline rework first sketch
* feat(server authz): add new server authz middleware poc implementation
* test(server authz): add unittests for the new server authz workflow
* feat(wip rework of fileuploads vs blob storage): add basim impl of separate blob storage service
* feat(fileimport service): refactored file import service to utilize the new asssetstorage service
* refactor(server errors): refactor server errors to use the shared module definitions
Now all the errors inherit from BaseError
* refactor(fileimport service): cleanup after refactor
* feat(frontend fileimports): use the new blob storage for downloading the original file
* refactor(server fileimports): clean up the remnants of S3 storage from file imports
* refactor(server authz): centralize generic authz pipeline configs
* refactor(server blob storage): refactor / rename everything to use the `blob-storage` name
* ci(circleci): add s3 objectstorage environment variables
* ci(circleci): fix missing env variables
* ci(circleci): add minio test container
* ci(circleci): fix minio app startup
* ci(circleci): enable circleci remote docker
* ci(circleci): fix minio startup
* ci(cirleci): detach and wait properly for minio to start
* ci(circleci): revert to additional minio img config, it only fails when the container is stopped ?!
* ci(circleci): disable file uploads
* fix(fileimports): update with blob storage refactor leftovers
* feat(server blob storage): add blob storage graphql api
* refactor(server errors): merge new errors to shared module
* fix(server comments rte): fix import for RTE error
* chore(fileimports): remove node-fetch from dependency
* chore(server): remove body parser dependency
* fix(server blob storage): fix gql api
* fix(frontend): fix fileupload item not loading the new upload status, cause of premature event fire
* feat(server blob storage): fix file size limit and allow for public streams
* Update packages/server/modules/blobstorage/graph/schemas/blobstorage.graphql
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
* chore(blobstorage): fix PR review issues
* fix(server): fix import bugs
* test(server): blob storage first test
* test(server blob services): add tests for blob storage services
* test(server blob storage): add service and rest api tests
* test(server blob storage): add server blob storage graphql api tests
* feat(server blob storage): store and make available blob fileHash attribute
* feat(server authz): add fatal failure option to server authz pipeline
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
* yarn first go
* fix frontend build cache loader
* yarn workspaces built server Docker
* build(yarn): add workspaces plugin config
* chore(package defs): clean package*.json -s
* chore(gitignore): ignore yarn error log
* build(yarn): update yarn lock
* build(preview-service webpack): add extra resolved path to preview service webpack config
because of yarn package hoisting, there are no package level node_modules folder anymore.
* build(docker): update dockerignore with yarn specific configs
* build(docker): update Dockerfiles for yarn workspaces utilization
* ci(circleci): update server test job to yarn
* ci(circle): disable cache restore
* ci(circleci): trying the node orb yarn-run
* ci(circleci): yarn-run again
* ci(circleci): disable node orb
* ci(circleci): change base node image for tests
* ci(circleci): add yarn cache
* ci(circleci): remove node install step
* ci(circleci): add server specific cache archives
* ci(circleci): test build and publish
* ci(circleci): change npm auth method to suit yarn
* ci(circleci): trying new builder image
* ci(circleci): another base image, maybe this works
* ci(circleci): force a specific docker engine version
* ci(circleci): add yarn version plugin and its changes
* ci(circleci): cleanup and remove temp branch config
* chore(package defs): moving from npm run to yarn
* explicitly specifying webpack4 as a frontend dep
* chore(package defs): replace npm with yarn everywhere
* docs(root readme): update with some yarn specific docs
* test(server comments gql): add wip server comments gql tests
* test(server comments graphql): add missing test operations and generate a bunch of testcases
* test(server comments graphql api): fix all authz test cases for comments
* test(server comments service): fix comments service failing test
* fix(tests): do not look inside
Co-authored-by: Fabians <fabis94@live.com>
Co-authored-by: Dimitrie Stefanescu <didimitrie@gmail.com>
Kristaps Fabians Geikins