* Demonstration of bug to test when middleware added
- Adding middleware, even no-op, causes test to fail
* Make middleware async, but introduce delay.
Revert test back to original.
* Revert tests
* Add a 1ms sleep to the test to reduce likelihood of flakiness
* Rate limiting on all express endpoints using middleware
* Adds all configuration for existing rate limited endpoints
* It is helpful to add the package to yarn first
* Implements respectsLimits using Redis rate limiter
* Fix for test `Should rate-limit user creation`
- if rate limit error, post to `/auth/local/register` will return a 429 status code
* All rate limiting provided by new ratelimiter.ts
* Consolidate typescript interfaces
* Amend signature of function to require source to be passed in, and not try to guess it from the request
* Rename respectsLimits to isWithinRateLimits
* Throw within catch of Promise
* Replace rejectsRequestWithRatelimitStatusIfNeeded throughout code
* Sending rate limit response should deal with other types of error
- Sentry notified of the error
* Express middleware rate limits by a 3 second burst or a daily rate
- Provide action when generating 429 response
* Prevent DOS of Redis
* Add 'Retry-After' for all cases when responding with 429 status code
- default of 1 day, but dynamic based on available information
* Generate rate limiters once, on init
- Improved and consistent handling of exit from functions
- fixed environment variable names
* WIP Refactor rate limiting setup
Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com>
* WIP: fixed references, now runs but tests fail
* Use getSourceFromRequest where possible
* WIP: unit tests for rate limiter
* Unit tests for ratelimiter
* feat(IFC): WIP IFC parser improvements
* Revert "feat(IFC): WIP IFC parser improvements"
This reverts commit 093089a2c4.
* refactor authz, rate limiting middleware to global
Co-authored-by: Kristaps Fabians Geikins <fabis94@users.noreply.github.com>
Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com>
* invites tests fix
* fix(server ratelimiter): export public interfaces
* Unit test for rate limiter use in memory rate limiter
- in memory rate limiter is configured with zero limit by default
* Fixed#1219 (#1221)
* WIP: improve auth test for rate limiting user creation
* ci(circleci config): publishing was broken when main branch was tagged (i.e. for releases) (#1224)
* Gitignore CPU profiles
* All tests are now passing locally
* Fixed an issue in the frontend which was causing the views not to work. Fixed an issue with object selection camera animation where the dolly lerp factor was much too high for smooth animation (#1225)
* feat(structured logging): implements structured logging for backend (#1217)
* each log line is a json object
* structured logging allows logs to be ingested by machines and the logs to be indexed and queried addresses #1105
* structured logging allows arbitrary properties to be appended to each log line, and ingestion of logs to remain robust
* Structured logging provided by `pino` library
* Add `express-pino-logger` dependency
* Remove `debug`, `morgan`, and `morgan-debug` and replace with structured logging
* `console.log` & `console.error` replaced with structured logging in backend
* Remove `DEBUG` environment variable and replace with `LOG_LEVEL`
- Note that there is a test which reads from a logged line on `stdout`. This is not robust, it would be better to use the childProcess.pid to look up the port number.
* Log errors at points we explicitly send error to Sentry
* Amend indentation of a couple of log messages to align indentation with others
* Revert "feat(structured logging): implements structured logging for backend (#1217)" (#1227)
This reverts commit 84cb74e8b3.
* Move error to core/errors
- augmented typescript types moved to type-augmentations
* Added a missing wait in the screenshot generation loop (#1228)
* refactor(server rest api): remove duplicate rate limit requests
* feat(server rate limits): increase rate limits for the upload endpoints
* chore(server rate limits): final cleanup
Co-authored-by: Gergő Jedlicska <gergo@jedlicska.com>
Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com>
Co-authored-by: Dimitrie Stefanescu <didimitrie@gmail.com>
Co-authored-by: Kristaps Fabians Geikins <fabis94@users.noreply.github.com>
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
Co-authored-by: Alexandru Popovici <alexandrupopoviciioan@gmail.com>
* each log line is a json object
* structured logging allows logs to be ingested by machines and the logs to be indexed and queried addresses #1105
* structured logging allows arbitrary properties to be appended to each log line, and ingestion of logs to remain robust
* Structured logging provided by `pino` library
* Add `express-pino-logger` dependency
* Remove `debug`, `morgan`, and `morgan-debug` and replace with structured logging
* `console.log` & `console.error` replaced with structured logging in backend
* Remove `DEBUG` environment variable and replace with `LOG_LEVEL`
- Note that there is a test which reads from a logged line on `stdout`. This is not robust, it would be better to use the childProcess.pid to look up the port number.
* Log errors at points we explicitly send error to Sentry
* Amend indentation of a couple of log messages to align indentation with others
* fix: some extra param validation for some API endpoints
* fix(server): potentially leaking internal error details
* fix: secure session cookie for ssl servers
* fix(server): fixing tests
* Update to new specklepy (#1173)
* Publish images for all branches but limit tagging
* only tag 'latest' and '2' when 'SHOULD_PUBLISH' variable is 'true'
* Publishing helm chart should check for `SHOULD_PUBLISH`
* Move blocking step to publish-helm chart, and allow images to be published
* Pin python requirements and bump to latest versions
* Fix EOL whitespace
* use valid version for psycopg2-binary (the clue is in the 2!)
* fix(fileimports): add exception printing to file imports
* fix(fileimports): bump specklepy version
move to a specklepy version that contains a fix for send without writing to disk
Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
* Fixes liveness and readiness checks to prevent CSRF error message (#1169)
- provides content-type header
- check that status code is 200
* Fixes broken helm template by adding quotation marks around liveness probe command (#1171)
* fix(server activities): make sure the stream events are properly dispatched
* feat(server webhooks): add scheduled orphaned webhook cleanup
* test(server webhooks): add test to webhook cleanup service
* feat(server webhooks): drop foreign key reference for webhooks schema to streams
* refactor(server req context): refactor req context to have the ip attribute for all requests
* feat(server objects rest api): add ratelimits to objects rest api endpoints
* fix(server rest api): properly handle returning 419
Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
* ci(snyk): snyk vulnerability scan of yarn package.json
* Run with strict-out-of-sync=false
* only runs for main branch to prevent pollution of Snyk's dashboard with branches
All previews were in the 600 to infinity bucket of the histogram, rendering it effectively useless.
This PR adds additional larger buckets at 1200 and 1800 ranges to provide better granularity.
* Publish images for all branches but limit tagging
* only tag 'latest' and '2' when 'SHOULD_PUBLISH' variable is 'true'
* Publishing helm chart should check for `SHOULD_PUBLISH`
* Move blocking step to publish-helm chart, and allow images to be published
* Pin python requirements and bump to latest versions
* Fix EOL whitespace
* use valid version for psycopg2-binary (the clue is in the 2!)
* fix(fileimports): add exception printing to file imports
* fix(fileimports): bump specklepy version
move to a specklepy version that contains a fix for send without writing to disk
Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
* Always build, except when branch (not main) does not have PR
* Logic fix
* Should fail the step
* Debug
* More circleci debugging
* More and more circleci debugging
* More more more debugging
* Attempt to explicitly exit
* 🤦 the script required a passing line in order to exit non-zero
* Started working on progressive sao
* Started working on the ao generation stage shader and pass
* Continued with progressive SAO. Added the accumulation pass
* Implemented progressive AO using the same estimator as for the dynamic one.
* First draft of progressive AO with multiple selectable estimators. WIP on final implementation
* Restructuring how the pipeline and it's passes work so we can extend/configure them more easily. Added the concept of SpecklePass
* A lot of changes. Pipeline rework is more or less complete. Added individual output for some pipeline passes that we might want to inspect inside the sandbox.
* Finished with displaying individual pipeline stages
* Intruduced accumulation frames properly at pipeline level. Cleaned up progressive AO shader and pass. Exposed the number of accumulation frames as well as the kernel size in the sandbox.
* Accumulation frame count is now reset after a resize
* In the end I just implemented my own stationary/dynamic detection routine since the events sent from the 'camera controller' are dogshit
* Implemented interpolating during frame accumulation between the dynamic SAO and the static AO. This makes the transition much more smooth and seamless than simply jumping straight to the static AO which begins accumulating from 0
* Added configurable intensity to static AO. Fixed an issue with static scenario detection where the accumulation frames were ran twice in succession, because the 'camera controller' is probably clamping the angles to X decimal points at the end of a camera motion cycle, which in turn triggers a camera move larger than our defined epsilon
* Dynamic AO at half res
* Some improvements to how we update the rendering pipeline and it's dynamic and progressive states
* Trying to fix the issue with artifacts on progressive AO on mac. seems to be depth encoding related. Changed the depth pass to render to 32 float texture to check if the artifacts are gone
* Trying out a new float->RGBA encoding and decoding
* Added ner and far camera planes controls to see how they're values change depth encoding precision
* Finally managed to get AO properly working with encoded linear depth
* Trying 24 bit encoding
* Fix for the macOS artifacts using linear depth buffer and a bias value
* Extended the camera controller class and added/changed a few things so that the controller doesn't mess up our camera movement and also we now rely on the events sent by it to determine stationary vs dynamic camera scenarios. This way we spare the rendered from having to determine camera movement deltas by itslef(only to have it ruined by the camera controller anyway)
* Added switching between perspective and linear depth when going from dynamic AO to progressive AO, so we don't have to integrate linear depth buffer sampling in the already existing features
* Disabled normal texture rendering for dynamic AO
* Some fixes and added dynamic AO back in at half res
* Fixed an issue with converging progressive pipeline when using a trackpad
* Implemented relative dynamic progressive AO kernel size
* Fixed compile error
* Fixed an issue with macOS and depth texture filtering
* Added half res depth buffer switching for dynamic AO
* Changed some params now that dynamic AO is at half res
* Made section boxes work properly with our new progressive pipeline
* Added correct kernel size computation for orthographic projection
* Patched the camera controls to send events when zooming in orthographic mode
* Some more integrations with camera views and fixed a bug with bad pipeline resets
* Small fix for zoom and startup reset
* Various fixes of issues I could find
* Removed longs
* preview-service now waits for the pipeline to convgerge before taking the screenshot
* Handled #1110. Added the concept of 'ObjectLayers' which uses three's existing implementation for selective layer rendering/lighting/picking. Also added the BaseSpecklePass abstract class which provides a default implementation for any subclassing pass that needs to use specific layers when rendering.
* Added more time between frames for the preview service. It seems to be generating the previews properly now
Co-authored-by: Alex <alex@192-168-0-159.rdsnet.ro>
Co-authored-by: Alex <alexandru.popovici@busymachines.com>
* ci(circleci): introduces manual holds to prevent resource wastage
We were building all commits unnecessarily. The logic was not consistent for versioning and when to
build and when to publish. This PR addresses these issues.
* Replaces `should-build` with `build-approval`
* Replace `should-publish` with `publish-approval`
* Clean up image tagging logic
- tags as latest when version is semver or semver-alpha.build_number
- other commits aren't tagged as latest, and instead tagged with their branch name
- move naming logic to common.sh script
* Push each tag individually to avoid pollution
- docker engine used is shared amongst all runs of speckle-server pipeline, so pushing all tags indiscrimanately pushes images built on other CircleCI runs 😬
* Improve logging
* Do not require approval for any tagged commit
Gergő Jedlicska