* feat(viewer-sandbox): Made a sandbox function that will only invoke the object-loader loading objects
* first pass of creating an objectloader2
* updated build + added vitest
* try to get viewer sandbox to use new code
* sandbox type fix
* refactor a bit
* can download root
* intermediate commit for downloader/caching queue
* can download stuff!
* refactor files
* intro isBase and fix isString
* move single download to downloader
* fix download
* PR feedback
* some intermediate commit
* do clean up and download better
* clean up promises and linting
* can generate values while downloading and caching
* add a finish method
* remove unused functions
* remove asBase
* add temporary docs
* add more docs with mermaid
* add more test models
* add response validation
* add tests and redo options
* add test for download batch
* fix downloader tests and change Item to have clearer Base items
* add tests and refactor a little
* use fetch in downloader as an option
* use optional in-memory indexdb instead of monkey patching the global one
* more refactors for options for objectloader2
* add tests for objectloader2
* adjust single download
* benchmark loading and adjust ol2 batches
* download more!
* adjust to use hash privates
* refactored again with renaming
* cleanup
* make setupCacheDb throw instead
* use BatchedPool for downloads!
* fix tests
* adjust timings and add adaptive waiting
* Only wait if queue wasn't empty and queue size was full
* fix tests
* fix file names and some private usage
* fix interval and private usage
* rename vars
* use params for methods
* fix params for constructors and tests
* fix params for constructors and tests again
* using dexie
* faster settings but doesn't end well
* fixed end, optimized and removed logs
* fix tests
* fix types?
* update lock with WSL
* add e2e small model test
* fix/update yarn.lock
* Remove unused eslint ignore to fix pre-commit
* prettier fixes
* fix real DB usage
* rename methods to better match OL1
* rename methods to better match OL1 again
* add extra header collection
* add headers correctly
* test getTotalObjectCount
* feat(viewer-lib): Replaced old object loader with Adam's objectloder2
* fix(viewer-lib): Removed the old object loader. Removed unneeded pause time in speckle loader
* Testing
* only deferred if not downloaded....don't save everything
* Lockfile
* pool isn't adjustable, adjust download buckets, dexie read is faster
* chore
* fix(viewer-lib): Fixed compiler errors
* fix getObject access with real indexeddb...adjust buffer for deferred access
* Fix disposal and pausing
* don't index item!
* fix dockerfiles to use OL2
* fix Dockerfile
* Fix dockerfile
* defer correctly and use record to add/lookup/remove to
* delete stuff correctly
* chore(sandbox): Enabled viewer loading
* use objects instead of arrays to avoid findIndex
* remove extra count
* add a found cache to avoid some db hits
* order matters for deferment
* move found map to deferment
* change option numbers
* 2 level cache with expiry
* defer everything, use loader to track what is requested....expire only found items
* add deferment disposal
* oops mismerge
* chore(sandbox): Default stream
* Beta version of CachePump and CacheReader
* Clean up initialization
* More clean up
* chore(objectloader2): Fixed CI compiler error
* chore(objectloader2): Fixed prettier
* add cachePump tests
* add cacheReader tests
* fixed more tests
* fixed final tests
* moving stuff around and lock return value
* try to move stuff out of objectloader2
* use a factory
* rename factory
* formatting
* eslist fixes
* try allocating no strings
* add comments
* small refactor and add another test
* fix deferment expiration and have test
* use byte size for max memory cache size
* fix deferment manager tests
* saved comment
* fix(viewer-sandbox): Fixed compiler error
* ignore tshy
* chore(frontend): Attempt to make viewer loading sequential
---------
Co-authored-by: Adam Hathcock <adam@hathcock.uk>
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
* webhooks perm minor fix
* tryna get fileimport service to work
* new comment policies - shared
* BE done?
* checks implemented in FE
* lint fix
* tests fix
* readme fix
* first pass of CLI ifc converter
* some updates
* closer
* yarn works
* can execute ifc?
* change exe
* remove extra venv needs
* invocation works
* fixed dockerfile and url
* refactor(fileimport): temp results path should not be hardcoded in parsers
* update importer to output stuff
* fix up argments
* remove dead code
* adjust dockerfile to have tini and workdir better
* fix node to a specific version
* Add shell statement and pin yarn version
* add ifc converter c# to ignore
* merge fix
* move ifc c#
* fix the api usage
* update the importer to new SDK
* Adds a feature flag `FF_FILEIMPORT_IFC_DOTNET_ENABLED` for enabling .Net IFC parser
* move directories
* put back ifc js
* use FF and reversions
* needs token too
* fix docker?
* one last copy fix
* adjust prettier ignore
* change to enable
* fix helm chart nesting
* Amend healthcheck node binary path
* Add FF_FILEIMPORT_IFC_DOTNET_ENABLED to feature flag parser
* Allow app to write to /.config directory
* fix: volume name has to be lower case
* update ifc importing
---------
Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
Co-authored-by: root <root@Clynelish>
* WIP create modal
* babababa
* create dialog looks ok
* FE largely there
* workss
* cleanup
* fixed up test plumbing to avoid deadlocks and simplify GQL calls
* test fix
* added all tests
* CI fix
* feat(db queries): adds db query scripts for determining db sizes
* Adds a new metric `speckle_db_tablesize` with db table sizes
* lower case all the file types to provide a combined metric
* group file status count by lower cased file type
* reinstate accidentally deleted metric
* Fix developer script
* Demonstration of bug to test when middleware added
- Adding middleware, even no-op, causes test to fail
* Make middleware async, but introduce delay.
Revert test back to original.
* Revert tests
* Add a 1ms sleep to the test to reduce likelihood of flakiness
* Rate limiting on all express endpoints using middleware
* Adds all configuration for existing rate limited endpoints
* It is helpful to add the package to yarn first
* Implements respectsLimits using Redis rate limiter
* Fix for test `Should rate-limit user creation`
- if rate limit error, post to `/auth/local/register` will return a 429 status code
* All rate limiting provided by new ratelimiter.ts
* Consolidate typescript interfaces
* Amend signature of function to require source to be passed in, and not try to guess it from the request
* Rename respectsLimits to isWithinRateLimits
* Throw within catch of Promise
* Replace rejectsRequestWithRatelimitStatusIfNeeded throughout code
* Sending rate limit response should deal with other types of error
- Sentry notified of the error
* Express middleware rate limits by a 3 second burst or a daily rate
- Provide action when generating 429 response
* Prevent DOS of Redis
* Add 'Retry-After' for all cases when responding with 429 status code
- default of 1 day, but dynamic based on available information
* Generate rate limiters once, on init
- Improved and consistent handling of exit from functions
- fixed environment variable names
* WIP Refactor rate limiting setup
Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com>
* WIP: fixed references, now runs but tests fail
* Use getSourceFromRequest where possible
* WIP: unit tests for rate limiter
* Unit tests for ratelimiter
* feat(IFC): WIP IFC parser improvements
* Revert "feat(IFC): WIP IFC parser improvements"
This reverts commit 093089a2c4.
* refactor authz, rate limiting middleware to global
Co-authored-by: Kristaps Fabians Geikins <fabis94@users.noreply.github.com>
Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com>
* invites tests fix
* fix(server ratelimiter): export public interfaces
* Unit test for rate limiter use in memory rate limiter
- in memory rate limiter is configured with zero limit by default
* Fixed#1219 (#1221)
* WIP: improve auth test for rate limiting user creation
* ci(circleci config): publishing was broken when main branch was tagged (i.e. for releases) (#1224)
* Gitignore CPU profiles
* All tests are now passing locally
* Fixed an issue in the frontend which was causing the views not to work. Fixed an issue with object selection camera animation where the dolly lerp factor was much too high for smooth animation (#1225)
* feat(structured logging): implements structured logging for backend (#1217)
* each log line is a json object
* structured logging allows logs to be ingested by machines and the logs to be indexed and queried addresses #1105
* structured logging allows arbitrary properties to be appended to each log line, and ingestion of logs to remain robust
* Structured logging provided by `pino` library
* Add `express-pino-logger` dependency
* Remove `debug`, `morgan`, and `morgan-debug` and replace with structured logging
* `console.log` & `console.error` replaced with structured logging in backend
* Remove `DEBUG` environment variable and replace with `LOG_LEVEL`
- Note that there is a test which reads from a logged line on `stdout`. This is not robust, it would be better to use the childProcess.pid to look up the port number.
* Log errors at points we explicitly send error to Sentry
* Amend indentation of a couple of log messages to align indentation with others
* Revert "feat(structured logging): implements structured logging for backend (#1217)" (#1227)
This reverts commit 84cb74e8b3.
* Move error to core/errors
- augmented typescript types moved to type-augmentations
* Added a missing wait in the screenshot generation loop (#1228)
* refactor(server rest api): remove duplicate rate limit requests
* feat(server rate limits): increase rate limits for the upload endpoints
* chore(server rate limits): final cleanup
Co-authored-by: Gergő Jedlicska <gergo@jedlicska.com>
Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com>
Co-authored-by: Dimitrie Stefanescu <didimitrie@gmail.com>
Co-authored-by: Kristaps Fabians Geikins <fabis94@users.noreply.github.com>
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
Co-authored-by: Alexandru Popovici <alexandrupopoviciioan@gmail.com>
In the event of an error in VSCode, the IDE produces log files in the .vscode file. These should
not be checked in to the project as they are specific to the error on the developer's local machine.
* feat(server): add server authz pipeline rework first sketch
* feat(server authz): add new server authz middleware poc implementation
* test(server authz): add unittests for the new server authz workflow
* feat(wip rework of fileuploads vs blob storage): add basim impl of separate blob storage service
* feat(fileimport service): refactored file import service to utilize the new asssetstorage service
* refactor(server errors): refactor server errors to use the shared module definitions
Now all the errors inherit from BaseError
* refactor(fileimport service): cleanup after refactor
* feat(frontend fileimports): use the new blob storage for downloading the original file
* refactor(server fileimports): clean up the remnants of S3 storage from file imports
* refactor(server authz): centralize generic authz pipeline configs
* refactor(server blob storage): refactor / rename everything to use the `blob-storage` name
* ci(circleci): add s3 objectstorage environment variables
* ci(circleci): fix missing env variables
* ci(circleci): add minio test container
* ci(circleci): fix minio app startup
* ci(circleci): enable circleci remote docker
* ci(circleci): fix minio startup
* ci(cirleci): detach and wait properly for minio to start
* ci(circleci): revert to additional minio img config, it only fails when the container is stopped ?!
* ci(circleci): disable file uploads
* fix(fileimports): update with blob storage refactor leftovers
* feat(server blob storage): add blob storage graphql api
* refactor(server errors): merge new errors to shared module
* fix(server comments rte): fix import for RTE error
* chore(fileimports): remove node-fetch from dependency
* chore(server): remove body parser dependency
* fix(server blob storage): fix gql api
* fix(frontend): fix fileupload item not loading the new upload status, cause of premature event fire
* feat(server blob storage): fix file size limit and allow for public streams
* Update packages/server/modules/blobstorage/graph/schemas/blobstorage.graphql
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
* chore(blobstorage): fix PR review issues
* fix(server): fix import bugs
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
* yarn first go
* fix frontend build cache loader
* yarn workspaces built server Docker
* build(yarn): add workspaces plugin config
* chore(package defs): clean package*.json -s
* chore(gitignore): ignore yarn error log
* build(yarn): update yarn lock
* build(preview-service webpack): add extra resolved path to preview service webpack config
because of yarn package hoisting, there are no package level node_modules folder anymore.
* build(docker): update dockerignore with yarn specific configs
* build(docker): update Dockerfiles for yarn workspaces utilization
* ci(circleci): update server test job to yarn
* ci(circle): disable cache restore
* ci(circleci): trying the node orb yarn-run
* ci(circleci): yarn-run again
* ci(circleci): disable node orb
* ci(circleci): change base node image for tests
* ci(circleci): add yarn cache
* ci(circleci): remove node install step
* ci(circleci): add server specific cache archives
* ci(circleci): test build and publish
* ci(circleci): change npm auth method to suit yarn
* ci(circleci): trying new builder image
* ci(circleci): another base image, maybe this works
* ci(circleci): force a specific docker engine version
* ci(circleci): add yarn version plugin and its changes
* ci(circleci): cleanup and remove temp branch config
* chore(package defs): moving from npm run to yarn
* explicitly specifying webpack4 as a frontend dep
* chore(package defs): replace npm with yarn everywhere
* docs(root readme): update with some yarn specific docs
* chore(root workspace): update dev scripts and package lock
* ci(circleci): enable package publish step with yarn
Co-authored-by: Fabians <fabis94@live.com>
huanld
Kristaps Fabians Geikins