Iain Sproat
a537d34dcc
* Demonstration of bug to test when middleware added - Adding middleware, even no-op, causes test to fail * Make middleware async, but introduce delay. Revert test back to original. * Revert tests * Add a 1ms sleep to the test to reduce likelihood of flakiness * Rate limiting on all express endpoints using middleware * Adds all configuration for existing rate limited endpoints * It is helpful to add the package to yarn first * Implements respectsLimits using Redis rate limiter * Fix for test `Should rate-limit user creation` - if rate limit error, post to `/auth/local/register` will return a 429 status code * All rate limiting provided by new ratelimiter.ts * Consolidate typescript interfaces * Amend signature of function to require source to be passed in, and not try to guess it from the request * Rename respectsLimits to isWithinRateLimits * Throw within catch of Promise * Replace rejectsRequestWithRatelimitStatusIfNeeded throughout code * Sending rate limit response should deal with other types of error - Sentry notified of the error * Express middleware rate limits by a 3 second burst or a daily rate - Provide action when generating 429 response * Prevent DOS of Redis * Add 'Retry-After' for all cases when responding with 429 status code - default of 1 day, but dynamic based on available information * Generate rate limiters once, on init - Improved and consistent handling of exit from functions - fixed environment variable names * WIP Refactor rate limiting setup Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com> * WIP: fixed references, now runs but tests fail * Use getSourceFromRequest where possible * WIP: unit tests for rate limiter * Unit tests for ratelimiter * feat(IFC): WIP IFC parser improvements * Revert "feat(IFC): WIP IFC parser improvements" This reverts commit093089a2c4. * refactor authz, rate limiting middleware to global Co-authored-by: Kristaps Fabians Geikins <fabis94@users.noreply.github.com> Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com> * invites tests fix * fix(server ratelimiter): export public interfaces * Unit test for rate limiter use in memory rate limiter - in memory rate limiter is configured with zero limit by default * Fixed #1219 (#1221) * WIP: improve auth test for rate limiting user creation * ci(circleci config): publishing was broken when main branch was tagged (i.e. for releases) (#1224) * Gitignore CPU profiles * All tests are now passing locally * Fixed an issue in the frontend which was causing the views not to work. Fixed an issue with object selection camera animation where the dolly lerp factor was much too high for smooth animation (#1225) * feat(structured logging): implements structured logging for backend (#1217) * each log line is a json object * structured logging allows logs to be ingested by machines and the logs to be indexed and queried addresses #1105 * structured logging allows arbitrary properties to be appended to each log line, and ingestion of logs to remain robust * Structured logging provided by `pino` library * Add `express-pino-logger` dependency * Remove `debug`, `morgan`, and `morgan-debug` and replace with structured logging * `console.log` & `console.error` replaced with structured logging in backend * Remove `DEBUG` environment variable and replace with `LOG_LEVEL` - Note that there is a test which reads from a logged line on `stdout`. This is not robust, it would be better to use the childProcess.pid to look up the port number. * Log errors at points we explicitly send error to Sentry * Amend indentation of a couple of log messages to align indentation with others * Revert "feat(structured logging): implements structured logging for backend (#1217)" (#1227) This reverts commit84cb74e8b3. * Move error to core/errors - augmented typescript types moved to type-augmentations * Added a missing wait in the screenshot generation loop (#1228) * refactor(server rest api): remove duplicate rate limit requests * feat(server rate limits): increase rate limits for the upload endpoints * chore(server rate limits): final cleanup Co-authored-by: Gergő Jedlicska <gergo@jedlicska.com> Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com> Co-authored-by: Dimitrie Stefanescu <didimitrie@gmail.com> Co-authored-by: Kristaps Fabians Geikins <fabis94@users.noreply.github.com> Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com> Co-authored-by: Alexandru Popovici <alexandrupopoviciioan@gmail.com>
Speckle | Server
Server and Web packages
Speckle is data infrastructure for the AEC industry.
About Speckle
What is Speckle? Check our
Features
- Object-based: say goodbye to files! Speckle is the first object based platform for the AEC industry
- Version control: Speckle is the Git & Hub for geometry and BIM data
- Collaboration: share your designs collaborate with others
- 3D Viewer: see your CAD and BIM models online, share and embed them anywhere
- Interoperability: get your CAD and BIM models into other software without exporting or importing
- Real time: get real time updates and notifications and changes
- GraphQL API: get what you need anywhere you want it
- Webhooks: the base for a automation and next-gen pipelines
- Built for developers: we are building Speckle with developers in mind and got tools for every stack
- Built for the AEC industry: Speckle connectors are plugins for the most common software used in the industry such as Revit, Rhino, Grasshopper, AutoCAD, Civil 3D, Excel, Unreal Engine, Unity, QGIS, Blender, ArchiCAD and more!
Try Speckle now!
Give Speckle a try in no time by:
⇒ creating an account at
⇒ deploying an instance in 1 click
Resources
for help, feature requests or just to hang with other speckle enthusiasts, check out our community forum!
our tutorials portal is full of resources to get you started using Speckle
reference on almost any end-user and developer functionality
Repo structure
This monorepo is the home of the Speckle v2 web packages:
packages/server: the Server, a nodejs app. Core external dependencies are a Redis and Postgresql db.packages/frontend: the Frontend, a static Vue app.packages/viewer: a threejs extension that allows you to display 3D datapackages/objectloader: a small js utility class that helps you stream an object and all its sub-components from the Speckle Server API.packages/preview-service: generates object previews for Speckle Objects headlessly. This package is meant to be called on by the server.packages/webhook-service: making external webhook callspackages/fileimport-service: parsing and importing files
Other repos
Make sure to also check and ⭐️ these other Speckle repositories:
speckle-sharp: .NET tooling, connectors and interoperabilityspecklepy: Python SDK 🐍speckle-excel: Excel connectorspeckle-unity: Unity 3D connectorspeckle-blender: Blender connectorspeckle-unreal: Unreal Engine connectorspeckle-qgis: QGIS connectorspeckle-powerbi: PowerBi connector- and more connectors & tooling!
Developing and Debugging
Have you checked our dev docs?
We have a detailed section on deploying a Speckle server. To get started developing locally, you can see the Local development environment page.
TL;DR;
We're using yarn and its workspaces functionalities to manage the monorepo. Make sure you are using nodejs 16+. To get started, run:
$ corepack enable
$ yarn
$ yarn build
After this, you can use the scripts in the individual packages or run all in dev mode:
$ yarn dev
For development you'll also want to run yarn dev:docker:up which will start up the docker containers you will need
to be able to run the apps on your machine.
IDE
The suggested IDE for working in this repo is VSCode, because the repo has special configs to improve the DX there (auto-format & auto-lint, recommended extensions etc.).
This repository relies on VSCode multi-root workspaces, so for the best DX and proper formatting/linting make sure you open the project using the workspace.code-workspace file instead of just opening the folder in VSCode.
Proper TypeScript support
To get proper TS support in standard TS files as well as Vue files, we rely on the Volar VSCode extension - not Vetur! When you open the workspace for the very first time, VSCode should already prompt you to install this extension. The default Volar setup can be quite resource intensive, however, because there end up being standard TS Language Server instances running as well as Vue TS Language server instances. To resolve this issue we rely on Volar's Take Over Mode.
Essentially, ensure you have Volar enabled, and the built in TypeScript and JavaScript Language Features extension disabled (for this workspace)!
Useful containers
Maildev
We have a Maildev container available that you can use to see all e-mails sent out from the app. Make sure your server .env file is configured properly to use it:
EMAIL=true
EMAIL_FROM="speckle@speckle.local"
EMAIL_HOST="localhost"
EMAIL_PORT="1025"
The web portal is available at localhost:1080 and it's listening for mail on port 1025.
Contributing
Please make sure you read the contribution guidelines for an overview of the best practices we try to follow.
When pushing commits to this repo, please follow the following guidelines:
- When ready to commit, run
$ yarn cz& follow the prompts.
When committing to git, an automated script will run. This will check that your commit conforms to Speckle's guidelines.
- To enable additional checks, you may wish to optionally install the following:
- Please note that these additional checks will be run by Speckle's automated CI environment and non-conformances, if any, may prevent your Pull Request from being merged.
- Please use the name of the package as the scope of your commit.
- Note that
ggshieldrequires you to register an account and then authorise with GitGuardian before it can be used:ggshield auth login
Security
For any security vulnerabilities or concerns, please contact us directly at security[at]speckle.systems.
License
Unless otherwise described, the code in this repository is licensed under the Apache-2.0 License. Please note that some modules, extensions or code herein might be otherwise licensed. This is indicated either in the root of the containing folder under a different license file, or in the respective file's header. If you have any questions, don't hesitate to get in touch with us via email.