Iain Sproat
a537d34dcc
* Demonstration of bug to test when middleware added - Adding middleware, even no-op, causes test to fail * Make middleware async, but introduce delay. Revert test back to original. * Revert tests * Add a 1ms sleep to the test to reduce likelihood of flakiness * Rate limiting on all express endpoints using middleware * Adds all configuration for existing rate limited endpoints * It is helpful to add the package to yarn first * Implements respectsLimits using Redis rate limiter * Fix for test `Should rate-limit user creation` - if rate limit error, post to `/auth/local/register` will return a 429 status code * All rate limiting provided by new ratelimiter.ts * Consolidate typescript interfaces * Amend signature of function to require source to be passed in, and not try to guess it from the request * Rename respectsLimits to isWithinRateLimits * Throw within catch of Promise * Replace rejectsRequestWithRatelimitStatusIfNeeded throughout code * Sending rate limit response should deal with other types of error - Sentry notified of the error * Express middleware rate limits by a 3 second burst or a daily rate - Provide action when generating 429 response * Prevent DOS of Redis * Add 'Retry-After' for all cases when responding with 429 status code - default of 1 day, but dynamic based on available information * Generate rate limiters once, on init - Improved and consistent handling of exit from functions - fixed environment variable names * WIP Refactor rate limiting setup Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com> * WIP: fixed references, now runs but tests fail * Use getSourceFromRequest where possible * WIP: unit tests for rate limiter * Unit tests for ratelimiter * feat(IFC): WIP IFC parser improvements * Revert "feat(IFC): WIP IFC parser improvements" This reverts commit093089a2c4. * refactor authz, rate limiting middleware to global Co-authored-by: Kristaps Fabians Geikins <fabis94@users.noreply.github.com> Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com> * invites tests fix * fix(server ratelimiter): export public interfaces * Unit test for rate limiter use in memory rate limiter - in memory rate limiter is configured with zero limit by default * Fixed #1219 (#1221) * WIP: improve auth test for rate limiting user creation * ci(circleci config): publishing was broken when main branch was tagged (i.e. for releases) (#1224) * Gitignore CPU profiles * All tests are now passing locally * Fixed an issue in the frontend which was causing the views not to work. Fixed an issue with object selection camera animation where the dolly lerp factor was much too high for smooth animation (#1225) * feat(structured logging): implements structured logging for backend (#1217) * each log line is a json object * structured logging allows logs to be ingested by machines and the logs to be indexed and queried addresses #1105 * structured logging allows arbitrary properties to be appended to each log line, and ingestion of logs to remain robust * Structured logging provided by `pino` library * Add `express-pino-logger` dependency * Remove `debug`, `morgan`, and `morgan-debug` and replace with structured logging * `console.log` & `console.error` replaced with structured logging in backend * Remove `DEBUG` environment variable and replace with `LOG_LEVEL` - Note that there is a test which reads from a logged line on `stdout`. This is not robust, it would be better to use the childProcess.pid to look up the port number. * Log errors at points we explicitly send error to Sentry * Amend indentation of a couple of log messages to align indentation with others * Revert "feat(structured logging): implements structured logging for backend (#1217)" (#1227) This reverts commit84cb74e8b3. * Move error to core/errors - augmented typescript types moved to type-augmentations * Added a missing wait in the screenshot generation loop (#1228) * refactor(server rest api): remove duplicate rate limit requests * feat(server rate limits): increase rate limits for the upload endpoints * chore(server rate limits): final cleanup Co-authored-by: Gergő Jedlicska <gergo@jedlicska.com> Co-authored-by: Iain Sproat <iainsproat@users.noreply.github.com> Co-authored-by: Dimitrie Stefanescu <didimitrie@gmail.com> Co-authored-by: Kristaps Fabians Geikins <fabis94@users.noreply.github.com> Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com> Co-authored-by: Alexandru Popovici <alexandrupopoviciioan@gmail.com>
Speckle Server
Status
Disclaimer
We're working to stabilize the 2.0 API, and until then there will be breaking changes.
Documentation
Comprehensive developer and user documentation can be found in our:
📚 Speckle Docs website
Introduction
The Speckle Server is a node application tested against v12.
The external dependencies are PostgreSQL and Redis. To get the dependencies running without any hassle, you can run them in docker containers as described in our Server deployment instructions (chapter Run your speckle-server fork, step 1)
NOTE: If you install PostgreSQL yourself or use an existing PostgreSQL instance, make sure to create a database and a user that can access it
After you have PostgreSQL and Redis running, in the packages/server folder:
- copy the
.env-examplefile to.env, - (if you plan to run tests) copy the
.env.test-examplefile to.env.test - If you have a custom setup, open and edit the
.env&.env.testfiles, filling in the required variables, - run
yarn install, - finally
yarn dev, - check
localhost:3000/graphqlout!
Developing
The server consists of several semi-related components, or modules. These can be found in /modules. Module composition:
- an
index.jsfile that exposes two functions,initandfinalize(mandatory) - a
graphfolder, with two subfolders, namelyresolversandschemas(optional - these will be picked up and merged).
TypeScript
This package has TypeScript support and you can use TS everywhere in it - modules, tests, migrations (read note about migrations below).
To run the app, build it first into /dist and then run it through ./bin/www. Or just run - yarn dev which will run the TS compiler in watch mode and also run the build app through nodemon.
Tests and the CLI, however, do not need an explicit build inside the /dist folder as they use ts-node to execute TS files directly. This is to improve the DX and allow you to iterate on tests faster, without having to run the TS compiler.
GraphQL types
Whenever a schema changes you can run yarn gqlgen to regenerate GraphQL types at @/modules/core/graph/generated/graphql.ts. This file will hold types for scalars, variables and most importantly - resolvers.
You can get the best DX by typing your resolvers with the Resolvers type and then you will get proper type checking for parent, arguments and so on in your resolvers.
Migrations
To create new migrations use yarn migrate create. Note that migrations are only ever read from the ./dist folder to avoid scenarious when both the TS and JS version of the same migration is executed, so if you ever create a new migration make sure
you build the app into /dist if you want it to be applied.
CLI
We've got a yargs based dev-only CLI that you can run and extend with useful commands. Run it through yarn cli and add new commands under ./modules/cli
Bull queue monitoring
Use yarn cli bull monitor to serve a Web UI for our Bull queues (e.g. Notifications queues). In the prod env we don't retain old jobs, but locally these older results aren't deleted and you'll see them in this Web UI.
Server & Apps
Frontend
-
In development mode, the Speckle Server will proxy the frontend from
localhost:3000tolocalhost:8080. If you don't see anything, ensure you've runyarn devin the frontend package. -
In production mode, the frontend is served by an
nginxcontainer that proxy server requests to the server (depending on the requested path). For more information about making a production deployment, check out our detailed guide
GraphIQL
A GraphIQL app is available for authenticated api exploration at localhost:3000/explorer. Note that for the authentication flow to work, you need to have the frontend running first.
GraphQL Playground
For non-authenticated api exploration, you can use the Graphql Playground which is available by default at localhost:3000/graphql.
Testing
To run all tests, simply run yarn test.
The recommended extensions for the workspace include a test explorer, that can run individual tests.
If you really want to run specific tests from a terminal, use the mocha --grep @subset syntax. For example:
mocha --grep @auth --watchto run tests pertaning to the auth module only in watch mode.mocha --grep @core-streams --watchto run tests pertaining to stream related services.
It's suggested to just run tests from the VSCode test explorer, however.
Integration tests with GraphQL
The best way to do integration tests is to actually invoke queries against an ApolloServer instance. To make this process even better you can rely on GraphQL Code Generator to properly generate types for the queries you write in your tests.
Put your test-specific queries/mutations in @/test/graphql and then run yarn gqlgen. This will generate a typings file at @/test/graphql/generated/graphql.ts which will contain query & variable types for the operations you've created.
You can then specify these types when running operations through executeOperation from @/test/graphqlHelper.ts (through the generic arguments), and then inside your TS test file you'll get properly typed response structures. Awesome!
Community
The Speckle Community hangs out on the forum, do join and introduce yourself & feel free to ask us questions!
License
Unless otherwise described, the code in this repository is licensed under the Apache-2.0 License. Please note that some modules, extensions or code herein might be otherwise licensed. This is indicated either in the root of the containing folder under a different license file, or in the respective file's header. If you have any questions, don't hesitate to get in touch with us via email.