Commit Graph

18 Commits

Author SHA1 Message Date
Iain Sproat 444d2ca7dd Structured logging (attempt 2) (#1234)
* Revert "Revert "feat(structured logging): implements structured logging for backend (#1217)" (#1227)"

This reverts commit 63e6581162.

* Use pino-http instead of express pino logger
* Use correct reference to knex and do not instantiate HttpLogger prematurely
* Adds missing dependency for pino to webhook-service
* Do not instantiate middleware when passed to express
* Refactor to move logging into shared
* Copy shared packages into dockerfiles
* Build shared workspace in docker build-stage for fileimport & webhook
2022-12-05 14:49:52 +00:00
Iain Sproat 63e6581162 Revert "feat(structured logging): implements structured logging for backend (#1217)" (#1227)
This reverts commit 84cb74e8b3.
2022-11-25 16:57:28 +00:00
Iain Sproat 84cb74e8b3 feat(structured logging): implements structured logging for backend (#1217)
* each log line is a json object
* structured logging allows logs to be ingested by machines and the logs to be indexed and queried addresses #1105
* structured logging allows arbitrary properties to be appended to each log line, and ingestion of logs to remain robust
* Structured logging provided by `pino` library
* Add `express-pino-logger` dependency
* Remove `debug`, `morgan`, and `morgan-debug` and replace with structured logging
* `console.log` & `console.error` replaced with structured logging in backend
* Remove `DEBUG` environment variable and replace with `LOG_LEVEL`
- Note that there is a test which reads from a logged line on `stdout`. This is not robust, it would be better to use the childProcess.pid to look up the port number.
* Log errors at points we explicitly send error to Sentry
* Amend indentation of a couple of log messages to align indentation with others
2022-11-25 16:05:05 +00:00
Iain Sproat d755136f19 Upgrade postgres in DO 1click template to 14.5 (#1090) 2022-10-21 09:51:50 +01:00
Iain Sproat c59084f4fd Upgrades redis to 7.0.5 in circleci & DO 1click configuration (#1087)
* Upgrades redis to 7.0.5 in docker-compose & circleci

* Upgrade redis on minikube to 7.0
2022-10-20 12:53:12 +01:00
Gergő Jedlicska 393a192940 gergo/emailDigestFixes (#1074)
* feat(server task scheduler): sketch out core task scheduler implementation

* feat(server weekly activity digests): add function lock duration to the weekly digest execution

* feat(server scheduled tasks): add scheduled tasks type definition, db schema and migration

* feat(server scheduled tasks): add scheduled tasks repository

* feat(server task scheduler): add task scheduler service implementation

* chore(server deps): add mocha type definitions

* refactor(server scheduled tasks): refactor scheduled tasks migration

* refactor(server scheduled tasks): refactor scheduled task db schema and type definitions

* feat(server scheduled tasks): implement db side lock acquire

* refactor(server scheduled tasks): refactor task scheduler with lock on query mechanism

* test(server scheduled tasks): add tests for scheduled tasks implementation

* refactor(server weekly activity digests): refactor to new task scheduler implementation

* feat(server weekly activity digest): switch to a 1000 seconds trigger period for testing purposes

* fix(server task scheduler): fix not catching lock acquire function errors

* feat(server weekly digest): switch weekly digest cron trigger to the prod ready value

* fix(nginx configs): fix missing static route proxy to backend

* fix(server email template): fix footer anchor tags not pointing to the right places
2022-10-05 10:09:24 +02:00
Iain Sproat da7dafe819 fix(fileimport service): s3 is not required by fileimport service (#924)
Fileimport service retreives blobs via the server storage API, and not directly from s3.  Fileimport
service no longer requires information or credentials about s3.
2022-08-15 15:49:10 +01:00
Peter Grainger 72d27b9a7c Allow save object to S3 in different region (#910)
* Allow save object to S3 in different region

* feat(helm & docker-compose): adds S3_REGION to helm chart & docker-compose

Explicitly adding the environment variable to deployment configuration files provides system operators with documentation of its existence.

Set to empty by default, which will result in the default value being used.

Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
2022-08-15 14:24:30 +02:00
Iain Sproat 427050a15d chore(pre-commit): Add shellcheck to pre-commit hook (#850)
* chore(pre-commit): adds shellcheck
* Resolves shellcheck linting issues in existing files
2022-08-09 15:00:51 +01:00
Iain Sproat 90847e422d Feat: configurable file limits (#835)
* Feat: configurable file limits

* ci(circleci): container build speed imporvements

* feat(frontend nginx): add file size limit configurability to frontend nginx

* feat(server blobstorage): use the new file size limit customization value

* feat(helm chart): implement the file size configuration in the helm chart

* fix(frontend docker): fix entrypoint script

* fix(server blobstorage): fix env var parsing NaN

* feat(fileimport-service): add customizable import timeout

* feat(helm chart): add fileimport service timeout value to helm chart

* feat(blobstorage): add server side blob storage size limits

* feat(docker-compose): add blob size limit env var to  docker-compose files

* refactor(frontend file uploads): refactor file uploads to use `useQuery`

* refactor(server env helper): move env helper to shared module

* refactor(blobstorage): use env helper for file size limit

* refactor(frontend file uploads): use generated query document

* fix(server blob sotrage): fix file size limit function call

Co-authored-by: Gergő Jedlicska <gergo@jedlicska.com>
2022-07-29 12:00:29 +02:00
Fabians abe6d5e086 chore: re-formatted everything with prettier 2022-03-30 12:54:24 +03:00
cristi8 e840060d8a Added FileImport Service to DigitalOcean 1click image 2021-11-05 12:33:09 +02:00
Cristian Balas 0723e70edc DO 1click: Fix setup.py executable delay at first boot 2021-10-20 13:54:48 +03:00
Cristian Balas 1d6fc56052 DO OneClickImage - added webhook service (#358) 2021-07-27 12:46:39 +03:00
Cristian Balas 37f18be68d Added memory limits to previewservice (#349) 2021-07-21 10:39:16 +03:00
Cristian Balas 3b227713da [nginx] disabled request buffering and set max-body-size at the server level (#332) 2021-07-13 11:40:41 +03:00
Cristian Balas 3840068cad diff endpoints + added version in ServerInfo (#235) 2021-05-11 20:23:42 +03:00
Cristian Balas 2172a06e4f Scripts for DigitalOcean oneclick image (#209) 2021-05-03 15:21:37 +03:00