Commit Graph

44 Commits

Author SHA1 Message Date
Gergő Jedlicska a5d624166b feat(backgroundjobs): add new background jobs module for file imports (#5041)
* feat(backgroundjobs): add new background jobs module for file imports
queueing

* fix(fileuploads): a merge gone wrong

* feat(backgroundjobs): rename rhino queue env var

* test(backgroundjob): use deep equal claude

* fix(fileuploads): sync PR review

* feat(backgroundjobs): add origin server url to the background jobs table

* test(backgroundjobs): make tests pass
2025-07-10 18:04:16 +01:00
Iain Sproat f769ff4572 fix(server/fileuploads): longer time out before expiry, from last converted time (#5039) 2025-07-08 09:42:36 +01:00
Kristaps Fabians Geikins 290fdb3e6d feat(server): dev-only endpoint for monitoring bull queues (#5000)
* feat(server): dev-only endpoint for monitoring bull queues

* cr comments
2025-06-30 16:12:45 +03:00
Kristaps Fabians Geikins 0ae4ba63a9 fix(server): externally managed DB connection for notifs (#5002)
* fix(server): externally managed DB connection for notifs

* weakMap based listener tracking
2025-06-27 17:31:32 +03:00
Kristaps Fabians Geikins c8bbab864a chore(server): improved db notif logging for issue troubleshooting (#5001) 2025-06-27 15:41:30 +03:00
Kristaps Fabians Geikins fd1a3cf341 fix(server): using project scoped db for fileupload sub handlers (#4994) 2025-06-26 17:56:15 +03:00
Kristaps Fabians Geikins ed507c265b fix(server): not firing upload created/processed subs in next gen uploads (#4991)
* fix(server): not firing upload created subs in next gen uploads

* lint fixes

* upload finished fixes

* test fixes

* fixed file_import_started
2025-06-26 15:33:11 +03:00
Iain Sproat 4784fdec4f chore(file imports): metrics for next gen file importer (#4977) 2025-06-25 09:36:28 +01:00
Gergő Jedlicska bb82c75745 fix(fileuploads): use the correct DB for tokens (#4971) 2025-06-20 16:26:13 +01:00
Iain Sproat d7d564d0d4 feat(file imports): next gen allows multiple file types (#4953) 2025-06-19 08:50:42 +01:00
Kristaps Fabians Geikins 2be1592341 feat: show historical model uploads (#4954)
* ensure modelId is always filled

* BE seems fine

* frontendd

* pagination fix

* table max height

* lint fix

* unset tiny limit

* test fix
2025-06-18 10:56:33 +03:00
Kristaps Fabians Geikins 441aa6e54d fix: create model before uploading file (#4942) 2025-06-16 10:17:17 +03:00
Kristaps Fabians Geikins f30ef2f9fa feat: set model name/description on file upload (#4937)
* WIP except description

* description implemented

* added defaultTo to migration
2025-06-13 13:44:40 +03:00
Iain Sproat d20cde3a68 feat(fileimport): provide admin dashboard for queue (#4904)
* feat(fileimport): provide admin dashboard for queue

* mitigate deprecation warning

* fix typo in GitHub Action
2025-06-09 13:36:43 +02:00
Daniel Gak Anagrov 23b61769b7 feat(fileupload): added eventBus and mixpanel monitoring 2025-06-02 08:22:20 +02:00
Daniel Gak Anagrov 82547b1411 feat(fileuploads): nexGenFileUpload populates messages to new job queue (#4702)
* add file import job to nextGen import endpoint

*  moved queue, organized code

* add tests, minor refactoring
2025-05-14 12:32:54 +02:00
Iain Sproat 8a74d9cbe3 feat(file importer): server has REST API to receive results about a file import (#4688)
* fix(server): remove redundant code from bad merge

* wrong router

* WIP: http api for receiving file import results

* Lints

* fix

* WIP: tests

* test fix

* auth middleware has implicit requirement for param named streamId

* complete tests and fixes

* can run the old way alongside the existing, no conflicts

* tidying

* correct error thrown

* incorporate review comments
2025-05-08 19:12:15 +02:00
Iain Sproat 4529b6b37b Merge branch 'main' into iain/refactor-blob-storage 2025-04-26 08:19:21 +01:00
Iain Sproat 74f54bde13 Address review comments 2025-03-31 13:43:49 +01:00
Iain Sproat f75b77e899 push work on to database, simplify application logic 2025-03-31 12:03:43 +01:00
Iain Sproat 5960788da1 add test 2025-03-31 09:53:36 +01:00
Iain Sproat 7376eb4298 Use constant for event name 2025-03-28 15:19:50 +00:00
Iain Sproat c19c7892f4 Refactor REST to different directory and register router 2025-03-28 15:13:59 +00:00
Iain Sproat d8ec2f12c4 feat(file import): expire failed file imports if taking too long 2025-03-28 15:05:44 +00:00
Iain Sproat 6499a11d02 Merge branch 'main' into iain/refactor-blob-storage 2025-03-06 09:12:20 +00:00
Iain Sproat b5201fe8d5 Restore incorrectly deleted db listener 2025-03-06 08:45:32 +00:00
Iain Sproat 5655b3e758 fix(metrics): pass registry in to modules (#4113)
- ensure metrics are added to the correct registry
2025-03-06 08:37:05 +00:00
Iain Sproat 89d61eedc0 misunderstood this event listener, reverted 2025-03-05 21:20:56 +00:00
Iain Sproat 7c6ebc95dc refactor fileuploads to avoid http request 2025-03-05 16:39:23 +00:00
Iain Sproat 6afad05430 refactor(blobstorage): separate REST components and services 2025-03-05 14:21:50 +00:00
Iain Sproat ce08e5e02d refactor(server): logging directory renamed to observability (#4077) 2025-02-28 08:43:54 +00:00
Kristaps Fabians Geikins 1d2a594f0a chore: upgrade TS 5.2 -> 5.7.3 & ESLint to 9.20.1 (#4032)
* chore: upgrade TS 5.2 -> 5.7.3

* vite dts fix

* lint fix

* resolutions fix

* ui comp build fix

* precommit fix?

* latest eslint version

* autoloader fix

* undo unnecessary viewer change

* eslint fixes fe2 + trying disabled type linting

* lint fixes
2025-02-20 14:18:18 +02:00
Kristaps Fabians Geikins fe3982b6cf undo ts check 2025-02-17 13:55:43 +02:00
Kristaps Fabians Geikins 4278e20467 chore(server): refactor activityStream invocations - batch #3 - branches 2025-02-17 13:55:43 +02:00
Iain Sproat 35eb2090bc fix(POST /api/file): REST API endpoint returns json content-type (#3463) 2025-01-02 10:15:39 +00:00
Kristaps Fabians Geikins 8d0cbad8b6 feat(server): multi region blob support (#3653)
Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
2024-12-10 09:05:18 +00:00
Chuck Driesler fd5f316af0 fix(automate): automate module multi region (#3531)
* refactor(automate): logs api can get the projectId from the path

* fix(automate): multiregion gql resolvers

* fix(automate): multiregion event listeners

* fix(automate): drop automationCount

* fix(automate): multiregion run status

* fix(automate): correctness

* fix(automate): actually finish event listeners

* chore(automate): fix tests fix tests

* fix(automate): fix tests but make it multiregion flavor

* fix(automate): logs endpoint

* chore(automate): globalDb to db

* fix(automate): inject projectid correctly

* fix(automate): debug log fetch failure

* chore(automate): fix tests for new message

---------

Co-authored-by: Gergő Jedlicska <gergo@jedlicska.com>
2024-11-27 15:26:09 +00:00
Gergő Jedlicska 68b8341945 gergo/multiRegionFileImport (#3474)
* feat(webhook-service): learn to speak multi region

* feat(fileimport-service): talk multi region to me

* feat(fileuploads, blobs): multi region

* feat(fileimport-service): multi region fixes

* feat(branchesAndCommits): multi region resolvers

* fix(fileimports): no need for ts ignore

* fix(fileimports): fix pr comments
2024-11-13 14:20:25 +01:00
Iain Sproat 0ec4a87c7c fix(file import): server port was broken when localhost was canonical url (#3360)
* fix(file import): server port was broken when localhost was canonical url
- use getServerOrigin() instead of directly calling process.env.CANONICAL_URL
- where server is calling itself, use 127.0.0.1 instead of CANONICAL_URL.
- where server is calling itself (i.e. 127.0.0.1, use the server's listening port and not default for protocol http/80, https/443
- tidy some areas where process.env was called directly; should use envHelper.

* fix tests
2024-10-22 21:12:18 +02:00
Alessandro Magionami a2e9693a6a chore(activitystream): addBranchCreatedActivity refactor multiregion 2024-10-22 09:52:47 +02:00
Kristaps Fabians Geikins 5b5dd201e2 chore(server): IoC 29 - getStream(s)Factory 2024-10-09 10:39:00 +03:00
Kristaps Fabians Geikins 0f27d41cae chore(server): core IoC 4 - getStreamBranch(es)ByNameFactory 2024-09-26 15:26:48 +03:00
Kristaps Fabians Geikins c1a87bd6ce chore(server): shared IoC 2 - authz pipelines 2024-09-25 15:59:41 +03:00
Kristaps Fabians Geikins 675bec46a0 module index ts refactor 2024-09-11 14:51:51 +03:00