68 Commits

Author SHA1 Message Date
Alexandru Popovici 06c5529024 Alex/ol2 no hardcode (#5730)
* fix(object-loader): WIP on removing the hardocoding.

* feat(object-loader-2): Object skipping by type is not added as an option. By default OL2 will not skip any objects. Added server filters and skippable objects to sandbox loading
2025-11-05 15:07:39 +01:00
Gergő Jedlicska 1a1ba3d592 feat(viewer): expose object loader attribute masking to the viewer (#5715) 2025-10-08 15:46:17 +02:00
Adam Hathcock 49a0faa050 fix test 2025-10-01 08:30:27 +01:00
Adam Hathcock 63a819ad60 add more tests for draining 2025-09-30 13:12:15 +01:00
Adam Hathcock 02ae8c38e9 fix tests 2025-09-30 13:05:13 +01:00
Adam Hathcock a657687b84 Merge remote-tracking branch 'origin/adam/fix-tests-and-batchingqueue' into adam/fix-batchingqueue-error
# Conflicts:
#	packages/objectloader2/src/queues/batchingQueue.test.ts
#	packages/objectloader2/src/queues/batchingQueue.ts
2025-09-30 13:00:57 +01:00
Adam Hathcock 3bd835f03a Erroring ends processing and adding, Processing just skips itself and Disposal does flush 2025-09-30 12:16:19 +01:00
Adam Hathcock d7f87076cc Fixes tests and batchingqueue during exceptions 2025-09-25 06:29:15 +01:00
Adam Hathcock 9658574b09 prettier 2025-09-23 13:10:48 +01:00
Adam Hathcock c5cbbe585a Fix TS errors 2025-09-23 12:58:59 +01:00
Adam Hathcock 9b03977bb8 fix build issues 2025-09-23 10:49:34 +01:00
Adam Hathcock 0e7d71dd70 add id and prettier 2025-09-23 10:33:30 +01:00
Adam Hathcock 194b5c9945 refactor and add test for getObject not found 2025-09-23 10:29:54 +01:00
Adam Hathcock db005b3fa8 Merge remote-tracking branch 'origin/main' into adam/add-ol2-options 2025-09-23 09:44:31 +01:00
Adam Hathcock afbe7e49d4 add assertion 2025-09-22 10:21:34 +01:00
Adam Hathcock f1e3b1d6ff BatchingQueue should still drain but not process when disposed 2025-09-22 10:17:25 +01:00
Adam Hathcock 872879518a add back masking option 2025-09-18 15:10:30 +01:00
Adam Hathcock 89fc312839 maybe fix boolean logic for cache 2025-09-18 15:05:10 +01:00
Gergő Jedlicska c3d452ad0c Gergo/cxpla 278 v2 server api endpoint with masking support (#5414)
* WIP: streaming masking endpoint

* feat(objects): working object streaming with filtering

* feat(object-loader): use the new objects endpoint

* fix(server): allow downloading of public project data

* feat(objectloader-2): include object masking in objectloader

* feat(objectloader2): make attribute masking optional

* fix(server): remove unused imports
2025-09-18 15:56:41 +02:00
Adam Hathcock f41719f7e6 make boolean logic correct 2025-09-18 14:48:06 +01:00
Adam Hathcock 3b2cd031ed Merge remote-tracking branch 'origin/gergo/cxpla-278-v2-server-api-endpoint-with-masking-support' into adam/add-ol2-options
# Conflicts:
#	packages/objectloader2/src/core/objectLoader2Factory.ts
2025-09-18 14:41:53 +01:00
Adam Hathcock ed29ead354 add more options to factory 2025-09-18 14:39:23 +01:00
Gergo Jedlicska b0ed356282 feat(objectloader2): make attribute masking optional 2025-09-17 18:00:59 +02:00
Gergo Jedlicska 2a43ce23d4 feat(objectloader-2): include object masking in objectloader 2025-09-17 17:56:40 +02:00
Gergo Jedlicska 7b5c2e47a2 Merge branch 'main' of github.com:specklesystems/speckle-server into gergo/cxpla-278-v2-server-api-endpoint-with-masking-support 2025-09-17 17:01:52 +02:00
Gergo Jedlicska 010f4abe37 feat(object-loader): use the new objects endpoint 2025-09-15 12:50:47 +02:00
Adam Hathcock 44d80203c1 handle errors when downloading in batchingQueue 2025-09-12 14:04:15 +01:00
Adam Hathcock 812dac9920 OL2(fix) Use BatchingQueue instead of BatchedPool as parsing download happens on main thread asap anyway (#5165)
* Use BatchingQueue instead of BatchedPool as parsing download happens on main thread anyway

* add tests for keyedqueue

* clean up

* more prettier fix
2025-08-26 12:07:02 +01:00
Adam Hathcock cab2a401db OL2 (fix) Simplify idb (#5174)
* Simplify idb usage and collapse the class

* fix tests

* fmt
2025-08-26 12:06:47 +01:00
Adam Hathcock 9220a3ee4b ol2(feat) remove dexie dependency (#5148)
* add new indexed db wrapper

* remove dexie and use new idb

* change storage durability

* allow undefined gets

* some clean up

* linting fixes

* add db close

* cleaner upgrade DB and more clean up

* fix database issues by deleting things instead of upgrading them
2025-07-28 15:57:07 +01:00
Adam Hathcock 7c1b9fa035 OL2(fix) Make a returned RawEncoding value return an empty Item instead of a undefined item (#5145)
* Make a returned RawEncoding value return an empty Item instead of a undefined item

* add generated tests and also cover RawEncoding case
2025-07-28 11:31:41 +01:00
Adam Hathcock 40e2b8148d OL2 (feat): dont return or save rawencoding type (#5143)
* Skip Objects.Other.RawEncoding when found

* ensure raw encoding is skipped

* Update packages/objectloader2/src/core/stages/serverDownloader.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Don't encode bytes if we don't have to

* oops, use const

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-25 14:37:55 +01:00
Adam Hathcock 167bbed7be (fix) debug mode for viewer (#5142)
* Fix the debug usage for the viewer

* formatting
2025-07-25 10:50:38 +01:00
Adam Hathcock 877266bca7 OL2 (feat): useCache option and flag clean up (#5133)
* add addAll

* change to useCache query option which defaults to true

* add documentation

* called query params feature flags and fixed usage

* fixed debug logging

* eslint and prettier fixes

* eslint and prettier fixes

* revert

* Update packages/viewer-sandbox/src/Sandbox.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-23 15:13:24 +01:00
Adam Hathcock cc54b42a70 OL2 (fix): undefer on write (#5127)
* undefer only on write, not download

* fix test and format
2025-07-22 11:41:03 +01:00
Adam Hathcock 67bd1341aa dispose cache as it should (#5119) 2025-07-21 14:59:58 +01:00
Adam Hathcock 477db6ef02 All BatchedQueues should drain when disposed (also adds query string for output: "debug=true") (#5098)
* ensure disposal is correct

* add tests for disposal of batching queue

* fixes for draining disposal

* Update packages/objectloader2/src/queues/batchingQueue.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix comment

* fix tests and build

* add query string inspection of debug parameter

* Update packages/objectloader2/src/queues/batchingQueue.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update packages/objectloader2/src/core/objectLoader2Factory.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix test

* fix AI

* export getQueryParameter to avoid dup code.  Sandbox uses it too

* add tests for functions

* prettier fix

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-21 12:01:27 +01:00
Adam Hathcock 390e587709 Cache By Reference count (#5096)
* we don't want to break eviction checking for first item...continue the check

* always include size when adding

* scan for references and count them...request when found and don't clean up if referenced

* add display of closure calculation

* add always output to sandbox

* formatting fixes

* removed tracking of deferments, moved caching to separate concern and class

* fixed cache checking and simplifed deferment

* add tests for new deferment and cache

* formatting

* Update packages/objectloader2/src/deferment/BaseCache.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update packages/objectloader2/src/deferment/BaseCache.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update packages/objectloader2/src/core/objectLoader2.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update packages/objectloader2/src/core/objectLoader2.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update packages/objectloader2/src/core/objectLoader2.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix AI suggestions

* rename BaseCache to MemoryCache

* use private method

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-21 11:31:21 +01:00
Adam Hathcock 005b6224d0 adjust waits and actually save the root last (#5056)
* adjust waits and actually save the root last

* Update packages/objectloader2/src/core/objectLoader2.ts

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-10 11:10:40 +01:00
Adam Hathcock f86893935f Change and fix BatchingQueue implementation (#5044)
* BatchingQueues allowed processing to happen more than one at a time.  Simplify and don't allow this.

* Use proper logging

* clean up and fix tests

* add more batchingqueue tests

* fixed according to AI

* fix: linting issues

* make deferment more intelligent and fix logging

* add to deferment tracking when bulk adding reads

* format

* fix tests
2025-07-10 10:13:58 +01:00
Adam Hathcock 1babe5859d safari fix: TextDecoder needs to be reset near 2 Gigs (#5048)
* safari fix: TextDecoder needs to be reset near 2 Gigs

* remove extra try/catch

* add back decoding for line error

* prettify
2025-07-10 10:13:24 +01:00
Adam Hathcock c5967a9616 (OL2) move files around to make more sense (#4950)
* Rename to saveBatch

* forgot a file

* first pass of cacheReader

* OL2 tests have infinite timeout

* OL2 refactor works

* fix for tests

* moved/removed types to make a more logical structure

* fixed imports

* rework loop to be in async generator for the expected count

* get rid of pumps and fix test

* lint fix

* redo mermaid diagrams

* add readme section on deferment

* always return root first

* fix linting

* revert the counting

* merge fixes

* remove unused var
2025-06-26 13:28:50 +01:00
Adam Hathcock 312f8083ab Progress for Traversal and Rendering alongside Data loading (#4899)
* Fix deferment logging

* Add count and total for traversal and rendering

* Prettier

* ESlint fixes

* Don't have a total for traversal

* revert lock change

* chore(viewer-lib): Added a comment abou render trees and reported node counts

* Traversal has no total

* ignore conversion total

* remove total for converted

* forgot a build breakage

* make the count callback optional

---------

Co-authored-by: AlexandruPopovici <alexandrupopoviciioan@gmail.com>
2025-06-19 10:08:38 +01:00
Adam Hathcock 2b828a5eeb (OL2) refactor read queue (#4948)
* Rename to saveBatch

* forgot a file

* first pass of cacheReader

* OL2 tests have infinite timeout

* OL2 refactor works

* fix for tests

* get rid of pumps and fix test

* lint fix

* redo mermaid diagrams

* add readme section on deferment
2025-06-17 08:24:05 +01:00
Adam Hathcock b11724eb6e fix(OL2): Deferment cleanup has to be a timer to not be concurrent (#4896)
* Deferment cleanup has to be a timer to not be concurrent

* Remove obsolete snapshot
2025-06-05 11:45:20 +01:00
Adam Hathcock 0b83f5e05d Deferment cleaning happens only on undefer and clean takes request count into account (#4886)
* Clean deferments on undefer

* Sort closures

* Track referenced item counts

* add tests

* run prettier again
2025-06-05 10:02:10 +01:00
Adam Hathcock 51dab0f4ee Remove getItem as we should use getItems with an array (#4862)
* Remove getItem as we should use getItems with an array

* remove getItem tests

* Update to latest and test

* Test memory downloader and memory db
2025-06-03 08:07:56 +01:00
Alexandru Popovici 3ca6757fdc fix(object-loader): Reduced max id request count to 25k to avoid 413 (#4871) 2025-06-02 15:05:04 +03:00
Adam Hathcock 36397a757c Add createFromUrl that can use memory database (#4870) 2025-06-02 12:57:30 +01:00
Adam Hathcock 9bb420c08b OL2: Dispose pump inline with the rest when disposing to kill loop (#4861) 2025-05-30 14:31:01 +01:00