* add new indexed db wrapper
* remove dexie and use new idb
* change storage durability
* allow undefined gets
* some clean up
* linting fixes
* add db close
* cleaner upgrade DB and more clean up
* fix database issues by deleting things instead of upgrading them
* Skip Objects.Other.RawEncoding when found
* ensure raw encoding is skipped
* Update packages/objectloader2/src/core/stages/serverDownloader.ts
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Don't encode bytes if we don't have to
* oops, use const
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* we don't want to break eviction checking for first item...continue the check
* always include size when adding
* scan for references and count them...request when found and don't clean up if referenced
* add display of closure calculation
* add always output to sandbox
* formatting fixes
* removed tracking of deferments, moved caching to separate concern and class
* fixed cache checking and simplifed deferment
* add tests for new deferment and cache
* formatting
* Update packages/objectloader2/src/deferment/BaseCache.ts
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update packages/objectloader2/src/deferment/BaseCache.ts
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update packages/objectloader2/src/core/objectLoader2.ts
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update packages/objectloader2/src/core/objectLoader2.ts
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update packages/objectloader2/src/core/objectLoader2.ts
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix AI suggestions
* rename BaseCache to MemoryCache
* use private method
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* BatchingQueues allowed processing to happen more than one at a time. Simplify and don't allow this.
* Use proper logging
* clean up and fix tests
* add more batchingqueue tests
* fixed according to AI
* fix: linting issues
* make deferment more intelligent and fix logging
* add to deferment tracking when bulk adding reads
* format
* fix tests
* Rename to saveBatch
* forgot a file
* first pass of cacheReader
* OL2 tests have infinite timeout
* OL2 refactor works
* fix for tests
* moved/removed types to make a more logical structure
* fixed imports
* rework loop to be in async generator for the expected count
* get rid of pumps and fix test
* lint fix
* redo mermaid diagrams
* add readme section on deferment
* always return root first
* fix linting
* revert the counting
* merge fixes
* remove unused var
* Fix deferment logging
* Add count and total for traversal and rendering
* Prettier
* ESlint fixes
* Don't have a total for traversal
* revert lock change
* chore(viewer-lib): Added a comment abou render trees and reported node counts
* Traversal has no total
* ignore conversion total
* remove total for converted
* forgot a build breakage
* make the count callback optional
---------
Co-authored-by: AlexandruPopovici <alexandrupopoviciioan@gmail.com>
* Rename to saveBatch
* forgot a file
* first pass of cacheReader
* OL2 tests have infinite timeout
* OL2 refactor works
* fix for tests
* get rid of pumps and fix test
* lint fix
* redo mermaid diagrams
* add readme section on deferment
* feat(viewer-sandbox): Made a sandbox function that will only invoke the object-loader loading objects
* first pass of creating an objectloader2
* updated build + added vitest
* try to get viewer sandbox to use new code
* sandbox type fix
* refactor a bit
* can download root
* intermediate commit for downloader/caching queue
* can download stuff!
* refactor files
* intro isBase and fix isString
* move single download to downloader
* fix download
* PR feedback
* some intermediate commit
* do clean up and download better
* clean up promises and linting
* can generate values while downloading and caching
* add a finish method
* remove unused functions
* remove asBase
* add temporary docs
* add more docs with mermaid
* add more test models
* add response validation
* add tests and redo options
* add test for download batch
* fix downloader tests and change Item to have clearer Base items
* add tests and refactor a little
* use fetch in downloader as an option
* use optional in-memory indexdb instead of monkey patching the global one
* more refactors for options for objectloader2
* add tests for objectloader2
* adjust single download
* benchmark loading and adjust ol2 batches
* download more!
* adjust to use hash privates
* refactored again with renaming
* cleanup
* make setupCacheDb throw instead
* use BatchedPool for downloads!
* fix tests
* adjust timings and add adaptive waiting
* Only wait if queue wasn't empty and queue size was full
* fix tests
* fix file names and some private usage
* fix interval and private usage
* rename vars
* use params for methods
* fix params for constructors and tests
* fix params for constructors and tests again
* using dexie
* faster settings but doesn't end well
* fixed end, optimized and removed logs
* fix tests
* fix types?
* update lock with WSL
* add e2e small model test
* fix/update yarn.lock
* Remove unused eslint ignore to fix pre-commit
* prettier fixes
* fix real DB usage
* rename methods to better match OL1
* rename methods to better match OL1 again
* add extra header collection
* add headers correctly
* test getTotalObjectCount
* feat(viewer-lib): Replaced old object loader with Adam's objectloder2
* fix(viewer-lib): Removed the old object loader. Removed unneeded pause time in speckle loader
* Testing
* only deferred if not downloaded....don't save everything
* Lockfile
* pool isn't adjustable, adjust download buckets, dexie read is faster
* chore
* fix(viewer-lib): Fixed compiler errors
* fix getObject access with real indexeddb...adjust buffer for deferred access
* Fix disposal and pausing
* don't index item!
* fix dockerfiles to use OL2
* fix Dockerfile
* Fix dockerfile
* defer correctly and use record to add/lookup/remove to
* delete stuff correctly
* chore(sandbox): Enabled viewer loading
* use objects instead of arrays to avoid findIndex
* remove extra count
* add a found cache to avoid some db hits
* order matters for deferment
* move found map to deferment
* change option numbers
* 2 level cache with expiry
* defer everything, use loader to track what is requested....expire only found items
* add deferment disposal
* oops mismerge
* chore(sandbox): Default stream
* Beta version of CachePump and CacheReader
* Clean up initialization
* More clean up
* chore(objectloader2): Fixed CI compiler error
* chore(objectloader2): Fixed prettier
* add cachePump tests
* add cacheReader tests
* fixed more tests
* fixed final tests
* moving stuff around and lock return value
* try to move stuff out of objectloader2
* use a factory
* rename factory
* formatting
* eslist fixes
* try allocating no strings
* add comments
* small refactor and add another test
* fix deferment expiration and have test
* use byte size for max memory cache size
* fix deferment manager tests
* use a set of keys to determine if the whole batch was gathered
* saved comment
* fix(viewer-sandbox): Fixed compiler error
* remove extra disposal
* simplify the error handling
---------
Co-authored-by: AlexandruPopovici <alexandrupopoviciioan@gmail.com>
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
* feat(viewer-sandbox): Made a sandbox function that will only invoke the object-loader loading objects
* first pass of creating an objectloader2
* updated build + added vitest
* try to get viewer sandbox to use new code
* sandbox type fix
* refactor a bit
* can download root
* intermediate commit for downloader/caching queue
* can download stuff!
* refactor files
* intro isBase and fix isString
* move single download to downloader
* fix download
* PR feedback
* some intermediate commit
* do clean up and download better
* clean up promises and linting
* can generate values while downloading and caching
* add a finish method
* remove unused functions
* remove asBase
* add temporary docs
* add more docs with mermaid
* add more test models
* add response validation
* add tests and redo options
* add test for download batch
* fix downloader tests and change Item to have clearer Base items
* add tests and refactor a little
* use fetch in downloader as an option
* use optional in-memory indexdb instead of monkey patching the global one
* more refactors for options for objectloader2
* add tests for objectloader2
* adjust single download
* benchmark loading and adjust ol2 batches
* download more!
* adjust to use hash privates
* refactored again with renaming
* cleanup
* make setupCacheDb throw instead
* use BatchedPool for downloads!
* fix tests
* adjust timings and add adaptive waiting
* Only wait if queue wasn't empty and queue size was full
* fix tests
* fix file names and some private usage
* fix interval and private usage
* rename vars
* use params for methods
* fix params for constructors and tests
* fix params for constructors and tests again
* using dexie
* faster settings but doesn't end well
* fixed end, optimized and removed logs
* fix tests
* fix types?
* update lock with WSL
* add e2e small model test
* fix/update yarn.lock
* Remove unused eslint ignore to fix pre-commit
* prettier fixes
* fix real DB usage
* rename methods to better match OL1
* rename methods to better match OL1 again
* add extra header collection
* add headers correctly
* test getTotalObjectCount
* feat(viewer-lib): Replaced old object loader with Adam's objectloder2
* fix(viewer-lib): Removed the old object loader. Removed unneeded pause time in speckle loader
* Testing
* only deferred if not downloaded....don't save everything
* Lockfile
* pool isn't adjustable, adjust download buckets, dexie read is faster
* chore
* fix(viewer-lib): Fixed compiler errors
* fix getObject access with real indexeddb...adjust buffer for deferred access
* Fix disposal and pausing
* don't index item!
* fix dockerfiles to use OL2
* fix Dockerfile
* Fix dockerfile
* defer correctly and use record to add/lookup/remove to
* delete stuff correctly
* chore(sandbox): Enabled viewer loading
* use objects instead of arrays to avoid findIndex
* remove extra count
* add a found cache to avoid some db hits
* order matters for deferment
* move found map to deferment
* change option numbers
* 2 level cache with expiry
* defer everything, use loader to track what is requested....expire only found items
* add deferment disposal
* oops mismerge
* chore(sandbox): Default stream
* Beta version of CachePump and CacheReader
* Clean up initialization
* More clean up
* chore(objectloader2): Fixed CI compiler error
* chore(objectloader2): Fixed prettier
* add cachePump tests
* add cacheReader tests
* fixed more tests
* fixed final tests
* moving stuff around and lock return value
* try to move stuff out of objectloader2
* use a factory
* rename factory
* formatting
* eslist fixes
* try allocating no strings
* add comments
* small refactor and add another test
* fix deferment expiration and have test
* use byte size for max memory cache size
* fix deferment manager tests
* saved comment
* fix(viewer-sandbox): Fixed compiler error
* ignore tshy
* chore(frontend): Attempt to make viewer loading sequential
---------
Co-authored-by: Adam Hathcock <adam@hathcock.uk>
Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com>
Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
Adam Hathcock