feat(viewer): objectloader2 integration (#4267)

* feat(viewer-sandbox): Made a sandbox function that will only invoke the object-loader loading objects * first pass of creating an objectloader2 * updated build + added vitest * try to get viewer sandbox to use new code * sandbox type fix * refactor a bit * can download root * intermediate commit for downloader/caching queue * can download stuff! * refactor files * intro isBase and fix isString * move single download to downloader * fix download * PR feedback * some intermediate commit * do clean up and download better * clean up promises and linting * can generate values while downloading and caching * add a finish method * remove unused functions * remove asBase * add temporary docs * add more docs with mermaid * add more test models * add response validation * add tests and redo options * add test for download batch * fix downloader tests and change Item to have clearer Base items * add tests and refactor a little * use fetch in downloader as an option * use optional in-memory indexdb instead of monkey patching the global one * more refactors for options for objectloader2 * add tests for objectloader2 * adjust single download * benchmark loading and adjust ol2 batches * download more! * adjust to use hash privates * refactored again with renaming * cleanup * make setupCacheDb throw instead * use BatchedPool for downloads! * fix tests * adjust timings and add adaptive waiting * Only wait if queue wasn't empty and queue size was full * fix tests * fix file names and some private usage * fix interval and private usage * rename vars * use params for methods * fix params for constructors and tests * fix params for constructors and tests again * using dexie * faster settings but doesn't end well * fixed end, optimized and removed logs * fix tests * fix types? * update lock with WSL * add e2e small model test * fix/update yarn.lock * Remove unused eslint ignore to fix pre-commit * prettier fixes * fix real DB usage * rename methods to better match OL1 * rename methods to better match OL1 again * add extra header collection * add headers correctly * test getTotalObjectCount * feat(viewer-lib): Replaced old object loader with Adam's objectloder2 * fix(viewer-lib): Removed the old object loader. Removed unneeded pause time in speckle loader * Testing * only deferred if not downloaded....don't save everything * Lockfile * pool isn't adjustable, adjust download buckets, dexie read is faster * chore * fix(viewer-lib): Fixed compiler errors * fix getObject access with real indexeddb...adjust buffer for deferred access * Fix disposal and pausing * don't index item! * fix dockerfiles to use OL2 * fix Dockerfile * Fix dockerfile * defer correctly and use record to add/lookup/remove to * delete stuff correctly * chore(sandbox): Enabled viewer loading * use objects instead of arrays to avoid findIndex * remove extra count * add a found cache to avoid some db hits * order matters for deferment * move found map to deferment * change option numbers * 2 level cache with expiry * defer everything, use loader to track what is requested....expire only found items * add deferment disposal * oops mismerge * chore(sandbox): Default stream * Beta version of CachePump and CacheReader * Clean up initialization * More clean up * chore(objectloader2): Fixed CI compiler error * chore(objectloader2): Fixed prettier * add cachePump tests * add cacheReader tests * fixed more tests * fixed final tests * moving stuff around and lock return value * try to move stuff out of objectloader2 * use a factory * rename factory * formatting * eslist fixes * try allocating no strings * add comments * small refactor and add another test * fix deferment expiration and have test * use byte size for max memory cache size * fix deferment manager tests * saved comment * fix(viewer-sandbox): Fixed compiler error * ignore tshy * chore(frontend): Attempt to make viewer loading sequential --------- Co-authored-by: Adam Hathcock <adam@hathcock.uk> Co-authored-by: Kristaps Fabians Geikins <fabis94@live.com> Co-authored-by: Iain Sproat <68657+iainsproat@users.noreply.github.com>
2025-05-21 10:05:50 +03:00
parent aaa4e1ab71
commit a385823b2d
51 changed files with 1317 additions and 782 deletions
@@ -1,7 +1,7 @@
-import Queue from './queue.js'
+import KeyedQueue from './keyedQueue.js'

-export default class BatchingQueue<T> implements Queue<T> {
-  #queue: T[] = []
+export default class BatchingQueue<T> {
+  #queue: KeyedQueue<string, T> = new KeyedQueue<string, T>()
  #batchSize: number
  #processFunction: (batch: T[]) => Promise<void>

@@ -30,28 +30,32 @@ export default class BatchingQueue<T> implements Queue<T> {
    await this.#processingLoop
  }

-  add(item: T): void {
-    this.#queue.push(item)
+  add(key: string, item: T): void {
+    this.#queue.enqueue(key, item)
+  }
+
+  get(id: string): T | undefined {
+    return this.#queue.get(id)
  }

  count(): number {
-    return this.#queue.length
+    return this.#queue.size
  }

  #getBatch(batchSize: number): T[] {
-    return this.#queue.splice(0, Math.min(batchSize, this.#queue.length))
+    return this.#queue.spliceValues(0, Math.min(batchSize, this.#queue.size))
  }

  async #loop(): Promise<void> {
    let interval = this.#baseInterval
-    while (!this.#finished || this.#queue.length > 0) {
+    while (!this.#finished || this.#queue.size > 0) {
      const startTime = performance.now()
-      if (this.#queue.length > 0) {
+      if (this.#queue.size > 0) {
        const batch = this.#getBatch(this.#batchSize)
        //console.log('running with queue size of ' + this.#queue.length)
        await this.#processFunction(batch)
      }
-      if (this.#queue.length < this.#batchSize / 2) {
+      if (this.#queue.size < this.#batchSize / 2) {
        //refigure interval
        const endTime = performance.now()
        const duration = endTime - startTime
@@ -62,7 +66,7 @@ export default class BatchingQueue<T> implements Queue<T> {
        }
        /*console.log(
          'queue is waiting ' +
-            interval / TIME_MS.second +
+            interval / 1000 +
            ' with queue size of ' +
            this.#queue.length
        )*/