speckle-server/docs/viewer/large-model-streaming-tasks.md

Large Model Streaming Tasks

Nguon plan: large-model-streaming-plan.md

Trang thai ngay 2026-05-07:

• Luu plan chi tiet trong repo.
• Tao derivative schema/API backend co status queued, processing, ready, failed.
• Converter sinh manifest va tile binary SPKTILE1.
• Worker upload artifact streaming theo chunk len server.
• Frontend dung derivative loader khi manifest ready.
• Chay end-to-end tren file IFC lon va ghi benchmark RAM/time.
• Runtime viewer chi load tile can thiet theo camera/frustum. Initial load now uses a bounded tile working set, but not camera-driven reconciliation yet.
• Evict tile/GPU memory theo budget. Initial load now has a configurable fetch/build budget, but no runtime GPU eviction yet.
• LOD generation that su.
• Metadata/property streaming theo selection.
• Cleanup/GC derivative artifacts cu.

Task 1: Make Derivative MVP Reliable

Goal: upload IFC lon, convert xong, vao viewer khong full-load object graph va render duoc derivative.

• Backend route GET manifest.
• Backend route PUT artifacts/*.
• Backend route POST publish.
• Backend route POST fail.
• Worker chunk upload artifact.
• Worker mark failed neu upload/publish loi.
• Add integration test cho repository + route status transitions.
• Validate object storage key safety va missing artifact behavior.
• Add logs/metrics cho artifact upload bytes, tile count, publish duration.

Task 2: Converter Memory And Tile Quality

Goal: converter khong can giu full geometry trong RAM managed va tile co kich thuoc hop ly.

• Native geometry probe.
• Native geometry stream callback.
• Multi-tile writer theo IFC_VIEWER_DERIVATIVE_TARGET_TILE_MB.
• Record per-tile bbox, mesh count, vertex count, index count.
• Split tile theo spatial grid/octree, khong chi theo byte threshold.
• Add metadata sidecar mapping expressId -> tile/mesh/globalId.
• Benchmark peak RSS tren file 250 MB.

Task 3: Viewer Runtime Streaming

Goal: browser khong tai toan bo derivative, chi tai tile can thiet.

• LargeModelStreamingLoader parse manifest va SPKTILE1.
• Frontend route ready derivative sang streaming loader.
• Parse tile data khong giu duplicate ArrayBuffer references lau hon can thiet.
• Load tile theo concurrency limit.
• Cancel/abort tile fetch khi unload model hoac route viewer doi resource.
• Frustum/section-box tile selection tu manifest bbox. Manifest now has tile bbox and initial-load ranking uses it, but not live camera/section culling yet.
• Tile unload + GPU dispose khi vuot memory budget. Loader now caps initial tile set by bytes/count; runtime eviction still pending.
• Progressive first render voi LOD thap.

Task 4: Selection And Metadata

Goal: click/select object tren derivative van ra duoc metadata can thiet.

• Preserve expressId/meshId trong render node id.
• Build metadata index sidecar.
• Add backend route GET metadata/:elementId.
• Frontend selection resolver cho derivative.

Task 5: Production Hardening

Goal: chay duoc tren server 16 GB RAM voi retry/cleanup.

• Feature flag cho derivative generation va derivative viewer.
• Retry-safe generation id/run id.
• GC artifacts khi version/model/project bi xoa.
• Admin/debug endpoint xem derivative status.
• Playwright benchmark: first render, peak JS heap, GPU memory proxy, FPS.