Commit Graph

107 Commits

Author SHA1 Message Date
Adam Hathcock 2352306269 add ability for objectloader to skip server too. (#228)
* add ability for objectloader to skip server too.  add test so both cache and server can't be skipped

* move testing project to correct dir

* remove extra file
2025-02-13 09:41:07 +00:00
Adam Hathcock 3072047129 Use priority scheduler like serialize process to constrain Task enqueuing (#223)
* Use priority scheduler like serialize process to constrain Task enqueuing

* renamespace

* Fix tests and format
2025-02-05 15:36:26 +00:00
Adam Hathcock 316379b04c Update to AwesomeAssertions and update verify (#224)
* Update to AwesomeAssertions and update verify

* attempt to solve disposal problem
2025-02-05 14:03:28 +00:00
Adam Hathcock b056d98645 don't dispose the blocking collection when cancelling (#222)
* don't dispose the blocking collection

* dispose deserialize process correctly

* don't use enumerable and token

* format
2025-02-04 13:14:31 +00:00
Adam Hathcock 3aa993cecb Add cancellation tests (#218)
* Don't log cancelling

* redo exception handling for receive

* remove null test case

* clean up with Id/Json and more cancels

* Change the exception stacks

* fix serialization test

* make a custom scrubber for internalized exceptions

* clean up

* fix namespaces again :(

* adjust the scrubber

* try to make tests more predictable

* rework exceptions again

* strip out compile files used

* formatting

* custom exception validation

* fix init

* Move serialization to own class

* save serialize test

* add deep clean

* add cancellation test on save to cache

* cancellation tests

* format

* do DI correctly

* receive cancel works
2025-01-30 13:42:15 +00:00
Jedd Morgan f0c7169be8 Small QOL tweaks to Version.Create and ProjectModelsFilter (#210)
* Version Create now returns a Version

* ProjectModelsFilter now has optional args
2025-01-24 12:19:54 +00:00
Adam Hathcock 93912d6712 Fix cancellation in schedular. It's no longer optional (#213)
* Fix cancellation in schedular.  It's no longer optional

* formatting

* fix tests because of stacktrace
2025-01-24 11:24:51 +00:00
Adam Hathcock f81fc97a91 Add exception handling for SerializeProcess with CancellationTokenSource (#211)
* Add exception handling for SerializeProcess with CancellationTokenSource

* formatting

* add exception test to make sure we handle a server exception

* add extra exception and handling to stop

* add comment and another test

* one last chance for user to cancel

* formatting
2025-01-23 17:06:08 +00:00
Jedd Morgan 3a71a10cf1 First pass increasing test coverage (#209)
* point tests

* Mesh Tests and removal of unused functions

* Vector tests

* more tests
2025-01-21 14:19:10 +00:00
Adam Hathcock bafd130ece Snapshot testing (#208)
* snapshot testing with verify

* formatting

* add back old serialization tests

* pass verify

* use json correctly

* formatting

* Don't use Quibble and order ourselves because ordering doesn't matter

* whitespace on snapshot

* Better json diffing?  Quibble is back

* add common project

* add object unit tests to see how verify would work

* format

* move random exes to new solution folder

* update lock files
2025-01-16 13:46:53 +00:00
Adam Hathcock 69167dc206 update some deps and add locks to versions for known packages (#207) 2025-01-15 16:11:04 +00:00
Adam Hathcock 4370af5376 Allow SDK to report requested downloaded objects and minor adjustments (#203)
* Allow SDK to report requested downloaded objects and minor adjustments

* Remove xyz get server info test
2025-01-13 12:48:28 +00:00
Adam Hathcock 14d959834f Convert to Xunit (#196)
* xunit unit tests

* most pass with formatting

* convert objects to xunit

* remove nunit

* format

* merge fixes

* switch objects to fluent assertions

* update to fluent assertions

* more FA

* convert all to FA

* Format

* Fix tests

* formatting

* hopefully made credential test better

* Catch more specific exception

* use another more specific exception

* Fix tests

* update to xunit

* update packages
2025-01-09 15:32:28 +00:00
Adam Hathcock 465f635142 Add details to sqlite exceptions (#198)
* add ServerObjectManagerFactory

* add usage of a command pool

* add more disposal

* save saving increase

* fix tests

* fixes

* push out concurrency and disposablity

* Add a custom task scheduler

* Better usage, don't wait to enqueue to save to channels

* Completely pre-cal batch size to avoid spinning issues

* Try to fix cache counting

* properly dispose things

* format

* clean up

* adjust count and save on current thread

* move batch it's own file

* update a few packages

* fix build and add batch tests

* revert and format

* Revert "save saving increase"

This reverts commit 3b50c857fb.

* revert change

* adjust and add tests

* Dispose sqlite manager properly

* Make Batch a IMemoryOwner to allow for pooling

* Fix tests

* Upgrade some deps

* try to make tests more explicit

* remove return value

* Add detailed SqLiteJsonCacheException

* details changes
2025-01-09 11:04:14 +00:00
Adam Hathcock ed5bdc91ed Sqlite pooling for connections and commands (#193)
* add ServerObjectManagerFactory

* add usage of a command pool

* add more disposal

* save saving increase

* fix tests

* fixes

* push out concurrency and disposablity

* Add a custom task scheduler

* Better usage, don't wait to enqueue to save to channels

* Completely pre-cal batch size to avoid spinning issues

* Try to fix cache counting

* properly dispose things

* format

* clean up

* adjust count and save on current thread

* move batch it's own file

* update a few packages

* fix build and add batch tests

* revert and format

* Revert "save saving increase"

This reverts commit 3b50c857fb.

* revert change

* adjust and add tests

* Dispose sqlite manager properly

* Make Batch a IMemoryOwner to allow for pooling

* Fix tests

* Upgrade some deps

* try to make tests more explicit

* remove return value

* Use named tuple for all objects
2025-01-08 11:04:32 +00:00
Adam Hathcock 1fe1a54dcf Custom task scheduler for serialization, fix batch size calc (#194)
* Add a custom task scheduler

* Better usage, don't wait to enqueue to save to channels

* Completely pre-cal batch size to avoid spinning issues

* Try to fix cache counting

* properly dispose things

* format

* clean up

* adjust count and save on current thread

* move batch it's own file

* update a few packages

* fix build and add batch tests
2025-01-03 12:26:19 +00:00
Adam Hathcock 991b31265f Fix data chunks getting written with closure tables (#191)
* Never write closures for datachunks

* add tests and format

* add another test with array and reuses reference

* Remove a writeline in tests
2024-12-17 16:49:17 +03:00
Adam Hathcock 675d896e0d Add method to replace in sqlite (#190) 2024-12-16 15:01:53 +03:00
Adam Hathcock defcee165a Closures are kept for children instead of global (#189)
* disable channels when skipping things

* pass child closures to current.  Current closures out to parent.

* fix build

* adjust options

* use a dictionary pool and pool correctly

* add pools for data chunks

* format
2024-12-13 11:00:21 +00:00
Adam Hathcock 722df50d34 Fix sending caching (#188)
* add list pool creation

* add back base cache for closures

* use bounded channels and tell them to wait

* format and add exception

* adding back json cache?

* nevermind

* format

* add option to skip total finding

* add some comments
2024-12-10 11:21:12 +00:00
Adam Hathcock a393f7ca8b add explicit interface tests (#186) 2024-12-05 11:29:44 +00:00
Claire Kuang d29450f0a9 Merge branch 'claire/cnx-849' of https://github.com/specklesystems/speckle-sharp-sdk into claire/cnx-849 2024-12-03 22:30:33 +00:00
Claire Kuang 1e5bf903a2 update tests 2024-12-03 22:30:24 +00:00
Adam Hathcock d904c68c10 Nullable fixes - nullability on Base (#184)
* Remove operator overloading and Equals/GetHashCode which caused problems

* properly have nullability annotations on Base

* redeux known nullables

* fmt

* more nullability

* fmt

* remove point equality operator

* Revert "Remove operator overloading and Equals/GetHashCode which caused problems"

This reverts commit 6b5d3f1462.

* Revert "remove point equality operator"

This reverts commit 9b14330147.

* revert nullable accounts

* fix build issues
2024-12-03 13:01:33 +00:00
Adam Hathcock daec64cf8e Remove the Base cache because it's not practically useful (#183) 2024-12-03 12:48:18 +00:00
Adam Hathcock 1844386235 Better options for serializer (#180)
* refactor options to use on factories, add dummies

* format

* reduce lists to yield return

* get primitives first to avoid to string

* fmt

* add cache base option
2024-11-29 15:43:10 +00:00
Adam Hathcock 05f90eae5c Use frozen collections (#178)
* add detail to exceptions

* Detachable nested objects work

* fmt

* Use frozen collections

* fix tests and formatting
2024-11-29 15:25:29 +00:00
Adam Hathcock 7bbc568022 Make ISqLiteJsonCacheManager and use it for new serialization and account manager (#179)
* Introduce ISqLiteJsonCacheManager with factory for cache manager and other uses

* AccountManager uses new sqlite

* format

* add extension
2024-11-27 12:14:16 +00:00
Adam Hathcock 7a1edbbf99 Fix externals refs again (#177)
* add detail to exceptions

* Detachable nested objects work

* fmt
2024-11-26 09:47:44 +00:00
Adam Hathcock 4b7c6e4641 fixed test to allow for new detachments 2024-11-22 09:52:59 +00:00
Adam Hathcock 2477423ba7 some tests adjusted 2024-11-22 09:12:10 +00:00
Claire Kuang 71ec9a6d11 rescopes data object classes 2024-11-21 17:53:37 +00:00
Adam Hathcock b343d9bd0d add Id and Json types (#165)
* Can debug dependencies

* Different exceptions

* Uses root id only after we found it to signal the end

* DataChunks are created later and need to be accounted for

* format

* use app ids in tests and references

* check sqlite cache after serialize

* use dummy to go through channels to end

* fmt

* Extend channel lib to batch by size

* fmt

* build fix

* adjust limits

* FIx sending

* Optimize reference generation

* more

* remove tolist

* rework closures to be constant and serializer only deals with current....references bases are cached

* fix chunk creation

* another bug fix

* clean up with factories

* add deserializer factory

* Needed to reference interface

* add Id and Json types

* Fix tests

* remove optional

* add another missing id write

* fix merge

* Fix new code
2024-11-20 09:51:10 +00:00
Adam Hathcock 604e141468 Update CSharpier with better new usage (#172) 2024-11-19 14:10:30 +00:00
Jedd Morgan 39dcc5da0b Refactor MaybeThrowGraphqlException to throw AggregateException with all GraphQL errors (#169)
* first pass

* Updated tests

* Removed path from message
2024-11-19 13:22:52 +00:00
Jedd Morgan af35edf0a2 Breaking API changes for comments API (#173)
* Fixed breaking change to comments api

* fixed stuff

* oops
2024-11-19 12:15:17 +00:00
Adam Hathcock 715bb7274a Batch by size, Closures and Detached items are computed differently (#164)
* Can debug dependencies

* Different exceptions

* Uses root id only after we found it to signal the end

* DataChunks are created later and need to be accounted for

* format

* use app ids in tests and references

* check sqlite cache after serialize

* use dummy to go through channels to end

* fmt

* Extend channel lib to batch by size

* fmt

* build fix

* adjust limits

* FIx sending

* Optimize reference generation

* more

* remove tolist

* rework closures to be constant and serializer only deals with current....references bases are cached

* fix chunk creation

* another bug fix

* clean up with factories

* add deserializer factory

* Needed to reference interface

* move around streamId

* some clean up

* Use StringBuilder pool on serialization to reduce memory pressure

* remove extra

* remove extra clears

* Fix a flaw in batchsize

* use default complete

* format

* loader should use 1 writer that is batched

* remove redundant ref gen

* Fix graphql commands by adding project id
2024-11-12 14:25:49 +00:00
Adam Hathcock 43445bc4ff Add SQLite transport2 (#162)
* Add SQLiteTransport2

* cleanup
2024-11-07 10:00:43 +00:00
Adam Hathcock 0240e3b49e Send Channel fixes (#155)
* Can debug dependencies

* Different exceptions

* Uses root id only after we found it to signal the end

* DataChunks are created later and need to be accounted for

* format

* use app ids in tests and references

* check sqlite cache after serialize

* use dummy to go through channels to end

* fmt

* fix tests and add more send events
2024-11-06 11:53:21 +00:00
Adam Hathcock 4cc78c4bc9 Serialize using a Channel (#146)
* Use a stack channel for deserialization

* multi-threaded

* add object dictionary pool

* more pooling

* adjust sqlite transport

* format

* Optimize IsPropNameValid

* object loader first pass

* save test

* add cache pre check

* save better deserialize

* mostly works

* uses tasks but slower at end

* rework to make more sense

* add check to avoid multi-deserialize

* modify max parallelism

* async enqueuing of tasks

* switch to more asyncenumerable

* fmt

* fmt

* cleanup sqlite

* make ServerObjectManager

* revert change

* add ability to skip cache check

* cache json to know what is loaded

* testing

* clean up usage

* clean up and added new op

* Fix exception handling

* fixing progress

* remove codejam

* Hides ObjectPool dependency

* fmt

* Use the 1.0 BCL async to try to be more compatible

* rename to dependencies

* Move Polly to internal dependencies

* format

* remove more old references

* remove stackchannel

* fixes for registration

* remove console writeline

* add cache check shortcut for root object

* start refactoring send

* recevie2 benchmark

* add test for deserialize new

* use channels for sending

* test and fixes

* Use same asyncinterfaces as Dynamo.  Merge fixes

* clean up

* fix download object progress

* put back from bad merge

* intermediate commit: separating get child function from serializer

* send didn't error

* add channels

* Use net48, netstandard2.1 and net8

* remove collection special case

* have to make a tree of tasks even though it may serialize things twice

* pre-id changing during serialize

* need AsyncInterfaces for net48 :(

* options changes

* revert to netstandard2.0 and net8.0

* fix totals

* revert httpcontext changes

* format

* clean up

* active tasks works when accounting for id not being stable

* add id tests

* more fixes

* works

* format

* Convert to BaseItem and use single SQLite checks to avoid locks

* use locks and batch sqlite operations

* hook up and handle null ids

* remove unused parameter

* remove progress from serializer itself

* invert has objects call

* readd object references

* format

* fix tests

* remove active tasks check

* bug fix for json cache

* remove locks from sqlite

* General Send test

* add childclosures

* redo extract all to be enumerable

* group tests in projects

* caching json does matter

* cache checking should be managed by channels

* format

* Merge pull request #152 from specklesystems/new-json-test

Uses a new objects test in Revit for serialization tests

* add skip

* add new roundtrip test

* fix finish

* clean up tests

* check happens in serialize...don't do it twice

* better progress reporting

* fix progress reporting

* only use detached properties when children gathering

* move detached tests

* add detached tests

* fix merge

* Fix progress change

* fix more tests

---------

Co-authored-by: Jedd Morgan <45512892+JR-Morgan@users.noreply.github.com>
Co-authored-by: Claire Kuang <kuang.claire@gmail.com>
2024-11-05 09:56:54 +00:00
Claire Kuang 8a148b892f refactor(objects): removes unused classes and constructors (#151)
* removes unused classes and gh schema constructors

* additional cleanup of backwards compatibility (v2)

* Update Arc.cs

* updates unit tests

* re-adds bbox to box

* updates tests

* pr fixes for backwards compatibility

* Uses a new objects test in Revit for serialization tests

* format

* updates revit model curve class to inherit from curve geometry, and adds deprecated v2 classes

* fixes transform unit tests

* fixes deprecated speckle type

* removes unnecessary new revit curve classes

---------

Co-authored-by: Adam Hathcock <adam@hathcock.uk>
Co-authored-by: Alan Rynne <alan@rynne.es>
2024-11-04 17:05:16 +00:00
Jedd Morgan fba0c46ed8 Added explicit nullability to GqlModels (#144)
* Added explicit nullability to GqlModels

* data and alignment

* ModelsWithVersions

* Models

* active user

* Invites and active user fixes

* Invite changes

* Subscriptions

* Fixed subscription tests

* corrected nullability of projectinput

* removed unused mutation responses

* format

---------

Co-authored-by: Alan Rynne <alan@rynne.es>
2024-11-04 16:52:15 +00:00
Adam Hathcock 27d3028880 Revert to SQlite 7.0.5 to exactly match V2's usage (#153)
* Revert to SQlite 7.0.5 to exactly match V2's usage

* update comment to be exact
2024-10-31 14:58:52 +00:00
Adam Hathcock f5a8d1a738 Add cache download channels (#149)
* Use a stack channel for deserialization

* multi-threaded

* add object dictionary pool

* more pooling

* adjust sqlite transport

* format

* Optimize IsPropNameValid

* object loader first pass

* save test

* add cache pre check

* save better deserialize

* mostly works

* uses tasks but slower at end

* rework to make more sense

* add check to avoid multi-deserialize

* modify max parallelism

* async enqueuing of tasks

* switch to more asyncenumerable

* fmt

* fmt

* cleanup sqlite

* make ServerObjectManager

* revert change

* add ability to skip cache check

* cache json to know what is loaded

* testing

* clean up usage

* clean up and added new op

* Fix exception handling

* fixing progress

* remove codejam

* Hides ObjectPool dependency

* fmt

* Use the 1.0 BCL async to try to be more compatible

* rename to dependencies

* Move Polly to internal dependencies

* format

* remove more old references

* remove stackchannel

* fixes for registration

* remove console writeline

* add cache check shortcut for root object

* recevie2 benchmark

* add test for deserialize new

* Use same asyncinterfaces as Dynamo.  Merge fixes

* clean up

* fix download object progress

* add channels

* Use net48, netstandard2.1 and net8

* need AsyncInterfaces for net48 :(

* options changes

* revert to netstandard2.0 and net8.0

* fix totals

* revert httpcontext changes

* format

* add active tasks collection

* format

* remove more changes and remove unused method

* remove more comment changes

---------

Co-authored-by: Jedd Morgan <45512892+JR-Morgan@users.noreply.github.com>
2024-10-25 14:28:33 +01:00
Adam Hathcock b6d2d01068 Tasks hide new dependencies (#145)
* Use a stack channel for deserialization

* multi-threaded

* add object dictionary pool

* more pooling

* adjust sqlite transport

* format

* Optimize IsPropNameValid

* object loader first pass

* save test

* add cache pre check

* save better deserialize

* mostly works

* uses tasks but slower at end

* rework to make more sense

* add check to avoid multi-deserialize

* modify max parallelism

* async enqueuing of tasks

* switch to more asyncenumerable

* fmt

* fmt

* cleanup sqlite

* make ServerObjectManager

* revert change

* add ability to skip cache check

* cache json to know what is loaded

* testing

* clean up usage

* clean up and added new op

* Fix exception handling

* fixing progress

* remove codejam

* Hides ObjectPool dependency

* fmt

* Use the 1.0 BCL async to try to be more compatible

* rename to dependencies

* Move Polly to internal dependencies

* format

* remove more old references

* remove stackchannel

* fixes for registration

* remove console writeline

* add cache check shortcut for root object

* recevie2 benchmark

* add test for deserialize new

* Use same asyncinterfaces as Dynamo.  Merge fixes

* clean up

* fix download object progress

---------

Co-authored-by: Jedd Morgan <45512892+JR-Morgan@users.noreply.github.com>
2024-10-24 15:23:16 +02:00
Adam Hathcock cca8828565 Using Tasks for Deserialization (#143)
* Use a stack channel for deserialization

* multi-threaded

* add object dictionary pool

* more pooling

* adjust sqlite transport

* format

* Optimize IsPropNameValid

* object loader first pass

* save test

* add cache pre check

* save better deserialize

* mostly works

* uses tasks but slower at end

* rework to make more sense

* add check to avoid multi-deserialize

* modify max parallelism

* async enqueuing of tasks

* switch to more asyncenumerable

* fmt

* fmt

* cleanup sqlite

* make ServerObjectManager

* revert change

* add ability to skip cache check

* cache json to know what is loaded

* testing

* clean up usage

* clean up and added new op

* Fix exception handling

* fixing progress

* remove codejam

* remove stackchannel

* remove console writeline

* add cache check shortcut for root object

* recevie2 benchmark

---------

Co-authored-by: Jedd Morgan <45512892+JR-Morgan@users.noreply.github.com>
2024-10-22 14:14:39 +01:00
Adam Hathcock e14913b8c3 Fix deserialization progress reporting (#137)
* Fix progress reporting for deserialization, remove children known count

* fix tests
2024-10-09 11:57:48 +01:00
Adam Hathcock 14359333b4 Nuget Updates (#132)
* update test packages

* update csharpier
2024-10-07 12:48:00 +02:00
Adam Hathcock 75d88c5f39 Use IProgress instead of custom type (#131)
* Use IProgress instead of custom type

* Format

* ProgressArgs are a readonly struct

* fix tests
2024-10-02 10:50:49 +01:00
Adam Hathcock d73bf365c2 Clean app plugin registration and all speckle plugin verisons (#123)
* Clean app plugin registration and all speckle plugin verisons

* fix naming
2024-09-20 09:32:29 +02:00