Commit Graph

74 Commits

Author SHA1 Message Date
Adam Hathcock 604e141468 Update CSharpier with better new usage (#172) 2024-11-19 14:10:30 +00:00
Jedd Morgan 39dcc5da0b Refactor MaybeThrowGraphqlException to throw AggregateException with all GraphQL errors (#169)
* first pass

* Updated tests

* Removed path from message
2024-11-19 13:22:52 +00:00
Jedd Morgan af35edf0a2 Breaking API changes for comments API (#173)
* Fixed breaking change to comments api

* fixed stuff

* oops
2024-11-19 12:15:17 +00:00
Adam Hathcock 715bb7274a Batch by size, Closures and Detached items are computed differently (#164)
* Can debug dependencies

* Different exceptions

* Uses root id only after we found it to signal the end

* DataChunks are created later and need to be accounted for

* format

* use app ids in tests and references

* check sqlite cache after serialize

* use dummy to go through channels to end

* fmt

* Extend channel lib to batch by size

* fmt

* build fix

* adjust limits

* FIx sending

* Optimize reference generation

* more

* remove tolist

* rework closures to be constant and serializer only deals with current....references bases are cached

* fix chunk creation

* another bug fix

* clean up with factories

* add deserializer factory

* Needed to reference interface

* move around streamId

* some clean up

* Use StringBuilder pool on serialization to reduce memory pressure

* remove extra

* remove extra clears

* Fix a flaw in batchsize

* use default complete

* format

* loader should use 1 writer that is batched

* remove redundant ref gen

* Fix graphql commands by adding project id
2024-11-12 14:25:49 +00:00
Adam Hathcock 43445bc4ff Add SQLite transport2 (#162)
* Add SQLiteTransport2

* cleanup
2024-11-07 10:00:43 +00:00
Adam Hathcock 0240e3b49e Send Channel fixes (#155)
* Can debug dependencies

* Different exceptions

* Uses root id only after we found it to signal the end

* DataChunks are created later and need to be accounted for

* format

* use app ids in tests and references

* check sqlite cache after serialize

* use dummy to go through channels to end

* fmt

* fix tests and add more send events
2024-11-06 11:53:21 +00:00
Adam Hathcock 4cc78c4bc9 Serialize using a Channel (#146)
* Use a stack channel for deserialization

* multi-threaded

* add object dictionary pool

* more pooling

* adjust sqlite transport

* format

* Optimize IsPropNameValid

* object loader first pass

* save test

* add cache pre check

* save better deserialize

* mostly works

* uses tasks but slower at end

* rework to make more sense

* add check to avoid multi-deserialize

* modify max parallelism

* async enqueuing of tasks

* switch to more asyncenumerable

* fmt

* fmt

* cleanup sqlite

* make ServerObjectManager

* revert change

* add ability to skip cache check

* cache json to know what is loaded

* testing

* clean up usage

* clean up and added new op

* Fix exception handling

* fixing progress

* remove codejam

* Hides ObjectPool dependency

* fmt

* Use the 1.0 BCL async to try to be more compatible

* rename to dependencies

* Move Polly to internal dependencies

* format

* remove more old references

* remove stackchannel

* fixes for registration

* remove console writeline

* add cache check shortcut for root object

* start refactoring send

* recevie2 benchmark

* add test for deserialize new

* use channels for sending

* test and fixes

* Use same asyncinterfaces as Dynamo.  Merge fixes

* clean up

* fix download object progress

* put back from bad merge

* intermediate commit: separating get child function from serializer

* send didn't error

* add channels

* Use net48, netstandard2.1 and net8

* remove collection special case

* have to make a tree of tasks even though it may serialize things twice

* pre-id changing during serialize

* need AsyncInterfaces for net48 :(

* options changes

* revert to netstandard2.0 and net8.0

* fix totals

* revert httpcontext changes

* format

* clean up

* active tasks works when accounting for id not being stable

* add id tests

* more fixes

* works

* format

* Convert to BaseItem and use single SQLite checks to avoid locks

* use locks and batch sqlite operations

* hook up and handle null ids

* remove unused parameter

* remove progress from serializer itself

* invert has objects call

* readd object references

* format

* fix tests

* remove active tasks check

* bug fix for json cache

* remove locks from sqlite

* General Send test

* add childclosures

* redo extract all to be enumerable

* group tests in projects

* caching json does matter

* cache checking should be managed by channels

* format

* Merge pull request #152 from specklesystems/new-json-test

Uses a new objects test in Revit for serialization tests

* add skip

* add new roundtrip test

* fix finish

* clean up tests

* check happens in serialize...don't do it twice

* better progress reporting

* fix progress reporting

* only use detached properties when children gathering

* move detached tests

* add detached tests

* fix merge

* Fix progress change

* fix more tests

---------

Co-authored-by: Jedd Morgan <45512892+JR-Morgan@users.noreply.github.com>
Co-authored-by: Claire Kuang <kuang.claire@gmail.com>
2024-11-05 09:56:54 +00:00
Claire Kuang 8a148b892f refactor(objects): removes unused classes and constructors (#151)
* removes unused classes and gh schema constructors

* additional cleanup of backwards compatibility (v2)

* Update Arc.cs

* updates unit tests

* re-adds bbox to box

* updates tests

* pr fixes for backwards compatibility

* Uses a new objects test in Revit for serialization tests

* format

* updates revit model curve class to inherit from curve geometry, and adds deprecated v2 classes

* fixes transform unit tests

* fixes deprecated speckle type

* removes unnecessary new revit curve classes

---------

Co-authored-by: Adam Hathcock <adam@hathcock.uk>
Co-authored-by: Alan Rynne <alan@rynne.es>
2024-11-04 17:05:16 +00:00
Jedd Morgan fba0c46ed8 Added explicit nullability to GqlModels (#144)
* Added explicit nullability to GqlModels

* data and alignment

* ModelsWithVersions

* Models

* active user

* Invites and active user fixes

* Invite changes

* Subscriptions

* Fixed subscription tests

* corrected nullability of projectinput

* removed unused mutation responses

* format

---------

Co-authored-by: Alan Rynne <alan@rynne.es>
2024-11-04 16:52:15 +00:00
Adam Hathcock 27d3028880 Revert to SQlite 7.0.5 to exactly match V2's usage (#153)
* Revert to SQlite 7.0.5 to exactly match V2's usage

* update comment to be exact
2024-10-31 14:58:52 +00:00
Adam Hathcock f5a8d1a738 Add cache download channels (#149)
* Use a stack channel for deserialization

* multi-threaded

* add object dictionary pool

* more pooling

* adjust sqlite transport

* format

* Optimize IsPropNameValid

* object loader first pass

* save test

* add cache pre check

* save better deserialize

* mostly works

* uses tasks but slower at end

* rework to make more sense

* add check to avoid multi-deserialize

* modify max parallelism

* async enqueuing of tasks

* switch to more asyncenumerable

* fmt

* fmt

* cleanup sqlite

* make ServerObjectManager

* revert change

* add ability to skip cache check

* cache json to know what is loaded

* testing

* clean up usage

* clean up and added new op

* Fix exception handling

* fixing progress

* remove codejam

* Hides ObjectPool dependency

* fmt

* Use the 1.0 BCL async to try to be more compatible

* rename to dependencies

* Move Polly to internal dependencies

* format

* remove more old references

* remove stackchannel

* fixes for registration

* remove console writeline

* add cache check shortcut for root object

* recevie2 benchmark

* add test for deserialize new

* Use same asyncinterfaces as Dynamo.  Merge fixes

* clean up

* fix download object progress

* add channels

* Use net48, netstandard2.1 and net8

* need AsyncInterfaces for net48 :(

* options changes

* revert to netstandard2.0 and net8.0

* fix totals

* revert httpcontext changes

* format

* add active tasks collection

* format

* remove more changes and remove unused method

* remove more comment changes

---------

Co-authored-by: Jedd Morgan <45512892+JR-Morgan@users.noreply.github.com>
2024-10-25 14:28:33 +01:00
Adam Hathcock b6d2d01068 Tasks hide new dependencies (#145)
* Use a stack channel for deserialization

* multi-threaded

* add object dictionary pool

* more pooling

* adjust sqlite transport

* format

* Optimize IsPropNameValid

* object loader first pass

* save test

* add cache pre check

* save better deserialize

* mostly works

* uses tasks but slower at end

* rework to make more sense

* add check to avoid multi-deserialize

* modify max parallelism

* async enqueuing of tasks

* switch to more asyncenumerable

* fmt

* fmt

* cleanup sqlite

* make ServerObjectManager

* revert change

* add ability to skip cache check

* cache json to know what is loaded

* testing

* clean up usage

* clean up and added new op

* Fix exception handling

* fixing progress

* remove codejam

* Hides ObjectPool dependency

* fmt

* Use the 1.0 BCL async to try to be more compatible

* rename to dependencies

* Move Polly to internal dependencies

* format

* remove more old references

* remove stackchannel

* fixes for registration

* remove console writeline

* add cache check shortcut for root object

* recevie2 benchmark

* add test for deserialize new

* Use same asyncinterfaces as Dynamo.  Merge fixes

* clean up

* fix download object progress

---------

Co-authored-by: Jedd Morgan <45512892+JR-Morgan@users.noreply.github.com>
2024-10-24 15:23:16 +02:00
Adam Hathcock cca8828565 Using Tasks for Deserialization (#143)
* Use a stack channel for deserialization

* multi-threaded

* add object dictionary pool

* more pooling

* adjust sqlite transport

* format

* Optimize IsPropNameValid

* object loader first pass

* save test

* add cache pre check

* save better deserialize

* mostly works

* uses tasks but slower at end

* rework to make more sense

* add check to avoid multi-deserialize

* modify max parallelism

* async enqueuing of tasks

* switch to more asyncenumerable

* fmt

* fmt

* cleanup sqlite

* make ServerObjectManager

* revert change

* add ability to skip cache check

* cache json to know what is loaded

* testing

* clean up usage

* clean up and added new op

* Fix exception handling

* fixing progress

* remove codejam

* remove stackchannel

* remove console writeline

* add cache check shortcut for root object

* recevie2 benchmark

---------

Co-authored-by: Jedd Morgan <45512892+JR-Morgan@users.noreply.github.com>
2024-10-22 14:14:39 +01:00
Adam Hathcock e14913b8c3 Fix deserialization progress reporting (#137)
* Fix progress reporting for deserialization, remove children known count

* fix tests
2024-10-09 11:57:48 +01:00
Adam Hathcock 14359333b4 Nuget Updates (#132)
* update test packages

* update csharpier
2024-10-07 12:48:00 +02:00
Adam Hathcock 75d88c5f39 Use IProgress instead of custom type (#131)
* Use IProgress instead of custom type

* Format

* ProgressArgs are a readonly struct

* fix tests
2024-10-02 10:50:49 +01:00
Adam Hathcock d73bf365c2 Clean app plugin registration and all speckle plugin verisons (#123)
* Clean app plugin registration and all speckle plugin verisons

* fix naming
2024-09-20 09:32:29 +02:00
Adam Hathcock 198f1a2564 Remove STJ dependency (#122) 2024-09-19 14:11:16 +01:00
Adam Hathcock 0f29aacf24 Making SDK non-static (#116)
* initial burn

* fix: Minor fixes, some pending

* fix: Minor fix + DeepClean target

* setup things to be injected

* things compile and activites are instanced

* compiles except tests

* tests compile

* fmt

* fix merge

* Activities are now an interface

* fmt

* fully remove logging DLL

* clean up

* fix tests

* more cleanup

* Fix integration tests

* clean up hash tests

* clean up path provider

* more cleanup for path

* merge fixes

* Merge fixes

* remove some extra classes

---------

Co-authored-by: Alan Rynne <alan@speckle.systems>
2024-09-17 15:51:50 +01:00
Jedd Morgan df5b151c90 Test Coverage (#117) 2024-09-16 08:41:26 +01:00
Jedd Morgan ccb4f54550 Jedd/cxpla 55 add required keyword for more geometry types (#101)
* Sdk

* Objects

* Supressed IDE warnings via editor config instead of nowarn

* Nullability and other warnings

* using

* Required keyword on lines and meshes

* Fixed test project

* fixed tests

* Proxies

* Fixed equality of Point

* IEquatable

* Fixed issue with serialization of detached lists

* Added tests for jsonIgnore affecting id calc

* removed comments

* Fixed issue with fallback to double on large values

* Fixed undocumented large number support
2024-09-13 14:47:29 +01:00
Adam Hathcock e07bfc18d5 Remove async from serializer (#113)
* Remove async from serializer

* Make GetId be sync

* fix chunking?

* Chunking Tests (#114)

* fmt

* Fixed speckle type

---------

Co-authored-by: Jedd Morgan <45512892+JR-Morgan@users.noreply.github.com>
2024-09-11 15:35:59 +01:00
Adam Hathcock 2ad14ca00b Fix serialization of ignored (#112)
* Fixed issue with serialization of detached lists

* Added tests for jsonIgnore affecting id calc

* use extra writer and pass tests

* avoid returning extra object

* fix tests

* Add span overload back

* fmt

---------

Co-authored-by: Jedd Morgan <45512892+JR-Morgan@users.noreply.github.com>
2024-09-10 13:17:46 +01:00
Jedd Morgan 450dbcce81 Sha256 hash with spans (#107)
* Sha256 hash with spans

* HashData

* NumericFormat

* Tests

* md5 test

* xml

* use utf16 encoding rather than utf8

---------

Co-authored-by: Alan Rynne <alan@rynne.es>
2024-09-09 12:43:32 +00:00
Adam Hathcock 0b1de566df Async GetObject and Async Deserialization (#95)
* Use Concurrent Dictionaries and ignore case on them to avoid ToLower

* remove single array allocation

* allocate GetClosures differently

* Use JsonReader for closures

* add comment

* use isdefined

* more readonly with less allocations

* sorts

* fmt

* use element type when making an array

* first pass

* Fix reading for object and array

* serialization works?

* Fix closure parsing

* things are way faster

* Deserialize is async

* fmt

* renames

* remove deserialize threads

* fmt

* faster to use ordinal compare

* serialization looks okay

* fix closure writing

* fixes

* use possibly different values for compute id

* use closure parser on download

* memory test for blobs

* decompose serialization

* fmt

* fmt

* for decomposing, values should be used instead of original

* set id after computing it

* redo more closure parsing

* fix memory test

* don't throw on try get deserialized

* fmt

* fix integration tests

* fix tests

* disable memory blob storage by default

* put back ?

* merge fixes and delete worker threads

* fmt

* serialization of old floats pass

* serialization of old floats pass

* rename class

* Use async/await on GetObject

* await deserialization

* fmt

* uncomment and fix tests

* don't allow things to exist in the closure table that doesn't exist

* detach blob tests

* rename serializer

* async more correct

* revert

* fix merge

* fix blob tests again

* more fixes

* Fix building

* async fixes

* more async fixes

* fix tests?

* rename GetId to GetIdAsync

* clean up

* more cleanup

* fmt

* fix test

* fix analyzer errors

* use ConcurrentDictionary to be thread safe
2024-09-04 13:45:56 +01:00
Jedd Morgan 5e0ea324c3 Re-introduced code analysers and fixed many violations (#92)
* Sdk

* Objects

* Supressed IDE warnings via editor config instead of nowarn

* Nullability and other warnings

* using

* Fixed warnings

* Important fix

* More fixes
2024-09-04 11:49:35 +00:00
Adam Hathcock 22f4945d71 Json reader deserialize (#88)
* Use Concurrent Dictionaries and ignore case on them to avoid ToLower

* remove single array allocation

* allocate GetClosures differently

* Use JsonReader for closures

* add comment

* use isdefined

* more readonly with less allocations

* sorts

* fmt

* use element type when making an array

* first pass

* Fix reading for object and array

* serialization works?

* Fix closure parsing

* things are way faster

* Deserialize is async

* fmt

* renames

* remove deserialize threads

* fmt

* faster to use ordinal compare

* serialization looks okay

* fix closure writing

* fixes

* use possibly different values for compute id

* use closure parser on download

* memory test for blobs

* decompose serialization

* fmt

* fmt

* for decomposing, values should be used instead of original

* set id after computing it

* redo more closure parsing

* fix memory test

* don't throw on try get deserialized

* fmt

* fix integration tests

* fix tests

* disable memory blob storage by default

* put back ?

* merge fixes and delete worker threads

* fmt

* serialization of old floats pass

* serialization of old floats pass

* rename class

* uncomment and fix tests

* don't allow things to exist in the closure table that doesn't exist

* detach blob tests

* rename serializer

* revert

* fix blob tests again
2024-09-03 15:53:50 +01:00
Jedd Morgan c0f8949705 Jedd/cxpla 33 move performance test project (#87)
* Moved performance test project

* register types

* perf tests
2024-09-02 15:41:48 +00:00
Jedd Morgan 2b46792a82 Fixed issue with Km unit scalling (#93) 2024-08-28 13:27:42 +02:00
Adam Hathcock 4c3e572bfe clean up from progress (#78)
* progress intermediate commit

* add progress for download

* remove unused code

* remove batch sent callbacks

* multi-threaded deserialize works

* Progress for download and deserialization

* Fix tests

* Have less indeterminate deserialization

* fix deserialization

* make download faster with buffered stream

* put local receive back

* remove unused callback

* fmt

* Progress for serialization and upload

* fix uploading

* clean up from progress

* merge fixes and fmt
2024-08-22 11:18:16 +01:00
Adam Hathcock 0f116ad847 Base caching on TypeLoader (#82)
* Base caching on TypeLoader

* fmt

* add back types
2024-08-22 11:11:53 +01:00
Jedd Morgan 9916048921 Long Send Resiliency (#89)
* LongSendTest

* Moved to separate project

* Server transport throws early

* raised timeouts

* Use poly for timeout

* Cleaned up some of the AsyncPolicy construction

* GraphQL Client back to 30 second timeout
2024-08-21 18:52:23 +01:00
Jedd Morgan b14c8db8d3 Required keyword on various types (#76)
* Made bounding boxes nullable

* Required keyword for Polyline

* Curve

* others

* Interval

* Breps!

* more changes

* removed resolved todo comment

* Fixed comment

* resolved alan's comments

* surfaces
2024-08-16 13:21:05 +01:00
Adam Hathcock dc4da49078 Add Progress for transfers (#74)
* progress intermediate commit

* add progress for download

* remove unused code

* remove batch sent callbacks

* multi-threaded deserialize works

* Progress for download and deserialization

* Fix tests

* Have less indeterminate deserialization

* fix deserialization

* make download faster with buffered stream

* put local receive back

* remove unused callback

* fmt
2024-08-15 10:43:41 +02:00
Adam Hathcock ea5ed874a3 Remove extra SaveObject (#75)
* Remove extra SaveObject

* fmt
2024-08-15 09:30:24 +02:00
Adam Hathcock d269ac73dd Disallow SpeckleType inheritance (#71)
* Disallow SpeckleType inheritance

* fmt

* more tests
2024-08-13 14:32:58 +00:00
Jedd Morgan 66b3c5e60c Added DUI3 Operations.Send changes to Sdk (#68)
* Added DUI3 Operations.Send changes to Sdk

* Tests

* more tests

---------

Co-authored-by: Dimitrie Stefanescu <didimitrie@gmail.com>
2024-08-12 18:28:35 +01:00
Jedd Morgan cfc4018bd3 removed a bunch of obsolete (#65)
* removed a bunch of obsolete

* Accidental trailing char
2024-08-12 11:49:43 +01:00
Jedd Morgan 4b74ca8607 Some tweaks r.e. JsonIgnore (#64)
* Removed total children count

* Some random tweaks

* JsonIgnore tests

* Ignore base
2024-08-12 09:03:56 +00:00
Jedd Morgan 78faa71592 Collection name space change (#66) 2024-08-08 12:32:41 +00:00
Adam Hathcock 6f5f044095 Adam/cxpla 6 kill remaining kit code in core (#59)
* compiles with relevant deletions

* Test fixes

* fix type loading

* type load for tests

* speckle objects renamespace

* rename Core to Sdk

* Fix test references

* tests renaming

* rename logging

* fmt

* start of adding an attribute to all base types

* convert all types and do basic test

* Fix most tests

* fix more tests

* fmt

* Build fix

* add changes and more tests

* Fix tests

* Fix integration tests
2024-08-08 10:52:19 +01:00
Adam Hathcock 7148aa0a6f Refining OTEL/Logging configuration (#57)
* More defined configuration

* more config changes and fmt

* fix usages of app versions

* add endpoints and remove seq

* forgot to commit

* Fixes around tracing

* use more strong typing

* add test
2024-08-06 16:24:55 +01:00
Adam Hathcock 77ffcead69 Speckle.Logging with ILRepack (#54)
* Revert SQLite to allow side by side

* Update Serilog to 3.1.1 and remove Sentry

* fmt

* Revert serilog change

* first pass

* downgrade to MELA 2.2 to avoid conflicts

* add otel

* Speckle.Logging proper

* move otel and logging to make core dependant

* readd configuration

* revert namespace and add console tracing

* fmt

* Remove extra usings

* readd serilog

* fix deps

* remove extras for build

* some reversions

* add more context

* Fix conversion

* More explicit

* Revert naming

* back to public loggerfactory

* Remove more obsolete things

* Drop ME.Logging dependency and expose logging interface

* restore integration test compose

* fmt

* remove the ME Logging dependency
2024-07-31 15:37:10 +01:00
Adam Hathcock e79716c5f0 Remove Sentry (#49)
* Revert SQLite to allow side by side

* Update Serilog to 3.1.1 and remove Sentry

* fmt

* Revert serilog change
2024-07-29 16:10:26 +01:00
Adam Hathcock 02f75c717c Add NotNull Empty and tests (#51) 2024-07-24 15:05:44 +01:00
Adam Hathcock 059253ff39 Revert SQLite to allow side by side (#48) 2024-07-24 11:38:34 +01:00
Adam Hathcock 5c7a96b960 ensure proper disposal of transports from di factory (#44)
* Rework servertransport to have proper interface and factory

* fmt

* rename class

* exclude factory from tests
2024-07-18 16:13:17 +01:00
Claire Kuang a16bc4eae0 updates tests 2024-07-17 14:28:51 +01:00
Claire Kuang 04446b9e79 changes namespace of collections and adds interfaces 2024-07-17 10:40:55 +01:00
Jedd Morgan baf5ab0b14 Fixed mistake (#32) 2024-07-16 10:47:56 +01:00