Commit Graph

16 Commits

Author SHA1 Message Date
Adam Hathcock 05f7353925 Revert "Merge pull request #335 from specklesystems/adam/cnx-1786-allow-multiple-sends-to-access-sqlite-in-a-non-locking-2" (#339)
This reverts commit 59019bf846, reversing
changes made to 3afaf61a1a.

Co-authored-by: Adam Hathcock <adam@Adams-Mac-mini.localdomain>
2025-06-11 15:32:06 +00:00
Adam Hathcock 46c067308e Fix DI dependency and tests 2025-06-10 11:39:42 +01:00
Adam Hathcock bc81c21e9d format 2025-06-10 11:15:30 +01:00
Adam Hathcock 7f8b59d348 Pool object savers instead of sqlite 2025-06-10 11:15:01 +01:00
Adam Hathcock 44ba61e4a5 Adjustments to avoid sqlite "database is locked" errors (#333)
* add new exception test

* Make memory tests and file path tests be explicit

* set the default write parallelism to 1

* set to single reader for caching channel

* format

* Try to have consistent DB locked error test

* always a single reader of the channel

* Remove extra snapshot

* Revert "Try to have consistent DB locked error test"

This reverts commit 93669c57a3.

* remove extra test that doesn't do anything
2025-06-09 16:24:39 +00:00
Adam Hathcock f163b2822e (feat) add memory serialize and make relevant tests use it (#252)
* First pass of ObjectSaver and better in-memory usage

* fix some tests

* add commit to match deserialize process

* correct more tests

* format

* make a deserialize factory

* fix tests? and format

* use distinct

* Fix mismerge

* Fix serialization issues with tests

* fix merges

* follow copilot suggestions

* remove disables
2025-04-08 10:21:47 +01:00
Adam Hathcock e5a09155a2 Serialize now waits for the scheduler to be completed before returning (#225)
* Serialize now waits for the scheduler to be completed before returning

* formatting

* wait in deserialize too

* Pass cancellation token to download, refactor how things are passed/created

* responses are now streamed and cancel them earlier

* format

* add manual empty message for sending that definitely knows when the channels are empty

* Fix configure awaits

* more configure await false
2025-02-17 10:59:10 +00:00
Adam Hathcock 3072047129 Use priority scheduler like serialize process to constrain Task enqueuing (#223)
* Use priority scheduler like serialize process to constrain Task enqueuing

* renamespace

* Fix tests and format
2025-02-05 15:36:26 +00:00
Adam Hathcock 3aa993cecb Add cancellation tests (#218)
* Don't log cancelling

* redo exception handling for receive

* remove null test case

* clean up with Id/Json and more cancels

* Change the exception stacks

* fix serialization test

* make a custom scrubber for internalized exceptions

* clean up

* fix namespaces again :(

* adjust the scrubber

* try to make tests more predictable

* rework exceptions again

* strip out compile files used

* formatting

* custom exception validation

* fix init

* Move serialization to own class

* save serialize test

* add deep clean

* add cancellation test on save to cache

* cancellation tests

* format

* do DI correctly

* receive cancel works
2025-01-30 13:42:15 +00:00
Adam Hathcock 93912d6712 Fix cancellation in schedular. It's no longer optional (#213)
* Fix cancellation in schedular.  It's no longer optional

* formatting

* fix tests because of stacktrace
2025-01-24 11:24:51 +00:00
Adam Hathcock f81fc97a91 Add exception handling for SerializeProcess with CancellationTokenSource (#211)
* Add exception handling for SerializeProcess with CancellationTokenSource

* formatting

* add exception test to make sure we handle a server exception

* add extra exception and handling to stop

* add comment and another test

* one last chance for user to cancel

* formatting
2025-01-23 17:06:08 +00:00
Adam Hathcock ed5bdc91ed Sqlite pooling for connections and commands (#193)
* add ServerObjectManagerFactory

* add usage of a command pool

* add more disposal

* save saving increase

* fix tests

* fixes

* push out concurrency and disposablity

* Add a custom task scheduler

* Better usage, don't wait to enqueue to save to channels

* Completely pre-cal batch size to avoid spinning issues

* Try to fix cache counting

* properly dispose things

* format

* clean up

* adjust count and save on current thread

* move batch it's own file

* update a few packages

* fix build and add batch tests

* revert and format

* Revert "save saving increase"

This reverts commit 3b50c857fb.

* revert change

* adjust and add tests

* Dispose sqlite manager properly

* Make Batch a IMemoryOwner to allow for pooling

* Fix tests

* Upgrade some deps

* try to make tests more explicit

* remove return value

* Use named tuple for all objects
2025-01-08 11:04:32 +00:00
Adam Hathcock 1844386235 Better options for serializer (#180)
* refactor options to use on factories, add dummies

* format

* reduce lists to yield return

* get primitives first to avoid to string

* fmt

* add cache base option
2024-11-29 15:43:10 +00:00
Adam Hathcock 7bbc568022 Make ISqLiteJsonCacheManager and use it for new serialization and account manager (#179)
* Introduce ISqLiteJsonCacheManager with factory for cache manager and other uses

* AccountManager uses new sqlite

* format

* add extension
2024-11-27 12:14:16 +00:00
Adam Hathcock b343d9bd0d add Id and Json types (#165)
* Can debug dependencies

* Different exceptions

* Uses root id only after we found it to signal the end

* DataChunks are created later and need to be accounted for

* format

* use app ids in tests and references

* check sqlite cache after serialize

* use dummy to go through channels to end

* fmt

* Extend channel lib to batch by size

* fmt

* build fix

* adjust limits

* FIx sending

* Optimize reference generation

* more

* remove tolist

* rework closures to be constant and serializer only deals with current....references bases are cached

* fix chunk creation

* another bug fix

* clean up with factories

* add deserializer factory

* Needed to reference interface

* add Id and Json types

* Fix tests

* remove optional

* add another missing id write

* fix merge

* Fix new code
2024-11-20 09:51:10 +00:00
Adam Hathcock 715bb7274a Batch by size, Closures and Detached items are computed differently (#164)
* Can debug dependencies

* Different exceptions

* Uses root id only after we found it to signal the end

* DataChunks are created later and need to be accounted for

* format

* use app ids in tests and references

* check sqlite cache after serialize

* use dummy to go through channels to end

* fmt

* Extend channel lib to batch by size

* fmt

* build fix

* adjust limits

* FIx sending

* Optimize reference generation

* more

* remove tolist

* rework closures to be constant and serializer only deals with current....references bases are cached

* fix chunk creation

* another bug fix

* clean up with factories

* add deserializer factory

* Needed to reference interface

* move around streamId

* some clean up

* Use StringBuilder pool on serialization to reduce memory pressure

* remove extra

* remove extra clears

* Fix a flaw in batchsize

* use default complete

* format

* loader should use 1 writer that is batched

* remove redundant ref gen

* Fix graphql commands by adding project id
2024-11-12 14:25:49 +00:00