Compare commits

..

6 Commits

Author SHA1 Message Date
izzy lyseggen 682afce05f fix(transport): call begin/end write in server (#195)
* fix(transport): call begin/end write in server

* style: remove unused code
2022-06-30 16:16:05 +01:00
izzy lyseggen 21b27e2f3b feat(metrics): add ids for unknown users (#194)
* feat(metrics): add `merge_ids` helper method

* fix(metrics): use alias ids instead

* fix(metrics): final cleanup for aliasing

* fix(metrics): lol jk scratch the aliasing
2022-06-28 12:23:19 +01:00
izzy lyseggen 69cd9706cf fix(base): remove default units to Base (#193)
* fix(base): remove default units to `Base`

change in sharp that wasn't propagated to py!

* fix(objects): add `None` to unit encodings
2022-06-22 17:03:50 +01:00
izzy lyseggen 98075fa2cf fix(metrics): remove unused prop (#192) 2022-06-22 14:52:49 +01:00
izzy lyseggen 782f70fb49 chore: drop python 3.6 and update ujson (#190)
* chore: depreciate python 3.6 support & upate ujson

after collection python version info metrics, we fount that only 2 users
are still using python 3.6. since it has been eol for 5 months now,
we believe it's safe to let it go.

rest easy 3.6 ⚰- you served us well 🫡

closes Please upgrade the ujson dependency, which has a CVE #160

* chore: upgrade and clean some deps
2022-06-20 12:19:09 +01:00
Gergő Jedlicska 52ab27e60f SQLite write batching (#188)
### SUMMARY
**sqlite transport**

This transport now batches and bulk inserts objects when writing resulting in huge performance improvements (100x).

**base object serializer**

Batching in the sqlite transport necessitated some refactoring here in order to safely call end_write when not using operations.send/receive. This has been resolved by turning traverse_base into a wrapper for _traverse_base which can take care of calling begin/end_write and resetting the writer at the top level. This is not breaking since the top level methods to call have not changed names and the original method has just been prepended with a _

Additionally, missing referenced child objects in the read transport used to raise a SpeckleException. However, using the gql client to call objects.get() will return an object with missing references by design thus throwing an error in serialization. This has been resolved by instead raising a SpeckleWarning when child objects can't be found and just returning the reference + id. ((this method of interacting with objects is discouraged so it is not surprising to me that this bug was lurking for so long - but an oopsie nonetheless!))

**ci / dev**

Updates for the ci config and the dev container to work with the recent changes in server.

NOTE: dev container seems to be pulling an older version of server -- not resolved yet

---

* quick and hacky sqlite batching

* feat(transports): batching sqlite inserts

* chore: upgrade gql3

also removed py-spy as it's not used and i was getting install errors :/

* ci: bump node version

* ci: formatting

* update CI versions

* update to new circleci redis baseimage

* update test fixture auth to non deprecated token based method

* add start and finish write method calls to base object serialize

* chore: dev container update

* fix(serialization): move end and begin write

* style: formatting

* fix(serializer): warn but don't throw if ref not found

this is _not_ an issue with the transports, but an issue with using the
graphql api to fetch objects. since you are only receiving one obj and none of
the children, the transport has no way to find them and should simply
return the reference as is. idk why anyone would really use `object.get`
so tbh i'm not surprised no one has found this bug yet lol

* fix(client): don't parse obj create response

* fix(serializer): wrap `traverse_base`

moving `begin` and `end_write` to the seriazlier due to the new
sqlite transport with batched writes necessitates a wrapper around
`traverse_base` so end/begin write can be called once at the top level.
just adding begin/end write to the original traversal method would make
tons of calls to `end_write` since the traversal is recursive

Co-authored-by: izzy lyseggen <izzy.lyseggen@gmail.com>
2022-06-20 12:00:09 +01:00
7 changed files with 433 additions and 486 deletions
+1 -1
View File
@@ -65,7 +65,7 @@ workflows:
- test:
matrix:
parameters:
tag: ["3.6", "3.7", "3.8", "3.9"]
tag: ["3.7", "3.8", "3.9", "3.10"]
filters:
tags:
only: /.*/
Generated
+410 -450
View File
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -11,11 +11,11 @@ homepage = "https://speckle.systems/"
[tool.poetry.dependencies]
python = "^3.6.5"
python = ">=3.7.0, <4.0"
pydantic = "^1.8.2"
appdirs = "^1.4.4"
gql = {extras = ["requests", "websockets"], version = "^3.3.0"}
ujson = "^4.3.0"
ujson = "^5.3.0"
Deprecated = "^1.2.13"
[tool.poetry.dev-dependencies]
+14 -10
View File
@@ -1,10 +1,13 @@
import socket
import sys
import queue
import hashlib
import getpass
import logging
import requests
import threading
import platform
import contextlib
"""
Anonymous telemetry to help us understand how to make a better Speckle.
@@ -12,7 +15,7 @@ This really helps us to deliver a better open source project and product!
"""
TRACK = True
HOST_APP = "python"
HOST_APP_VERSION = f"python {'.'.join(map(str, sys.version_info[:3]))}"
HOST_APP_VERSION = f"python {'.'.join(map(str, sys.version_info[:2]))}"
PLATFORMS = {"win32": "Windows", "cygwin": "Windows", "darwin": "Mac OS X"}
LOG = logging.getLogger(__name__)
@@ -75,8 +78,7 @@ def track(action: str, account: "Account" = None, custom_props: dict = None):
METRICS_TRACKER.queue.put_nowait(event_params)
except Exception as ex:
# wrapping this whole thing in a try except as we never want a failure here to annoy users!
LOG.error("Error queueing metrics request: " + str(ex))
LOG.error(f"Error queueing metrics request: {str(ex)}")
def initialise_tracker(account: "Account" = None):
global METRICS_TRACKER
@@ -101,8 +103,7 @@ class Singleton(type):
class MetricsTracker(metaclass=Singleton):
analytics_url = "https://analytics.speckle.systems/track?ip=1"
analytics_token = "acd87c5a50b56df91a795e999812a3a4"
user_ip = None
last_user = None
last_user = ""
last_server = None
platform = None
sending_thread = None
@@ -114,12 +115,15 @@ class MetricsTracker(metaclass=Singleton):
)
self.platform = PLATFORMS.get(sys.platform, "linux")
self.sending_thread.start()
self.user_ip = socket.gethostbyname(socket.gethostname())
with contextlib.suppress(Exception):
node, user = platform.node(), getpass.getuser()
if node and user:
self.last_user = f"@{self.hash(f'{node}-{user}')}"
def set_last_user(self, email: str):
if not email:
return
self.last_user = "@" + self.hash(email)
self.last_user = f"@{self.hash(email)}"
def set_last_server(self, server: str):
if not server:
@@ -137,6 +141,6 @@ class MetricsTracker(metaclass=Singleton):
try:
session.post(self.analytics_url, json=event_params)
except Exception as ex:
LOG.error("Error sending metrics request: " + str(ex))
LOG.error(f"Error sending metrics request: {str(ex)}")
self.queue.task_done()
self.queue.task_done()
+1 -1
View File
@@ -142,7 +142,7 @@ class Base(_RegisteringBase):
id: Optional[str] = None
totalChildrenCount: Optional[int] = None
applicationId: Optional[str] = None
_units: str = "m"
_units: str = None
# dict of chunkable props and their max chunk size
_chunkable: Dict[str, int] = {}
_chunk_size_default: int = 1000
+3 -4
View File
@@ -16,6 +16,7 @@ UNITS_STRINGS = {
}
UNITS_ENCODINGS = {
None: 0,
"none": 0,
"mm": 1,
"cm": 2,
@@ -58,7 +59,5 @@ def get_units_from_encoding(unit: int):
def get_encoding_from_units(unit: str):
try:
return UNITS_ENCODINGS[unit]
except KeyError:
raise SpeckleException(
message=f"No encoding exists for unit {unit}. Please enter a valid unit to encode (eg {UNITS_ENCODINGS})."
)
except KeyError as e:
raise SpeckleException(message=f"No encoding exists for unit {unit}. Please enter a valid unit to encode (eg {UNITS_ENCODINGS}).") from e
+2 -18
View File
@@ -156,29 +156,13 @@ class ServerTransport(AbstractTransport):
lines = r.iter_lines(decode_unicode=True)
# iter through returned objects saving them as we go
target_transport.begin_write()
for line in lines:
if line:
hash, obj = line.split("\t")
target_transport.save_object(hash, obj)
target_transport.save_object(id, root_obj_serialized)
target_transport.end_write()
return root_obj_serialized
# async def stream_res(self, endpoint: str) -> str:
# data = b""
# async with aiohttp.ClientSession() as session:
# session.headers.update(
# {
# "Authorization": f"{self.session.headers['Authorization']}",
# "Accept": "text/plain",
# }
# )
# async with session.get(endpoint) as res:
# while True:
# chunk = await res.content.read(self.chunk_size)
# if not chunk:
# break
# data += chunk
# return data.decode("utf-8")