Compare commits

...

21 Commits

Author SHA1 Message Date
izzy lyseggen 69cd9706cf fix(base): remove default units to Base (#193)
* fix(base): remove default units to `Base`

change in sharp that wasn't propagated to py!

* fix(objects): add `None` to unit encodings
2022-06-22 17:03:50 +01:00
izzy lyseggen 98075fa2cf fix(metrics): remove unused prop (#192) 2022-06-22 14:52:49 +01:00
izzy lyseggen 782f70fb49 chore: drop python 3.6 and update ujson (#190)
* chore: depreciate python 3.6 support & upate ujson

after collection python version info metrics, we fount that only 2 users
are still using python 3.6. since it has been eol for 5 months now,
we believe it's safe to let it go.

rest easy 3.6 ⚰- you served us well 🫡

closes Please upgrade the ujson dependency, which has a CVE #160

* chore: upgrade and clean some deps
2022-06-20 12:19:09 +01:00
Gergő Jedlicska 52ab27e60f SQLite write batching (#188)
### SUMMARY
**sqlite transport**

This transport now batches and bulk inserts objects when writing resulting in huge performance improvements (100x).

**base object serializer**

Batching in the sqlite transport necessitated some refactoring here in order to safely call end_write when not using operations.send/receive. This has been resolved by turning traverse_base into a wrapper for _traverse_base which can take care of calling begin/end_write and resetting the writer at the top level. This is not breaking since the top level methods to call have not changed names and the original method has just been prepended with a _

Additionally, missing referenced child objects in the read transport used to raise a SpeckleException. However, using the gql client to call objects.get() will return an object with missing references by design thus throwing an error in serialization. This has been resolved by instead raising a SpeckleWarning when child objects can't be found and just returning the reference + id. ((this method of interacting with objects is discouraged so it is not surprising to me that this bug was lurking for so long - but an oopsie nonetheless!))

**ci / dev**

Updates for the ci config and the dev container to work with the recent changes in server.

NOTE: dev container seems to be pulling an older version of server -- not resolved yet

---

* quick and hacky sqlite batching

* feat(transports): batching sqlite inserts

* chore: upgrade gql3

also removed py-spy as it's not used and i was getting install errors :/

* ci: bump node version

* ci: formatting

* update CI versions

* update to new circleci redis baseimage

* update test fixture auth to non deprecated token based method

* add start and finish write method calls to base object serialize

* chore: dev container update

* fix(serialization): move end and begin write

* style: formatting

* fix(serializer): warn but don't throw if ref not found

this is _not_ an issue with the transports, but an issue with using the
graphql api to fetch objects. since you are only receiving one obj and none of
the children, the transport has no way to find them and should simply
return the reference as is. idk why anyone would really use `object.get`
so tbh i'm not surprised no one has found this bug yet lol

* fix(client): don't parse obj create response

* fix(serializer): wrap `traverse_base`

moving `begin` and `end_write` to the seriazlier due to the new
sqlite transport with batched writes necessitates a wrapper around
`traverse_base` so end/begin write can be called once at the top level.
just adding begin/end write to the original traversal method would make
tons of calls to `end_write` since the traversal is recursive

Co-authored-by: izzy lyseggen <izzy.lyseggen@gmail.com>
2022-06-20 12:00:09 +01:00
izzy lyseggen 61e7ebeabd fix(client): add faves count to mutation return (#186) 2022-04-25 11:45:54 +01:00
izzy lyseggen 3a8121c306 feat(client): stream model update + favoriting (#185)
* feat(models): a quickie lil update

- adds fave and comment count
(mutations to come)
- adds source app to commit within stream query

* feat(client): add favorite mutation
2022-04-25 11:32:24 +01:00
izzy lyseggen 209a95879f fix(metrics): patch app version check 2022-04-22 11:46:27 +01:00
izzy lyseggen 4f829d9908 feat(metrics): some cleanup and updates (#184)
- add metrics for client init / auth
- add server metrics
- remove incompatible server check in client
(at this point, it's been long enough that I think it's fine and will
save time on request / esp in places like blender)
2022-04-22 11:26:28 +01:00
luzpaz ac5345f528 Fix various typos (#181) 2022-04-21 17:56:11 +01:00
izzy lyseggen 1142481d89 fix(geometry): int(index vals) for curve encoding (#183)
* fix(geometry): int(index vals) for curve encoding

* fix(client): update poss invalid token check

server now returns `None` instead of a `GraphqlExcetion` when asking for
the user with an invalid token (or no scopes token)
2022-04-21 17:50:43 +01:00
izzy lyseggen b4690f082f feat(objects): revit params in objects for blender (#179) 2022-04-01 11:58:49 +01:00
izzy lyseggen 81a98ea938 feat(client): stream and user activity (#176)
* feat(models): `ActivityCollection` and `Activity`

* feat(client): stream activity method

* test(client): test for stream activity

* refactor(client): use datetime args for activity

* docs(client): clean up stream activity docstring

* feat/test(client): user activity
2022-03-23 17:34:10 +00:00
izzy lyseggen 9b387da77a feat(serialisation): enums (#175)
note that this won't re-serialise dynamic members as enums.
they will come back as ints for consistency w sharp

closes 🃏 Enum serialisation bug #174
2022-03-23 11:49:16 +00:00
izzy lyseggen d0724c7d06 fix(objects): brep display val fix (#170)
* fix(client): auth fix

* fix(objects): temp displayValue prop setter

will be removed in the future, but keeping it now for backwards compat
2022-03-02 14:17:04 +00:00
izzy lyseggen 1414a3611b fix(wrapper): use full url for creating shell account (#169)
used for creating a transport if you don't have a local account
for the specified server
2022-03-01 10:33:47 +00:00
izzy lyseggen a553c17c43 fix/test(serialization): null values in dicts (#168) 2022-02-24 11:31:04 +00:00
izzy lyseggen 0be3fac6ab docs: update streamwrapper docstring 2022-02-23 16:52:23 +00:00
izzy lyseggen 944e70221e refactor(auth&metrics): use accounts everywhere and switch metrics (#166)
* feat(metrics): wip

* refactor(auth): use accts instead of tokens

* fix(wrapper): delayed auth bug

* refactor(memory): quick fix

* fix(creds): change incompatible py 3.8+ syntax

* feat(anatylics): updates tracking

* fix(credentials): catch when no accts

* fix(metrics): remove unused field

* feat(wrapper): raise exception for old import

* feat(analytics): consolidate names
2022-02-23 11:00:04 +00:00
izzy lyseggen 21f13c4750 chore: lgtm fixes (#164)
* chore: clean imports

* chore: more lil fixes
2022-02-21 11:19:06 +00:00
izzy lyseggen be85ddd159 feat(server): force utf-8 encoding (#163)
objects were being split on non-english characters causing receive fails

will also get fixed server side, but this will act as a double check
and an immediate fix for people dealing with this now
2022-02-18 10:33:25 +00:00
izzy lyseggen 77c538ced9 feat(server): allow unauth server transport (#162)
for receiving public objects
2022-02-17 10:10:25 +00:00
47 changed files with 1752 additions and 1128 deletions
+9 -8
View File
@@ -1,16 +1,16 @@
version: 2.1
orbs:
python: circleci/python@1.3.2
python: circleci/python@2.0.3
codecov: codecov/codecov@3.2.2
jobs:
test:
docker:
- image: "cimg/python:<<parameters.tag>>"
- image: 'cimg/node:14.18'
- image: 'circleci/redis:6'
- image: 'cimg/postgres:12.8'
- image: "cimg/node:16.15"
- image: "cimg/redis:6.2"
- image: "cimg/postgres:14.2"
environment:
POSTGRES_DB: speckle2_test
POSTGRES_PASSWORD: speckle
@@ -27,6 +27,7 @@ jobs:
STRATEGY_LOCAL: "true"
CANONICAL_URL: "http://localhost:3000"
WAIT_HOSTS: localhost:5432, localhost:6379
DISABLE_FILE_UPLOADS: "true"
parameters:
tag:
default: "3.8"
@@ -51,7 +52,7 @@ jobs:
deploy:
docker:
- image: "circleci/python:3.8"
- image: "cimg/python:3.8"
steps:
- checkout
- run: python patch_version.py $CIRCLE_TAG
@@ -60,17 +61,17 @@ jobs:
workflows:
main:
jobs:
jobs:
- test:
matrix:
parameters:
tag: ["3.6", "3.7", "3.8", "3.9"]
tag: ["3.7", "3.8", "3.9", "3.10"]
filters:
tags:
only: /.*/
- deploy:
requires:
- test
- test
filters:
tags:
only: /[0-9]+(\.[0-9]+)*/
+3 -2
View File
@@ -1,7 +1,7 @@
version: "3.3" # optional since v1.27.0
services:
postgres:
image: circleci/postgres:12
image: cimg/postgres:14.2
environment:
POSTGRES_DB: speckle2_test
POSTGRES_PASSWORD: speckle
@@ -10,7 +10,7 @@ services:
# - "5432:5432"
network_mode: host
redis:
image: circleci/redis:6
image: cimg/redis:6.2
# ports:
# - "6379:6379"
network_mode: host
@@ -27,6 +27,7 @@ services:
STRATEGY_LOCAL: "true"
CANONICAL_URL: "http://localhost:3000"
WAIT_HOSTS: localhost:5432, localhost:6379
DISABLE_FILE_UPLOADS: "true"
# ports:
# - "3000:3000"
network_mode: host
+6 -2
View File
@@ -4,6 +4,7 @@
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Python: Current File",
"type": "python",
@@ -13,10 +14,13 @@
"justMyCode": false
},
{
"name": "Python: Test debug config",
"name": "Pytest",
"type": "python",
"request": "test",
"request": "launch",
"program": "poetry",
"args": ["run", "pytest"],
"console": "integratedTerminal",
"justMyCode": true
}
]
}
+63
View File
@@ -0,0 +1,63 @@
from typing import List
from specklepy.objects import Base
from specklepy.api import operations
from specklepy.transports.sqlite import SQLiteTransport
import time
from pathlib import Path
import os
import string
import random
class Sub(Base):
bar: List[str]
def random_string():
letters = string.ascii_lowercase
return "".join(random.choice(letters) for _ in range(10))
BASE_PATH = SQLiteTransport.get_base_path("Speckle")
def clean_db():
os.remove(Path(BASE_PATH, "Objects.db"))
def one_pass(clean: bool, randomize: bool, child_count: int):
foo = Base()
for i in range(child_count):
stuff = random_string() if randomize else "stuff"
foo[f"@child_{i}"] = Sub(bar=["asdf", "bar", i, stuff])
if clean:
clean_db()
transport = SQLiteTransport()
start = time.time()
hash = operations.send(base=foo, transports=[transport])
send_time = time.time() - start
receive_start = time.time()
operations.receive(hash, transport)
receive_time = time.time() - receive_start
return send_time, receive_time
if __name__ == "__main__":
sample_size = 4
test_permutations = [
(True, True),
(False, False),
(False, True),
(True, False),
]
for clean, randomize in test_permutations:
print(f"CLEAN: {clean}, RANDOMIZE: {randomize}")
for child_count in [10, 100, 1000, 10000]:
print(f"\tCHILD COUNT: {child_count}")
for _ in range(sample_size):
send_time, receive_time = one_pass(clean, randomize, child_count)
print(f"\t\tSend: {send_time} Receive: {receive_time}")
+2 -2
View File
@@ -50,10 +50,10 @@ if __name__ == "__main__":
)
# support for dynamic attributes
custom_sub.extra_extra = "what is this?"
debug(custom_sub.json())
debug(custom_sub)
serialized = operations.serialize(custom_sub)
deserialized = operations.deserialize(serialized)
# the only difference should be between the two data is that the deserialized
# instance id attribute is not None.
debug(deserialized.json())
debug(deserialized)
Generated
+569 -506
View File
File diff suppressed because it is too large Load Diff
+5 -3
View File
@@ -11,11 +11,12 @@ homepage = "https://speckle.systems/"
[tool.poetry.dependencies]
python = "^3.6.5"
python = ">=3.7.0, <4.0"
pydantic = "^1.8.2"
appdirs = "^1.4.4"
gql = {version = ">=3.0.0b1", extras = ["all"], allow-prereleases = true}
ujson = "^4.3.0"
gql = {extras = ["requests", "websockets"], version = "^3.3.0"}
ujson = "^5.3.0"
Deprecated = "^1.2.13"
[tool.poetry.dev-dependencies]
black = "^20.8b1"
@@ -23,6 +24,7 @@ isort = "^5.7.0"
pytest = "^6.2.2"
pytest-ordering = "^0.6"
pytest-cov = "^3.0.0"
devtools = "^0.8.0"
[tool.black]
+67 -30
View File
@@ -1,7 +1,9 @@
import re
from warnings import warn
from deprecated import deprecated
from specklepy.api.credentials import Account, get_account_from_token
from specklepy.logging import metrics
from specklepy.logging.exceptions import (
GraphQLException,
SpeckleException,
SpeckleWarning,
)
@@ -39,9 +41,9 @@ class SpeckleClient:
client = SpeckleClient(host="speckle.xyz") # or whatever your host is
# client = SpeckleClient(host="localhost:3000", use_ssl=False) or use local server
# authenticate the client with a token (account has been added in Speckle Manager)
# authenticate the client with an account (account has been added in Speckle Manager)
account = get_default_account()
client.authenticate(token=account.token)
client.authenticate_with_account(account)
# create a new stream. this returns the stream id
new_stream_id = client.stream.create(name="a shiny new stream")
@@ -55,6 +57,7 @@ class SpeckleClient:
USE_SSL = True
def __init__(self, host: str = DEFAULT_HOST, use_ssl: bool = USE_SSL) -> None:
metrics.track(metrics.CLIENT, custom_props={"name": "create"})
ws_protocol = "ws"
http_protocol = "http"
@@ -66,9 +69,9 @@ class SpeckleClient:
host = re.sub(r"((^\w+:|^)\/\/)|(\/$)", "", host)
self.url = f"{http_protocol}://{host}"
self.graphql = self.url + "/graphql"
self.graphql = f"{self.url}/graphql"
self.ws_url = f"{ws_protocol}://{host}/graphql"
self.me = None
self.account = Account()
self.httpclient = Client(
transport=RequestsHTTPTransport(url=self.graphql, verify=True, retries=3)
@@ -77,21 +80,25 @@ class SpeckleClient:
self._init_resources()
# Check compatibility with the server
try:
serverInfo = self.server.get()
if isinstance(serverInfo, Exception):
raise serverInfo
if not isinstance(serverInfo, ServerInfo):
raise Exception("Couldn't get ServerInfo")
except Exception as ex:
raise SpeckleException(f"{self.url} is not a compatible Speckle Server", ex)
# ? Check compatibility with the server - i think we can skip this at this point? save a request
# try:
# server_info = self.server.get()
# if isinstance(server_info, Exception):
# raise server_info
# if not isinstance(server_info, ServerInfo):
# raise Exception("Couldn't get ServerInfo")
# except Exception as ex:
# raise SpeckleException(
# f"{self.url} is not a compatible Speckle Server", ex
# ) from ex
def __repr__(self):
return (
f"SpeckleClient( server: {self.url}, authenticated: {self.me is not None} )"
)
return f"SpeckleClient( server: {self.url}, authenticated: {self.account.token is not None} )"
@deprecated(
version="2.6.0",
reason="Renamed: please use `authenticate_with_account` or `authenticate_with_token` instead.",
)
def authenticate(self, token: str) -> None:
"""Authenticate the client using a personal access token
The token is saved in the client object and a synchronous GraphQL entrypoint is created
@@ -99,9 +106,35 @@ class SpeckleClient:
Arguments:
token {str} -- an api token
"""
self.me = {"token": token}
self.authenticate_with_token(token)
self._set_up_client()
def authenticate_with_token(self, token: str) -> None:
"""Authenticate the client using a personal access token
The token is saved in the client object and a synchronous GraphQL entrypoint is created
Arguments:
token {str} -- an api token
"""
self.account = get_account_from_token(token, self.url)
metrics.track(metrics.CLIENT, self.account, {"name": "authenticate with token"})
self._set_up_client()
def authenticate_with_account(self, account: Account) -> None:
"""Authenticate the client using an Account object
The account is saved in the client object and a synchronous GraphQL entrypoint is created
Arguments:
account {Account} -- the account object which can be found with `get_default_account` or `get_local_accounts`
"""
metrics.track(metrics.CLIENT, account, {"name": "authenticate with account"})
self.account = account
self._set_up_client()
def _set_up_client(self) -> None:
metrics.track(metrics.CLIENT, self.account, {"name": "set up client"})
headers = {
"Authorization": f"Bearer {self.me['token']}",
"Authorization": f"Bearer {self.account.token}",
"Content-Type": "application/json",
}
httptransport = RequestsHTTPTransport(
@@ -109,17 +142,17 @@ class SpeckleClient:
)
wstransport = WebsocketsTransport(
url=self.ws_url,
init_payload={"Authorization": f"Bearer {self.me['token']}"},
init_payload={"Authorization": f"Bearer {self.account.token}"},
)
self.httpclient = Client(transport=httptransport)
self.wsclient = Client(transport=wstransport)
self._init_resources()
if isinstance(self.user.get(), GraphQLException):
if self.user.get() is None:
warn(
SpeckleWarning(
f"Invalid token - could not authenticate Speckle Client for server {self.url}"
f"Possibly invalid token - could not authenticate Speckle Client for server {self.url}"
)
)
@@ -128,23 +161,25 @@ class SpeckleClient:
def _init_resources(self) -> None:
self.stream = stream.Resource(
me=self.me, basepath=self.url, client=self.httpclient
account=self.account, basepath=self.url, client=self.httpclient
)
self.commit = commit.Resource(
me=self.me, basepath=self.url, client=self.httpclient
account=self.account, basepath=self.url, client=self.httpclient
)
self.branch = branch.Resource(
me=self.me, basepath=self.url, client=self.httpclient
account=self.account, basepath=self.url, client=self.httpclient
)
self.object = object.Resource(
me=self.me, basepath=self.url, client=self.httpclient
account=self.account, basepath=self.url, client=self.httpclient
)
self.server = server.Resource(
me=self.me, basepath=self.url, client=self.httpclient
account=self.account, basepath=self.url, client=self.httpclient
)
self.user = user.Resource(
account=self.account, basepath=self.url, client=self.httpclient
)
self.user = user.Resource(me=self.me, basepath=self.url, client=self.httpclient)
self.subscribe = subscriptions.Resource(
me=self.me,
account=self.account,
basepath=self.ws_url,
client=self.wsclient,
)
@@ -152,7 +187,9 @@ class SpeckleClient:
def __getattr__(self, name):
try:
attr = getattr(resources, name)
return attr.Resource(me=self.me, basepath=self.url, client=self.httpclient)
return attr.Resource(
account=self.account, basepath=self.url, client=self.httpclient
)
except:
raise SpeckleException(
f"Method {name} is not supported by the SpeckleClient class"
+49 -148
View File
@@ -1,29 +1,25 @@
import os
from warnings import warn
from pydantic import BaseModel
from pydantic import BaseModel, Field
from typing import List, Optional
from urllib.parse import urlparse, unquote
from specklepy.logging import metrics
from specklepy.api.models import ServerInfo
from specklepy.api.client import SpeckleClient
from specklepy.transports.sqlite import SQLiteTransport
from specklepy.transports.server.server import ServerTransport
from specklepy.logging.exceptions import SpeckleException, SpeckleWarning
from specklepy.logging.exceptions import SpeckleException
class UserInfo(BaseModel):
name: str
email: str
name: Optional[str]
email: Optional[str]
company: Optional[str]
id: str
id: Optional[str]
class Account(BaseModel):
isDefault: bool = None
token: str
isDefault: bool = False
token: str = None
refreshToken: str = None
serverInfo: ServerInfo
userInfo: UserInfo
serverInfo: ServerInfo = Field(default_factory=ServerInfo)
userInfo: UserInfo = Field(default_factory=UserInfo)
id: str = None
def __repr__(self) -> str:
@@ -32,6 +28,12 @@ class Account(BaseModel):
def __str__(self) -> str:
return self.__repr__()
@classmethod
def from_token(cls, token: str, server_url: str = None):
acct = cls(token=token)
acct.serverInfo.url = server_url
return acct
def get_local_accounts(base_path: str = None) -> List[Account]:
"""Gets all the accounts present in this environment
@@ -42,7 +44,6 @@ def get_local_accounts(base_path: str = None) -> List[Account]:
Returns:
List[Account] -- list of all local accounts or an empty list if no accounts were found
"""
metrics.track(metrics.ACCOUNT_LIST)
account_storage = SQLiteTransport(scope="Accounts", base_path=base_path)
json_path = os.path.join(account_storage._base_path, "Accounts")
os.makedirs(json_path, exist_ok=True)
@@ -63,6 +64,13 @@ def get_local_accounts(base_path: str = None) -> List[Account]:
"Invalid json accounts could not be read. Please fix or remove them.",
ex,
)
metrics.track(
metrics.ACCOUNTS,
next(
(acc for acc in accounts if acc.isDefault),
accounts[0] if accounts else None,
),
)
return accounts
@@ -75,7 +83,6 @@ def get_default_account(base_path: str = None) -> Account:
Returns:
Account -- the default account or None if no local accounts were found
"""
metrics.track(metrics.ACCOUNT_DEFAULT)
accounts = get_local_accounts(base_path=base_path)
if not accounts:
return None
@@ -84,147 +91,41 @@ def get_default_account(base_path: str = None) -> Account:
if not default:
default = accounts[0]
default.isDefault = True
metrics.initialise_tracker(default)
return default
class StreamWrapper:
def get_account_from_token(token: str, server_url: str = None) -> Account:
"""Gets the local account for the token if it exists
Arguments:
token {str} -- the api token
Returns:
Account -- the local account with this token or a shell account containing just the token and url if no local account is found
"""
The `StreamWrapper` gives you some handy helpers to deal with urls and get authenticated clients and transports.
accounts = get_local_accounts()
if not accounts:
return Account.from_token(token, server_url)
Construct a `StreamWrapper` with a stream, branch, commit, or object URL. The corresponding ids will be stored
in the wrapper. If you have local accounts on the machine, you can use the `get_account` and `get_client` methods
to get a local account for the server. You can also pass a token into `get_client` if you don't have a corresponding
local account for the server.
acct = next((acc for acc in accounts if acc.token == token), None)
if acct:
return acct
```py
from specklepy.api.credentials import StreamWrapper
# provide any stream, branch, commit, object, or globals url
wrapper = StreamWrapper("https://speckle.xyz/streams/3073b96e86/commits/604bea8cc6")
# get an authenticated SpeckleClient if you have a local account for the server
client = wrapper.get_client()
# get an authenticated ServerTransport if you have a local account for the server
transport = wrapper.get_transport()
```
"""
stream_url: str = None
use_ssl: bool = True
host: str = None
stream_id: str = None
commit_id: str = None
object_id: str = None
branch_name: str = None
_client: SpeckleClient = None
_account: Account = None
def __repr__(self):
return f"StreamWrapper( server: {self.host}, stream_id: {self.stream_id}, type: {self.type} )"
def __str__(self) -> str:
return self.__repr__()
@property
def type(self) -> str:
if self.object_id:
return "object"
elif self.commit_id:
return "commit"
elif self.branch_name:
return "branch"
else:
return "stream" if self.stream_id else "invalid"
def __init__(self, url: str) -> None:
metrics.track("streamwrapper")
self.stream_url = url
parsed = urlparse(url)
self.host = parsed.netloc
self.use_ssl = parsed.scheme == "https"
segments = parsed.path.strip("/").split("/", 3)
if not segments or len(segments) < 2:
raise SpeckleException(
f"Cannot parse {url} into a stream wrapper class - invalid URL provided."
)
while segments:
segment = segments.pop(0)
if segments and segment.lower() == "streams":
self.stream_id = segments.pop(0)
elif segments and segment.lower() == "commits":
self.commit_id = segments.pop(0)
elif segments and segment.lower() == "branches":
self.branch_name = unquote(segments.pop(0))
elif segments and segment.lower() == "objects":
self.object_id = segments.pop(0)
elif segment.lower() == "globals":
self.branch_name = "globals"
if segments:
self.commit_id = segments.pop(0)
else:
raise SpeckleException(
f"Cannot parse {url} into a stream wrapper class - invalid URL provided."
)
if not self.stream_id:
raise SpeckleException(
f"Cannot parse {url} into a stream wrapper class - no stream id found."
)
def get_account(self) -> Account:
"""
Gets an account object for this server from the local accounts db (added via Speckle Manager or a json file)
"""
if self._account:
return self._account
self._account = next(
(a for a in get_local_accounts() if self.host in a.serverInfo.url),
None,
if server_url:
url = server_url.lower()
acct = next(
(acc for acc in accounts if url in acc.serverInfo.url.lower()), None
)
if acct:
return acct
return self._account
return Account.from_token(token, server_url)
def get_client(self, token: str = None) -> SpeckleClient:
"""
Gets an authenticated client for this server. You may provide a token if there aren't any local accounts on this
machine. If no account is found and no token is provided, an unauthenticated client is returned.
Arguments:
token {str} -- optional token if no local account is available (defaults to None)
Returns:
SpeckleClient -- authenticated with a corresponding local account or the provided token
"""
if self._client and token is None:
return self._client
if not self._account:
self.get_account()
if not self._client:
self._client = SpeckleClient(host=self.host, use_ssl=self.use_ssl)
if self._account is None and token is None:
warn(f"No local account found for server {self.host}", SpeckleWarning)
return self._client
self._client.authenticate(self._account.token if self._account else token)
return self._client
def get_transport(self, token: str = None) -> ServerTransport:
"""
Gets a server transport for this stream using an authenticated client. If there is no local account for this
server and the client was not authenticated with a token, this will throw an exception.
Returns:
ServerTransport -- constructed for this stream with a pre-authenticated client
"""
if not self._client or not self._client.me:
self.get_client(token)
return ServerTransport(self.stream_id, self._client)
class StreamWrapper:
def __init__(self, url: str = None) -> None:
raise SpeckleException(
message="The StreamWrapper has moved as of v2.6.0! Please import from specklepy.api.wrapper",
exception=DeprecationWarning,
)
+39 -6
View File
@@ -23,7 +23,7 @@ class Commit(BaseModel):
authorId: Optional[str]
authorAvatar: Optional[str]
branchName: Optional[str]
createdAt: Optional[str]
createdAt: Optional[datetime]
sourceApplication: Optional[str]
referencedObject: Optional[str]
totalChildrenCount: Optional[int]
@@ -38,7 +38,7 @@ class Commit(BaseModel):
class Commits(BaseModel):
totalCount: Optional[int]
cursor: Optional[Any]
cursor: Optional[datetime]
items: List[Commit] = []
@@ -47,7 +47,7 @@ class Object(BaseModel):
speckleType: Optional[str]
applicationId: Optional[str]
totalChildrenCount: Optional[int]
createdAt: Optional[str]
createdAt: Optional[datetime]
class Branch(BaseModel):
@@ -66,14 +66,18 @@ class Branches(BaseModel):
class Stream(BaseModel):
id: Optional[str]
name: Optional[str]
description: Optional[str]
role: Optional[str]
isPublic: Optional[bool]
createdAt: Optional[str]
updatedAt: Optional[str]
description: Optional[str]
createdAt: Optional[datetime]
updatedAt: Optional[datetime]
collaborators: List[Collaborator] = []
branches: Optional[Branches]
commit: Optional[Commit]
object: Optional[Object]
commentCount: Optional[int]
favoritedDate: Optional[datetime]
favoritesCount: Optional[int]
def __repr__(self):
return f"Stream( id: {self.id}, name: {self.name}, description: {self.description}, isPublic: {self.isPublic})"
@@ -106,6 +110,35 @@ class User(BaseModel):
return self.__repr__()
class Activity(BaseModel):
actionType: Optional[str]
info: Optional[dict]
userId: Optional[str]
streamId: Optional[str]
resourceId: Optional[str]
resourceType: Optional[str]
message: Optional[str]
time: Optional[datetime]
def __repr__(self) -> str:
return f"Activity( streamId: {self.streamId}, actionType: {self.actionType}, message: {self.message}, userId: {self.userId} )"
def __str__(self) -> str:
return self.__repr__()
class ActivityCollection(BaseModel):
totalCount: Optional[int]
items: Optional[List[Activity]]
cursor: Optional[datetime]
def __repr__(self) -> str:
return f"ActivityCollection( totalCount: {self.totalCount}, items: {len(self.items) if self.items else 0}, cursor: {self.cursor.isoformat()} )"
def __str__(self) -> str:
return self.__repr__()
class ServerInfo(BaseModel):
name: Optional[str]
company: Optional[str]
+14 -11
View File
@@ -2,6 +2,7 @@ from typing import List
from specklepy.logging import metrics
from specklepy.objects.base import Base
from specklepy.transports.sqlite import SQLiteTransport
from specklepy.transports.server import ServerTransport
from specklepy.logging.exceptions import SpeckleException
from specklepy.transports.abstract_transport import AbstractTransport
from specklepy.serialization.base_object_serializer import BaseObjectSerializer
@@ -9,7 +10,7 @@ from specklepy.serialization.base_object_serializer import BaseObjectSerializer
def send(
base: Base,
transports: List[AbstractTransport] = [],
transports: List[AbstractTransport] = None,
use_default_cache: bool = True,
):
"""Sends an object via the provided transports. Defaults to the local cache.
@@ -22,24 +23,26 @@ def send(
Returns:
str -- the object id of the sent object
"""
metrics.track(metrics.SEND)
if not transports and not use_default_cache:
raise SpeckleException(
message="You need to provide at least one transport: cannot send with an empty transport list and no default cache"
)
if transports is None:
metrics.track(metrics.SEND)
transports = []
else:
metrics.track(metrics.SEND, getattr(transports[0], "account", None))
if use_default_cache:
transports.insert(0, SQLiteTransport())
serializer = BaseObjectSerializer(write_transports=transports)
for t in transports:
t.begin_write()
hash, _ = serializer.write_json(base=base)
obj_hash, _ = serializer.write_json(base=base)
for t in transports:
t.end_write()
return hash
return obj_hash
def receive(
@@ -58,13 +61,13 @@ def receive(
Returns:
Base -- the base object
"""
metrics.track(metrics.RECEIVE)
metrics.track(metrics.RECEIVE, getattr(remote_transport, "account", None))
if not local_transport:
local_transport = SQLiteTransport()
serializer = BaseObjectSerializer(read_transport=local_transport)
# try local transport first. if the parent is there, we assume all the children are there and continue wth deserialisation using the local transport
# try local transport first. if the parent is there, we assume all the children are there and continue with deserialisation using the local transport
obj_string = local_transport.get_object(obj_id)
if obj_string:
return serializer.read_json(obj_string=obj_string)
+3 -2
View File
@@ -1,3 +1,4 @@
from specklepy.api.credentials import Account
from specklepy.transports.sqlite import SQLiteTransport
from typing import Dict, List
from gql.client import Client
@@ -10,13 +11,13 @@ from specklepy.serialization.base_object_serializer import BaseObjectSerializer
class ResourceBase(object):
def __init__(
self,
me: Dict,
account: Account,
basepath: str,
client: Client,
name: str,
methods: list,
) -> None:
self.me = me
self.account = account
self.basepath = basepath
self.client = client
self.name = name
+13 -9
View File
@@ -1,9 +1,7 @@
from specklepy.api.resources import stream
from typing import List, Optional
from gql import gql
from pydantic.main import BaseModel
from specklepy.api.resource import ResourceBase
from specklepy.api.models import Branch
from specklepy.logging import metrics
NAME = "branch"
METHODS = ["create"]
@@ -12,9 +10,13 @@ METHODS = ["create"]
class Resource(ResourceBase):
"""API Access class for branches"""
def __init__(self, me, basepath, client) -> None:
def __init__(self, account, basepath, client) -> None:
super().__init__(
me=me, basepath=basepath, client=client, name=NAME, methods=METHODS
account=account,
basepath=basepath,
client=client,
name=NAME,
methods=METHODS,
)
self.schema = Branch
@@ -30,7 +32,7 @@ class Resource(ResourceBase):
Returns:
id {str} -- the newly created branch's id
"""
metrics.track(metrics.BRANCH, self.account, {"name": "create"})
query = gql(
"""
mutation BranchCreate($branch: BranchCreateInput!) {
@@ -61,7 +63,7 @@ class Resource(ResourceBase):
Returns:
Branch -- the fetched branch with its latest commits
"""
metrics.track(metrics.BRANCH, self.account, {"name": "get"})
query = gql(
"""
query BranchGet($stream_id: String!, $name: String!, $commits_limit: Int!) {
@@ -109,6 +111,7 @@ class Resource(ResourceBase):
Returns:
List[Branch] -- the branches on the stream
"""
metrics.track(metrics.BRANCH, self.account, {"name": "get"})
query = gql(
"""
query BranchesGet($stream_id: String!, $branches_limit: Int!, $commits_limit: Int!) {
@@ -161,8 +164,9 @@ class Resource(ResourceBase):
description {str} -- optional: the updated branch description
Returns:
bool -- True if update is successfull
bool -- True if update is successful
"""
metrics.track(metrics.BRANCH, self.account, {"name": "update"})
query = gql(
"""
mutation BranchUpdate($branch: BranchUpdateInput!) {
@@ -196,7 +200,7 @@ class Resource(ResourceBase):
Returns:
bool -- True if deletion is successful
"""
metrics.track(metrics.BRANCH, self.account, {"name": "delete"})
query = gql(
"""
mutation BranchDelete($branch: BranchDeleteInput!) {
+12 -2
View File
@@ -2,6 +2,7 @@ from typing import Optional, List
from gql import gql
from specklepy.api.resource import ResourceBase
from specklepy.api.models import Commit
from specklepy.logging import metrics
NAME = "commit"
@@ -11,9 +12,13 @@ METHODS = []
class Resource(ResourceBase):
"""API Access class for commits"""
def __init__(self, me, basepath, client) -> None:
def __init__(self, account, basepath, client) -> None:
super().__init__(
me=me, basepath=basepath, client=client, name=NAME, methods=METHODS
account=account,
basepath=basepath,
client=client,
name=NAME,
methods=METHODS,
)
self.schema = Commit
@@ -66,6 +71,7 @@ class Resource(ResourceBase):
Returns:
List[Commit] -- a list of the most recent commit objects
"""
metrics.track(metrics.COMMIT, self.account, {"name": "get"})
query = gql(
"""
query Commits($stream_id: String!, $limit: Int!) {
@@ -119,6 +125,7 @@ class Resource(ResourceBase):
Returns:
str -- the id of the created commit
"""
metrics.track(metrics.COMMIT, self.account, {"name": "create"})
query = gql(
"""
mutation CommitCreate ($commit: CommitCreateInput!){ commitCreate(commit: $commit)}
@@ -152,6 +159,7 @@ class Resource(ResourceBase):
Returns:
bool -- True if the operation succeeded
"""
metrics.track(metrics.COMMIT, self.account, {"name": "update"})
query = gql(
"""
mutation CommitUpdate($commit: CommitUpdateInput!){ commitUpdate(commit: $commit)}
@@ -176,6 +184,7 @@ class Resource(ResourceBase):
Returns:
bool -- True if the operation succeeded
"""
metrics.track(metrics.COMMIT, self.account, {"name": "delete"})
query = gql(
"""
mutation CommitDelete($commit: CommitDeleteInput!){ commitDelete(commit: $commit)}
@@ -197,6 +206,7 @@ class Resource(ResourceBase):
"""
Mark a commit object a received by the source application.
"""
metrics.track(metrics.COMMIT, self.account, {"name": "received"})
query = gql(
"""
mutation CommitReceive($receivedInput:CommitReceivedInput!){
+9 -4
View File
@@ -1,6 +1,5 @@
from typing import Dict, List
from gql import gql
from graphql.language import parser
from specklepy.api.resource import ResourceBase
from specklepy.objects.base import Base
@@ -11,9 +10,13 @@ METHODS = []
class Resource(ResourceBase):
"""API Access class for objects"""
def __init__(self, me, basepath, client) -> None:
def __init__(self, account, basepath, client) -> None:
super().__init__(
me=me, basepath=basepath, client=client, name=NAME, methods=METHODS
account=account,
basepath=basepath,
client=client,
name=NAME,
methods=METHODS,
)
self.schema = Base
@@ -49,7 +52,9 @@ class Resource(ResourceBase):
params = {"stream_id": stream_id, "object_id": object_id}
return self.make_request(
query=query, params=params, return_type=["stream", "object", "data"]
query=query,
params=params,
return_type=["stream", "object", "data"],
)
def create(self, stream_id: str, objects: List[Dict]) -> str:
+11 -3
View File
@@ -1,8 +1,8 @@
from typing import Dict, List
from gql import gql
from gql.client import Client
from specklepy.api.models import ServerInfo
from specklepy.api.resource import ResourceBase
from specklepy.logging import metrics
NAME = "server"
@@ -12,9 +12,13 @@ METHODS = ["get", "apps"]
class Resource(ResourceBase):
"""API Access class for the server"""
def __init__(self, me, basepath, client) -> None:
def __init__(self, account, basepath, client) -> None:
super().__init__(
me=me, basepath=basepath, client=client, name=NAME, methods=METHODS
account=account,
basepath=basepath,
client=client,
name=NAME,
methods=METHODS,
)
def get(self) -> ServerInfo:
@@ -23,6 +27,7 @@ class Resource(ResourceBase):
Returns:
dict -- the server info in dictionary form
"""
metrics.track(metrics.SERVER, self.account, {"name": "get"})
query = gql(
"""
query Server {
@@ -62,6 +67,7 @@ class Resource(ResourceBase):
Returns:
dict -- a dictionary of apps registered on the server
"""
metrics.track(metrics.SERVER, self.account, {"name": "apps"})
query = gql(
"""
query Apps {
@@ -95,6 +101,7 @@ class Resource(ResourceBase):
Returns:
str -- the new API token. note: this is the only time you'll see the token!
"""
metrics.track(metrics.SERVER, self.account, {"name": "create_token"})
query = gql(
"""
mutation TokenCreate($token: ApiTokenCreateInput!) {
@@ -120,6 +127,7 @@ class Resource(ResourceBase):
Returns:
bool -- True if the token was successfully deleted
"""
metrics.track(metrics.SERVER, self.account, {"name": "revoke_token"})
query = gql(
"""
mutation TokenRevoke($token: String!) {
+219 -108
View File
@@ -1,27 +1,26 @@
from datetime import datetime, timezone
from gql import gql
from typing import Dict, List, Optional
from typing import List
from specklepy.logging import metrics
from specklepy.api.models import Stream
from specklepy.api.models import ActivityCollection, Stream
from specklepy.api.resource import ResourceBase
from specklepy.logging.exceptions import SpeckleException
NAME = "stream"
METHODS = [
"list",
"create",
"get",
"update",
"delete",
"search",
]
METHODS = ["list", "create", "get", "update", "delete", "search", "activity"]
class Resource(ResourceBase):
"""API Access class for streams"""
def __init__(self, me, basepath, client) -> None:
def __init__(self, account, basepath, client) -> None:
super().__init__(
me=me, basepath=basepath, client=client, name=NAME, methods=METHODS
account=account,
basepath=basepath,
client=client,
name=NAME,
methods=METHODS,
)
self.schema = Stream
@@ -37,45 +36,49 @@ class Resource(ResourceBase):
Returns:
Stream -- the retrieved stream
"""
metrics.track(metrics.STREAM_GET)
metrics.track(metrics.STREAM, self.account, {"name": "get"})
query = gql(
"""
query Stream($id: String!, $branch_limit: Int!, $commit_limit: Int!) {
stream(id: $id) {
id
name
description
isPublic
createdAt
updatedAt
collaborators {
id
name
role
avatar
}
branches(limit: $branch_limit) {
totalCount
cursor
items {
stream(id: $id) {
id
name
role
description
commits(limit: $commit_limit) {
totalCount
cursor
items {
isPublic
createdAt
updatedAt
commentCount
favoritesCount
collaborators {
id
referencedObject
message
authorName
authorId
createdAt
}
name
role
avatar
}
}
branches(limit: $branch_limit) {
totalCount
cursor
items {
id
name
description
commits(limit: $commit_limit) {
totalCount
cursor
items {
id
message
authorId
createdAt
authorName
referencedObject
sourceApplication
}
}
}
}
}
}
}
"""
)
@@ -93,38 +96,41 @@ class Resource(ResourceBase):
Returns:
List[Stream] -- A list of Stream objects
"""
metrics.track(metrics.STREAM_LIST)
metrics.track(metrics.STREAM, self.account, {"name": "get"})
query = gql(
"""
query User($stream_limit: Int!) {
user {
id
email
name
bio
company
avatar
verified
profiles
role
streams(limit: $stream_limit) {
totalCount
cursor
items {
user {
id
bio
name
description
isPublic
createdAt
updatedAt
collaborators {
id
name
role
}
email
avatar
company
verified
profiles
role
streams(limit: $stream_limit) {
totalCount
cursor
items {
id
name
role
isPublic
createdAt
updatedAt
description
commentCount
favoritesCount
collaborators {
id
name
role
}
}
}
}
}
}
}
"""
)
@@ -151,7 +157,7 @@ class Resource(ResourceBase):
Returns:
id {str} -- the id of the newly created stream
"""
metrics.track(metrics.STREAM_CREATE)
metrics.track(metrics.STREAM, self.account, {"name": "create"})
query = gql(
"""
mutation StreamCreate($stream: StreamCreateInput!) {
@@ -182,11 +188,11 @@ class Resource(ResourceBase):
Returns:
bool -- whether the stream update was successful
"""
metrics.track(metrics.STREAM_UPDATE)
metrics.track(metrics.STREAM, self.account, {"name": "update"})
query = gql(
"""
mutation StreamUpdate($stream: StreamUpdateInput!) {
streamUpdate(stream: $stream)
streamUpdate(stream: $stream)
}
"""
)
@@ -213,13 +219,13 @@ class Resource(ResourceBase):
Returns:
bool -- whether the deletion was successful
"""
metrics.track(metrics.STREAM_DELETE)
metrics.track(metrics.STREAM, self.account, {"name": "delete"})
query = gql(
"""
mutation StreamDelete($id: String!) {
streamDelete(id: $id)
streamDelete(id: $id)
}
"""
"""
)
params = {"id": id}
@@ -246,47 +252,48 @@ class Resource(ResourceBase):
Returns:
List[Stream] -- a list of Streams that match the search query
"""
metrics.track(metrics.STREAM_SEARCH)
metrics.track(metrics.STREAM, self.account, {"name": "search"})
query = gql(
"""
query StreamSearch($search_query: String!,$limit: Int!, $branch_limit:Int!, $commit_limit:Int!) {
streams(query: $search_query, limit: $limit) {
items {
id
name
description
isPublic
createdAt
updatedAt
collaborators {
id
name
role
avatar
}
branches(limit: $branch_limit) {
totalCount
cursor
streams(query: $search_query, limit: $limit) {
items {
id
name
description
commits(limit: $commit_limit) {
totalCount
cursor
items {
id
referencedObject
message
authorName
authorId
createdAt
id
name
role
description
isPublic
createdAt
updatedAt
collaborators {
id
name
role
avatar
}
branches(limit: $branch_limit) {
totalCount
cursor
items {
id
name
description
commits(limit: $commit_limit) {
totalCount
cursor
items {
id
referencedObject
message
authorName
authorId
createdAt
}
}
}
}
}
}
}
}
}
}
"""
)
@@ -302,6 +309,39 @@ class Resource(ResourceBase):
query=query, params=params, return_type=["streams", "items"]
)
def favorite(self, stream_id: str, favorited: bool = True):
"""Favorite or unfavorite the given stream.
Arguments:
stream_id {str} -- the id of the stream to favorite / unfavorite
favorited {bool} -- whether to favorite (True) or unfavorite (False) the stream
Returns:
Stream -- the stream with its `id`, `name`, and `favoritedDate`
"""
metrics.track(metrics.STREAM, self.account, {"name": "favorite"})
query = gql(
"""
mutation StreamFavorite($stream_id: String!, $favorited: Boolean!) {
streamFavorite(streamId: $stream_id, favorited: $favorited) {
id
name
favoritedDate
favoritesCount
}
}
"""
)
params = {
"stream_id": stream_id,
"favorited": favorited,
}
return self.make_request(
query=query, params=params, return_type=["streamFavorite"]
)
def grant_permission(self, stream_id: str, user_id: str, role: str):
"""Grant permissions to a user on a given stream
@@ -313,6 +353,7 @@ class Resource(ResourceBase):
Returns:
bool -- True if the operation was successful
"""
metrics.track(metrics.PERMISSION, self.account, {"name": "add", "role": role})
query = gql(
"""
mutation StreamGrantPermission($permission_params: StreamGrantPermissionInput !) {
@@ -346,6 +387,7 @@ class Resource(ResourceBase):
Returns:
bool -- True if the operation was successful
"""
metrics.track(metrics.PERMISSION, self.account, {"name": "revoke"})
query = gql(
"""
mutation StreamRevokePermission($permission_params: StreamRevokePermissionInput !) {
@@ -362,3 +404,72 @@ class Resource(ResourceBase):
return_type="streamRevokePermission",
parse_response=False,
)
def activity(
self,
stream_id: str,
action_type: str = None,
limit: int = 20,
before: datetime = None,
after: datetime = None,
cursor: datetime = None,
):
"""
Get the activity from a given stream in an Activity collection. Step into the activity `items` for the list of activity.
Note: all timestamps arguments should be `datetime` of any tz as they will be converted to UTC ISO format strings
stream_id {str} -- the id of the stream to get activity from
action_type {str} -- filter results to a single action type (eg: `commit_create` or `commit_receive`)
limit {int} -- max number of Activity items to return
before {datetime} -- latest cutoff for activity (ie: return all activity _before_ this time)
after {datetime} -- oldest cutoff for activity (ie: return all activity _after_ this time)
cursor {datetime} -- timestamp cursor for pagination
"""
query = gql(
"""
query StreamActivity($stream_id: String!, $action_type: String, $before:DateTime, $after: DateTime, $cursor: DateTime, $limit: Int){
stream(id: $stream_id) {
activity(actionType: $action_type, before: $before, after: $after, cursor: $cursor, limit: $limit) {
totalCount
cursor
items {
actionType
info
userId
streamId
resourceId
resourceType
message
time
}
}
}
}
"""
)
try:
params = {
"stream_id": stream_id,
"limit": limit,
"action_type": action_type,
"before": before.astimezone(timezone.utc).isoformat()
if before
else before,
"after": after.astimezone(timezone.utc).isoformat() if after else after,
"cursor": cursor.astimezone(timezone.utc).isoformat()
if cursor
else cursor,
}
except AttributeError as e:
raise SpeckleException(
"Could not get stream activity - `before`, `after`, and `cursor` must be in `datetime` format if provided",
ValueError,
) from e
return self.make_request(
query=query,
params=params,
return_type=["stream", "activity"],
schema=ActivityCollection,
)
+8 -4
View File
@@ -1,9 +1,9 @@
from typing import Callable, Dict, List, Optional, Any
from typing import Callable, Dict, List
from functools import wraps
from gql import gql
from specklepy.api.resource import ResourceBase
from specklepy.api.resources.stream import Stream
from specklepy.logging.exceptions import GraphQLException, SpeckleException
from specklepy.logging.exceptions import SpeckleException
NAME = "subscribe"
METHODS = [
@@ -29,9 +29,13 @@ def check_wsclient(function):
class Resource(ResourceBase):
"""API Access class for subscriptions"""
def __init__(self, me, basepath, client) -> None:
def __init__(self, account, basepath, client) -> None:
super().__init__(
me=me, basepath=basepath, client=client, name=NAME, methods=METHODS
account=account,
basepath=basepath,
client=client,
name=NAME,
methods=METHODS,
)
@check_wsclient
+76 -6
View File
@@ -1,20 +1,25 @@
from datetime import datetime, timezone
from specklepy.logging import metrics
from specklepy.logging.exceptions import SpeckleException
from typing import List, Optional
from typing import List
from gql import gql
from pydantic.main import BaseModel
from specklepy.api.resource import ResourceBase
from specklepy.api.models import User
from specklepy.api.models import ActivityCollection, User
NAME = "user"
METHODS = ["get"]
METHODS = ["get", "search", "update", "activity"]
class Resource(ResourceBase):
"""API Access class for users"""
def __init__(self, me, basepath, client) -> None:
def __init__(self, account, basepath, client) -> None:
super().__init__(
me=me, basepath=basepath, client=client, name=NAME, methods=METHODS
account=account,
basepath=basepath,
client=client,
name=NAME,
methods=METHODS,
)
self.schema = User
@@ -27,6 +32,7 @@ class Resource(ResourceBase):
Returns:
User -- the retrieved user
"""
metrics.track(metrics.USER, self.account, {"name": "get"})
query = gql(
"""
query User($id: String) {
@@ -63,6 +69,7 @@ class Resource(ResourceBase):
message="User search query must be at least 3 characters"
)
metrics.track(metrics.USER, self.account, {"name": "search"})
query = gql(
"""
query UserSearch($search_query: String!, $limit: Int!) {
@@ -99,6 +106,7 @@ class Resource(ResourceBase):
Returns:
bool -- True if your profile was updated successfully
"""
metrics.track(metrics.USER, self.account, {"name": "update"})
query = gql(
"""
mutation UserUpdate($user: UserUpdateInput!) {
@@ -118,3 +126,65 @@ class Resource(ResourceBase):
return self.make_request(
query=query, params=params, return_type="userUpdate", parse_response=False
)
def activity(
self,
user_id: str = None,
limit: int = 20,
action_type: str = None,
before: datetime = None,
after: datetime = None,
cursor: datetime = None,
):
"""
Get the activity from a given stream in an Activity collection. Step into the activity `items` for the list of activity.
If no id argument is provided, will return the current authenticated user's activity (as extracted from the authorization header).
Note: all timestamps arguments should be `datetime` of any tz as they will be converted to UTC ISO format strings
user_id {str} -- the id of the user to get the activity from
action_type {str} -- filter results to a single action type (eg: `commit_create` or `commit_receive`)
limit {int} -- max number of Activity items to return
before {datetime} -- latest cutoff for activity (ie: return all activity _before_ this time)
after {datetime} -- oldest cutoff for activity (ie: return all activity _after_ this time)
cursor {datetime} -- timestamp cursor for pagination
"""
query = gql(
"""
query UserActivity($user_id: String, $action_type: String, $before:DateTime, $after: DateTime, $cursor: DateTime, $limit: Int){
user(id: $user_id) {
activity(actionType: $action_type, before: $before, after: $after, cursor: $cursor, limit: $limit) {
totalCount
cursor
items {
actionType
info
userId
streamId
resourceId
resourceType
message
time
}
}
}
}
"""
)
params = {
"user_id": user_id,
"limit": limit,
"action_type": action_type,
"before": before.astimezone(timezone.utc).isoformat() if before else before,
"after": after.astimezone(timezone.utc).isoformat() if after else after,
"cursor": cursor.astimezone(timezone.utc).isoformat() if cursor else cursor,
}
return self.make_request(
query=query,
params=params,
return_type=["user", "activity"],
schema=ActivityCollection,
)
+1 -1
View File
@@ -445,7 +445,7 @@ input ServerInfoUpdateInput {
stream( id: String! ): Stream
"""
All the streams of the current user, pass in the `query` parameter to seach by name, description or ID.
All the streams of the current user, pass in the `query` parameter to search by name, description or ID.
"""
streams( query: String, limit: Int = 25, cursor: String ): StreamCollection
@hasScope(scope: "streams:read")
+166
View File
@@ -0,0 +1,166 @@
from warnings import warn
from urllib.parse import urlparse, unquote
from specklepy.api.credentials import (
Account,
get_account_from_token,
get_local_accounts,
)
from specklepy.logging import metrics
from specklepy.api.client import SpeckleClient
from specklepy.transports.server.server import ServerTransport
from specklepy.logging.exceptions import SpeckleException, SpeckleWarning
class StreamWrapper:
"""
The `StreamWrapper` gives you some handy helpers to deal with urls and get authenticated clients and transports.
Construct a `StreamWrapper` with a stream, branch, commit, or object URL. The corresponding ids will be stored
in the wrapper. If you have local accounts on the machine, you can use the `get_account` and `get_client` methods
to get a local account for the server. You can also pass a token into `get_client` if you don't have a corresponding
local account for the server.
```py
from specklepy.api.wrapper import StreamWrapper
# provide any stream, branch, commit, object, or globals url
wrapper = StreamWrapper("https://speckle.xyz/streams/3073b96e86/commits/604bea8cc6")
# get an authenticated SpeckleClient if you have a local account for the server
client = wrapper.get_client()
# get an authenticated ServerTransport if you have a local account for the server
transport = wrapper.get_transport()
```
"""
stream_url: str = None
use_ssl: bool = True
host: str = None
stream_id: str = None
commit_id: str = None
object_id: str = None
branch_name: str = None
_client: SpeckleClient = None
_account: Account = None
def __repr__(self):
return f"StreamWrapper( server: {self.host}, stream_id: {self.stream_id}, type: {self.type} )"
def __str__(self) -> str:
return self.__repr__()
@property
def type(self) -> str:
if self.object_id:
return "object"
elif self.commit_id:
return "commit"
elif self.branch_name:
return "branch"
else:
return "stream" if self.stream_id else "invalid"
def __init__(self, url: str) -> None:
self.stream_url = url
parsed = urlparse(url)
self.host = parsed.netloc
self.use_ssl = parsed.scheme == "https"
segments = parsed.path.strip("/").split("/", 3)
metrics.track(metrics.STREAM_WRAPPER, self.get_account())
if not segments or len(segments) < 2:
raise SpeckleException(
f"Cannot parse {url} into a stream wrapper class - invalid URL provided."
)
while segments:
segment = segments.pop(0)
if segments and segment.lower() == "streams":
self.stream_id = segments.pop(0)
elif segments and segment.lower() == "commits":
self.commit_id = segments.pop(0)
elif segments and segment.lower() == "branches":
self.branch_name = unquote(segments.pop(0))
elif segments and segment.lower() == "objects":
self.object_id = segments.pop(0)
elif segment.lower() == "globals":
self.branch_name = "globals"
if segments:
self.commit_id = segments.pop(0)
else:
raise SpeckleException(
f"Cannot parse {url} into a stream wrapper class - invalid URL provided."
)
if not self.stream_id:
raise SpeckleException(
f"Cannot parse {url} into a stream wrapper class - no stream id found."
)
@property
def server_url(self):
return f"{'https' if self.use_ssl else 'http'}://{self.host}"
def get_account(self, token: str = None) -> Account:
"""
Gets an account object for this server from the local accounts db (added via Speckle Manager or a json file)
"""
if self._account and self._account.token:
return self._account
self._account = next(
(a for a in get_local_accounts() if self.host in a.serverInfo.url),
None,
)
if not self._account:
self._account = get_account_from_token(token, self.server_url)
if self._client:
self._client.authenticate_with_account(self._account)
return self._account
def get_client(self, token: str = None) -> SpeckleClient:
"""
Gets an authenticated client for this server. You may provide a token if there aren't any local accounts on this
machine. If no account is found and no token is provided, an unauthenticated client is returned.
Arguments:
token {str} -- optional token if no local account is available (defaults to None)
Returns:
SpeckleClient -- authenticated with a corresponding local account or the provided token
"""
if self._client and token is None:
return self._client
if not self._account or not self._account.token:
self.get_account(token)
if not self._client:
self._client = SpeckleClient(host=self.host, use_ssl=self.use_ssl)
if self._account.token is None and token is None:
warn(f"No local account found for server {self.host}", SpeckleWarning)
return self._client
if self._account.token:
self._client.authenticate_with_account(self._account)
else:
self._client.authenticate_with_token(token)
return self._client
def get_transport(self, token: str = None) -> ServerTransport:
"""
Gets a server transport for this stream using an authenticated client. If there is no local account for this
server and the client was not authenticated with a token, this will throw an exception.
Returns:
ServerTransport -- constructed for this stream with a pre-authenticated client
"""
if not self._account or not self._account.token:
self.get_account(token)
return ServerTransport(self.stream_id, account=self._account)
+70 -55
View File
@@ -1,10 +1,10 @@
import os
import socket
import sys
import queue
import hashlib
import logging
import requests
import threading
from requests.sessions import session
from specklepy.transports.sqlite import SQLiteTransport
"""
Anonymous telemetry to help us understand how to make a better Speckle.
@@ -12,25 +12,25 @@ This really helps us to deliver a better open source project and product!
"""
TRACK = True
HOST_APP = "python"
HOST_APP_VERSION = f"python {'.'.join(map(str, sys.version_info[:3]))}"
PLATFORMS = {"win32": "Windows", "cygwin": "Windows", "darwin": "Mac OS X"}
LOG = logging.getLogger(__name__)
METRICS_TRACKER = None
# actions
RECEIVE = "receive"
SEND = "send"
STREAM_CREATE = "stream/create"
STREAM_GET = "stream/get"
STREAM_UPDATE = "stream/update"
STREAM_DELETE = "stream/delete"
STREAM_DETAILS = "stream/details"
STREAM_LIST = "stream/list"
STREAM_VIEW = "stream/view"
STREAM_SEARCH = "stream/search"
RECEIVE = "Receive"
SEND = "Send"
STREAM = "Stream Action"
PERMISSION = "Permission Action"
COMMIT = "Commit Action"
BRANCH = "Branch Action"
USER = "User Action"
SERVER = "Server Action"
CLIENT = "Speckle Client"
STREAM_WRAPPER = "Stream Wrapper"
ACCOUNT_DEFAULT = "account/default"
ACCOUNT_DETAILS = "account/details"
ACCOUNT_LIST = "account/list"
ACCOUNTS = "Get Local Accounts"
SERIALIZE = "serialization/serialize"
DESERIALIZE = "serialization/deserialize"
@@ -41,44 +41,54 @@ def disable():
TRACK = False
def set_host_app(host_app: str):
global HOST_APP
def enable():
global TRACK
TRACK = True
def set_host_app(host_app: str, host_app_version: str = None):
global HOST_APP, HOST_APP_VERSION
HOST_APP = host_app
HOST_APP_VERSION = host_app_version or HOST_APP_VERSION
def track(action: str):
def track(action: str, account: "Account" = None, custom_props: dict = None):
if not TRACK:
return
try:
global METRICS_TRACKER
if not METRICS_TRACKER:
METRICS_TRACKER = MetricsTracker()
page_params = {
"rec": 1,
"idsite": METRICS_TRACKER.site_id,
"uid": METRICS_TRACKER.suuid,
"action_name": action,
"url": f"http://connectors/{HOST_APP}/{action}",
"urlref": f"http://connectors/{HOST_APP}/{action}",
"_cvar": {"1": ["hostApplication", HOST_APP]},
}
initialise_tracker(account)
event_params = {
"rec": 1,
"idsite": METRICS_TRACKER.site_id,
"uid": MetricsTracker.suuid,
"_cvar": {"1": ["hostApplication", HOST_APP]},
"e_c": HOST_APP,
"e_a": action,
"event": action,
"properties": {
"distinct_id": METRICS_TRACKER.last_user,
"server_id": METRICS_TRACKER.last_server,
"token": METRICS_TRACKER.analytics_token,
"hostApp": HOST_APP,
"hostAppVersion": HOST_APP_VERSION,
"$os": METRICS_TRACKER.platform,
"type": "action",
},
}
if custom_props:
event_params["properties"].update(custom_props)
METRICS_TRACKER.queue.put_nowait([event_params, page_params])
METRICS_TRACKER.queue.put_nowait(event_params)
except Exception as ex:
# wrapping this whole thing in a try except as we never want a failure here to annoy users!
LOG.error("Error queueing metrics request: " + str(ex))
def initialise_tracker(account: "Account" = None):
global METRICS_TRACKER
if not METRICS_TRACKER:
METRICS_TRACKER = MetricsTracker()
if account and account.userInfo.email:
METRICS_TRACKER.set_last_user(account.userInfo.email)
if account and account.serverInfo.url:
METRICS_TRACKER.set_last_server(account.userInfo.email)
class Singleton(type):
_instances = {}
@@ -89,10 +99,11 @@ class Singleton(type):
class MetricsTracker(metaclass=Singleton):
matomo_url = "https://speckle.matomo.cloud/matomo.php"
site_id = 2
host_app = "python"
suuid = None
analytics_url = "https://analytics.speckle.systems/track?ip=1"
analytics_token = "acd87c5a50b56df91a795e999812a3a4"
last_user = None
last_server = None
platform = None
sending_thread = None
queue = queue.Queue(1000)
@@ -100,25 +111,29 @@ class MetricsTracker(metaclass=Singleton):
self.sending_thread = threading.Thread(
target=self._send_tracking_requests, daemon=True
)
self.set_suuid()
self.platform = PLATFORMS.get(sys.platform, "linux")
self.sending_thread.start()
def set_suuid(self):
try:
file_path = os.path.join(SQLiteTransport.get_base_path("Speckle"), "suuid")
with open(file_path, "r") as file:
self.suuid = file.read()
except:
self.suuid = "unknown-suuid"
def set_last_user(self, email: str):
if not email:
return
self.last_user = "@" + self.hash(email)
def set_last_server(self, server: str):
if not server:
return
self.last_server = self.hash(server)
def hash(self, value: str):
return hashlib.md5(value.lower().encode("utf-8")).hexdigest().upper()
def _send_tracking_requests(self):
session = requests.Session()
while True:
params = self.queue.get()
event_params = [self.queue.get()]
try:
session.post(self.matomo_url, params=params[0])
session.post(self.matomo_url, params=params[1])
session.post(self.analytics_url, json=event_params)
except Exception as ex:
LOG.error("Error sending metrics request: " + str(ex))
+12 -8
View File
@@ -1,15 +1,17 @@
import typing
from typing import (
Any,
Callable,
ClassVar,
Dict,
List,
Optional,
Union,
Set,
Type,
get_type_hints,
)
import contextlib
from enum import EnumMeta
from warnings import warn
from specklepy.logging.exceptions import SpeckleException
@@ -140,7 +142,7 @@ class Base(_RegisteringBase):
id: Optional[str] = None
totalChildrenCount: Optional[int] = None
applicationId: Optional[str] = None
_units: str = "m"
_units: str = None
# dict of chunkable props and their max chunk size
_chunkable: Dict[str, int] = {}
_chunk_size_default: int = 1000
@@ -249,17 +251,20 @@ class Base(_RegisteringBase):
types = getattr(self, "_attr_types", {})
t = types.get(name, None)
if t is None:
if t is None or t is Any:
return value
if value is None:
return None
if isinstance(t, EnumMeta) and (value in t._value2member_map_):
return t(value)
if t.__module__ == "typing":
origin = getattr(t, "__origin__")
t = (
tuple(getattr(sub_t, "__origin__", sub_t) for sub_t in t.__args__)
if origin is typing.Union
if origin is Union
else origin
)
@@ -276,13 +281,11 @@ class Base(_RegisteringBase):
if isinstance(t, tuple):
t = t[0]
try:
with contextlib.suppress(ValueError):
if t is float:
return float(value)
if t is str and value:
return str(value)
except ValueError:
pass
raise SpeckleException(
f"Cannot set '{self.__class__.__name__}.{name}': it expects type '{t.__name__}', but received type '{type(value).__name__}'"
@@ -406,4 +409,5 @@ class DataChunk(Base, speckle_type="Speckle.Core.Models.DataChunk"):
data: List[Any] = None
def __init__(self) -> None:
super().__init__()
self.data = []
+1 -1
View File
@@ -74,7 +74,7 @@ class ObjectArray:
index = 0
while index < len(data):
item_length = data[index]
item_length = int(data[index])
item_start = index + 1
item_end = item_start + item_length
item_data = data[item_start:item_end]
+10 -1
View File
@@ -1,5 +1,6 @@
from specklepy.objects.geometry import Point
from enum import Enum
from typing import List
from specklepy.objects.geometry import Point
from .base import Base
@@ -19,11 +20,19 @@ class FakeGeo(Base, chunkable={"dots": 50}, detachable={"pointslist"}):
dots: List[int] = None
class FakeDirection(Enum):
NORTH = 1
EAST = 2
SOUTH = 3
WEST = 4
class FakeMesh(FakeGeo, chunkable=CHUNKABLE_PROPS, detachable=DETACHABLE):
vertices: List[float] = None
faces: List[int] = None
colors: List[int] = None
textureCoordinates: List[float] = None
cardinal_dir: FakeDirection = None
test_bases: List[Base] = None
detach_this: Base = None
detached_list: List[Base] = None
+17 -5
View File
@@ -293,9 +293,9 @@ class Curve(
@classmethod
def from_list(cls, args: List[Any]) -> "Curve":
point_count = args[7]
weights_count = args[8]
knots_count = args[9]
point_count = int(args[7])
weights_count = int(args[8])
knots_count = int(args[9])
points_start = 10
weights_start = 10 + point_count
@@ -303,7 +303,7 @@ class Curve(
knots_end = knots_start + knots_count
return cls(
degree=args[1],
degree=int(args[1]),
periodic=bool(args[2]),
rational=bool(args[3]),
closed=bool(args[4]),
@@ -632,7 +632,7 @@ class Brep(
bbox: Box = None
area: float = None
volume: float = None
displayValue: Mesh = None
_displayValue: List[Mesh] = None
Surfaces: List[Surface] = None
Curve3D: List[Base] = None
Curve2D: List[Base] = None
@@ -648,6 +648,18 @@ class Brep(
child._Brep = self
return children
# set as prop for now for backwards compatibility
@property
def displayValue(self) -> List[Mesh]:
return self._displayValue
@displayValue.setter
def displayValue(self, value):
if isinstance(value, Mesh):
self._displayValue = [value]
elif isinstance(value, list):
self._displayValue = value
@property
def Edges(self) -> List[BrepEdge]:
return self._inject_self_into_children(self._Edges)
+19 -4
View File
@@ -1,4 +1,4 @@
from typing import List
from typing import Any, List
from specklepy.objects.geometry import Point, Vector
from .base import Base
@@ -55,10 +55,11 @@ class Transform(
def value(self, value: List[float]) -> None:
try:
value = [float(x) for x in value]
except (ValueError, TypeError):
except (ValueError, TypeError) as error:
raise ValueError(
f"Could not create a Transform object with the requested value. Input must be a 16 element list of numbers. Value provided: {value}"
)
) from error
if len(value) != 16:
raise ValueError(
f"Could not create a Transform object: input list should be 16 floats long, but was {len(value)} long"
@@ -196,4 +197,18 @@ class BlockInstance(
Base, speckle_type=OTHER + "BlockInstance", detachable={"blockDefinition"}
):
blockDefinition: BlockDefinition = None
transform: Transform = None
transform: Transform = None
# TODO: prob move this into a built elements module, but just trialling this for now
class RevitParameter(Base, speckle_type="Objects.BuiltElements.Revit.Parameter"):
name: str = None
value: Any = None
applicationUnitType: str = None # eg UnitType UT_Length
applicationUnit: str = None # DisplayUnitType eg DUT_MILLIMITERS
applicationInternalName: str = (
None # BuiltInParameterName or GUID for shared parameter
)
isShared: bool = False
isReadOnly: bool = False
isTypeParameter: bool = False
+1 -3
View File
@@ -1,6 +1,4 @@
from enum import Enum
import enum
from typing import Any, List, Optional
from typing import List
from ..base import Base
from ..geometry import *
+1 -2
View File
@@ -1,6 +1,5 @@
from enum import Enum
import enum
from typing import Any, List, Optional
from typing import List
from ..base import Base
from ..geometry import *
+1 -1
View File
@@ -1,5 +1,5 @@
from enum import Enum
from typing import Any, List, Optional
from typing import List
from ..base import Base
from .geometry import *
-1
View File
@@ -1,5 +1,4 @@
from enum import Enum
from typing import Any, List, Optional
from ..base import Base
@@ -1,6 +1,4 @@
from enum import Enum
from typing import Any, List, Optional
from ..base import Base
+1 -3
View File
@@ -1,6 +1,4 @@
from enum import Enum
import enum
from typing import Any, List, Optional
from typing import List
from ..base import Base
from ..geometry import *
+3 -4
View File
@@ -16,6 +16,7 @@ UNITS_STRINGS = {
}
UNITS_ENCODINGS = {
None: 0,
"none": 0,
"mm": 1,
"cm": 2,
@@ -58,7 +59,5 @@ def get_units_from_encoding(unit: int):
def get_encoding_from_units(unit: str):
try:
return UNITS_ENCODINGS[unit]
except KeyError:
raise SpeckleException(
message=f"No encoding exists for unit {unit}. Please enter a valid unit to encode (eg {UNITS_ENCODINGS})."
)
except KeyError as e:
raise SpeckleException(message=f"No encoding exists for unit {unit}. Please enter a valid unit to encode (eg {UNITS_ENCODINGS}).") from e
@@ -1,16 +1,19 @@
import re
import ujson
import hashlib
import re
import warnings
from uuid import uuid4
from enum import Enum
from warnings import warn
from typing import Any, Dict, List, Tuple
from specklepy.objects.base import Base, DataChunk
from specklepy.logging.exceptions import (
SerializationException,
SpeckleException,
SpeckleWarning,
)
from specklepy.transports.abstract_transport import AbstractTransport
# import for serialization
import specklepy.objects.geometry
import specklepy.objects.other
@@ -49,9 +52,16 @@ class BaseObjectSerializer:
self.read_transport = read_transport
def write_json(self, base: Base):
self.__reset_writer()
self.detach_lineage = [True]
"""Serializes a given base object into a json string
Arguments:
base {Base} -- the base object to be decomposed and serialized
Returns:
(str, str) -- a tuple containing the hash (id) of the base object and the serialized object string
"""
hash, obj = self.traverse_base(base)
return hash, ujson.dumps(obj)
def traverse_base(self, base: Base) -> Tuple[str, Dict]:
@@ -63,6 +73,21 @@ class BaseObjectSerializer:
Returns:
(str, dict) -- a tuple containing the hash (id) of the base object and the constructed serializable dictionary
"""
self.__reset_writer()
if self.write_transports:
for wt in self.write_transports:
wt.begin_write()
hash, obj = self._traverse_base(base)
if self.write_transports:
for wt in self.write_transports:
wt.end_write()
return hash, obj
def _traverse_base(self, base: Base) -> Tuple[str, Dict]:
if not self.detach_lineage:
self.detach_lineage = [True]
@@ -111,6 +136,11 @@ class BaseObjectSerializer:
object_builder[prop] = value
continue
# NOTE: for dynamic props, this won't be re-serialised as an enum but as an int
if isinstance(value, Enum):
object_builder[prop] = value.value
continue
# 2. handle Base objects
elif isinstance(value, Base):
child_obj = self.traverse_value(value, detach=detach)
@@ -135,7 +165,7 @@ class BaseObjectSerializer:
chunk_refs = []
for c in chunks:
self.detach_lineage.append(detach)
ref_hash, _ = self.traverse_base(c)
ref_hash, _ = self._traverse_base(c)
ref_obj = self.detach_helper(ref_hash=ref_hash)
chunk_refs.append(ref_obj)
object_builder[prop] = chunk_refs
@@ -182,6 +212,10 @@ class BaseObjectSerializer:
if isinstance(obj, PRIMITIVES):
return obj
# NOTE: for dynamic props, this won't be re-serialised as an enum but as an int
if isinstance(obj, Enum):
return obj.value
elif isinstance(obj, (list, tuple, set)):
if not detach:
return [self.traverse_value(o) for o in obj]
@@ -190,7 +224,7 @@ class BaseObjectSerializer:
for o in obj:
if isinstance(o, Base):
self.detach_lineage.append(detach)
hash, _ = self.traverse_base(o)
hash, _ = self._traverse_base(o)
detached_list.append(self.detach_helper(ref_hash=hash))
else:
detached_list.append(self.traverse_value(o, detach))
@@ -198,7 +232,7 @@ class BaseObjectSerializer:
elif isinstance(obj, dict):
for k, v in obj.items():
if isinstance(v, PRIMITIVES):
if isinstance(v, PRIMITIVES) or v is None:
continue
else:
obj[k] = self.traverse_value(v)
@@ -206,17 +240,18 @@ class BaseObjectSerializer:
elif isinstance(obj, Base):
self.detach_lineage.append(detach)
_, base_obj = self.traverse_base(obj)
_, base_obj = self._traverse_base(obj)
return base_obj
else:
try:
return obj.dict()
except:
SerializationException(
message=f"Failed to handle {type(obj)} in `BaseObjectSerializer.traverse_value`",
object=obj,
warn(
f"Failed to handle {type(obj)} in `BaseObjectSerializer.traverse_value`",
SpeckleWarning,
)
return str(obj)
def detach_helper(self, ref_hash: str) -> Dict[str, str]:
@@ -244,7 +279,7 @@ class BaseObjectSerializer:
def __reset_writer(self) -> None:
"""Reinitializes the lineage, and other variables that get used during the json writing process"""
self.detach_lineage = []
self.detach_lineage = [True]
self.lineage = []
self.family_tree = {}
self.closure_table = {}
@@ -310,12 +345,15 @@ class BaseObjectSerializer:
elif "referencedId" in value:
ref_hash = value["referencedId"]
ref_obj_str = self.read_transport.get_object(id=ref_hash)
if not ref_obj_str:
raise SpeckleException(
f"Could not find the referenced child object of id `{ref_hash}` in the given read transport: {self.read_transport.name}"
if ref_obj_str:
ref_obj = safe_json_loads(ref_obj_str, ref_hash)
base.__setattr__(prop, self.recompose_base(obj=ref_obj))
else:
warnings.warn(
f"Could not find the referenced child object of id `{ref_hash}` in the given read transport: {self.read_transport.name}",
SpeckleWarning,
)
ref_obj = safe_json_loads(ref_obj_str, ref_hash)
base.__setattr__(prop, self.recompose_base(obj=ref_obj))
base.__setattr__(prop, self.handle_value(value))
# 3. handle all other cases (base objects, lists, and dicts)
else:
@@ -369,8 +407,10 @@ class BaseObjectSerializer:
ref_hash = obj["referencedId"]
ref_obj_str = self.read_transport.get_object(id=ref_hash)
if not ref_obj_str:
raise SpeckleException(
f"Could not find the referenced child object of id `{ref_hash}` in the given read transport: {self.read_transport.name}"
warnings.warn(
f"Could not find the referenced child object of id `{ref_hash}` in the given read transport: {self.read_transport.name}",
SpeckleWarning,
)
return obj
return safe_json_loads(ref_obj_str, ref_hash)
+1 -1
View File
@@ -1,5 +1,5 @@
from abc import ABC, abstractmethod
from typing import Any, Optional, List, Dict
from typing import Optional, List, Dict
from pydantic import BaseModel
from pydantic.main import Extra
+1 -6
View File
@@ -1,6 +1,4 @@
import json
from typing import Any, List, Dict
from specklepy.logging.exceptions import SpeckleException
from specklepy.transports.abstract_transport import AbstractTransport
@@ -28,10 +26,7 @@ class MemoryTransport(AbstractTransport):
raise NotImplementedError
def get_object(self, id: str) -> str or None:
if id in self.objects:
return self.objects[id]
else:
return None
return self.objects[id] if id in self.objects else None
def has_objects(self, id_list: List[str]) -> Dict[str, bool]:
return {id: (id in self.objects) for id in id_list}
@@ -2,7 +2,6 @@ import json
import logging
import threading
import queue
import time
import gzip
import requests
+27 -19
View File
@@ -1,12 +1,12 @@
import json
import time
import requests
from warnings import warn
from typing import Any, Dict, List, Type
from typing import Any, Dict, List
from specklepy.api.client import SpeckleClient
from specklepy.logging.exceptions import SpeckleException
from specklepy.api.credentials import Account, get_account_from_token
from specklepy.logging.exceptions import SpeckleException, SpeckleWarning
from specklepy.transports.abstract_transport import AbstractTransport
from .batch_sender import BatchSender
@@ -19,7 +19,8 @@ class ServerTransport(AbstractTransport):
The `ServerTransport` can be authenticted two different ways:
1. by providing a `SpeckleClient`
2. by providing a `token` and `url`
2. by providing an `Account`
3. by providing a `token` and `url`
```py
from specklepy.api import operations
@@ -46,6 +47,7 @@ class ServerTransport(AbstractTransport):
_name = "RemoteTransport"
url: str = None
stream_id: str = None
account: Account = None
saved_obj_count: int = 0
session: requests.Session = None
@@ -53,35 +55,43 @@ class ServerTransport(AbstractTransport):
self,
stream_id: str,
client: SpeckleClient = None,
account: Account = None,
token: str = None,
url: str = None,
**data: Any,
) -> None:
super().__init__(**data)
# TODO: replace client with account or some other auth avenue
if client is None and token is None and url is None:
if client is None and account is None and token is None and url is None:
raise SpeckleException(
"You must provide either a client or a token and url to construct a ServerTransport."
)
if client:
if not client.me:
raise SpeckleException(
"The provided SpeckleClient was not authenticated."
)
token = client.me["token"]
if account:
self.account = account
url = account.serverInfo.url
elif client:
url = client.url
if not client.account.token:
warn(
SpeckleWarning(
f"Unauthenticated Speckle Client provided to Server Transport for {self.url}. Receiving from private streams will fail."
)
)
else:
self.account = client.account
else:
self.account = get_account_from_token(token, url)
self.stream_id = stream_id
self.url = url
self._batch_sender = BatchSender(
self.url, self.stream_id, token, max_batch_size_mb=1
self.url, self.stream_id, self.account.token, max_batch_size_mb=1
)
self.session = requests.Session()
self.session.headers.update(
{"Authorization": f"Bearer {token}", "Accept": "text/plain"}
{"Authorization": f"Bearer {self.account.token}", "Accept": "text/plain"}
)
def begin_write(self) -> None:
@@ -120,8 +130,7 @@ class ServerTransport(AbstractTransport):
) -> str:
endpoint = f"{self.url}/objects/{self.stream_id}/{id}/single"
r = self.session.get(endpoint)
if r.encoding is None:
r.encoding = "utf-8"
r.encoding = "utf-8"
if r.status_code != 200:
raise SpeckleException(
@@ -143,8 +152,7 @@ class ServerTransport(AbstractTransport):
r = self.session.post(
endpoint, data={"objects": json.dumps(new_children_ids)}, stream=True
)
if r.encoding is None:
r.encoding = "utf-8"
r.encoding = "utf-8"
lines = r.iter_lines(decode_unicode=True)
# iter through returned objects saving them as we go
+35 -61
View File
@@ -3,7 +3,7 @@ import sys
import time
import sched
import sqlite3
from typing import Any, List, Dict
from typing import Any, List, Dict, Tuple
from appdirs import user_data_dir
from contextlib import closing
from specklepy.transports.abstract_transport import AbstractTransport
@@ -14,25 +14,29 @@ class SQLiteTransport(AbstractTransport):
_name = "SQLite"
_base_path: str = None
_root_path: str = None
_is_writing: bool = False
_scheduler = sched.scheduler(time.time, time.sleep)
_polling_interval = 0.5 # seconds
__connection: sqlite3.Connection = None
app_name: str = ""
scope: str = ""
saved_obj_count: int = 0
max_size: int = None
_current_batch: List[Tuple[str, str]] = None
_current_batch_size: int = None
def __init__(
self,
base_path: str = None,
app_name: str = None,
scope: str = None,
max_batch_size_mb: float = 10.0,
**data: Any,
) -> None:
super().__init__(**data)
self.app_name = app_name or "Speckle"
self.scope = scope or "Objects"
self._base_path = base_path or self.get_base_path(self.app_name)
self.max_size = int(max_batch_size_mb * 1000 * 1000)
self._current_batch = []
self._current_batch_size = 0
try:
os.makedirs(self._base_path, exist_ok=True)
@@ -50,12 +54,6 @@ class SQLiteTransport(AbstractTransport):
def __repr__(self) -> str:
return f"SQLiteTransport(app: '{self.app_name}', scope: '{self.scope}')"
# def __write_timer_elapsed(self):
# print("WRITE TIMER ELAPSED")
# proc = Process(target=_run_queue, args=(self.__queue, self._root_path))
# proc.start()
# proc.join()
@staticmethod
def get_base_path(app_name):
# from appdirs https://github.com/ActiveState/appdirs/blob/master/appdirs.py
@@ -74,36 +72,6 @@ class SQLiteTransport(AbstractTransport):
path = os.path.expanduser("~/.config/")
return os.path.join(path, app_name)
# def __consume_queue(self):
# if self._is_writing or self.__queue.empty():
# return
# print("CONSUME QUEUE")
# self._is_writing = True
# while not self.__queue.empty():
# data = self.__queue.get()
# self.save_object(data[0], data[1])
# self._is_writing = False
# self._scheduler.enter(
# delay=self._polling_interval, priority=1, action=self.__consume_queue
# )
# self._scheduler.run(blocking=True)
# def save_object(self, id: str, serialized_object: str) -> None:
# """Adds an object to the queue and schedules it to be saved.
# Arguments:
# id {str} -- the object id
# serialized_object {str} -- the full string representation of the object
# """
# print("SAVE OBJECT")
# self.__queue.put((id, serialized_object))
# self._scheduler.enter(
# delay=self._polling_interval, priority=1, action=self.__consume_queue
# )
# self._scheduler.run(blocking=True)
def save_object_from_transport(
self, id: str, source_transport: AbstractTransport
) -> None:
@@ -117,23 +85,41 @@ class SQLiteTransport(AbstractTransport):
self.save_object(id, serialized_object)
def save_object(self, id: str, serialized_object: str) -> None:
"""Directly saves an object into the database.
"""
Adds an object to the current batch to be written to the db. If the current batch is full,
the batch is written to the db and the current batch is reset.
Arguments:
id {str} -- the object id
serialized_object {str} -- the full string representation of the object
"""
obj_size = len(serialized_object)
if (
not self._current_batch
or self._current_batch_size + obj_size < self.max_size
):
self._current_batch.append((id, serialized_object))
self._current_batch_size += obj_size
return
self.save_current_batch()
self._current_batch = [(id, serialized_object)]
self._current_batch_size = obj_size
def save_current_batch(self) -> None:
"""Save the current batch of objects to the local db"""
self.__check_connection()
try:
with closing(self.__connection.cursor()) as c:
c.execute(
c.executemany(
"INSERT OR IGNORE INTO objects(hash, content) VALUES(?,?)",
(id, serialized_object),
self._current_batch,
)
self.__connection.commit()
except Exception as ex:
raise SpeckleException(
f"Could not save the object to the local db. Inner exception: {ex}", ex
f"Could not save the batch of objects to the local db. Inner exception: {ex}",
ex,
)
def get_object(self, id: str) -> str or None:
@@ -156,10 +142,14 @@ class SQLiteTransport(AbstractTransport):
return ret
def begin_write(self):
self._object_cache = []
self.saved_obj_count = 0
def end_write(self):
pass
if self._current_batch:
self.save_current_batch()
self._current_batch = []
self._current_batch_size = 0
def copy_object_and_children(
self, id: str, target_transport: AbstractTransport
@@ -199,19 +189,3 @@ class SQLiteTransport(AbstractTransport):
def __del__(self):
self.__connection.close()
# def _run_queue(queue: Queue, root_path: str):
# if queue.empty():
# return
# print("RUN QUEUE")
# conn = sqlite3.connect(root_path)
# while not queue.empty():
# data = queue.get()
# with closing(conn.cursor()) as c:
# c.execute(
# "INSERT OR IGNORE INTO objects(hash, content) VALUES(?,?)",
# (data[0], data[1]),
# )
# conn.commit()
# conn.close()
+8 -3
View File
@@ -6,7 +6,7 @@ from specklepy.api.models import Stream
from specklepy.api.client import SpeckleClient
from specklepy.objects.base import Base
from specklepy.objects.geometry import Point
from specklepy.objects.fakemesh import FakeMesh
from specklepy.objects.fakemesh import FakeDirection, FakeMesh
from specklepy.logging import metrics
metrics.disable()
@@ -61,7 +61,7 @@ def second_user_dict(host):
@pytest.fixture(scope="session")
def client(host, user_dict):
client = SpeckleClient(host=host, use_ssl=False)
client.authenticate(user_dict["token"])
client.authenticate_with_token(user_dict["token"])
return client
@@ -81,9 +81,10 @@ def mesh():
mesh = FakeMesh()
mesh.name = "my_mesh"
mesh.vertices = [random.uniform(0, 10) for _ in range(1, 210)]
mesh.faces = [i for i in range(1, 210)]
mesh.faces = list(range(1, 210))
mesh["@(100)colours"] = [random.uniform(0, 10) for _ in range(1, 210)]
mesh["@()default_chunk"] = [random.uniform(0, 10) for _ in range(1, 210)]
mesh.cardinal_dir = FakeDirection.WEST
mesh.test_bases = [Base(name=f"test {i}") for i in range(1, 22)]
mesh.detach_this = Base(name="predefined detached base")
mesh["@detach"] = Base(name="detached base")
@@ -102,6 +103,10 @@ def base():
base = Base()
base.name = "my_base"
base.units = "millimetres"
base.null_val = None
base.null_dict = {"a null val": None}
base.tuple = (1, 2, "3")
base.set = {1, 2, "3"}
base.vertices = [random.uniform(0, 10) for _ in range(1, 120)]
base.test_bases = [Base(name=i) for i in range(1, 22)]
base["@detach"] = Base(name="detached base")
+12 -1
View File
@@ -1,5 +1,6 @@
from contextlib import ExitStack as does_not_raise
from enum import Enum
from typing import Dict, List, Optional
from contextlib import ExitStack as does_not_raise
import pytest
from specklepy.api import operations
@@ -95,6 +96,12 @@ def test_base_of_custom_speckle_type() -> None:
assert b1.name == "Tweety's Crib"
class DietaryRestrictions(Enum):
VEGAN = 1
GLUTEN_FREE = 2
NUT_FREE = 3
class FrozenYoghurt(Base):
"""Testing type checking"""
@@ -103,6 +110,7 @@ class FrozenYoghurt(Base):
customer: str
add_ons: Optional[Dict[str, float]] # dict item types won't be checked
price: float = 0.0
dietary: DietaryRestrictions
def test_type_checking() -> None:
@@ -111,6 +119,7 @@ def test_type_checking() -> None:
order.servings = 2
order.price = "7" # will get converted
order.customer = "izzy"
order.dietary = DietaryRestrictions.VEGAN
with pytest.raises(SpeckleException):
order.flavours = "not a list"
@@ -118,6 +127,8 @@ def test_type_checking() -> None:
order.servings = "five"
with pytest.raises(SpeckleException):
order.add_ons = ["sprinkles"]
with pytest.raises(SpeckleException):
order.dietary = "no nuts plz"
order.add_ons = {"sprinkles": 0.2, "chocolate": 1.0}
order.flavours = ["strawberry", "lychee", "peach", "pineapple"]
+20 -3
View File
@@ -3,6 +3,7 @@ from specklepy.api import operations
from specklepy.api.client import SpeckleClient
from specklepy.objects.base import Base
from specklepy.transports.server import ServerTransport
from specklepy.api.credentials import Account, get_account_from_token
from specklepy.logging.exceptions import SpeckleException, SpeckleWarning
@@ -10,12 +11,12 @@ def test_invalid_authentication():
client = SpeckleClient()
with pytest.warns(SpeckleWarning):
client.authenticate("fake token")
client.authenticate_with_token("fake token")
def test_invalid_send():
client = SpeckleClient()
client.me = {"token": "fake token"}
client.account = Account(token="fake_token")
transport = ServerTransport("3073b96e86", client)
with pytest.raises(SpeckleException):
@@ -24,8 +25,24 @@ def test_invalid_send():
def test_invalid_receive():
client = SpeckleClient()
client.me = {"token": "fake token"}
client.account = Account(token="fake_token")
transport = ServerTransport("fake stream", client)
with pytest.raises(SpeckleException):
operations.receive("fake object", transport)
def test_account_from_token():
token = "fake token"
acct = get_account_from_token(token)
assert acct.token == token
def test_account_from_token_and_url():
token = "fake token"
url = "fake.server"
acct = get_account_from_token(token, url)
assert acct.token == token
assert acct.serverInfo.url == url
+1 -1
View File
@@ -55,7 +55,7 @@ class TestSerialization:
# also try constructing server transport with token and url
transport = ServerTransport(
stream_id=sample_stream.id, token=client.me["token"], url=client.url
stream_id=sample_stream.id, token=client.account.token, url=client.url
)
# use a fresh memory transport to force receiving from remote
received = operations.receive(
+27 -2
View File
@@ -1,5 +1,7 @@
import pytest
from specklepy.api.models import Stream
from datetime import datetime
from specklepy.api.models import ActivityCollection, Activity, Stream
from specklepy.api.client import SpeckleClient
from specklepy.logging.exceptions import GraphQLException
@@ -67,6 +69,16 @@ class TestStream:
assert len(search_results) == 1
assert search_results[0].name == updated_stream.name
def test_stream_favorite(self, client, stream):
favorited = client.stream.favorite(stream.id)
assert isinstance(favorited, Stream)
assert favorited.favoritedDate is not None
unfavorited = client.stream.favorite(stream.id, False)
assert isinstance(favorited, Stream)
assert unfavorited.favoritedDate is None
def test_stream_grant_permission(self, client, stream, second_user_dict):
granted = client.stream.grant_permission(
stream_id=stream.id,
@@ -87,9 +99,22 @@ class TestStream:
fetched_stream = client.stream.get(stream.id)
assert revoked == True
assert revoked is True
assert len(fetched_stream.collaborators) == 1
def test_stream_activity(self, client: SpeckleClient, stream: Stream):
activity = client.stream.activity(stream.id)
older_activity = client.stream.activity(
stream.id, before=activity.items[0].time
)
assert isinstance(activity, ActivityCollection)
assert isinstance(older_activity, ActivityCollection)
assert older_activity.totalCount < activity.totalCount
assert activity.items is not None
assert isinstance(activity.items[0], Activity)
def test_stream_delete(self, client, stream):
deleted = client.stream.delete(stream.id)
+17 -2
View File
@@ -1,6 +1,7 @@
from specklepy.logging.exceptions import SpeckleException
from specklepy.api.models import User
import pytest
from specklepy.api.client import SpeckleClient
from specklepy.api.models import Activity, ActivityCollection, User
from specklepy.logging.exceptions import SpeckleException
@pytest.mark.run(order=1)
@@ -43,3 +44,17 @@ class TestUser:
assert isinstance(failed_update, SpeckleException)
assert updated is True
assert me.bio == bio
def test_user_activity(self, client: SpeckleClient, second_user_dict):
my_activity = client.user.activity(limit=10)
their_activity = client.user.activity(second_user_dict["id"])
assert isinstance(my_activity, ActivityCollection)
assert isinstance(my_activity.items[0], Activity)
assert my_activity.totalCount > 0
assert isinstance(their_activity, ActivityCollection)
older_activity = client.user.activity(before=my_activity.items[0].time)
assert isinstance(older_activity, ActivityCollection)
assert older_activity.totalCount < my_activity.totalCount
+64 -62
View File
@@ -1,79 +1,81 @@
import pytest
from specklepy.api.credentials import StreamWrapper
from specklepy.api.wrapper import StreamWrapper
class TestWrapper:
def test_parse_stream(self):
wrap = StreamWrapper("https://testing.speckle.dev/streams/a75ab4f10f")
assert wrap.type == "stream"
def test_parse_stream():
wrap = StreamWrapper("https://testing.speckle.dev/streams/a75ab4f10f")
assert wrap.type == "stream"
def test_parse_branch(self):
wacky_wrap = StreamWrapper(
"https://testing.speckle.dev/streams/4c3ce1459c/branches/%F0%9F%8D%95%E2%AC%85%F0%9F%8C%9F%20you%20wat%3F"
)
wrap = StreamWrapper(
"https://testing.speckle.dev/streams/4c3ce1459c/branches/next%20level"
)
assert wacky_wrap.type == "branch"
assert wacky_wrap.branch_name == "🍕⬅🌟 you wat?"
assert wrap.type == "branch"
def test_parse_nested_branch(self):
wrap = StreamWrapper(
"https://testing.speckle.dev/streams/4c3ce1459c/branches/izzy/dev"
)
def test_parse_branch():
wacky_wrap = StreamWrapper(
"https://testing.speckle.dev/streams/4c3ce1459c/branches/%F0%9F%8D%95%E2%AC%85%F0%9F%8C%9F%20you%20wat%3F"
)
wrap = StreamWrapper(
"https://testing.speckle.dev/streams/4c3ce1459c/branches/next%20level"
)
assert wacky_wrap.type == "branch"
assert wacky_wrap.branch_name == "🍕⬅🌟 you wat?"
assert wrap.type == "branch"
assert wrap.branch_name == "izzy/dev"
assert wrap.type == "branch"
def test_parse_commit(self):
wrap = StreamWrapper(
"https://testing.speckle.dev/streams/4c3ce1459c/commits/8b9b831792"
)
assert wrap.type == "commit"
def test_parse_nested_branch():
wrap = StreamWrapper(
"https://testing.speckle.dev/streams/4c3ce1459c/branches/izzy/dev"
)
def test_parse_object(self):
wrap = StreamWrapper(
"https://testing.speckle.dev/streams/a75ab4f10f/objects/5530363e6d51c904903dafc3ea1d2ec6"
)
assert wrap.type == "object"
assert wrap.branch_name == "izzy/dev"
assert wrap.type == "branch"
def test_parse_globals_as_branch(self):
wrap = StreamWrapper("https://testing.speckle.dev/streams/0c6ad366c4/globals/")
assert wrap.type == "branch"
def test_parse_globals_as_commit(self):
wrap = StreamWrapper(
"https://testing.speckle.dev/streams/0c6ad366c4/globals/abd3787893"
)
assert wrap.type == "commit"
def test_parse_commit():
wrap = StreamWrapper(
"https://testing.speckle.dev/streams/4c3ce1459c/commits/8b9b831792"
)
assert wrap.type == "commit"
#! NOTE: the following three tests may not pass locally if you have a `speckle.xyz` account in manager
def test_get_client_without_auth(self):
wrap = StreamWrapper(
"https://speckle.xyz/streams/4c3ce1459c/commits/8b9b831792"
)
client = wrap.get_client()
assert client is not None
def test_parse_object():
wrap = StreamWrapper(
"https://testing.speckle.dev/streams/a75ab4f10f/objects/5530363e6d51c904903dafc3ea1d2ec6"
)
assert wrap.type == "object"
def test_get_new_client_with_token(self):
wrap = StreamWrapper(
"https://speckle.xyz/streams/4c3ce1459c/commits/8b9b831792"
)
client = wrap.get_client()
client = wrap.get_client(token="super-secret-token")
assert client.me["token"] == "super-secret-token"
def test_parse_globals_as_branch():
wrap = StreamWrapper("https://testing.speckle.dev/streams/0c6ad366c4/globals/")
assert wrap.type == "branch"
def test_get_transport_with_token(self):
wrap = StreamWrapper(
"https://speckle.xyz/streams/4c3ce1459c/commits/8b9b831792"
)
client = wrap.get_client()
assert not client.me # unauthenticated bc no local accounts
transport = wrap.get_transport(token="super-secret-token")
def test_parse_globals_as_commit():
wrap = StreamWrapper(
"https://testing.speckle.dev/streams/0c6ad366c4/globals/abd3787893"
)
assert wrap.type == "commit"
assert transport is not None
assert client.me["token"] == "super-secret-token"
#! NOTE: the following three tests may not pass locally if you have a `speckle.xyz` account in manager
def test_get_client_without_auth():
wrap = StreamWrapper("https://speckle.xyz/streams/4c3ce1459c/commits/8b9b831792")
client = wrap.get_client()
assert client is not None
def test_get_new_client_with_token():
wrap = StreamWrapper("https://speckle.xyz/streams/4c3ce1459c/commits/8b9b831792")
client = wrap.get_client()
client = wrap.get_client(token="super-secret-token")
assert client.account.token == "super-secret-token"
def test_get_transport_with_token():
wrap = StreamWrapper("https://speckle.xyz/streams/4c3ce1459c/commits/8b9b831792")
client = wrap.get_client()
assert not client.account.token # unauthenticated bc no local accounts
transport = wrap.get_transport(token="super-secret-token")
assert transport is not None
assert client.account.token == "super-secret-token"