Merge pull request #125 from specklesystems/izzy/metrics

metrics 🛰
fix(metrics): try catch whole track method
2021-10-12 11:38:40 +02:00 · 2021-10-12 10:36:38 +01:00 · 2021-10-11 18:10:53 +01:00 · 2021-10-11 18:10:33 +01:00 · 2021-10-11 18:09:50 +01:00 · 2021-10-04 09:39:55 +01:00
18 changed files with 1345 additions and 208 deletions
@@ -0,0 +1,78 @@
+name: Update issue Status
+
+on:
+  issues:
+    types: [closed]
+
+jobs:
+  update_issue:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Get project data
+        env:
+          GITHUB_TOKEN: ${{secrets.GHPROJECT_TOKEN}}
+          ORGANIZATION: specklesystems
+          PROJECT_NUMBER: 9
+        run: |
+          gh api graphql --header 'GraphQL-Features: projects_next_graphql' -f query='
+            query($org: String!, $number: Int!) {
+              organization(login: $org){
+                projectNext(number: $number) {
+                  id
+                  fields(first:20) {
+                    nodes {
+                      id
+                      name
+                      settings
+                    }
+                  }
+                }
+              }
+            }' -f org=$ORGANIZATION -F number=$PROJECT_NUMBER > project_data.json
+
+          echo 'PROJECT_ID='$(jq '.data.organization.projectNext.id' project_data.json) >> $GITHUB_ENV
+          echo 'STATUS_FIELD_ID='$(jq '.data.organization.projectNext.fields.nodes[] | select(.name== "Status") | .id' project_data.json) >> $GITHUB_ENV
+          
+          echo "$PROJECT_ID"
+          echo "$STATUS_FIELD_ID"
+          
+          echo 'DONE_ID='$(jq '.data.organization.projectNext.fields.nodes[] | select(.name== "Status") | .settings | fromjson | .options[] | select(.name== "Done") | .id' project_data.json) >> $GITHUB_ENV
+          echo "$DONE_ID"
+
+      - name: Add Issue to project #it's already in the project, but we do this to get its node id!
+        env:
+          GITHUB_TOKEN: ${{secrets.GHPROJECT_TOKEN}}
+          ISSUE_ID: ${{ github.event.issue.node_id }}
+        run: |
+          item_id="$( gh api graphql --header 'GraphQL-Features: projects_next_graphql' -f query='
+            mutation($project:ID!, $id:ID!) {
+              addProjectNextItem(input: {projectId: $project, contentId: $id}) {
+                projectNextItem {
+                  id
+                }
+              }
+            }' -f project=$PROJECT_ID -f id=$ISSUE_ID --jq '.data.addProjectNextItem.projectNextItem.id')"
+          
+          echo 'ITEM_ID='$item_id >> $GITHUB_ENV
+          
+      - name: Update Status
+        env:
+          GITHUB_TOKEN: ${{secrets.GHPROJECT_TOKEN}}
+          ISSUE_ID: ${{ github.event.issue.node_id }}
+        run: |
+          gh api graphql --header 'GraphQL-Features: projects_next_graphql' -f query='
+            mutation($project:ID!, $status:ID!, $id:ID!, $value:String!) {
+              set_status: updateProjectNextItemField(
+                input: {
+                  projectId: $project
+                  itemId: $id
+                  fieldId: $status
+                  value: $value
+                }
+              ) {
+                projectNextItem {
+                  id
+                }
+              }
+            }' -f project=$PROJECT_ID -f status=$STATUS_FIELD_ID -f id=$ITEM_ID -f value=${{ env.DONE_ID }}
+          
@@ -0,0 +1,50 @@
+name: Move new issues into Project
+
+on:
+  issues:
+    types: [opened]
+
+jobs:
+  track_issue:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Get project data
+        env:
+          GITHUB_TOKEN: ${{secrets.GHPROJECT_TOKEN}}
+          ORGANIZATION: specklesystems
+          PROJECT_NUMBER: 9
+        run: |
+          gh api graphql --header 'GraphQL-Features: projects_next_graphql' -f query='
+            query($org: String!, $number: Int!) {
+              organization(login: $org){
+                projectNext(number: $number) {
+                  id
+                  fields(first:20) {
+                    nodes {
+                      id
+                      name
+                      settings
+                    }
+                  }
+                }
+              }
+            }' -f org=$ORGANIZATION -F number=$PROJECT_NUMBER > project_data.json
+
+          echo 'PROJECT_ID='$(jq '.data.organization.projectNext.id' project_data.json) >> $GITHUB_ENV
+          echo 'STATUS_FIELD_ID='$(jq '.data.organization.projectNext.fields.nodes[] | select(.name== "Status") | .id' project_data.json) >> $GITHUB_ENV
+     
+      - name: Add Issue to project
+        env:
+          GITHUB_TOKEN: ${{secrets.GHPROJECT_TOKEN}}
+          ISSUE_ID: ${{ github.event.issue.node_id }}
+        run: |
+          item_id="$( gh api graphql --header 'GraphQL-Features: projects_next_graphql' -f query='
+            mutation($project:ID!, $id:ID!) {
+              addProjectNextItem(input: {projectId: $project, contentId: $id}) {
+                projectNextItem {
+                  id
+                }
+              }
+            }' -f project=$PROJECT_ID -f id=$ISSUE_ID --jq '.data.addProjectNextItem.projectNextItem.id')"
+          
+          echo 'ITEM_ID='$item_id >> $GITHUB_ENV
@@ -1,17 +1,53 @@
-# speckle-py 🥧
+<h1 align="center">
+  <img src="https://user-images.githubusercontent.com/2679513/131189167-18ea5fe1-c578-47f6-9785-3748178e4312.png" width="150px"/><br/>
+  Speckle | specklepy 🐍
+</h1>
+<h3 align="center">
+    The Python SDK
+</h3>
+<p align="center"><b>Speckle</b> is the data infrastructure for the AEC industry.</p><br/>

-[![Twitter Follow](https://img.shields.io/twitter/follow/SpeckleSystems?style=social)](https://twitter.com/SpeckleSystems) [![Community forum users](https://img.shields.io/discourse/users?server=https%3A%2F%2Fdiscourse.speckle.works&style=flat-square&logo=discourse&logoColor=white)](https://discourse.speckle.works) [![website](https://img.shields.io/badge/https://-speckle.systems-royalblue?style=flat-square)](https://speckle.systems) [![docs](https://img.shields.io/badge/docs-speckle.guide-orange?style=flat-square&logo=read-the-docs&logoColor=white)](https://speckle.guide/dev/)
+<p align="center"><a href="https://twitter.com/SpeckleSystems"><img src="https://img.shields.io/twitter/follow/SpeckleSystems?style=social" alt="Twitter Follow"></a> <a href="https://speckle.community"><img src="https://img.shields.io/discourse/users?server=https%3A%2F%2Fspeckle.community&amp;style=flat-square&amp;logo=discourse&amp;logoColor=white" alt="Community forum users"></a> <a href="https://speckle.systems"><img src="https://img.shields.io/badge/https://-speckle.systems-royalblue?style=flat-square" alt="website"></a> <a href="https://speckle.guide/dev/"><img src="https://img.shields.io/badge/docs-speckle.guide-orange?style=flat-square&amp;logo=read-the-docs&amp;logoColor=white" alt="docs"></a></p>
+<p align="center"><a href="https://github.com/specklesystems/specklepy/"><img src="https://circleci.com/gh/specklesystems/specklepy.svg?style=svg&amp;circle-token=76eabd350ea243575cbb258b746ed3f471f7ac29" alt="Speckle-Next"></a> </p>

-## Introduction
+# About Speckle

-> ⚠ This is the start of the Python client for Speckle 2.0. It is currently quite nebulous and may be trashed and rebuilt at any moment! It is compatible with Python 3.6+ ⚠
-> 
+What is Speckle? Check our ![YouTube Video Views](https://img.shields.io/youtube/views/B9humiSpHzM?label=Speckle%20in%201%20minute%20video&style=social)

-## Documentation
+### Features

-Comprehensive developer and user documentation can be found in our:
+- **Object-based:** say goodbye to files! Speckle is the first object based platform for the AEC industry
+- **Version control:** Speckle is the Git & Hub for geometry and BIM data
+- **Collaboration:** share your designs collaborate with others
+- **3D Viewer:** see your CAD and BIM models online, share and embed them anywhere
+- **Interoperability:** get your CAD and BIM models into other software without exporting or importing
+- **Real time:** get real time updates and notifications and changes
+- **GraphQL API:** get what you need anywhere you want it
+- **Webhooks:** the base for a automation and next-gen pipelines
+- **Built for developers:** we are building Speckle with developers in mind and got tools for every stack
+- **Built for the AEC industry:** Speckle connectors are plugins for the most common software used in the industry such as Revit, Rhino, Grasshopper, AutoCAD, Civil 3D, Excel, Unreal Engine, Unity, QGIS, Blender and more!

-#### 📚 [Speckle Docs website](https://speckle.guide/dev/)
+### Try Speckle now!
+
+Give Speckle a try in no time by:
+
+- [![speckle XYZ](https://img.shields.io/badge/https://-speckle.xyz-0069ff?style=flat-square&logo=hackthebox&logoColor=white)](https://speckle.xyz) ⇒ creating an account at our public server
+- [![create a droplet](https://img.shields.io/badge/Create%20a%20Droplet-0069ff?style=flat-square&logo=digitalocean&logoColor=white)](https://marketplace.digitalocean.com/apps/speckle-server?refcode=947a2b5d7dc1) ⇒ deploying an instance in 1 click 
+
+### Resources
+
+- [![Community forum users](https://img.shields.io/badge/community-forum-green?style=for-the-badge&logo=discourse&logoColor=white)](https://speckle.community) for help, feature requests or just to hang with other speckle enthusiasts, check out our community forum!
+- [![website](https://img.shields.io/badge/tutorials-speckle.systems-royalblue?style=for-the-badge&logo=youtube)](https://speckle.systems) our tutorials portal is full of resources to get you started using Speckle
+- [![docs](https://img.shields.io/badge/docs-speckle.guide-orange?style=for-the-badge&logo=read-the-docs&logoColor=white)](https://speckle.guide/dev/) reference on almost any end-user and developer functionality
+
+
+# Repo structure
+
+## Usage
+
+Send and receive data from a Speckle Server with `operations`, interact with the Speckle API with the `SpeckleClient`, create and extend your own custom Speckle Objects with `Base`, and more! 
+
+Head to the [**📚 specklepy docs**](https://speckle.guide/dev/python.html) for more information and usage examples.

 ## Developing & Debugging

@@ -34,109 +70,6 @@ It may be helpful to know where the local accounts and object cache dbs are stor
 - Linux: `$XDG_DATA_HOME` or by default `~/.local/share/Speckle`
 - Mac: `~/.config/Speckle`

-## Overview of functionality 
-
-The `SpeckleClient` is the entry point for interacting with the GraphQL API. You'll need to have a running server to use this.
-
-```py
-from specklepy.api.client import SpeckleClient
-from specklepy.api.credentials import get_default_account, get_local_accounts
-
-all_accounts = get_local_accounts() # get back a list
-account = get_default_account()
-
-client = SpeckleClient(host="speckle.xyz")
-# client = SpeckleClient(host="yourserver.com") or whatever your host is
-
-client.authenticate(account.token)
-```
-
-Interacting with streams is meant to be intuitive and evocative of PySpeckle 1.0
-
-```py
-# get your streams
-stream_list = client.stream.list()
-
-# search your streams
-results = client.user.search("mech")
-
-# create a stream
-new_stream_id = client.stream.create(name="a shiny new stream")
-
-# get a stream
-new_stream = client.stream.get(id=new_stream_id)
-```
-
-New in 2.0: commits! Here are some basic commit interactions.
-
-```py
-# get list of commits
-commits = client.commit.list("stream id")
-
-# get a specific commit
-commit = client.commit.get("stream id", "commit id")
-
-# create a commit
-commit_id = client.commit.create("stream id", "object id", "this is a commit message to describe the commit")
-
-# delete a commit
-deleted = client.commit.delete("stream id", "commit id")
-```
-
-The `BaseObjectSerializer` is used for decomposing and serializing `Base` objects so they can be sent / received to the server. You can use it directly to get the id (hash) and a serializable object representation of the decomposed `Base`. You can learn more about the Speckle `Base` object [here](https://discourse.speckle.works/t/core-2-0-the-base-object/782) and the decomposition API [here](https://discourse.speckle.works/t/core-2-0-decomposition-api/911).
-
-```py
-from specklepy.objects.base import Base
-from specklepy.serialization.base_object_serializer import BaseObjectSerializer
-
-detached_base = Base()
-detached_base.name = "this will get detached"
-
-base_obj = Base()
-base_obj.name = "my base"
-base_obj["@nested"] = detached_base
-
-serializer = BaseObjectSerializer()
-hash, obj_dict = serializer.traverse_base(base_obj)
-```
-
-If you use the `operations`, you will not need to interact with the serializer directly as this will be taken care of for you. You will just need to provide a transport to indicate where the objects should be sent / received from. At the moment, just the `MemoryTransport` and the `ServerTransport` are fully functional at the moment. If you'd like to learn more about Transports in Speckle 2.0, have a look [here](https://discourse.speckle.works/t/core-2-0-transports/919).
-
-```py
-from specklepy.transports.memory import MemoryTransport
-from specklepy.api import operations
-
-transport = MemoryTransport()
-
-# this serialises the object and sends it to the transport
-hash = operations.send(base=base_obj, transports=[transport])
-
-# if the object had detached objects, you can see these as well
-saved_objects = transport.objects # a dict with the obj hash as the key
-
-# this receives and object from the given transport, deserialises it, and recomposes it into a base object
-received_base = operations.receive(obj_id=hash, remote_transport=transport)
-```
-
-You can also use the GraphQL API to send and receive objects.
-
-```py
-# create a test base object
-test_base = Base()
-test_base.testing = "a test base obj"
-
-# run it through the serialiser
-s = BaseObjectSerializer()
-hash, obj = s.traverse_base(test_base)
-
-# send it to the server
-objCreate = client.object.create(stream_id="stream id", objects=[obj])
-
-received_base = client.object.get("stream id", hash)
-```
-
-This doc is not complete - there's more to see so have a dive into the code and play around! Please feel free to provide feedback, submit issues, or discuss new features ✨
-
 ## Contributing

 Please make sure you read the [contribution guidelines](.github/CONTRIBUTING.md) for an overview of the best practices we try to follow.
@@ -1,12 +1,13 @@
 import os
-from specklepy.transports.server.server import ServerTransport
 from warnings import warn
 from pydantic import BaseModel
 from typing import List, Optional
 from urllib.parse import urlparse, unquote
+from specklepy.logging import metrics
 from specklepy.api.models import ServerInfo
 from specklepy.api.client import SpeckleClient
 from specklepy.transports.sqlite import SQLiteTransport
+from specklepy.transports.server.server import ServerTransport
 from specklepy.logging.exceptions import SpeckleException, SpeckleWarning


@@ -41,6 +42,7 @@ def get_local_accounts(base_path: str = None) -> List[Account]:
    Returns:
        List[Account] -- list of all local accounts or an empty list if no accounts were found
    """
+    metrics.track(metrics.ACCOUNT_LIST)
    account_storage = SQLiteTransport(scope="Accounts", base_path=base_path)
    json_path = os.path.join(account_storage._base_path, "Accounts")
    os.makedirs(json_path, exist_ok=True)
@@ -73,6 +75,7 @@ def get_default_account(base_path: str = None) -> Account:
    Returns:
        Account -- the default account or None if no local accounts were found
    """
+    metrics.track(metrics.ACCOUNT_DEFAULT)
    accounts = get_local_accounts(base_path=base_path)
    if not accounts:
        return None
@@ -114,6 +117,7 @@ class StreamWrapper:
            return "stream" if self.stream_id else "invalid"

    def __init__(self, url: str) -> None:
+        metrics.track("streamwrapper")
        self.stream_url = url
        parsed = urlparse(url)
        self.host = parsed.netloc
@@ -1,7 +1,7 @@
 from typing import List
+from specklepy.logging import metrics
 from specklepy.objects.base import Base
 from specklepy.transports.sqlite import SQLiteTransport
-from specklepy.transports.server import ServerTransport
 from specklepy.logging.exceptions import SpeckleException
 from specklepy.transports.abstract_transport import AbstractTransport
 from specklepy.serialization.base_object_serializer import BaseObjectSerializer
@@ -22,6 +22,7 @@ def send(
    Returns:
        str -- the object id of the sent object
    """
+    metrics.track(metrics.RECEIVE)
    if not transports and not use_default_cache:
        raise SpeckleException(
            message="You need to provide at least one transport: cannot send with an empty transport list and no default cache"
@@ -57,7 +58,7 @@ def receive(
    Returns:
        Base -- the base object
    """
-
+    metrics.track(metrics.RECEIVE)
    if not local_transport:
        local_transport = SQLiteTransport()

@@ -92,6 +93,7 @@ def serialize(base: Base, write_transports: List[AbstractTransport] = []) -> str
    Returns:
        str -- the serialized object
    """
+    metrics.track(metrics.SERIALIZE)
    serializer = BaseObjectSerializer(write_transports=write_transports)

    return serializer.write_json(base)[1]
@@ -109,6 +111,7 @@ def deserialize(obj_string: str, read_transport: AbstractTransport = None) -> Ba
    Returns:
        Base -- the deserialized object
    """
+    metrics.track(metrics.DESERIALIZE)
    if not read_transport:
        read_transport = SQLiteTransport()

@@ -1,7 +1,9 @@
-from typing import Dict, List, Optional
 from gql import gql
-from specklepy.api.resource import ResourceBase
+from typing import Dict, List, Optional
+from specklepy.logging import metrics
 from specklepy.api.models import Stream
+from specklepy.api.resource import ResourceBase
+

 NAME = "stream"
 METHODS = [
@@ -35,6 +37,7 @@ class Resource(ResourceBase):
        Returns:
            Stream -- the retrieved stream
        """
+        metrics.track(metrics.STREAM_GET)
        query = gql(
            """
            query Stream($id: String!, $branch_limit: Int!, $commit_limit: Int!) {
@@ -90,6 +93,7 @@ class Resource(ResourceBase):
        Returns:
            List[Stream] -- A list of Stream objects
        """
+        metrics.track(metrics.STREAM_LIST)
        query = gql(
            """
            query User($stream_limit: Int!) {
@@ -147,6 +151,7 @@ class Resource(ResourceBase):
        Returns:
            id {str} -- the id of the newly created stream
        """
+        metrics.track(metrics.STREAM_CREATE)
        query = gql(
            """
            mutation StreamCreate($stream: StreamCreateInput!) {
@@ -177,6 +182,7 @@ class Resource(ResourceBase):
        Returns:
            bool -- whether the stream update was successful
        """
+        metrics.track(metrics.STREAM_UPDATE)
        query = gql(
            """
            mutation StreamUpdate($stream: StreamUpdateInput!) {
@@ -207,6 +213,7 @@ class Resource(ResourceBase):
        Returns:
            bool -- whether the deletion was successful
        """
+        metrics.track(metrics.STREAM_DELETE)
        query = gql(
            """
            mutation StreamDelete($id: String!) {
@@ -239,6 +246,7 @@ class Resource(ResourceBase):
        Returns:
            List[Stream] -- a list of Streams that match the search query
        """
+        metrics.track(metrics.STREAM_SEARCH)
        query = gql(
            """
            query StreamSearch($search_query: String!,$limit: Int!, $branch_limit:Int!, $commit_limit:Int!) {
@@ -0,0 +1,125 @@
+import os
+import queue
+import logging
+import requests
+import threading
+from requests.sessions import session
+from specklepy.transports.sqlite import SQLiteTransport
+
+"""
+Anonymous telemetry to help us understand how to make a better Speckle.
+This really helps us to deliver a better open source project and product!
+"""
+TRACK = True
+HOST_APP = "python"
+
+LOG = logging.getLogger(__name__)
+METRICS_TRACKER = None
+
+# actions
+RECEIVE = "receive"
+SEND = "send"
+STREAM_CREATE = "stream/create"
+STREAM_GET = "stream/get"
+STREAM_UPDATE = "stream/update"
+STREAM_DELETE = "stream/delete"
+STREAM_DETAILS = "stream/details"
+STREAM_LIST = "stream/list"
+STREAM_VIEW = "stream/view"
+STREAM_SEARCH = "stream/search"
+
+ACCOUNT_DEFAULT = "account/default"
+ACCOUNT_DETAILS = "account/details"
+ACCOUNT_LIST = "account/list"
+
+SERIALIZE = "serialization/serialize"
+DESERIALIZE = "serialization/deserialize"
+
+
+def disable():
+    global TRACK
+    TRACK = False
+
+
+def set_host_app(host_app: str):
+    global HOST_APP
+    HOST_APP = host_app
+
+
+def track(action: str):
+    if not TRACK:
+        return
+    try:
+        global METRICS_TRACKER
+        if not METRICS_TRACKER:
+            METRICS_TRACKER = MetricsTracker()
+
+        page_params = {
+            "rec": 1,
+            "idsite": METRICS_TRACKER.site_id,
+            "uid": METRICS_TRACKER.suuid,
+            "action_name": action,
+            "url": f"http://connectors/{HOST_APP}/{action}",
+            "urlref": f"http://connectors/{HOST_APP}/{action}",
+            "_cvar": {"1": ["hostApplication", HOST_APP]},
+        }
+
+        event_params = {
+            "rec": 1,
+            "idsite": METRICS_TRACKER.site_id,
+            "uid": MetricsTracker.suuid,
+            "_cvar": {"1": ["hostApplication", HOST_APP]},
+            "e_c": HOST_APP,
+            "e_a": action,
+        }
+
+        METRICS_TRACKER.queue.put_nowait([event_params, page_params])
+    except Exception as ex:
+        # wrapping this whole thing in a try except as we never want a failure here to annoy users!
+        LOG.error("Error queueing metrics request: " + str(ex))
+
+
+class Singleton(type):
+    _instances = {}
+
+    def __call__(cls, *args, **kwargs):
+        if cls not in cls._instances:
+            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
+        return cls._instances[cls]
+
+
+class MetricsTracker(metaclass=Singleton):
+    matomo_url = "https://speckle.matomo.cloud/matomo.php"
+    site_id = 2
+    host_app = "python"
+    suuid = None
+    sending_thread = None
+    queue = queue.Queue(1000)
+
+    def __init__(self) -> None:
+        self.sending_thread = threading.Thread(
+            target=self._send_tracking_requests, daemon=True
+        )
+        self.set_suuid()
+        self.sending_thread.start()
+
+    def set_suuid(self):
+        try:
+            file_path = os.path.join(SQLiteTransport.get_base_path("Speckle"), "suuid")
+            with open(file_path, "r") as file:
+                self.suuid = file.read()
+        except:
+            self.suuid = "unknown-suuid"
+
+    def _send_tracking_requests(self):
+        session = requests.Session()
+        while True:
+            params = self.queue.get()
+
+            try:
+                session.post(self.matomo_url, params=params[0])
+                session.post(self.matomo_url, params=params[1])
+            except Exception as ex:
+                LOG.error("Error sending metrics request: " + str(ex))
+
+            self.queue.task_done()
@@ -1,11 +1,11 @@
 import typing
+from typing import (Any, Callable, ClassVar, Dict, List, Optional, Set, Type,
+                    get_type_hints)
 from warnings import warn
-from typing import get_type_hints
-from typing import ClassVar, Dict, List, Optional, Any, Set, Type
-from specklepy.transports.memory import MemoryTransport
+
 from specklepy.logging.exceptions import SpeckleException
 from specklepy.objects.units import get_units_from_string
-
+from specklepy.transports.memory import MemoryTransport

 PRIMITIVES = (int, float, str, bool)

@@ -62,6 +62,8 @@ REMOVE_FROM_DIR = {
    "to_dict",
    "update_forward_refs",
    "validate_prop_name",
+    "from_list",
+    "to_list",
 }


@@ -93,6 +95,7 @@ class _RegisteringBase:
        speckle_type: str = None,
        chunkable: Dict[str, int] = None,
        detachable: Set[str] = None,
+        serialize_ignore: Set[str] = None,
        **kwargs: Dict[str, Any],
    ):
        """
@@ -115,10 +118,14 @@ class _RegisteringBase:
        except Exception:
            cls._attr_types = getattr(cls, "__annotations__", {})
        if chunkable:
-            chunkable = {k: v for k, v in chunkable.items() if isinstance(v, int)}
+            chunkable = {k: v for k, v in chunkable.items()
+                         if isinstance(v, int)}
            cls._chunkable = dict(cls._chunkable, **chunkable)
        if detachable:
            cls._detachable = cls._detachable.union(detachable)
+        if serialize_ignore:
+            cls._serialize_ignore = cls._serialize_ignore.union(
+                serialize_ignore)
        super().__init_subclass__(**kwargs)


@@ -127,9 +134,11 @@ class Base(_RegisteringBase):
    totalChildrenCount: Optional[int] = None
    applicationId: Optional[str] = None
    _units: str = "m"
-    _chunkable: Dict[str, int] = {}  # dict of chunkable props and their max chunk size
+    # dict of chunkable props and their max chunk size
+    _chunkable: Dict[str, int] = {}
    _chunk_size_default: int = 1000
    _detachable: Set[str] = set()  # list of defined detachable props
+    _serialize_ignore: Set[str] = set()

    def __init__(self, **kwargs) -> None:
        super().__init__()
@@ -206,13 +215,15 @@ class Base(_RegisteringBase):
        try:
            cls._attr_types = get_type_hints(cls)
        except Exception as e:
-            warn(f"Could not update forward refs for class {cls.__name__}: {e}")
+            warn(
+                f"Could not update forward refs for class {cls.__name__}: {e}")

    @classmethod
    def validate_prop_name(cls, name: str) -> None:
        """Validator for dynamic attribute names."""
        if name in {"", "@"}:
-            raise ValueError("Invalid Name: Base member names cannot be empty strings")
+            raise ValueError(
+                "Invalid Name: Base member names cannot be empty strings")
        if name.startswith("@@"):
            raise ValueError(
                "Invalid Name: Base member names cannot start with more than one '@'",
@@ -288,36 +299,8 @@ class Base(_RegisteringBase):
    def units(self, value: str):
        self._units = get_units_from_string(value)

-    # def to_dict(self) -> Dict[str, Any]:
-    #     """Convenience method to view the whole base object as a dict"""
-    #     base_dict = self.__dict__
-    #     for key, value in base_dict.items():
-    #         if not value or isinstance(value, PRIMITIVES):
-    #             continue
-    #         else:
-    #             base_dict[key] = self.__dict_helper(value)
-    #     return base_dict
-
-    # def __dict_helper(self, obj: Any) -> Any:
-    #     if not obj or isinstance(obj, PRIMITIVES):
-    #         return obj
-    #     if isinstance(obj, Base):
-    #         return self.__dict_helper(obj.__dict__)
-    #     if isinstance(obj, (list, set)):
-    #         return [self.__dict_helper(v) for v in obj]
-    #     if not isinstance(obj, dict):
-    #         raise SpeckleException(
-    #             message=f"Could not convert to dict due to unrecognized type: {type(obj)}"
-    #         )
-
-    #     for k, v in obj.items():
-    #         if v and not isinstance(obj, PRIMITIVES):
-    #             obj[k] = self.__dict_helper(v)
-    #     return obj
-
    def get_member_names(self) -> List[str]:
        """Get all of the property names on this object, dynamic or not"""
-        # attrs = set(self.__dict__.keys())
        attr_dir = list(set(dir(self)) - REMOVE_FROM_DIR)
        return [
            name
@@ -325,6 +308,10 @@ class Base(_RegisteringBase):
            if not name.startswith("_") and not callable(getattr(self, name))
        ]

+    def get_serializable_attributes(self) -> List[str]:
+        """Get the attributes that should be serialized"""
+        return list(set(self.get_member_names()) - self._serialize_ignore)
+
    def get_typed_member_names(self) -> List[str]:
        """Get all of the names of the defined (typed) properties of this object"""
        return list(self._attr_types.keys())
@@ -350,9 +337,8 @@ class Base(_RegisteringBase):
        Returns:
            str -- the hash (id) of the fully serialized object
        """
-        from specklepy.serialization.base_object_serializer import (
-            BaseObjectSerializer,
-        )
+        from specklepy.serialization.base_object_serializer import \
+            BaseObjectSerializer

        serializer = BaseObjectSerializer()
        if decompose:
@@ -0,0 +1,127 @@
+from enum import Enum
+from typing import Any, Callable, List, Type
+
+from specklepy.logging.exceptions import SpeckleException
+from specklepy.objects.base import Base
+
+
+class CurveTypeEncoding(int, Enum):
+    Arc = 0
+    Circle = 1
+    Curve = 2
+    Ellipse = 3
+    Line = 4
+    Polyline = 5
+    Polycurve = 6
+
+    @property
+    def object_class(self) -> Type:
+        from . import geometry
+        if self == self.Arc:
+            return geometry.Arc
+        elif self == self.Circle:
+            return geometry.Circle
+        elif self == self.Curve:
+            return geometry.Curve
+        elif self == self.Ellipse:
+            return geometry.Ellipse
+        elif self == self.Line:
+            return geometry.Line
+        elif self == self.Polyline:
+            return geometry.Polyline
+        elif self == self.Polycurve:
+            return geometry.Polycurve
+        raise SpeckleException(
+            f'No corresponding object class for CurveTypeEncoding: {self}')
+
+
+def curve_from_list(args: List[float]):
+    curve_type = CurveTypeEncoding(args[0])
+    return curve_type.object_class.from_list(args)
+
+
+class ObjectArray:
+
+    def __init__(self) -> None:
+        self.data = []
+
+    @classmethod
+    def from_objects(cls, objects: List[Base]) -> 'ObjectArray':
+        data_chunk = cls()
+        if len(objects) == 0:
+            return data_chunk
+
+        speckle_type = objects[0].speckle_type
+
+        for obj in objects:
+            if speckle_type != obj.speckle_type:
+                raise SpeckleException(
+                    'All objects in chunk should have the same speckle_type. '
+                    f'Found {speckle_type} and {obj.speckle_type}'
+                )
+            data_chunk.encode_object(object=obj)
+
+        return data_chunk
+
+    @staticmethod
+    def decode_data(data: List[Any], decoder: Callable[[List[Any]], Base]) -> List[Base]:
+        index = 0
+        unchunked_data = []
+        while index < len(data):
+            chunk_length = data[index]
+            chunk_start = int(index + 1)
+            chunk_end = int(chunk_start + chunk_length)
+            chunk_data = data[chunk_start:chunk_end]
+            decoded_data = decoder(chunk_data)
+            unchunked_data.append(decoded_data)
+            index = chunk_end
+        return unchunked_data
+
+    def decode(self, decoder: Callable[[List[Any]], Any]):
+        return self.decode_data(data=self.data, decoder=decoder)
+
+    def encode_object(self, object: Base):
+        chunk = object.to_list()
+        chunk.insert(0, len(chunk))
+        self.data.extend(chunk)
+
+
+class CurveArray(ObjectArray):
+
+    @classmethod
+    def from_curve(cls, curve: Base) -> 'CurveArray':
+        crv_array = cls()
+        crv_array.data = curve.to_list()
+        return crv_array
+
+    @classmethod
+    def from_curves(cls, curves: List[Base]) -> 'CurveArray':
+        data = []
+        for curve in curves:
+            curve_list = curve.to_list()
+            curve_list.insert(0, len(curve_list))
+            data.extend(curve_list)
+        crv_array = cls()
+        crv_array.data = data
+        return crv_array
+
+    @staticmethod
+    def curve_from_list(args: List[float]) -> Base:
+        curve_type = CurveTypeEncoding(args[0])
+        return curve_type.object_class.from_list(args)
+
+    @property
+    def type(self) -> CurveTypeEncoding:
+        return CurveTypeEncoding(self.data[0])
+
+    def to_curve(self) -> Base:
+        return self.type.object_class.from_list(self.data)
+
+    @classmethod
+    def _curve_decoder(cls, data: List[float]) -> Base:
+        crv_array = cls()
+        crv_array.data = data
+        return crv_array.to_curve()
+
+    def to_curves(self) -> List[Base]:
+        return self.decode(decoder=self._curve_decoder)
@@ -1,5 +1,9 @@
+from enum import Enum
+from typing import Any, List, Optional
+
 from .base import Base
-from typing import Any, List
+from .encoding import CurveArray, CurveTypeEncoding, ObjectArray
+from .units import get_encoding_from_units, get_units_from_encoding

 GEOMETRY = "Objects.Geometry."

@@ -11,6 +15,13 @@ class Interval(Base, speckle_type="Objects.Primitive.Interval"):
    def length(self):
        return abs(self.start - self.end)

+    @classmethod
+    def from_list(cls, args: List[Any]) -> "Interval":
+        return cls(start=args[0], end=args[1])
+
+    def to_list(self) -> List[Any]:
+        return [self.start, self.end]
+

 class Point(Base, speckle_type=GEOMETRY + "Point"):
    x: float = 0.0
@@ -21,7 +32,14 @@ class Point(Base, speckle_type=GEOMETRY + "Point"):
        return f"{self.__class__.__name__}(x: {self.x}, y: {self.y}, z: {self.z}, id: {self.id}, speckle_type: {self.speckle_type})"

    @classmethod
-    def from_coords(x: float = 0.0, y: float = 0.0, z: float = 0.0):
+    def from_list(cls, args: List[float]) -> "Point":
+        return cls(x=args[0], y=args[1], z=args[2])
+
+    def to_list(self) -> List[Any]:
+        return [self.x, self.y, self.z]
+
+    @classmethod
+    def from_coords(cls, x: float = 0.0, y: float = 0.0, z: float = 0.0):
        pt = Point()
        pt.x, pt.y, pt.z = x, y, z
        return pt
@@ -41,6 +59,23 @@ class Plane(Base, speckle_type=GEOMETRY + "Plane"):
    xdir: Vector = Vector()
    ydir: Vector = Vector()

+    @classmethod
+    def from_list(cls, args: List[Any]) -> "Plane":
+        return cls(
+            origin=Point.from_list(args[0:3]),
+            normal=Vector.from_list(args[3:6]),
+            xdir=Vector.from_list(args[6:9]),
+            ydir=Vector.from_list(args[9:12]),
+        )
+
+    def to_list(self) -> List[Any]:
+        encoded = []
+        encoded.extend(self.origin.to_list())
+        encoded.extend(self.normal.to_list())
+        encoded.extend(self.xdir.to_list())
+        encoded.extend(self.ydir.to_list())
+        return encoded
+

 class Box(Base, speckle_type=GEOMETRY + "Box"):
    basePlane: Plane = Plane()
@@ -58,6 +93,21 @@ class Line(Base, speckle_type=GEOMETRY + "Line"):
    bbox: Box = None
    length: float = None

+    @classmethod
+    def from_list(cls, args: List[Any]) -> "Line":
+        return cls(
+            start=Point.from_list(args[0:3]),
+            end=Point.from_list(args[3:6]),
+            domain=Interval.from_list(args[6:9]),
+        )
+
+    def to_list(self) -> List[Any]:
+        encoded = []
+        encoded.extend(self.start.to_list())
+        encoded.extend(self.end.to_list())
+        encoded.extend(self.domain.to_list())
+        return encoded
+

 class Arc(Base, speckle_type=GEOMETRY + "Arc"):
    radius: float = None
@@ -73,6 +123,30 @@ class Arc(Base, speckle_type=GEOMETRY + "Arc"):
    area: float = None
    length: float = None

+    @classmethod
+    def from_list(cls, args: List[Any]) -> "Arc":
+        return cls(
+            radius=args[1],
+            startAngle=args[2],
+            endAngle=args[3],
+            angleRadians=args[4],
+            domain=Interval.from_list(args[5:7]),
+            plane=Plane.from_list(args[7:20]),
+            units=get_units_from_encoding(args[-1]),
+        )
+
+    def to_list(self) -> List[Any]:
+        encoded = []
+        encoded.append(CurveTypeEncoding.Arc.value)
+        encoded.append(self.radius)
+        encoded.append(self.startAngle)
+        encoded.append(self.endAngle)
+        encoded.append(self.angleRadians)
+        encoded.extend(self.domain.to_list())
+        encoded.extend(self.plane.to_list())
+        encoded.append(get_encoding_from_units(self.units))
+        return encoded
+

 class Circle(Base, speckle_type=GEOMETRY + "Circle"):
    radius: float = None
@@ -82,6 +156,24 @@ class Circle(Base, speckle_type=GEOMETRY + "Circle"):
    area: float = None
    length: float = None

+    @classmethod
+    def from_list(cls, args: List[Any]) -> "Circle":
+        return cls(
+            radius=args[1],
+            domain=Interval.from_list(args[2:4]),
+            plane=Plane.from_list(args[4:17]),
+            units=get_units_from_encoding(args[-1]),
+        )
+
+    def to_list(self) -> List[Any]:
+        encoded = []
+        encoded.append(CurveTypeEncoding.Circle.value)
+        encoded.append(self.radius),
+        encoded.extend(self.domain.to_list())
+        encoded.extend(self.plane.to_list())
+        encoded.append(get_encoding_from_units(self.units))
+        return encoded
+

 class Ellipse(Base, speckle_type=GEOMETRY + "Ellipse"):
    firstRadius: float = None
@@ -93,6 +185,26 @@ class Ellipse(Base, speckle_type=GEOMETRY + "Ellipse"):
    area: float = None
    length: float = None

+    @classmethod
+    def from_list(cls, args: List[Any]) -> "Ellipse":
+        return cls(
+            firstRadius=args[1],
+            secondRadius=args[2],
+            domain=Interval.from_list(args[3:5]),
+            plane=Plane.from_list(args[5:18]),
+            units=get_units_from_encoding(args[-1]),
+        )
+
+    def to_list(self) -> List[Any]:
+        encoded = []
+        encoded.append(CurveTypeEncoding.Ellipse.value)
+        encoded.append(self.firstRadius)
+        encoded.append(self.secondRadius)
+        encoded.extend(self.domain.to_list())
+        encoded.extend(self.plane.to_list())
+        encoded.append(get_encoding_from_units(self.units))
+        return encoded
+

 class Polyline(Base, speckle_type=GEOMETRY + "Polyline", chunkable={"value": 20000}):
    value: List[float] = None
@@ -111,15 +223,25 @@ class Polyline(Base, speckle_type=GEOMETRY + "Polyline", chunkable={"value": 200
            polyline.value.extend([point.x, point.y, point.z])
        return polyline

-    # @property
-    # def value(self) -> List[float]:
-    #     return self._value
+    @classmethod
+    def from_list(cls, args: List[Any]) -> "Polyline":
+        point_count = args[4]
+        return cls(
+            closed=bool(args[1]),
+            domain=Interval.from_list(args[2:4]),
+            value=args[5 : 5 + point_count],
+            units=get_units_from_encoding(args[-1]),
+        )

-    # @value.setter
-    # def value(self, coords) -> None:
-    #     if len(coords) % 3:
-    #         coords.extend([0] * (3 - len(coords) % 3))
-    #     self._value = coords
+    def to_list(self) -> List[Any]:
+        encoded = []
+        encoded.append(CurveTypeEncoding.Polyline.value)
+        encoded.append(int(self.closed))
+        encoded.extend(self.domain.to_list())
+        encoded.append(len(self.value))
+        encoded.extend(self.value)
+        encoded.append(get_encoding_from_units(self.units))
+        return encoded

    def as_points(self) -> List[Point]:
        """Converts the `value` attribute to a list of Points"""
@@ -166,6 +288,46 @@ class Curve(
            Point(x=v, y=next(values), z=next(values), units=self.units) for v in values
        ]

+    @classmethod
+    def from_list(cls, args: List[Any]) -> "Curve":
+        point_count = args[7]
+        weights_count = args[8]
+        knots_count = args[9]
+
+        points_start = 10
+        weights_start = 10 + point_count
+        knots_start = weights_start + weights_count
+        knots_end = knots_start + knots_count
+
+        return cls(
+            degree=args[1],
+            periodic=bool(args[2]),
+            rational=bool(args[3]),
+            closed=bool(args[4]),
+            domain=Interval.from_list(args[5:7]),
+            points=args[points_start:weights_start],
+            weights=args[weights_start:knots_start],
+            knots=args[knots_start:knots_end],
+            units=get_units_from_encoding(args[-1]),
+        )
+
+    def to_list(self) -> List[Any]:
+        encoded = []
+        encoded.append(CurveTypeEncoding.Curve.value)
+        encoded.append(self.degree)
+        encoded.append(int(self.periodic))
+        encoded.append(int(self.rational))
+        encoded.append(int(self.closed))
+        encoded.extend(self.domain.to_list())
+        encoded.append(len(self.points))
+        encoded.append(len(self.weights))
+        encoded.append(len(self.knots))
+        encoded.extend(self.points)
+        encoded.extend(self.weights)
+        encoded.extend(self.knots)
+        encoded.append(get_encoding_from_units(self.units))
+        return encoded
+

 class Polycurve(Base, speckle_type=GEOMETRY + "Polycurve"):
    segments: List[Base] = None
@@ -175,6 +337,27 @@ class Polycurve(Base, speckle_type=GEOMETRY + "Polycurve"):
    area: float = None
    length: float = None

+    @classmethod
+    def from_list(cls, args: List[Any]) -> "Polycurve":
+        curve_arrays = CurveArray()
+        curve_arrays.data = args[4:-1]
+        return cls(
+            closed=bool(args[1]),
+            domain=Interval.from_list(args[2:4]),
+            segments=curve_arrays.to_curves(),
+            units=get_units_from_encoding(args[-1]),
+        )
+
+    def to_list(self) -> List[Any]:
+        encoded = []
+        encoded.append(CurveTypeEncoding.Polycurve.value)
+        encoded.append(int(self.closed))
+        encoded.extend(self.domain.to_list())
+        curve_array = CurveArray.from_curves(self.segments)
+        encoded.extend(curve_array.data)
+        encoded.append(get_encoding_from_units(self.units))
+        return encoded
+

 class Extrusion(Base, speckle_type=GEOMETRY + "Extrusion"):
    capped: bool = None
@@ -218,6 +401,58 @@ class Surface(Base, speckle_type=GEOMETRY + "Surface"):
    countU: int = None
    countV: int = None
    bbox: Box = None
+    closedU: bool = None
+    closedV: bool = None
+    domainU: Interval = None
+    domainV: Interval = None
+    knotsU: List[float] = None
+    knotsV: List[float] = None
+
+    @classmethod
+    def from_list(cls, args: List[Any]) -> "Surface":
+        point_count = int(args[11])
+        knots_u_count = int(args[12])
+        knots_v_count = int(args[13])
+
+        start_point_data = 14
+        start_knots_u = start_point_data + point_count
+        start_knots_v = start_knots_u + knots_u_count
+
+        return cls(
+            degreeU=int(args[0]),
+            degreeV=int(args[1]),
+            countU=int(args[2]),
+            countV=int(args[3]),
+            rational=bool(args[4]),
+            closedU=bool(args[5]),
+            closedV=bool(args[6]),
+            domainU=Interval(start=args[7], end=args[8]),
+            domainV=Interval(start=args[9], end=args[10]),
+            pointData=args[start_point_data:start_knots_u],
+            knotsU=args[start_knots_u:start_knots_v],
+            knotsV=args[start_knots_v : start_knots_v + knots_v_count],
+            units=get_units_from_encoding(args[-1]),
+        )
+
+    def to_list(self) -> List[Any]:
+        encoded = []
+        encoded.append(self.degreeU)
+        encoded.append(self.degreeV)
+        encoded.append(self.countU)
+        encoded.append(self.countV)
+        encoded.append(int(self.rational))
+        encoded.append(int(self.closedU))
+        encoded.append(int(self.closedV))
+        encoded.extend(self.domainU.to_list())
+        encoded.extend(self.domainV.to_list())
+        encoded.append(len(self.pointData))
+        encoded.append(len(self.knotsU))
+        encoded.append(len(self.knotsV))
+        encoded.extend(self.pointData)
+        encoded.extend(self.knotsU)
+        encoded.extend(self.knotsV)
+        encoded.append(get_encoding_from_units(self.units))
+        return encoded


 class BrepFace(Base, speckle_type=GEOMETRY + "BrepFace"):
@@ -284,6 +519,17 @@ class BrepLoop(Base, speckle_type=GEOMETRY + "BrepLoop"):
            return [self._Brep.Trims[i] for i in self.TrimIndices]


+class BrepTrimTypeEnum(int, Enum):
+    Unknown = 0
+    Boundary = 1
+    Mated = 2
+    Seam = 3
+    Singular = 4
+    CurveOnSurface = 5
+    PointOnSurface = 6
+    Slit = 7
+
+
 class BrepTrim(Base, speckle_type=GEOMETRY + "BrepTrim"):
    _Brep: "Brep" = None
    EdgeIndex: int = None
@@ -313,21 +559,49 @@ class BrepTrim(Base, speckle_type=GEOMETRY + "BrepTrim"):
    def _curve_2d(self):
        return self._Brep.Curve2D[self.CurveIndex]

+    @classmethod
+    def from_list(cls, args: List[Any]) -> "BrepTrim":
+        return cls(
+            EdgeIndex=args[0],
+            StartIndex=args[1],
+            EndIndex=args[2],
+            FaceIndex=args[3],
+            LoopIndex=args[4],
+            CurveIndex=args[5],
+            IsoStatus=args[6],
+            TrimType=BrepTrimTypeEnum(args[7]).name,
+            IsReversed=bool(args[8]),
+        )
+
+    def to_list(self) -> List[Any]:
+        encoded = []
+        encoded.append(self.EdgeIndex)
+        encoded.append(self.StartIndex)
+        encoded.append(self.EndIndex)
+        encoded.append(self.FaceIndex)
+        encoded.append(self.LoopIndex)
+        encoded.append(self.CurveIndex)
+        encoded.append(self.IsoStatus)
+        encoded.append(getattr(BrepTrimTypeEnum, self.TrimType).value)
+        encoded.append(self.IsReversed)
+        return encoded
+

 class Brep(
    Base,
    speckle_type=GEOMETRY + "Brep",
    chunkable={
-        "Surfaces": 200,
-        "Curve3D": 200,
-        "Curve2D": 200,
-        "Vertices": 5000,
+        "SurfacesValue": 200,
+        "Curve3DValues": 200,
+        "Curve2DValues": 200,
+        "VerticesValue": 5000,
        "Edges": 5000,
        "Loops": 5000,
-        "Trims": 5000,
+        "TrimsValue": 5000,
        "Faces": 5000,
    },
    detachable={"displayValue"},
+    serialize_ignore={"Surfaces", "Curve3D", "Curve2D", "Vertices", "Trims"},
 ):
    provenance: str = None
    bbox: Box = None
@@ -338,20 +612,121 @@ class Brep(
    Curve3D: List[Base] = None
    Curve2D: List[Base] = None
    Vertices: List[Point] = None
-    Edges: List[BrepEdge] = None
-    Loops: List[BrepLoop] = None
-    Trims: List[BrepTrim] = None
-    Faces: List[BrepFace] = None
    IsClosed: bool = None
    Orientation: int = None

-    def __setattr__(self, name: str, value: Any) -> None:
-        if not value:
-            return
-        if name in {"Edges", "Loops", "Trims", "Faces"}:
-            for val in value:
-                val._Brep = self
-        super().__setattr__(name, value)
+    def _inject_self_into_children(self, children: Optional[List[Base]]) -> List[Base]:
+        if children is None:
+            return children
+
+        for child in children:
+            child._Brep = self
+        return children
+
+    @property
+    def Edges(self) -> List[BrepEdge]:
+        return self._inject_self_into_children(self._Edges)
+
+    @Edges.setter
+    def Edges(self, value: List[BrepEdge]):
+        self._Edges = value
+
+    @property
+    def Loops(self) -> List[BrepLoop]:
+        return self._inject_self_into_children(self._Loops)
+
+    @Loops.setter
+    def Loops(self, value: List[BrepLoop]):
+        self._Loops = value
+
+    @property
+    def Faces(self) -> List[BrepFace]:
+        return self._inject_self_into_children(self._Faces)
+
+    @Faces.setter
+    def Faces(self, value: List[BrepFace]):
+        self._Faces = value
+
+    @property
+    def SurfacesValue(self) -> List[float]:
+        if self.Surfaces is None:
+            return None
+        return ObjectArray.from_objects(self.Surfaces).data
+
+    @SurfacesValue.setter
+    def SurfacesValue(self, value: List[float]):
+        self.Surfaces = ObjectArray.decode_data(value, Surface.from_list)
+
+    @property
+    def Curve3DValues(self) -> List[float]:
+        if self.Curve3D is None:
+            return None
+        return CurveArray.from_curves(self.Curve3D).data
+
+    @Curve3DValues.setter
+    def Curve3DValues(self, value: List[float]):
+        crv_array = CurveArray()
+        crv_array.data = value
+        self.Curve3D = crv_array.to_curves()
+
+    @property
+    def Curve2DValues(self) -> List[Base]:
+        if self.Curve2D is None:
+            return None
+        return CurveArray.from_curves(self.Curve2D).data
+
+    @Curve2DValues.setter
+    def Curve2DValues(self, value: List[float]):
+        crv_array = CurveArray()
+        crv_array.data = value
+        self.Curve2D = crv_array.to_curves()
+
+    @property
+    def VerticesValue(self) -> List[Point]:
+        if self.Vertices is None:
+            return None
+        encoded_unit = get_encoding_from_units(self.Vertices[0].units)
+        values = [encoded_unit]
+        for vertex in self.Vertices:
+            values.extend(vertex.to_list())
+        return values
+
+    @VerticesValue.setter
+    def VerticesValue(self, value: List[float]):
+        value = value.copy()
+        units = get_units_from_encoding(value.pop(0))
+
+        vertices = []
+
+        for i in range(0, len(value), 3):
+            vertex = Point.from_list(value[i : i + 3])
+            vertex._units = units
+            vertices.append(vertex)
+
+        self.Vertices = vertices
+
+    @property
+    def Trims(self) -> List[BrepTrim]:
+        return self._inject_self_into_children(self._Trims)
+
+    @Trims.setter
+    def Trims(self, value: List[BrepTrim]):
+        self._Trims = value
+
+    @property
+    def TrimsValue(self) -> List[float]:
+        if self.Trims is None:
+            return None
+        values = []
+        for trim in self.Trims:
+            values.extend(trim.to_list())
+        return values
+
+    @TrimsValue.setter
+    def TrimsValue(self, value: List[float]):
+        self.Trims = [
+            BrepTrim.from_list(value[i : i + 9]) for i in range(0, len(value), 9)
+        ]


 BrepEdge.update_forward_refs()
@@ -14,6 +14,17 @@ UNITS_STRINGS = {
    "none": ["none", "null"],
 }

+UNITS_ENCODINGS = {
+    "mm": 1,
+    "cm": 2,
+    "m": 3,
+    "km": 4,
+    "in": 5,
+    "ft": 6,
+    "yd": 7,
+    "mi": 8,
+}
+

 def get_units_from_string(unit: str):
    unit = str.lower(unit)
@@ -24,3 +35,22 @@ def get_units_from_string(unit: str):
    raise SpeckleException(
        message=f"Could not understand what unit {unit} is referring to. Please enter a valid unit (eg {UNITS})."
    )
+
+
+def get_units_from_encoding(unit: int):
+    for name, encoding in UNITS_ENCODINGS.items():
+        if unit == encoding:
+            return name
+
+    raise SpeckleException(
+        message=f"Could not understand what unit {unit} is referring to. Please enter a valid unit encoding (eg {UNITS_ENCODINGS})."
+    )
+
+
+def get_encoding_from_units(unit: str):
+    try:
+        return UNITS_ENCODINGS[unit]
+    except KeyError:
+        raise SpeckleException(
+            message=f"No encoding exists for unit {unit}. Please enter a valid unit to encode (eg {UNITS_ENCODINGS})."
+        )
@@ -52,7 +52,7 @@ class BaseObjectSerializer:
        self.lineage.append(uuid4().hex)
        object_builder = {"id": "", "speckle_type": "Base", "totalChildrenCount": 0}
        object_builder.update(speckle_type=base.speckle_type)
-        obj, props = base, base.get_member_names()
+        obj, props = base, base.get_serializable_attributes()

        while props:
            prop = props.pop(0)
@@ -102,7 +102,9 @@ class BatchSender(object):
        new_objects = [obj[1] for obj in batch if obj[0] in new_object_ids]

        if not new_objects:
-            LOG.info(f"Uploading batch of {len(batch)} objects: all objects are already in the server")
+            LOG.info(
+                f"Uploading batch of {len(batch)} objects: all objects are already in the server"
+            )
            return

        upload_data = "[" + ",".join(new_objects) + "]"
@@ -112,24 +114,30 @@ class BatchSender(object):
            % (len(batch), len(new_objects), len(upload_data), len(upload_data_gzip))
        )

-        r = session.post(
-            url=f"{self.server_url}/objects/{self.stream_id}",
-            files={"batch-1": ("batch-1", upload_data_gzip, "application/gzip")},
-        )
-        if r.status_code != 201:
-            LOG.warning("Upload server response: %s", r.text)
-            raise SpeckleException(
-                message=f"Could not save the object to the server - status code {r.status_code}"
+        try:
+            r = session.post(
+                url=f"{self.server_url}/objects/{self.stream_id}",
+                files={"batch-1": ("batch-1", upload_data_gzip, "application/gzip")},
+            )
+            if r.status_code != 201:
+                LOG.warning("Upload server response: %s", r.text)
+                raise SpeckleException(
+                    message=f"Could not save the object to the server - status code {r.status_code}"
+                )
+        except json.JSONDecodeError as error:
+            return SpeckleException(
+                f"Failed to send objects to {self.server_url}. Please ensure this stream ({self.stream_id}) exists on this server and that you have permission to send to it.",
+                error,
            )

    def _create_threads(self):
-        for i in range(self.thread_count):
+        for _ in range(self.thread_count):
            t = threading.Thread(target=self._sending_thread_main, daemon=True)
            t.start()
            self._send_threads.append(t)

    def _delete_threads(self):
-        for i in range(len(self._send_threads)):
+        for _ in range(len(self._send_threads)):
            self._batches.put(None)

        for thread in self._send_threads:
@@ -32,7 +32,7 @@ class SQLiteTransport(AbstractTransport):
        super().__init__(**data)
        self.app_name = app_name or "Speckle"
        self.scope = scope or "Objects"
-        self._base_path = base_path or self.__get_base_path()
+        self._base_path = base_path or self.get_base_path(self.app_name)

        try:
            os.makedirs(self._base_path, exist_ok=True)
@@ -56,7 +56,8 @@ class SQLiteTransport(AbstractTransport):
    #     proc.start()
    #     proc.join()

-    def __get_base_path(self):
+    @staticmethod
+    def get_base_path(app_name):
        # from appdirs https://github.com/ActiveState/appdirs/blob/master/appdirs.py
        # default mac path is not the one we use (we use unix path), so using special case for this
        system = sys.platform
@@ -68,10 +69,10 @@ class SQLiteTransport(AbstractTransport):
                system = "darwin"

        if system != "darwin":
-            return user_data_dir(appname=self.app_name, appauthor=False, roaming=True)
+            return user_data_dir(appname=app_name, appauthor=False, roaming=True)

        path = os.path.expanduser("~/.config/")
-        return os.path.join(path, self.app_name)
+        return os.path.join(path, app_name)

    # def __consume_queue(self):
    #     if self._is_writing or self.__queue.empty():
@@ -7,6 +7,9 @@ from specklepy.api.client import SpeckleClient
 from specklepy.objects.base import Base
 from specklepy.objects.geometry import Point
 from specklepy.objects.fakemesh import FakeMesh
+from specklepy.logging import metrics
+
+metrics.disable()


@pytest.fixture(scope="session")
@@ -1,9 +1,10 @@
-import pytest
-from typing import Dict, List
-from specklepy.objects import Base
-from specklepy.api import operations
 from contextlib import ExitStack as does_not_raise
+from typing import Dict, List
+
+import pytest
+from specklepy.api import operations
 from specklepy.logging.exceptions import SpeckleException
+from specklepy.objects.base import Base, DataChunk


@pytest.mark.parametrize(
@@ -110,3 +111,5 @@ def test_type_checking() -> None:
    order.flavours = ["strawberry", "lychee", "peach", "pineapple"]

    assert order.price == 7.0
+
+
@@ -0,0 +1,389 @@
+import json
+from typing import Callable
+
+import pytest
+from specklepy.api import operations
+from specklepy.objects.base import Base
+from specklepy.objects.encoding import CurveArray, ObjectArray
+from specklepy.objects.geometry import (Arc, Box, Brep, BrepEdge, BrepFace,
+                                        BrepLoop, BrepTrim, BrepTrimTypeEnum,
+                                        Circle, Curve, Ellipse, Interval, Line,
+                                        Mesh, Plane, Point, Polycurve,
+                                        Polyline, Surface, Vector)
+from specklepy.transports.memory import MemoryTransport
+
+
+@pytest.fixture()
+def interval():
+    return Interval(start=0, end=5)
+
+
+@pytest.fixture()
+def point():
+    return Point(x=1, y=10, z=0)
+
+
+@pytest.fixture()
+def vector():
+    return Vector(x=1, y=32, z=10)
+
+
+@pytest.fixture()
+def plane(point, vector):
+    return Plane(
+        origin=point,
+        normal=vector,
+        xdir=vector,
+        ydir=vector,
+    )
+
+
+@pytest.fixture()
+def box(plane, interval):
+    return Box(
+        basePlane=plane,
+        ySize=interval,
+        zSize=interval,
+        xSize=interval,
+        area=20.4,
+        volume=44.2,
+    )
+
+
+@pytest.fixture()
+def line(point, interval):
+    return Line(
+        start=point,
+        end=point,
+        domain=interval,
+        # These attributes are not handled in C#
+        # bbox=None,
+        # length=None
+    )
+
+
+@pytest.fixture()
+def arc(plane, interval):
+    return Arc(
+        radius=2.3,
+        startAngle=22.1,
+        endAngle=44.5,
+        angleRadians=33,
+        plane=plane,
+        domain=interval,
+        units='m',
+        # These attributes are not handled in C#
+        # bbox=None,
+        # area=None,
+        # length=None,
+        # startPoint=None,
+        # midPoint=None,
+        # endPoint=None,
+    )
+
+
+@pytest.fixture()
+def circle(plane, interval):
+    return Circle(
+        radius=22,
+        plane=plane,
+        domain=interval,
+        units='m',
+        # These attributes are not handled in C#
+        # bbox=None,
+        # area=None,
+        # length=None,
+    )
+
+
+@pytest.fixture()
+def ellipse(plane, interval):
+    return Ellipse(
+        firstRadius=34,
+        secondRadius=22,
+        plane=plane,
+        domain=interval,
+        units='m',
+        # These attributes are not handled in C#
+        # trimDomain=None,
+        # bbox=None,
+        # area=None,
+        # length=None,
+    )
+
+
+@pytest.fixture()
+def polyline(interval):
+    return Polyline(
+        value=[22, 44, 54.3, 99, 232, 21],
+        closed=True,
+        domain=interval,
+        units='m',
+        # These attributes are not handled in C#
+        # bbox=None,
+        # area=None,
+        # length=None,
+    )
+
+
+@pytest.fixture()
+def curve(interval):
+    return Curve(
+        degree=90,
+        periodic=True,
+        rational=False,
+        closed=True,
+        domain=interval,
+        points=[23, 21, 44, 43, 56, 76, 1, 3, 2],
+        weights=[23, 11, 23],
+        knots=[22, 45, 76, 11],
+        units='m',
+        # These attributes are not handled in C#
+        # displayValue=None,
+        # bbox=None,
+        # area=None,
+        # length=None,
+    )
+
+
+@pytest.fixture()
+def polycurve(interval, curve, polyline):
+    return Polycurve(
+        segments=[curve, polyline],
+        domain=interval,
+        closed=True,
+        units='m',
+        # These attributes are not handled in C#
+        # bbox=None,
+        # area=None,
+        # length=None
+    )
+
+
+@pytest.fixture()
+def mesh(box):
+    return Mesh(
+        vertices=[2, 1, 2, 4, 77.3, 5, 33, 4, 2],
+        faces=[1, 2, 3, 4, 5, 6, 7],
+        colors=[111, 222, 333, 444, 555, 666, 777],
+        bbox=box,
+        area=233,
+        volume=232.2,
+    )
+
+
+@pytest.fixture()
+def surface(interval):
+    return Surface(
+        degreeU=33,
+        degreeV=44,
+        rational=True,
+        pointData=[1, 2.2, 3, 4, 5, 6, 7, 8, 9],
+        countU=3,
+        countV=4,
+        closedU=True,
+        closedV=False,
+        domainU=interval,
+        domainV=interval,
+        knotsU=[1.1, 2.2, 3.3, 4.4],
+        knotsV=[9, 8, 7, 6, 5, 4.4],
+        units='m',
+        # These attributes are not handled in C#
+        # bbox=None,
+        # area=None,
+    )
+
+
+@pytest.fixture()
+def brep_face():
+    return BrepFace(
+        SurfaceIndex=3,
+        LoopIndices=[1, 2, 3, 4],
+        OuterLoopIndex=2,
+        OrientationReversed=False,
+    )
+
+
+@pytest.fixture()
+def brep_edge(interval):
+    return BrepEdge(
+        Curve3dIndex=2,
+        TrimIndices=[4, 5, 6, 7],
+        StartIndex=2,
+        EndIndex=6,
+        ProxyCurveIsReversed=True,
+        Domain=interval,
+    )
+
+
+@pytest.fixture()
+def brep_loop():
+    return BrepLoop(
+        FaceIndex=5,
+        TrimIndices=[3, 4, 5],
+        Type='unknown'
+    )
+
+
+@pytest.fixture()
+def brep_trim():
+    return BrepTrim(
+        EdgeIndex=3,
+        StartIndex=4,
+        EndIndex=6,
+        FaceIndex=1,
+        LoopIndex=4,
+        CurveIndex=7,
+        IsoStatus=6,
+        TrimType='Mated',
+        IsReversed=False,
+        # These attributes are not handled in C#
+        # Domain=None,
+    )
+
+
+@pytest.fixture
+def brep(mesh, box, surface, curve, polyline, circle, point,
+         brep_edge, brep_loop, brep_trim, brep_face):
+    return Brep(
+        provenance='pytest',
+        bbox=box,
+        area=32,
+        volume=54,
+        displayValue=mesh,
+        Surfaces=[surface, surface, surface],
+        Curve3D=[curve, polyline],
+        Curve2D=[circle],
+        Vertices=[point, point, point, point],
+        Edges=[brep_edge],
+        Loops=[brep_loop, brep_loop],
+        Trims=[brep_trim],
+        Faces=[brep_face, brep_face],
+        IsClosed=False,
+        Orientation=3,
+    )
+
+
+@pytest.fixture
+def geometry_objects_dict(point, vector, plane, line, arc,
+                          circle, ellipse, polyline, curve,
+                          polycurve, surface, brep_trim):
+    return {
+        'point': point,
+        'vector': vector,
+        'plane': plane,
+        'line': line,
+        'arc': arc,
+        'circle': circle,
+        'ellipse': ellipse,
+        'polyline': polyline,
+        'curve': curve,
+        'polycurve': polycurve,
+        'surface': surface,
+        'brep_trim': brep_trim
+    }
+
+
+@pytest.mark.parametrize('object_name', [
+    'point', 'vector', 'plane', 'line', 'arc', 'circle',
+    'ellipse', 'polyline', 'curve', 'polycurve', 'surface', 'brep_trim'
+])
+def test_to_and_from_list(object_name: str, geometry_objects_dict):
+    object = geometry_objects_dict[object_name]
+    assert hasattr(object, 'to_list')
+    assert hasattr(object, 'from_list')
+
+    chunks = object.to_list()
+    assert isinstance(chunks, list)
+
+    object_class = object.__class__
+    decoded_object: Base = object_class.from_list(chunks)
+    assert decoded_object.get_id() == object.get_id()
+
+
+def test_brep_surfaces_value_serialization(surface):
+    brep = Brep()
+    assert brep.Surfaces == None
+    assert brep.SurfacesValue == None
+    brep.Surfaces = [surface, surface]
+    assert brep.SurfacesValue == ObjectArray.from_objects(
+        [surface, surface]).data
+
+    brep.SurfacesValue = ObjectArray.from_objects([surface]).data
+    assert len(brep.Surfaces) == 1
+    assert brep.Surfaces[0].get_id() == surface.get_id()
+
+
+def test_brep_curve2d_values_serialization(curve, polyline, circle):
+    brep = Brep()
+    assert brep.Curve2D == None
+    assert brep.Curve2DValues == None
+    brep.Curve2D = [curve, polyline]
+    assert brep.Curve2DValues == CurveArray.from_curves([curve, polyline]).data
+
+    brep.Curve2DValues = CurveArray.from_curves([circle]).data
+    assert len(brep.Curve2D) == 1
+    assert brep.Curve2D[0].get_id() == circle.get_id()
+
+
+def test_brep_curve3d_values_serialization(curve, polyline, circle):
+    brep = Brep()
+    assert brep.Curve3D == None
+    assert brep.Curve3DValues == None
+    brep.Curve3D = [curve, polyline]
+    assert brep.Curve3DValues == CurveArray.from_curves([curve, polyline]).data
+
+    brep.Curve3DValues = CurveArray.from_curves([circle]).data
+    assert len(brep.Curve3D) == 1
+    assert brep.Curve3D[0].get_id() == circle.get_id()
+
+
+def test_brep_vertices_values_serialization():
+    brep = Brep()
+    brep.VerticesValue = [1, 1, 1, 1, 2, 2, 2, 3, 3, 3]
+    brep.Vertices[0].get_id() == Point(x=1, y=1, z=1, _units='mm').get_id()
+    brep.Vertices[1].get_id() == Point(x=2, y=2, z=2, _units='mm').get_id()
+    brep.Vertices[2].get_id() == Point(x=3, y=3, z=3, _units='mm').get_id()
+
+
+def test_trims_value_serialization():
+    brep = Brep()
+    brep.TrimsValue = [
+        0, 0, 0, 0, 0, 0, 1, 1, 1,
+        1, 0, 0, 0, 0, 1, 2, 1, 0,
+    ]
+
+    brep.Trims[0].get_id() == BrepTrim(
+        EdgeIndex=0,
+        StartIndex=0,
+        EndIndex=0,
+        FaceIndex=0,
+        LoopIndex=0,
+        CurveIndex=0,
+        IsoStatus=1,
+        TrimType=BrepTrimTypeEnum.Boundary,
+        IsReversed=False,
+    ).get_id()
+
+    brep.Trims[1].get_id() == BrepTrim(
+        EdgeIndex=1,
+        StartIndex=0,
+        EndIndex=0,
+        FaceIndex=0,
+        LoopIndex=0,
+        CurveIndex=1,
+        IsoStatus=2,
+        TrimType=BrepTrimTypeEnum.Boundary,
+        IsReversed=True,
+    ).get_id()
+
+
+def test_serialized_brep_attributes(brep: Brep):
+    transport = MemoryTransport()
+    serialized = operations.serialize(brep, [transport])
+    serialized_dict = json.loads(serialized)
+
+    removed_keys = ['Surfaces', 'Curve3D', 'Curve2D', 'Vertices', 'Trims']
+
+    for k in removed_keys:
+        assert k not in serialized_dict.keys()
@@ -1,9 +1,9 @@
-from specklepy.transports.sqlite import SQLiteTransport
-from specklepy.objects import Base
-from specklepy.transports.memory import MemoryTransport
-from specklepy.api.models import Stream
-from specklepy.serialization.base_object_serializer import BaseObjectSerializer
 import pytest
+from specklepy.api.models import Stream
+from specklepy.objects import Base
+from specklepy.objects.encoding import ObjectArray
+from specklepy.serialization.base_object_serializer import BaseObjectSerializer
+from specklepy.transports.sqlite import SQLiteTransport


 class TestObject:
@@ -21,9 +21,11 @@ class TestObject:

    def test_object_create(self, client, stream, base):
        transport = SQLiteTransport()
-        s = BaseObjectSerializer(write_transports=[transport], read_transport=transport)
+        s = BaseObjectSerializer(
+            write_transports=[transport], read_transport=transport)
        _, base_dict = s.traverse_base(base)
-        obj_id = client.object.create(stream_id=stream.id, objects=[base_dict])[0]
+        obj_id = client.object.create(
+            stream_id=stream.id, objects=[base_dict])[0]

        assert isinstance(obj_id, str)
        assert base_dict["@detach"]["speckle_type"] == "reference"
@@ -38,3 +40,15 @@ class TestObject:
        assert fetched_base.name == base.name
        assert isinstance(fetched_base.vertices, list)
        # assert fetched_base["@detach"]["speckle_type"] == "reference"
+
+    def test_object_array_decoder(self):
+        array = ObjectArray()
+        array.data = [
+            5, 1, 1, 1, 1, 1,
+            4, 1, 1, 1, 1,
+            3, 1, 1, 1,
+            2, 1, 1,
+            1, 1
+        ]
+
+        assert array.decode(decoder=sum) == [5, 4, 3, 2, 1]
Author	SHA1	Message	Date
izzy lyseggen	81680ed766	Merge pull request #125 from specklesystems/izzy/metrics metrics 🛰	2021-10-12 11:38:40 +02:00
izzy lyseggen	c934720bb0	fix(metrics): try catch whole track method	2021-10-12 10:36:38 +01:00
izzy lyseggen	9297a5df49	feat(metrics): disable in tests	2021-10-11 18:10:53 +01:00
izzy lyseggen	7b8bf49769	feat(metrics): track creds, ops, and stream events	2021-10-11 18:10:33 +01:00
izzy lyseggen	c834496b72	feat(metrics): add them! 🛰	2021-10-11 18:09:50 +01:00
izzy lyseggen	f49491611f	Merge pull request #124 from specklesystems/izzy/objects fix(objects): quick geo and style edits	2021-10-04 09:39:55 +01:00
izzy lyseggen	19b83ba191	fix(objects): quick geo and style edits	2021-10-04 09:37:42 +01:00
izzy lyseggen	8d81aab1ac	Merge pull request #123 from specklesystems/izzy/server-errors fix(batch sender): improve error messages	2021-10-04 08:53:40 +01:00
izzy lyseggen	16868fbf3b	fix(batch sender): improve error messages for when send fails completely. previously user only got a json decode error	2021-10-04 08:53:02 +01:00
Matteo Cominetti	00892fc838	Create close-issue.yml	2021-10-02 17:13:24 +01:00
Matteo Cominetti	4987b33de2	Create open-issue.yml	2021-10-02 17:13:09 +01:00
izzy lyseggen	766f1fa840	Merge pull request #120 from AntoineDao/geometry-chunks-serialization Geometry chunks serialization	2021-09-30 11:59:28 +01:00
AntoineDao	69a5248abb	fix(brep): fix Curve2DValues.setter	2021-09-19 10:08:57 +00:00
AntoineDao	5c93e4f9dc	test(geometry): rewrite tests so they can fail I noticed that the Curv2D setter had a bug which wasn't caught by tests so I rewrote them to fail first	2021-09-19 10:08:28 +00:00
AntoineDao	e20b9b73c9	fix(brep): update vertices and trim values serialization It turns out that those are not serialized like the other list attributes... Which is not super helpful or consistent but oh well :D	2021-09-18 22:54:58 +00:00
AntoineDao	c06b20a963	style(comments): remove commented out code...	2021-09-18 21:56:59 +00:00
AntoineDao	5bc6b8c4ed	style(brep): use getter/setter on XValues attributes	2021-09-18 21:54:58 +00:00
AntoineDao	3005e421a6	refactor(base): use get_serializable_attributes	2021-09-18 11:47:47 +00:00
AntoineDao	8fb03972d5	fix(serialization): fix some bugs I introduced by not testing before committing...	2021-09-18 11:29:00 +00:00
AntoineDao	02702190c9	fix(object): move array encoding from datachunk to its own class	2021-09-15 23:30:59 +01:00
AntoineDao	2bd31ae954	style(lint): clean up code and update/re-order imports	2021-09-15 23:00:41 +01:00
AntoineDao	d0f8f95e4e	feat(brep): use list serializers	2021-09-12 17:43:12 +00:00
AntoineDao	fc3ae3b98e	feat(base): add 'ignore_serialize' attribute This enables users to specify attributes which should not be included in object serialization but should still be public members	2021-09-12 17:42:00 +00:00
AntoineDao	a6b19025e6	feat(base): add data chunk encoding/decoding methods These are helpful to iterate over chunked lists of encoded geometry objects and decode them into base objects (and vice versa)	2021-09-12 17:38:56 +00:00
AntoineDao	2be82f0874	feat(geometry): add list encoding serialization Most geometry objects can be encoded and decoded from list of floats that look like protobuffs in C#. This commit simply reproduces those methods	2021-09-12 17:36:40 +00:00
izzy lyseggen	70191b97a2	docs: update readme to align with new format	2021-09-03 16:20:15 +01:00
izzy lyseggen	dd2825272d	docs: update readme to align with new format	2021-09-03 16:18:18 +01:00