11 Commits

Author SHA1 Message Date
Jonathon Broughton b264e63627 Refactor matchers and clean up helpers
build and deploy Speckle functions / publish-automate-function-version (push) Has been cancelled
- Moved EmailMatcher and PatternChecker to a new matchers module.
- Introduced ParameterMatcher as an abstract class for matching strategies.
- Added PrefixMatcher and PatternMatcher classes for specific matching logic.
- Removed redundant code from the helpers module.
2025-03-25 01:00:39 +00:00
Jonathon Broughton 71e48c6005 Remove rule processor and rules files
Deleted the rule processing logic and associated rules. This clears out unused code that was no longer needed for the project.
2025-03-25 01:00:26 +00:00
Jonathon Broughton 002dd3de50 Add developer guide and enhance user instructions
- Introduced a comprehensive developer guide for the Data Shield function.
- Detailed deployment workflows, including release creation and dependency management.
- Explained core components like parameter matching, traversal systems, and actions.
- Updated user guide with sanitization modes and usage instructions.
- Added troubleshooting tips for common issues.
2025-03-25 01:00:12 +00:00
Jonathon Broughton 8191c3c743 Update sanitization mode logging
Changed the way sanitization mode is logged to use its value instead of the object directly. This improves clarity in the success message.
2025-03-25 00:20:29 +00:00
Jonathon Broughton ce2aea85bd Refactor parameter processing logic
- Updated the main function for better parameter sanitisation.
- Removed old ParameterProcessor class and integrated its functionality directly.
- Added new methods to handle v3-style properties and legacy v2 parameters.
- Improved email anonymisation logic in helper classes.
2025-03-25 00:19:32 +00:00
Jonathon Broughton f289891374 Refactor parameter actions and sanitisation logic
- Enhanced removal action to handle both dicts and Base objects.
- Improved error handling for dynamic attribute removal in Base objects.
- Updated anonymization action to support Revit-style parameters.
- Streamlined function inputs processing in the main automate function.
- Added new method for processing Revit parameters specifically.
2025-03-25 00:11:28 +00:00
Jonathon Broughton fe80f95a19 Update Ruff configuration options
- Enabled formatting with Ruff
- Kept existing settings for running on save and server usage
2025-03-25 00:11:17 +00:00
Jonathon Broughton 5295f8165d Add email anonymization action
- Introduced `AnonymizationAction` class for email handling.
- Added `EmailMatcher` to identify and anonymize emails.
- Updated `ParameterProcessor` to support value checking.
- Modified main function to create an anonymization action.
- Removed placeholder for the anonymization action.
2025-03-24 23:56:09 +00:00
Jonathon Broughton 42565839f9 Update success message with more details
- Enhanced the success message to include sanitisation mode.
- Added info on whether strict mode is active.
2025-03-24 23:16:45 +00:00
Jonathon Broughton f3bce1b753 Fix strict mode matching logic
Updated the strict mode in the matcher to check if the parameter name starts with the match value instead of being an exact match. This improves flexibility while still maintaining case insensitivity for non-strict matches.
2025-03-24 23:16:36 +00:00
Jonathon Broughton a676205cfb Add title for strict mode in inputs
- Added a title for the strict mode field
- Clarified case sensitivity in the description
2025-03-24 23:01:33 +00:00
12 changed files with 781 additions and 286 deletions
+1
View File
@@ -2,6 +2,7 @@
<project version="4">
<component name="RuffConfigService">
<option name="runRuffOnSave" value="true" />
<option name="useRuffFormat" value="true" />
<option name="useRuffServer" value="true" />
</component>
</project>
+273
View File
@@ -0,0 +1,273 @@
# Data Shield - Developer Guide
This document provides technical information for developers working on the Data Shield Speckle Automate function. It covers deployment workflows, core components, and guidance for extending the function.
## Deployment to Speckle Automate
### Creating a Release
The function is automatically deployed to Speckle Automate when a new release is created on GitHub:
1. Ensure your changes are committed and pushed to the main branch
2. Create a `requirements.txt` file (see next section) and commit to main branch
3. Create a new GitHub release:
- Go to your repository on GitHub
- Navigate to "Releases" under the repository name
- Click "Draft a new release"
- Create a **new tag** (e.g., `v1.0.1`)
- Write a descriptive title and release notes
- Click "Publish release"
Creating a new release triggers the GitHub Actions workflow defined in `main.yml`, which builds and publishes the function to Speckle Automate.
### Managing Dependencies
You can use any dependency management tool of your choice for local development (Poetry, pip, uv, etc.), but Speckle Automate requires a `requirements.txt` file for deployment.
**Important**: You must create and commit the `requirements.txt` file to the repository **before** creating a release. The deployment workflow relies on this file being present in the repository.
To generate and commit the requirements file based on your local environment:
- With standard pip: `pip freeze > requirements.txt`
- With uv: `uv pip freeze > requirements.txt`
- With Poetry: `poetry export -f requirements.txt --output requirements.txt --without-hashes`
- Or manually create/edit the file to include necessary dependencies
Then commit the updated file:
```bash
git add requirements.txt
git commit -m "Update requirements.txt"
git push
```
Only after the requirements.txt is committed should you create a new release as described above.
Note that during deployment, the GitHub Actions workflow uses `uv` to install the dependencies, but your local development environment can use any tool you prefer.
### Deployment Workflow Details
The deployment workflow:
1. Checks out the repository
2. Sets up Python 3.13
3. Installs dependencies from `requirements.txt`
4. Extracts the function schema
5. Uses the Speckle Automate GitHub composite action to:
- Build a Docker image with the function
- Push the image to the Speckle Automate registry
- Update the function in Speckle Automate
## Core Components
### Parameter Matching System
The function uses a strategy pattern for parameter matching, allowing flexible and extensible matching rules:
#### ParameterMatcher Classes
* `ParameterMatcher` (ABC): Abstract base class for all matchers
* `PrefixMatcher`: Matches parameters by prefix (with optional case sensitivity)
* `PatternMatcher`: Uses regex/glob patterns for more complex matching
```python
# Example: Creating a custom matcher
class SuffixMatcher(ParameterMatcher):
"""Matches parameters by suffix."""
def matches(self, param_name: str) -> bool:
"""Check if the parameter name ends with the match value."""
if self.strict_mode:
return param_name.endswith(self.match_value)
return param_name.lower().endswith(self.match_value.lower())
```
#### Pattern Checking
The `PatternChecker` class handles both glob-style patterns (e.g., `speckle_*`) and regular expressions (e.g., `/^speckle_\d+$/i`):
* Glob patterns use `fnmatch` for simple wildcard matching
* Regex patterns must be wrapped in slashes (`/pattern/`)
* Case sensitivity is controlled by:
- The global `strict_mode` parameter
- The `/i` flag for regex patterns (overrides `strict_mode`)
### Traversal System
The function uses Speckle's graph traversal system to navigate the complex object hierarchy:
1. `GraphTraversal` from `specklepy.objects.graph_traversal.traversal` defines rules for how to navigate objects
2. `TraversalRule` objects define:
- Conditions for when a rule applies to an object
- Methods to extract the next objects to traverse
3. Our custom rules in `traversal.py` focus on:
- `display_value_rule`: For objects with displayValue/elements properties
- `default_rule`: General fallback for traversing all object members
The traversal system provides contexts that contain:
- The current object being traversed
- The path taken to reach that object
- Other metadata used during traversal
### Parameter Actions
Actions implement the logic for what to do when a parameter match is found:
#### ParameterAction Classes
* `ParameterAction` (ABC): Abstract base class for all actions
* `RemovalAction`: Removes matching parameters from objects
* `AnonymizationAction`: Masks email addresses in parameter values
Each action implements:
- `check()`: Determines if the action should be applied
- `apply()`: Performs the action on a matching parameter
- `report()`: Generates feedback for the Automate context
```python
# Example: Creating a custom action
class TransformAction(ParameterAction):
"""Action to transform parameter values based on a rule."""
def __init__(self, matcher: ParameterMatcher, transform_func) -> None:
"""Initialize with a matcher strategy and transform function."""
super().__init__()
self.matcher = matcher
self.transform_func = transform_func
def check(self, param_name: str) -> bool:
"""Check if parameter matches using the provided matcher."""
return self.matcher.matches(param_name)
def apply(self, parameter, parent_object, containing_dict, parameter_key) -> None:
"""Transform the parameter value."""
param_name = parameter.get("name", parameter_key)
object_id = getattr(parent_object, "id", None)
if "value" in parameter and isinstance(parameter["value"], str):
parameter["value"] = self.transform_func(parameter["value"])
# Track affected object and parameter
self.affected_parameters[object_id].append(param_name)
def report(self, automate_context: AutomationContext) -> None:
"""Report the transformed parameters."""
if not self.affected_parameters:
return
transformed_params = set(param for params in self.affected_parameters.values() for param in params)
message = f"Transformed {len(transformed_params)} parameters"
automate_context.attach_info_to_objects(
category="Transformed_Parameters",
object_ids=list(self.affected_parameters.keys()),
message=message,
)
```
#### Parameter Processing
The `ParameterProcessor` class orchestrates the application of actions:
1. Takes an action and a flag indicating whether to check parameter names or values
2. Processes traversal contexts by examining properties and parameters
3. Handles both modern (v3) and legacy (v2) Speckle objects
4. Applies the action to matching parameters
5. Tracks processed objects for reporting
### Adding New Sanitization Modes
To add a new sanitization mode:
1. Update the `SanitizationMode` enum in `inputs.py`:
```python
class SanitizationMode(Enum):
PREFIX_MATCHING = "Prefix Matching"
PATTERN_MATCHING = "Pattern Matching"
ANONYMIZATION = "Anonymization"
NEW_MODE = "Your New Mode" # Add your new mode here
```
2. Create any necessary new matchers or actions in `actions.py`
3. Update the `automate_function` in `function.py` to handle the new mode:
```python
if function_inputs.sanitization_mode == SanitizationMode.NEW_MODE:
# Add specific validation for your new mode
action = create_your_new_action() # Create a factory function for your action
```
## Function Flow
The main function flow is:
1. User selects a sanitization mode and provides parameters via the UI
2. Function creates the appropriate action based on the mode
3. Version data is received from Speckle
4. Traversal rules navigate through the object tree
5. Parameters are processed with the selected action
6. Results are reported back to the Automate context
7. A new sanitized version is created
## Additional Resources
- [Speckle Automate Documentation](https://automate.speckle.dev/)
- [Speckle Python SDK Documentation](https://speckle.guide/dev/python.html)
- [Pydantic Documentation](https://docs.pydantic.dev/) (for function inputs)
## Testing
### Local Testing with pytest
pytest is the recommended way to test Speckle Automate functions locally. This allows you to verify your function works correctly before deploying it.
1. Set up your test environment by creating a `.env` file with your Speckle credentials:
```
SPECKLE_TOKEN="9a110400812dc32b57e524c9c6f1a2000ebabec1c9"
SPECKLE_SERVER_URL="https://app.speckle.systems/"
SPECKLE_PROJECT_ID="d94c63b75d"
SPECKLE_AUTOMATION_ID="99896f98b6"
```
2. Run the tests with your preferred method:
```bash
# Using pytest directly
python -m pytest
# Or if using a virtual environment tool
# poetry run pytest
```
The tests in `test_function.py` provide examples of how to set up the automation context and run the function with different inputs.
### Setting Up a Test Automation
To properly test your function, you should:
1. Create a test automation in Speckle Automate
2. Use the provided IDs and token in your `.env` file
3. This allows your tests to interact with actual Speckle objects and verify the function's behavior
The `speckle-automate` package provides fixtures that help with loading these environment variables and setting up the test context automatically.
Example test setup:
```python
def test_function_run(test_automation_run_data: AutomationRunData, test_automation_token: str) -> None:
"""Run an integration test for the automate function."""
automation_context = AutomationContext.initialize(test_automation_run_data, test_automation_token)
# Run your function with test inputs
automate_sdk = run_function(
automation_context,
automate_function,
FunctionInputs(sanitization_mode=SanitizationMode.PATTERN_MATCHING, parameter_input="test_*", strict_mode=True),
)
# Verify the results
assert automate_sdk.run_status == AutomationStatus.SUCCEEDED
```
The fixtures `test_automation_run_data` and `test_automation_token` are provided by the `speckle-automate` package and automatically use the values from your `.env` file.
+80 -63
View File
@@ -1,100 +1,117 @@
# Speckle Automate function template - Python
# 🛡️ Data Shield — User Guide
This template repository is for a Speckle Automate function written in Python
using the [specklepy](https://pypi.org/project/specklepy/) SDK to interact with Speckle data.
**Data Shield** is a Speckle Automate function that helps you keep your model data clean, safe, and share-ready. Whether you're sending models to clients, collaborators, or just tidying up before archiving — Data Shields got your back.
This template contains the full scaffolding required to publish a function to the Automate environment.
It also has some sane defaults for development environment setups.
---
## Getting started
## ✨ What Data Shield Does
1. Use this template repository to create a new repository in your own / organization's profile.
Data Shield scans your Speckle model for parameters youd rather not share and takes care of them for you. It creates a fresh, sanitized version of your model while keeping the original intact.
Register the function
### Why youll love it:
- **Privacy Protection** — Say goodbye to accidentally sharing sensitive data.
- **Data Compliance** — Stay on the right side of data protection policies.
- **Confident Collaboration** — Share models without oversharing.
### Add new dependencies
---
To add new Python package dependencies to the project, use the following:
`$ poetry add pandas`
## Sanitization Modes
### Change launch variables
We know one size doesnt fit all, so Data Shield offers three modes to suit your style:
Describe how the launch.json should be edited.
### Prefix Matching
> **Best for:** Simple, predictable naming conventions.
### Github Codespaces
Remove parameters that start with a specific prefix.
> Example: Want to remove everything starting with `secret_`? Just set that prefix and Data Shield does the rest.
Create a new repo from this template, and use the create new code.
**Setup**:
- Add your prefix (like `internal_`, `private_`, or `secret_`)
- Toggle strict mode for case sensitivity (on or off — your call)
### Using this Speckle Function
---
1. [Create](https://automate.speckle.dev/) a new Speckle Automation.
1. Select your Speckle Project and Speckle Model.
1. Select the deployed Speckle Function.
1. Enter a phrase to use in the comment.
1. Click `Create Automation`.
### Pattern Matching
> **Best for:** Wildcards, regex fans, and complex patterns.
## Getting Started with Creating Your Own Speckle Function
Get fancy and use `*`, `?`, or full regular expressions.
1. [Register](https://automate.speckle.dev/) your Function with [Speckle Automate](https://automate.speckle.dev/) and select the Python template.
1. A new repository will be created in your GitHub account.
1. Make changes to your Function in `main.py`. See below for the Developer Requirements and instructions on how to test.
1. To create a new version of your Function, create a new [GitHub release](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository) in your repository.
**Examples**:
- `client_*` matches anything that starts with `client_`
- `?_internal` matches `a_internal`, `b_internal`
- `/^(secret|private)_.*$/i` matches parameters starting with `secret_` or `private_`, ignoring case
## Developer Requirements
---
1. Install the following:
- [Python 3](https://www.python.org/downloads/)
- [Poetry](https://python-poetry.org/docs/#installing-with-the-official-installer)
1. Run `poetry shell && poetry install` to install the required Python packages.
### Anonymization
> **Best for:** Keeping the structure, hiding the details.
## Building and Testing
Automatically detect email addresses inside parameter values and anonymize them.
> Example: `john.doe@example.com` becomes `j***@example.com`
The code can be tested locally by running `poetry run pytest`.
No setup needed. Just select and go.
### Building and running the Docker Container Image
---
Running and testing your code on your machine is a great way to develop your Function; the following instructions are a bit more in-depth and only required if you are having issues with your Function in GitHub Actions or on Speckle Automate.
## How to Use Data Shield
#### Building the Docker Container Image
1. **Set up your automation:**
- In your Speckle project, head to **Automations**
- Click **Add Automation** and choose **Data Shield**
- Set your trigger (like “on new commit”)
The GitHub Action packages your code into the format required by Speckle Automate. This is done by building a Docker Image, which Speckle Automate runs. You can attempt to build the Docker Image locally to test the building process.
2. **Configure your mode:**
- Choose Prefix, Pattern, or Anonymization
- Add your prefix or pattern if needed
- Toggle strict mode if you want case sensitivity
To build the Docker Container Image, you must have [Docker](https://docs.docker.com/get-docker/) installed.
3. **Run it:**
- Itll run automatically when triggered — or you can manually run on specific commits
Once you have Docker running on your local machine:
4. **Check results:**
- Sanitized models show up under the `processed/` branch
- Youll get a run report showing what got cleaned
- Highlighted changes can be seen directly in the viewer
1. Open a terminal
1. Navigate to the directory in which you cloned this repository
1. Run the following command:
::: 💡 Tips & Tricks
```bash
docker build -f ./Dockerfile -t speckle_automate_python_example .
```
- **Test first!** — Run it on a small test model before going full production.
- **Start simple.** Use prefix matching for clear conventions, pattern matching for complexity, or anonymization for safe sharing.
- **Regex pro tip:**
- Wrap your regex in `/`
- Add `i` for case-insensitive matching
- Use `^` (start) and `$` (end) for tighter control
:::
#### Running the Docker Container Image
Once the GitHub Action has built the image, it is sent to Speckle Automate. When Speckle Automate runs your Function as part of an Automation, it will run the Docker Container Image. You can test that your Docker Container Image runs correctly locally.
## 📚 Example Workflows
1. To then run the Docker Container Image, run the following command:
### → Prepping for external sharing
- Use pattern matching with `/^(internal|private|confidential)_.*$/i`
- Run before sending out models
- Share confidently!
```bash
docker run --rm speckle_automate_python_example \
python -u main.py run \
'{"projectId": "1234", "modelId": "1234", "branchName": "myBranch", "versionId": "1234", "speckleServerUrl": "https://speckle.xyz", "automationId": "1234", "automationRevisionId": "1234", "automationRunId": "1234", "functionId": "1234", "functionName": "my function", "functionLogo": "base64EncodedPng"}' \
'{}' \
yourSpeckleServerAuthenticationToken
```
### → Anonymizing client data
- Select Anonymization mode
- Run on any models with contact details
- Use sanitized versions for demos, public decks, or sales pitches
Let's explain this in more detail:
### → Stripping out project-specific baggage
- Prefix matching with something like `projectX_`
- Clean your models before turning them into templates
`docker run—-rm speckle_automate_python_example` tells Docker to run the Docker Container Image we built earlier. `speckle_automate_python_example` is the name of the Docker Container Image. The `--rm` flag tells Docker to remove the container after it has finished running, freeing up space on your machine.
---
The line `python -u main.py run` is the command run inside the Docker Container Image. The rest of the command is the arguments passed to the command. The arguments are:
## 🛠️ Troubleshooting
- `'{"projectId": "1234", "modelId": "1234", "branchName": "myBranch", "versionId": "1234", "speckleServerUrl": "https://speckle.xyz", "automationId": "1234", "automationRevisionId": "1234", "automationRunId": "1234", "functionId": "1234", "functionName": "my function", "functionLogo": "base64EncodedPng"}'` - the metadata that describes the automation and the function.
- `{}` - the input parameters for the function the Automation creator can set. Here, they are blank, but you can add your parameters to test your function.
- `yourSpeckleServerAuthenticationToken`—the authentication token for the Speckle Server that the Automation can connect to. This is required to interact with the Speckle Server, for example, to get data from the Model.
- **Not matching anything?** Double-check your pattern or prefix.
- **Case mismatch?** Try turning off strict mode.
- **Only partly sanitized?** Some complex models might need multiple passes.
- **Errors?** Check run logs in the automation report for clues.
## Resources
---
- [Learn](https://speckle.guide/dev/python.html) more about SpecklePy and interacting with Speckle from Python.
## 🤔 Still stuck?
No worries — weve got your back.
👉 Post your questions in the [Speckle Community Forum](https://speckle.community) and someone from the team (or one of our awesome community members) will help you out!
+136 -24
View File
@@ -1,5 +1,5 @@
"""Module for parameter actions and matching strategies."""
import re
from abc import ABC, abstractmethod
from collections import defaultdict
from typing import Any
@@ -7,7 +7,7 @@ from typing import Any
from speckle_automate import AutomationContext
from specklepy.objects import Base
from data_shield.helpers import PatternChecker
from data_shield.matchers import EmailMatcher, PatternChecker
class ParameterMatcher(ABC):
@@ -30,7 +30,7 @@ class PrefixMatcher(ParameterMatcher):
def matches(self, param_name: str) -> bool:
"""Check if the parameter name starts with the match value."""
if self.strict_mode:
return param_name == self.match_value
return param_name.startswith(self.match_value)
return param_name.lower().startswith(self.match_value.lower())
@@ -79,29 +79,60 @@ class RemovalAction(ParameterAction):
return self.matcher.matches(param_name)
def apply(
self,
parameter: dict[str, Any],
parent_object: Base,
containing_dict: dict[str, Any],
parameter_key: str
self, parameter: dict[str, Any], parent_object: Base, containing_dict: dict[str, Any] | Base, parameter_key: str
) -> None:
"""Remove the parameter from the containing dictionary if it matches."""
param_name = parameter.get("name", parameter_key)
"""Remove the parameter from the containing dictionary if it matches.
# Remove from the containing dictionary
containing_dict.pop(parameter_key, None)
This method handles both dictionary-style containers and Base objects with attributes.
Args:
parameter: The parameter dictionary or object
parent_object: The parent Speckle object
containing_dict: The container (dict or Base object) holding the parameter
parameter_key: The key or attribute name of the parameter
"""
param_name = parameter.get("name", parameter_key)
object_id = getattr(parent_object, "id", None)
# Handle removal based on the container type
if isinstance(containing_dict, dict):
# Standard dictionary - just pop the key
containing_dict.pop(parameter_key, None)
elif isinstance(containing_dict, Base):
# For Base objects like Revit parameters, try to remove using __dict__
try:
if hasattr(containing_dict, "__dict__") and parameter_key in containing_dict.__dict__:
containing_dict.__dict__.pop(parameter_key)
else:
# If not in __dict__, try using dynamic attribute removal
containing_dict.__dict__.pop(parameter_key, None)
except (AttributeError, KeyError, TypeError):
# Fallback to alternative methods if direct dict manipulation fails
try:
delattr(containing_dict, parameter_key)
except (AttributeError, TypeError):
try:
setattr(containing_dict, parameter_key, None)
except (AttributeError, TypeError):
# If all removal attempts fail, try one more approach specific to Speckle Base objects
if (
hasattr(containing_dict, "get_dynamic_member_names")
and parameter_key in containing_dict.get_dynamic_member_names()
):
# This is a workaround for dynamic properties in Speckle Base objects
application_name = parameter.get("applicationInternalName", parameter_key)
if application_name in containing_dict.__dict__:
containing_dict.__dict__.pop(application_name)
# Track affected object and parameter
self.affected_parameters[getattr(parent_object, "id", None)].append(param_name)
self.affected_parameters[object_id].append(param_name)
def report(self, automate_context: AutomationContext) -> None:
"""Provide feedback based on the action's results."""
if not self.affected_parameters:
return
removed_params = set(
param for params in self.affected_parameters.values() for param in params
)
removed_params = set(param for params in self.affected_parameters.values() for param in params)
message = f"The following parameters were removed: {', '.join(removed_params)}"
@@ -112,6 +143,91 @@ class RemovalAction(ParameterAction):
)
class AnonymizationAction(ParameterAction):
"""Action to anonymize email addresses in parameter values."""
def __init__(self) -> None:
"""Initialize the anonymization action with an email matcher."""
super().__init__()
self.email_matcher = EmailMatcher()
# Count of anonymized parameters for reporting
self.anonymized_count = 0
def check(self, param_value: str) -> bool:
"""Check if parameter value contains an email address.
Args:
param_value: The parameter value to check
Returns:
bool: True if the parameter value contains an email address, False otherwise
"""
return self.email_matcher.contains_email(param_value)
def apply(
self, parameter: dict[str, Any], parent_object: Base, containing_dict: dict[str, Any] | Base, parameter_key: str
) -> None:
"""Anonymize email addresses in the parameter value.
Args:
parameter: The parameter dictionary
parent_object: The parent Speckle object
containing_dict: The container (dict or Base object) holding the parameter
parameter_key: The key or attribute name of the parameter
"""
if "value" not in parameter or not isinstance(parameter["value"], str):
return
param_name = parameter.get("name", parameter_key)
original_value = parameter["value"]
object_id = getattr(parent_object, "id", None)
# Anonymize email addresses in the parameter value
anonymized_value = self.email_matcher.anonymize_email(original_value)
# Only track changes if something was actually anonymized
if anonymized_value != original_value:
# Update the parameter value in place
parameter["value"] = anonymized_value
# If we're dealing with a Base object parameter (like in Revit),
# update the actual value property of the parameter object
if isinstance(containing_dict, Base):
try:
# Try to get the parameter object using __getitem__ first (Revit v2 style)
param_obj = containing_dict.__getitem__(parameter_key)
if param_obj is not None and hasattr(param_obj, "value"):
setattr(param_obj, "value", anonymized_value)
except (AttributeError, KeyError, TypeError):
# Fallback to standard attribute access
param_obj = getattr(containing_dict, parameter_key, None)
if param_obj is not None and hasattr(param_obj, "value"):
setattr(param_obj, "value", anonymized_value)
# Track affected object and parameter
self.affected_parameters[object_id].append(param_name)
self.anonymized_count += 1
def report(self, automate_context: AutomationContext) -> None:
"""Provide feedback based on the action's results.
Args:
automate_context: The automation context
"""
if not self.affected_parameters:
return
anonymized_params = set(param for params in self.affected_parameters.values() for param in params)
message = f"Email addresses were anonymized in {len(anonymized_params)} parameters"
automate_context.attach_info_to_objects(
category="Anonymized_Parameters",
object_ids=list(self.affected_parameters.keys()),
message=message,
)
# Factory functions to create specific actions with the right matcher
def create_prefix_removal_action(forbidden_prefix: str, strict_mode: bool = False) -> RemovalAction:
"""Create a removal action that matches by prefix."""
@@ -125,11 +241,7 @@ def create_pattern_removal_action(pattern: str, strict_mode: bool = False) -> Re
return RemovalAction(matcher)
# Placeholder for future anonymization action
def create_anonymization_action() -> None:
"""Create an action that anonymizes email addresses in parameter values.
This is a placeholder for future implementation.
"""
# To be implemented
return None
# Factory function to create anonymization action
def create_anonymization_action() -> AnonymizationAction:
"""Create an action that anonymizes email addresses in parameter values."""
return AnonymizationAction()
+24 -74
View File
@@ -1,69 +1,20 @@
"""Main Automate function for parameter sanitization."""
from speckle_automate import AutomationContext
from specklepy.objects import Base
"""Updated main Automate function for parameter sanitization."""
from data_shield.actions import ParameterAction, create_pattern_removal_action, create_prefix_removal_action
from speckle_automate import AutomationContext
from data_shield.actions import (
create_anonymization_action,
create_pattern_removal_action,
create_prefix_removal_action,
)
from data_shield.helpers import ParameterProcessor
from data_shield.inputs import FunctionInputs, SanitizationMode
from data_shield.traversal import get_data_traversal_rules
class ParameterProcessor:
"""Class to handle parameter processing with a removal action."""
def __init__(self, action: ParameterAction):
"""Initialize the parameter processor with a removal action.
Args:
action: The parameter action to apply
"""
self.action = action
self.processed_objects = set()
def process_context(self, context):
"""Process a traversal context to handle parameters and properties.
Args:
context: The traversal context containing the current object
"""
current_object = context.current
# Prioritise v3
if hasattr(current_object, "properties") and current_object.properties is not None:
properties_dict = (
current_object.properties.__dict__
if isinstance(current_object.properties, Base)
else current_object.properties
)
self.process_properties_dict(properties_dict, current_object)
# Legacy placeholder for v2, ready for later
if hasattr(current_object, "parameters") and current_object.parameters is not None:
pass # Add v2 handling when ready
def process_properties_dict(self, properties_dict, current_object):
"""Recursively process v3-style properties dictionary to find and apply the action to parameters.
Args:
properties_dict: The properties dictionary to process
current_object: The current object being processed
"""
for key, value in list(properties_dict.items()): # Safe iteration during mutation
if isinstance(value, dict) and "value" in value:
param_name = value.get("name", key)
# Check if parameter matches our criteria
if self.action.check(param_name):
self.action.apply(value, current_object, properties_dict, key)
self.processed_objects.add(current_object.id)
elif isinstance(value, dict):
# Recurse into nested dictionaries
self.process_properties_dict(value, current_object)
def automate_function(
automate_context: AutomationContext,
function_inputs: FunctionInputs,
automate_context: AutomationContext,
function_inputs: FunctionInputs,
) -> None:
"""Main function for parameter sanitization.
@@ -73,37 +24,32 @@ def automate_function(
"""
# Create appropriate action based on sanitization mode
action = None
check_values = False
if function_inputs.sanitization_mode == SanitizationMode.PREFIX_MATCHING:
if not function_inputs.parameter_input:
automate_context.mark_run_failed("No parameter prefix has been set for PREFIX_MATCHING mode.")
return
action = create_prefix_removal_action(
function_inputs.parameter_input,
function_inputs.strict_mode
)
action = create_prefix_removal_action(function_inputs.parameter_input, function_inputs.strict_mode)
elif function_inputs.sanitization_mode == SanitizationMode.PATTERN_MATCHING:
if not function_inputs.parameter_input:
automate_context.mark_run_failed("No parameter pattern has been set for PATTERN_MATCHING mode.")
return
action = create_pattern_removal_action(
function_inputs.parameter_input,
function_inputs.strict_mode
)
action = create_pattern_removal_action(function_inputs.parameter_input, function_inputs.strict_mode)
elif function_inputs.sanitization_mode == SanitizationMode.ANONYMIZATION:
# Anonymization doesn't require a parameter input
# Add anonymization action here when implemented
automate_context.mark_run_failed("ANONYMIZATION mode not yet implemented.")
return
# Anonymization doesn't require a parameter input as it automatically detects emails
action = create_anonymization_action()
# For anonymization, we check values, not names
check_values = True
if not action:
automate_context.mark_run_failed("Failed to create a valid action.")
return
# Process the model with the selected action
processor = ParameterProcessor(action)
processor = ParameterProcessor(action, check_values)
version_root_object = automate_context.receive_version()
speckle_data = get_data_traversal_rules()
@@ -142,4 +88,8 @@ def automate_function(
# We can pin the result view to the specific version we created.
automate_context.set_context_view([f"{new_model_id}@{new_version_id}"], False)
automate_context.mark_run_success("Parameters processed successfully.")
automate_context.mark_run_success(
f"Parameters processed successfully with shield function "
f"{function_inputs.sanitization_mode.value}"
f"{' running in strict mode' if function_inputs.strict_mode else ''}."
)
+113 -35
View File
@@ -1,44 +1,122 @@
"""Helper classes and functions for the parameter checker."""
import fnmatch
import re
from specklepy.objects import Base
from data_shield.actions import ParameterAction
class PatternChecker:
"""Checks if a parameter name matches a user-defined pattern."""
# Modified ParameterProcessor class imported from processor_update.py
class ParameterProcessor:
"""Class to handle parameter processing with various actions."""
def __init__(self, pattern: str, strict: bool = True):
"""Initializes the pattern checker.
def __init__(self, action: ParameterAction, check_values: bool = False):
"""Initialize the parameter processor with an action.
Args:
pattern: User-defined pattern. Glob by default; /regex/ for regex; /regex/i for ignore-case.
strict: Switches case-insensitive matching for both glob and regex (unless overridden by /i in regex).
action: The parameter action to apply
check_values: If True, check parameter values instead of names
"""
self.is_regex = pattern.startswith('/') and (pattern.rstrip('i').endswith('/'))
self.user_strict = strict
self.action = action
self.check_values = check_values
self.processed_objects = set()
if self.is_regex:
# Check for inline ignore-case flag
if pattern.endswith('/i'):
self.ignore_case = True
pattern_body = pattern[1:-2]
def process_context(self, context):
"""Process a traversal context to handle parameters and properties.
Args:
context: The traversal context containing the current object
"""
current_object = context.current
# Prioritise v3
if hasattr(current_object, "properties") and current_object.properties is not None:
properties_dict = (
current_object.properties.__dict__
if isinstance(current_object.properties, Base)
else current_object.properties
)
self.process_properties_dict(properties_dict, current_object)
# Legacy placeholder for v2, ready for later
if hasattr(current_object, "parameters") and current_object.parameters is not None:
pass # Add v2 handling when ready
def process_properties_dict(self, properties_dict, current_object):
"""Recursively process v3-style properties dictionary to find and apply the action to parameters.
Args:
properties_dict: The properties dictionary to process
current_object: The current object being processed
"""
for key, value in list(properties_dict.items()): # Safe iteration during mutation
if isinstance(value, dict) and "value" in value:
param_name = value.get("name", key)
# Check based on mode (name or value)
if self.check_values:
# For value-based actions (like anonymization)
if self.action.check(value.get("value", "")):
self.action.apply(value, current_object, properties_dict, key)
self.processed_objects.add(current_object.id)
else:
# For name-based actions (like removal)
if self.action.check(param_name):
self.action.apply(value, current_object, properties_dict, key)
self.processed_objects.add(current_object.id)
elif isinstance(value, dict):
# Recurse into nested dictionaries
self.process_properties_dict(value, current_object)
def process_revit_parameters(self, current_object):
"""Process v2 Revit-style parameters to find and apply the action.
Revit parameters are stored as Base objects with speckle_type 'Objects.BuiltElements.Revit.Parameter'
and can be accessed via current_object.parameters.
Args:
current_object: The current object being processed
"""
if not hasattr(current_object, "parameters") or current_object.parameters is None:
return
parameters = current_object.parameters
# Use get_dynamic_member_names() to get all parameter keys
for parameter_key in parameters.get_dynamic_member_names():
# Get the parameter object using __getitem__
try:
param_obj = parameters.__getitem__(f"{parameter_key}")
except KeyError:
continue
# Check if it's a Revit parameter
if (
not isinstance(param_obj, Base)
or getattr(param_obj, "speckle_type", "") != "Objects.BuiltElements.Revit.Parameter"
):
continue
# For name-based checks, we need to check both the name property and applicationInternalName
name_to_check = getattr(param_obj, "name", "")
value_to_check = getattr(param_obj, "value", "")
# Create a parameter dict to pass to the action
param_dict = {
"name": name_to_check,
"value": value_to_check,
"applicationInternalName": parameter_key,
}
# Check based on mode (name or value)
if self.check_values:
# For value-based actions (like anonymization)
if isinstance(value_to_check, str) and self.action.check(value_to_check):
# Apply the action
self.action.apply(param_dict, current_object, parameters, parameter_key)
self.processed_objects.add(current_object.id)
else:
self.ignore_case = not strict # fallback to global strict setting if no /i flag
pattern_body = pattern[1:-1]
flags = re.IGNORECASE if self.ignore_case else 0
self.regex = re.compile(pattern_body, flags)
self.pattern = pattern_body
else:
self.regex = None
self.pattern = pattern
self.ignore_case = not strict
def check(self, param_name: str) -> bool:
"""Checks if the parameter name matches the user-defined pattern."""
if self.is_regex:
return self.regex.search(param_name) is not None
# For glob: emulate strict or non-strict
if self.ignore_case:
return fnmatch.fnmatch(param_name.lower(), self.pattern.lower())
else:
return fnmatch.fnmatchcase(param_name, self.pattern)
# For name-based actions (like removal)
if self.action.check(name_to_check):
# Apply the action
self.action.apply(param_dict, current_object, parameters, parameter_key)
self.processed_objects.add(current_object.id)
+1
View File
@@ -51,6 +51,7 @@ class FunctionInputs(AutomateBase):
)
strict_mode: bool = Field(
title="Case Sensitivity Strict Mode",
default=False,
description="If checked, matching is case-sensitive. If unchecked, case-insensitive."
)
+146
View File
@@ -0,0 +1,146 @@
"""Module for parameter matching strategies and pattern checking."""
import fnmatch
import re
from abc import ABC, abstractmethod
from re import Pattern
class ParameterMatcher(ABC):
"""Strategy interface for parameter matching logic."""
def __init__(self, match_value: str, strict_mode: bool = False):
"""Initialize with a value to match against and a strict mode flag."""
self.match_value = match_value
self.strict_mode = strict_mode
@abstractmethod
def matches(self, param_name: str) -> bool:
"""Check if parameter name matches according to this strategy."""
pass
class PrefixMatcher(ParameterMatcher):
"""Matches parameters by prefix."""
def matches(self, param_name: str) -> bool:
"""Check if the parameter name starts with the match value."""
if self.strict_mode:
return param_name.startswith(self.match_value)
return param_name.lower().startswith(self.match_value.lower())
class PatternMatcher(ParameterMatcher):
"""Matches parameters by regex pattern."""
def matches(self, param_name: str) -> bool:
"""Check if the parameter name matches the regex pattern."""
pattern_checker = PatternChecker(self.match_value, self.strict_mode)
return pattern_checker.check(param_name)
class PatternChecker:
"""Checks if a parameter name matches a user-defined pattern."""
def __init__(self, pattern: str, strict: bool = True):
"""Initializes the pattern checker.
Args:
pattern: User-defined pattern. Glob by default; /regex/ for regex; /regex/i for ignore-case.
strict: Switches case-insensitive matching for both glob and regex (unless overridden by /i in regex).
"""
self.is_regex = pattern.startswith("/") and (pattern.rstrip("i").endswith("/"))
self.user_strict = strict
if self.is_regex:
# Check for inline ignore-case flag
if pattern.endswith("/i"):
self.ignore_case = True
pattern_body = pattern[1:-2]
else:
self.ignore_case = not strict # fallback to global strict setting if no /i flag
pattern_body = pattern[1:-1]
flags = re.IGNORECASE if self.ignore_case else 0
self.regex = re.compile(pattern_body, flags)
self.pattern = pattern_body
else:
self.regex = None
self.pattern = pattern
self.ignore_case = not strict
def check(self, param_name: str) -> bool:
"""Checks if the parameter name matches the user-defined pattern."""
if self.is_regex:
return self.regex.search(param_name) is not None
# For glob: emulate strict or non-strict
if self.ignore_case:
return fnmatch.fnmatch(param_name.lower(), self.pattern.lower())
else:
return fnmatch.fnmatchcase(param_name, self.pattern)
class EmailMatcher:
"""Class for identifying and anonymizing email addresses in parameter values."""
# Email regex pattern - basic pattern to identify email addresses
EMAIL_PATTERN = r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"
def __init__(self):
"""Initialize with a compiled regex pattern for email matching."""
self.pattern: Pattern = re.compile(self.EMAIL_PATTERN)
def contains_email(self, value: str) -> bool:
"""Check if a string contains an email address.
Args:
value: The string to check for email addresses
Returns:
bool: True if the string contains an email address, False otherwise
"""
if not isinstance(value, str):
return False
return bool(self.pattern.search(value))
def anonymize_email(self, value: str) -> str:
"""Anonymize email addresses in a string.
The function replaces the local part of each email address with the
first character followed by asterisks, preserving the domain part.
Example: "email@example.com" becomes "e****@example.com"
Args:
value: The string containing email addresses to anonymize
Returns:
str: The string with anonymized email addresses
"""
if not isinstance(value, str):
return value
def replace_email(match_obj):
"""Replace function for regex sub to anonymize matched emails."""
email = match_obj.group(0)
# Split the email into local part and domain part
local, domain = email.split("@", 1)
# Anonymize the local part: keep first and last character, replace rest with asterisks
if len(local) > 2:
# For longer local parts, keep first and last characters
anonymized_local = local[0] + "*" * (len(local) - 2) + local[-1]
elif len(local) == 2:
# For 2-character local parts, show first character and one asterisk
anonymized_local = local[0] + "*"
else:
# For 1-character local parts, just use an asterisk
anonymized_local = "*"
# Return the anonymized email
return f"{anonymized_local}@{domain}"
# Replace all email addresses in the string
return self.pattern.sub(replace_email, value)
-12
View File
@@ -1,12 +0,0 @@
# rule_processor.py
def process_rules(speckle_objects: list[Base], rules: list[dict], action_handler) -> dict:
"""Process rules against objects and apply actions."""
results = {}
for obj in speckle_objects:
for rule in rules:
if evaluate_rule(obj, rule):
action_handler.apply_action(obj, rule)
results[obj.id] = rule["action_type"]
return results
-70
View File
@@ -1,70 +0,0 @@
"""A collection of rules for processing parameters in Speckle objects."""
from collections.abc import Callable
from specklepy.objects import Base
# We're going to define a set of rules that will allow us to filter and
# process parameters in our Speckle objects. These rules will be encapsulated
# in a class called `ParameterRules`.
class ParameterRules:
"""A collection of rules for processing parameters in Speckle objects.
This class provides static methods that return lambda functions. These
lambda functions serve as filters or conditions we can use in our main
processing logic. By encapsulating these rules, we can easily extend
or modify them in the future.
"""
@staticmethod
def speckle_type_rule(desired_type: str) -> Callable[[Base], bool]:
"""Rule: Check if a parameter's speckle_type matches the desired type."""
return (
lambda parameter: getattr(parameter, "speckle_type", None) == desired_type
)
@staticmethod
def forbidden_prefix_rule(given_prefix: str) -> Callable[[Base], bool]:
"""Rule: check if a parameter's name starts with a given prefix.
This is a simple check, but there could be more complex naming rules for parameters of
different types. For example, a rule that checks if a parameter's name starts with a given string
exists particularly within IFC where parameters are often prefixed with "Ifc" or "PSet".
"""
return lambda parameter: parameter.name.startswith(given_prefix)
# This example Automate function is for prefixed parameter removal. Additional example rules below follow the same
# pattern, but with different logic. In some instances there is a strong coupling between the action and the
# checking logic, and in others there is a looser coupling. Which is why I have defined the actions separately from
# the checking logic.
@staticmethod
def has_missing_value(parameter: dict[str, str]) -> bool:
"""Rule: Missing Value Check.
The AEC industry often requires all parameters to have meaningful values.
This rule checks if a parameter is missing its value, potentially indicating
an oversight during data entry or transfer.
"""
return not parameter.get("value")
@staticmethod
def has_default_value(parameter: dict[str, str]) -> bool:
"""Rule: Default Value Check.
Default values can sometimes creep into final datasets due to software defaults.
This rule identifies parameters that still have their default values, helping
to highlight areas where real, meaningful values need to be provided.
"""
return parameter.get("value") == "Default"
@staticmethod
def parameter_exists(parameter_name: str, parent_object: dict[str, str]) -> bool:
"""Rule: Parameter Existence Check.
For certain critical parameters, their mere presence (or lack thereof) is vital.
This rule verifies if a specific parameter exists within an object, allowing
teams to ensure that key data points are always present.
"""
return parameter_name in parent_object.get("parameters", {})
+5 -6
View File
@@ -1,9 +1,10 @@
"""This module defines a function that generates traversal rules for navigating."""
from specklepy.objects.graph_traversal.traversal import GraphTraversal, TraversalRule
def get_data_traversal_rules() -> GraphTraversal:
"""
Generates traversal rules for navigating Speckle data structures.
"""Generates traversal rules for navigating Speckle data structures.
This function defines and returns traversal rules tailored for Speckle data.
These rules are used to navigate and extract specific data properties
@@ -33,9 +34,7 @@ def get_data_traversal_rules() -> GraphTraversal:
display_value_rule = TraversalRule(
[
lambda o: any(
getattr(o, alias, None) for alias in display_value_property_aliases
),
lambda o: any(getattr(o, alias, None) for alias in display_value_property_aliases),
lambda o: "Geometry" in o.speckle_type,
],
lambda o: elements_property_aliases,
@@ -46,4 +45,4 @@ def get_data_traversal_rules() -> GraphTraversal:
lambda o: o.get_member_names(),
)
return GraphTraversal([display_value_rule, default_rule])
return GraphTraversal([display_value_rule, default_rule])
+2 -2
View File
@@ -20,8 +20,8 @@ class TestFunction:
automation_context,
automate_function,
FunctionInputs(
sanitization_mode=SanitizationMode.PATTERN_MATCHING,
parameter_input="/.*?peckl.*/i",
sanitization_mode=SanitizationMode.ANONYMIZATION,
parameter_input="",
strict_mode=True,
),
)