Gergő Jedlicska
2e86a723c6
* feat(fileimport-service): add next gen file importer * feat(fileimports): integrate server and fileimporter * chore(dui3): remove leftover artifacts * fix(server): test typing fixes * fix(fileimports): test and pr comment fixes * feat(fileimports: moare test fixes * fix(fileimports): tests and yarn dedupe
File Import Service
Description of how this works
A micro-service which polls a Postgres database table file_uploads for new records and processes them.
It retrieves a referenced file from an S3 bucket and stores it in a local directory for parsing.
The File Import service can parse either STL, OBJ, or IFC files using external packages, written in either .Net or Python (note, there is a legacy IFC parser written in Node.js). These external packages are controlled via shell commands.
The parsers are responsible for extracting the necessary data from the files and storing it in the database. They are also responsible for creating a new Speckle model if necessary.
The service is then responsible for updating the status of the file_uploads table, and for posting a Postgres notification.
Dev setup
Building/Running the .NET importer
Requirements:
- Ubuntu 24+
Do this on Ubuntu/OSX to install dotnet:
# Add microsoft package repo
sudo apt update && sudo apt install -y wget apt-transport-https
wget https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
# Install dotnet sdk 8
sudo apt update
sudo apt install -y dotnet-sdk-8.0
# Verify version
dotnet --version
Do this to build:
cd ./packages/fileimport-service/src/ifc-dotnet
dotnet publish ifc-converter.csproj -c Release -o output/