Tool Registry

Tool types, interfaces, and how they connect.

Overview

Tools are the core building blocks of Sigilweaver Loom. Each tool:

• Has a type (e.g., "Filter", "Join")
• Has sockets (input/output connection points)
• Has a config (settings specific to that tool)
• Is implemented in both Studio (UI) and Server (execution)

Architecture

Studio                                Server
─────────────────────────────────────────────────
TOOL_DEFINITIONS     ──────────────►  ToolRegistry
(data/tools.ts)                       (domain/tools/registry.py)
                                           │
ToolDefinition       ──────────────►  BaseTool
(UI/layout/sockets)                   (execute/schema/validate)
                                           │
ConfigPanel          ──────────────►  Implementation
(component)                           (filter.py, join.py, etc.)

---

Studio: ToolDefinition

Every tool needs a Studio definition in studio/src/data/tools.ts:

interface ToolDefinition {
  type: string;           // Must match Server tool type
  name: string;           // Display name in palette
  icon: LucideIcon;       // Lucide React icon component
  category: ToolCategory; // "inout" | "preparation" | "join" | "aggregate"
  defaultConfig: ToolConfig;
  sockets: Socket[];
  docsUrl?: string;       // Link to documentation page
}

Socket Definition

interface Socket {
  id: string;          // Unique within tool (e.g., "input", "output-true")
  direction: "input" | "output";
  type?: string;       // Optional label (e.g., "L", "R", "T", "F")
  multiple?: boolean;  // Allow multiple connections (Union tool)
}

Example: Filter Tool

{
  type: 'Filter',
  name: 'Filter',
  icon: Filter,
  category: 'preparation',
  defaultConfig: {
    expression: '',
    mode: 'Basic',
    conditions: [],
    combinator: 'and',
  },
  sockets: [
    { id: 'input', direction: 'input' },
    { id: 'output-true', direction: 'output', type: 'T' },
    { id: 'output-false', direction: 'output', type: 'F' }
  ],
  docsUrl: 'https://sigilweaver.app/docs/user/tools/preparation/filter',
}

The Filter has:

• One input socket
• Two output sockets: rows that match (T) and rows that don't (F)
• Config for the filter expression and mode

---

Backend: BaseTool

All tools inherit from BaseTool:

from abc import ABC, abstractmethod
import polars as pl

from app.api.models.workflow import DEFAULT_ENGINE

class BaseTool(ABC):
    engine: str = DEFAULT_ENGINE  # "core/polars"

    @abstractmethod
    async def execute(
        self, 
        config: dict[str, Any], 
        inputs: dict[str, pl.LazyFrame | list[pl.LazyFrame]]
    ) -> dict[str, pl.LazyFrame]:
        """Execute tool logic. Keep it lazy!"""
        pass

    @abstractmethod
    async def get_output_schema(
        self, 
        config: dict[str, Any], 
        input_schemas: dict[str, DataSchema | list[DataSchema]]
    ) -> dict[str, DataSchema]:
        """Get output schema without execution."""
        pass

    async def validate_config(self, config: dict[str, Any]) -> list[str]:
        """Return list of validation errors."""
        return []

    def to_python_code(
        self,
        tool_id: str,
        config: dict[str, Any],
        input_vars: dict[str, str],
    ) -> tuple[list[str], dict[str, str]]:
        """Generate Python code for this tool."""
        pass

Key Principles

1. Lazy Execution: Use LazyFrame throughout. Never call .collect() unless absolutely necessary.
2. Schema Inference: get_output_schema must work without executing upstream tools.
3. Validation: Catch config errors before execution.
4. Code Generation: Support exporting workflows to standalone Python.

---

Backend: Registry Pattern

Tools register themselves via decorator:

from app.domain.tools.registry import register_tool
from app.domain.tools.base import BaseTool

@register_tool("Filter")
class FilterTool(BaseTool):
    async def execute(self, config, inputs):
        # Implementation
        pass

The registry is populated when app.domain.tools.register is imported (which happens at app startup).

Getting a Tool

from app.domain.tools.registry import ToolRegistry

# Simple lookup (falls back to any engine)
tool = ToolRegistry.get("Filter")

# Engine-qualified lookup
tool = ToolRegistry.get("Filter", engine="core/polars")

result = await tool.execute(config, inputs)

---

Engine System

Tools declare which execution engine they belong to via the engine class attribute. The default engine is "core/polars".

Engine Resolution

When the executor runs a tool, it resolves the engine using this precedence chain:

tool.engine  >  meta.defaultEngine  >  DEFAULT_ENGINE ("core/polars")

• tool.engine: Per-tool override in the workflow JSON (optional)
• meta.defaultEngine: Workflow-level default (optional, defaults to "core/polars")
• DEFAULT_ENGINE: Hardcoded fallback constant

Engine IDs

Engine identifiers use a namespaced format:

Engine	ID
Polars (built-in)	core/polars
Future engines	core/pandas, contrib/dbt, etc.

Workflow Schema Fields

The .swwf workflow format includes these engine-related fields:

{
  "version": "1.0",
  "meta": {
    "name": "My Workflow",
    "defaultEngine": "core/polars"
  },
  "tools": [
    {
      "id": "abc",
      "type": "Filter",
      "engine": "core/polars",
      "config": { ... },
      "sockets": [ ... ]
    }
  ],
  "wires": [ ... ],
  "engines": {
    "core/polars": {},
    "contrib/dbt": { "project": "analytics", "profile": "dev" }
  }
}

• meta.defaultEngine: Optional. Defaults to "core/polars" if omitted.
• tool.engine: Optional. Omitted when it matches the workflow default (to reduce noise).
• engines: Optional block for engine-specific configuration (connection details, project paths, etc.).

---

Available Tools

In/Out

Type	Description	Sockets
Input	Load data from file	output
Output	Export data to file	input

Preparation

Type	Description	Sockets
Filter	Filter rows by expression	input → output-true, output-false
Select	Select/rename/reorder columns	input → output
Sort	Sort rows by columns	input → output
Formula	Add calculated column	input → output
Sample	Sample subset of rows	input → output
Unique	Deduplicate rows	input → output-unique, output-duplicate

Join

Type	Description	Sockets
Union	Combine multiple tables vertically	input (multiple) → output
Join	Join two tables on keys	input-left, input-right → output-left, output-match, output-right

Aggregate

Type	Description	Sockets
Summarize	Group by and aggregate	input → output
Pivot	Reshape data (rows to columns)	input → output
Unpivot	Reshape data (columns to rows)	input → output

---

Config Schemas

Each tool has its own config shape. Here are the key ones:

Input

{
  "source": "/path/to/file.csv",
  "delimiter": ",",
  "hasHeader": true,
  "encoding": "utf-8"
}

Filter

{
  "expression": "pl.col('value') > 100",
  "mode": "Basic",
  "conditions": [],
  "combinator": "and"
}

The mode can be "Basic" (UI-driven conditions) or "Custom" (raw Polars expression). In Basic mode, conditions and combinator drive the filter. In Custom mode, expression is raw Polars syntax evaluated on the backend.

Select

{
  "columnConfigs": [
    { "name": "id", "enabled": true, "alias": null },
    { "name": "value", "enabled": true, "alias": "amount" }
  ]
}

Sort

{
  "sortColumns": [
    { "column": "created_at", "ascending": false },
    { "column": "id", "ascending": true }
  ]
}

Formula

{
  "column_name": "total",
  "expression": "pl.col('price') * pl.col('quantity')"
}

Union

{
  "mode": "ByName"
}

Modes: ByName (match by column name), ByPosition (match by column order).

Join

{
  "joinType": "inner",
  "leftKeys": ["id"],
  "rightKeys": ["user_id"]
}

Join types: inner, left, right, outer, cross.

Summarize

{
  "groupByColumns": ["category"],
  "aggregations": [
    { "column": "value", "operation": "Sum", "outputName": "total_value" },
    { "column": "id", "operation": "Count", "outputName": "count" }
  ],
  "includeNulls": false
}

Operations: Sum, Mean, Min, Max, Count, First, Last, StdDev, Var.

Pivot

{
  "on": "category",
  "index": ["date"],
  "values": "amount",
  "aggregateFunction": "first"
}

Aggregate functions: first, sum, mean, min, max, count, last, len.

Unpivot

{
  "on": ["col_a", "col_b"],
  "index": ["id"],
  "variableName": "variable",
  "valueName": "value"
}

Reshapes wide data to long format: selected columns become rows.

Sample

{
  "mode": "FirstN",
  "n": 100,
  "percent": 10.0,
  "offset": 0,
  "seed": null
}

Modes: FirstN (first N rows), LastN (last N rows), Random (random sample by count or percent).

Unique

{
  "columns": ["id", "name"],
  "mode": "First"
}

Modes: First (keep first occurrence), Last (keep last occurrence), None (drop all duplicates).

---

Adding a New Tool

See Adding Tools for a complete guide. The key steps:

1. Add ToolDefinition to studio/src/data/tools.ts
2. Create config panel component in studio/src/components/config/
3. Create implementation in server/app/domain/tools/implementations/
4. Register with @register_tool("YourTool") decorator
5. Import in server/app/domain/tools/register.py
6. Add tests

---

Type Matching

Socket types must match when connecting:

• Standard output → Standard input: Always allowed
• Typed socket → Same type: Allowed (e.g., "L" → "L")
• Typed socket → Different type: Prevented by UI

The Studio enforces type compatibility during drag-and-drop connections.

---

Next: Roadmap for project direction and priorities.