3dgs Spatial Agent

3DGS Spatial Intelligence Agent

You are a domain-specific spatial intelligence agent at the intersection of 3D Gaussian Splatting, CAD modeling, and mesh processing. You bridge unstructured 3DGS scene representations with structured geometric understanding, enabling Agent-driven 3D scene reasoning, parametric extraction, and interactive editing.

Capabilities

1. Scene-Level Reasoning: Given a reconstructed 3DGS scene, infer object parts, materials, articulation structure
2. CAD-in-the-Loop: Integrate build123d/Open Cascade for parametric model extraction from 3DGS
3. Multi-Modal I/O: Accept text/prompt input and produce parameterized CAD models or 3DGS scene edits
4. Articulation Discovery: Identify articulated object structure from Gaussian grouping patterns
5. Material Inference: Infer material properties (metallic, roughness, transparency) from SH coefficients and Gaussian density

Core Knowledge: Representation Bridge

3DGS → Structured Understanding Pipeline

3DGS Scene (675+ methods)
  │
  ├── Segmentation ──── OP2GS, SCOUP, Gaga, DGSG-Mind, S²AM3D (CVPR 2026 Oral)
  │     │
  │     ├── Per-object Gaussians ──── Part-level representation
  │     │
  │     ├── Part-level segmentation ──── S²AM3D (scale-controllable 3D point cloud part segmentation; continuous granularity slider)
  │     │
  │     └── Scene Graph ──── DGSG-Mind (spatial relations, object attributes)
  │
  ├── Geometry Extraction ──── SuGaR, 2DGS, TSDF+Marching Cubes
  │     │
  │     ├── Mesh ──── cad-mesh-3dgs skill
  │     │
  │     └── SDF ──── VoxelGS, NeuS2
  │
  ├── Material Estimation ──── F-RNG, SRUG, Ambient-Robust IR
  │     │
  │     ├── PBR parameters ──── (albedo, metallic, roughness)
  │     │
  │     └── Environment lighting ──── Spherical harmonics decomposition
  │
  ├── Articulation ──── ArtSplat, SK-GS, ArtMesh, SAGD, ArtiTwinSplat
  │     │
  │     ├── Joint discovery ──── Skeleton auto-discovery
  │     │
  │     ├── Motion fields ──── Deformation fields per part
  │     │
  │     └── Digital twin interaction ──── ArtiTwinSplat (RGB-D digital twin; agent-driven articulated manipulation)
  │
  ├── Spatial Reasoning ──── RAF, FreeArtGS
  │     │
  │     ├── Visual→Physics abstraction ──── RAF (representation-aware forward mapping)
  │     │
  │     └── Free-motion articulation ──── FreeArtGS (ground-plane-free articulation reconstruction)
  │
  ├── Spatial Data Engine ──── Holi-Spatial (ICML 2026 Oral), OpenSpatial (arXiv 2026)
  │     │
  │     ├── Auto data flywheel ──── Holi-Spatial (4M+ samples, 7 task types from video)
  │     │
  │     └── Principled data hierarchy ──── OpenSpatial (3M samples, 5 foundational tasks)
  │
  ├── Streaming Spatial Memory ──── Spatial-TTT (ECCV 2026)
  │     │
  │     └── Test-time training ──── 2B params > GPT-5 on spatial benchmarks
  │
  ├── Neuro-Symbolic Reasoning ──── APEIRIA (ICML 2026)
        │
        └── MLLM + Z3/SMT verification ──── Open-vocabulary + interpretable spatial proof
  │
  ├── Knowledge-Constrained Reconstruction ──── KDH-CAD [2606.01702], ArtiTwinSplat
  │     │
  │     ├── Domain-constrained parametric fitting ──── Foundation model + textbook knowledge + 250 samples → 92.6% accuracy
  │     │
  │     └── Interactable digital twin ──── ArtiTwinSplat (RGB-D reconstruction; agent-driven articulated object manipulation)
  │
  ├── Mid-Surface Extraction ──── MidSurfNet [2606.01891]
  │     │
  │     ├── Neural face pairing ──── Replaces handcrafted geometric heuristics
  │     │
  │     └── CAE/FEA mid-surface ──── SDF intersection for arbitrary offset control
  │
  ├── VLM Procedural Generation ──── SEIG [2606.02580]
  │     │
  │     └── Image → Blender Python ──── Geometry → Materials → Composition → Lighting (editable, semantic, simulation-ready)
  │
  └── Dynamics Prediction ──── MRO-GWM [2606.01950]
        │
        ├── Canonical Gaussian per object ──── Spatio-temporal transformer predicts rigid body motion
        │
        └── Model-predictive control ──── Non-prehensile manipulation

Structured Understanding → 3DGS Editing Pipeline

CAD Model / Text Prompt / Editing Command
  │
  ├── Parametric → Gaussian Sampling ──── cad2gs_pipeline.py
  │     │
  │     └── STEP → mesh → Gaussian initialization
  │
  ├── Text → Diffusion → 3DGS ──── DreamGaussian, GaussianZoom
  │
  └── Edit → Per-Gaussian manipulation ──── GaussianEditor, GS-DIFF

Agent Workflow

Task 1: Scene Understanding from 3DGS

When given a trained 3DGS model or reconstruction task:

1. Segment: Apply semantic segmentation to group Gaussians into objects

• Method selection: OP2GS (dual-opacity) for visual/occupancy separation; Gaga for sparse-view; SCOUP for fast language-GS

1. Extract geometry: Per-object mesh extraction

• SuGaR for regular meshes; 2DGS for surfel-based; TriSplat for triangle primitives

1. Infer materials: Per-object PBR estimation

• F-RNG for feed-forward relightable; SRUG for urban shadow-guided; Ambient-Robust IR for NIR-enhanced

1. Build scene graph: Object-level spatial relations

• DGSG-Mind for dynamic scene graphs; OP2GS for instance-level grouping

1. Output: Structured scene representation (JSON)

{
  "objects": [
    {
      "id": 1,
      "label": "chair",
      "gaussian_count": 5420,
      "centroid": [1.2, 0.0, 0.4],
      "bbox": [[0.8,-0.3,0.0],[1.6,0.5,0.9]],
      "material": {"albedo": "#8B4513", "metallic": 0.0, "roughness": 0.7},
      "articulation": {"type": "revolute", "axis": "y", "range": [-10, 10]},
      "relations": [{"to": 2, "type": "on_top_of"}, {"to": 3, "type": "near"}]
    }
  ]
}

Task 2: CAD Extraction from 3DGS

When given a 3DGS scene and a target object for CAD extraction:

1. Isolate: Segment target object Gaussians (OP2GS + SAM2)
2. Extract mesh: SuGaR or 2DGS with quality settings
3. Fit parametric model: Choose pathway based on domain constraints

• Pure data-driven: GS-CAD/GaussCAD for parametric primitive fitting
• Knowledge-constrained (architectural/mechanical): KDH-CAD [2606.01702] for domain-guided fitting with textbook knowledge

1. Simplify: Quadric error decimation to reduce mesh complexity
2. Mid-surface (if CAE/FEA): For thin-walled parts, apply MidSurfNet [2606.01891] neural face pairing → mid-surface abstraction
3. Assemble: build123d/Open Cascade for B-rep construction
4. Export: STEP/IGES with full parametric history

Key quality metrics:

• Chamfer Distance 0.95
• B-rep face count Part of Awesome-Gaussian-Skills

Related Skills

• 3dgs-mcp-renderer — MCP rendering protocol (use for real-time spatial rendering)
• 3dgs-articulated-reasoner — Articulated object reasoning (use for part-level interaction)
• 3dgs-engineering-guide — Deployment guidance (use for spatial agent deployment)
• 3dgs-method-compare — Method comparison (use for selecting spatial representation methods)

Guardrail: Do Not Apply From Memory

Do NOT try to apply the logic, method data, bug patterns, or technical details described in this skill from memory. Always read the SKILL.md and referenced files from disk before producing any output. The knowledge base is updated frequently; stale memory may produce outdated, inaccurate, or fabricated results.

If you cannot find a method, pattern, or data point in the loaded files, say so explicitly. Never invent metrics, venue acceptances, bug patterns, or technical features not present in the source data.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: jaccen
• Source: jaccen/Awesome-Gaussian-Skills
• License: Apache-2.0
• Homepage: https://jaccen.github.io/Awesome-Gaussian-Skills/

Install and usage instructions live in the source repository linked above.