Building a 3D System Inside a Node-Based Prototyping Tool

Building a 3D System Inside a Node-Based Prototyping Tool

Building RealityKit-powered 3D nodes for a visual prototyping tool, and the container entity pattern that made transforms predictable.

What is Stitch?

Stitch is an open-source visual prototyping tool in the spirit of Origami Studio and Quartz Composer. Instead of writing code, designers and developers wire together nodes in a graph to build interactive prototypes. One big area I focused on was figuring out how to bring 3D content into this node-based world – building everything from the first layer node for displaying 3D assets, to a complete RealityKit-based suite of nodes with lighting, physics forces, geometry primitives, and gesture-driven manipulation. Active development on Stitch has since wound down; the code remains open source.

The Foundation: Model3D Layer Node

Every 3D system starts with getting a single 3D object on screen. In Stitch, that meant I needed to create the Model3DLayerNode –a node that graph authors could drop into their prototype to import and display .usdz models.

My initial implementation was straightforward: load a USDZ, create a ModelEntity, render it in a RealityView that displayed the content and afforded ports so a user could maniuplate the USDZ asset.

I exposed a full suite of ports on the Model3DLayerNode: entity anchoring, 3D transforms, animation toggles, interaction flags, and layer effects. This meant a graph author could drag in a .usdz file, wire up transform controls, and immediately start manipulating 3D content without writing a line of code.

The Hidden Root Transform Problem

One issue that we encountered early on was that RealityKit quietly injects a hidden root transform whenever it imports a .usdz or other 3D asset whose units don’t match RealityKit’s meters-based coordinate space. Modeling apps typically work in centimeters and don’t bother translating all vertices into the target coordinate system – they just apply a correction at the root. So a USDZ file might arrive with a quaternion like (0.7, 0.7, 0.7, ...) at the root, representing a 90-degree rotation on the X axis, plus a scale factor converting centimeters to meters.

The result: any transform the user applied from a Stitch graph was multiplied against this unknown scale/rotation. Dragging a model a few centimeters could suddenly launch it across the scene; models would jump, resize, or rotate unexpectedly because I was fighting the engine’s automatic corrections without knowing they existed

Container Entity Pattern

The Solution

The solution for this ended up being to wrap an entity inside of another entity that acted as a container. The container’s transform would be manipulated, leaving the actual entity asset alone.

When a USDZ asset gets imported, an empty Entity() is created as a containerEntity. The real asset gets attached to the containerEntity as a child. The container entity is ultimately what gets manipulated by Stitch’s interaction patches. The RealityKit-generated root transform stays untouched inside the child.

Cascading Benefits

This pattern had effects far beyond the original bug:

• Collision shapes are generated on the container entity (updateCollisionBounds), so gestures, hit testing, and anchor placement all act on the same transform hierarchy the user sees
• Animation playback still runs on the inner entity, preserving timing – no more “animation resets scale to 0.01” bugs during edits
• Non-AR contexts benefit too: any RealityView or 3D layer gets a predictable container, making this foundational for interaction nodes beyond the AR stack

The container entity pattern turned out to be the single highest-leverage architectural decision in the entire 3D system. It solved transform unpredictability, enabled consistent gestures, preserved animations, and worked across AR and non-AR contexts alike.

Escaping the Matrix

It became a goal to eliminate the burden of having to work with or reason about matrcies from the user and instead present them with a human-friendly Transform abstraction.

Why Matrices Weren’t Working

My early prototypes relied on legacy matrix pack/unpack nodes inherited from the Quartz Composer era. The StitchMatrix type was a simd_float4x4 – a 4x4 grid of floating-point numbers. This made it effectively impossible to:

• Reason about position, rotation, and scale independently
• Bind device motion data directly into RealityKit patches
• Give users intuitive controls (a “rotation knob” that behaves like a rotation knob)

The data type was opaque. Almost none of the RealityKit utilities could digest it. Debugging sessions where “my model rotated 90 degrees when I expected 15” were routine.

Quaternions and Euler Angles

In order to create a more intuitive representation for how a user would actually want to interact with 3D interaction prototypes, I needed to go down a bit of a linear algebra refresh; relearning about Quaternions, Euler Angles, and Gimbal lock:

• Quaternions: XYZ components are imaginary, representing an axis/direction in space (when normalized) multiplied by the sine of half the rotation angle. W is the cosine of half the rotation angle. The null rotation (identity) is (0, 0, 0, 1). You can’t manipulate quaternion components individually without re-normalizing.
• Euler Angles: Human-readable but ambiguous. Composing two rotations can yield different sets of Euler angles – the answer might surprise you, but it’s not incorrect.
• Gimbal lock: The degenerate case where two rotation axes align, collapsing a degree of freedom. A threat whenever you store or extract Euler angles.

The Transform Pack/Unpack Innovation

The breakthrough was the TransformPackNode: nine discrete numeric inputs for position (X/Y/Z), scale (X/Y/Z), and rotation (X/Y/Z) that emit a single .transform value. This replaced the old matrix pack approach with something designers could actually reason about.

The complementary MatrixUnpackNode breaks transforms back into editable components, giving users both high-level and granular control over 3D objects.

Degree-Based Rotation

For a better user experience, we decided to make it so that rotation inputs were of a Degree type, not Radians. The simd_float4x4.init(rotationZYX:) initializer converts UI-provided degree values to radians internally and rebuilds the rotation matrix. Turning a rotation knob by 15 degrees would spin RealityKit content by 15 degrees – no more guessing whether the control was radians, reverse-radians, or gimbal-locked.

Matching the unit system to human intuition eliminated an entire class of “why does my rotation look wrong?” bugs

Reflections

The result is a system where a designer can wire together nodes to import 3D models, manipulate them with gestures, light them, and connect it all to the rest of their prototype — without writing code.

Looking back, almost every hard-won fix was the same move: insert a layer the user can reason about between them and the engine’s opaque internals. The container entity hid the injected root transform. The Transform pack/unpack hid the raw matrix. Degrees hid radians. None of these changed what RealityKit could do — they changed whether a designer could predict what it would do. Doing that inside a tool designers already used for UI prototyping is what made it new: 3D became just another layer in the graph;, not a separate world bolted onto the side.

This is one of four posts about my work at Stitch. See also: Bringing AR to Stitch, Computer Vision Nodes in Stitch, and Building StitchAI.