Building StitchAI: From Prompt to Graph

Attempting to build a Copilot-like experience that turns natural language prompts into node graphs

What is Stitch?

Stitch is an open-source visual prototyping tool in the spirit of Origami Studio and Quartz Composer. Instead of writing code, designers and developers wire together nodes in a graph to build interactive prototypes. For example, a draggable card might be made from a visual layer node, a drag interaction node, math nodes that calculate position, and connections between their inputs and outputs.

The goal of adding AI to Stitch was to make a Copilot-like experience where a user could describe the graph they wanted, and the AI would generate it. Instead of manually searching for nodes, setting values, and drawing connections, the model would generate the same graph-building events a person would have produced in the app.

The Stitch code remains open source.

The First Bet: Graph-Building Events

The first step was getting a flow working where a user would enter a prompt, the LLM returns structured graph actions, and Stitch applies those actions to the canvas.

The important part was a grammar for how we could describe the creation of a graph with a small set of primitives. That vocabulary consisted of five step types:

• add_node – create a patch or layer node
• set_input – write a typed value into a node input
• connect_nodes – connect one node’s output to another node’s input
• change_value_type – change the value type for a patch node input
• sidebar_group_created – put layer nodes into a sidebar group

The first demo was a small Python script from September 2024. It called gpt-4o-2024-08-06 through OpenAI’s then-new structured outputs API, asked for those five graph steps, and printed JSON that we could paste into the app and create a graph.

We moved that prompt flow into the app itself so we could get a feel for what the prompting —> generation experience actually felt like:

Fall 2024 to January 2025: A Tiny Language for Graphs

We experimented with enhancing the system prompt - we built a dynamic graph-generation prompt builder that assembled the structured-output schema, Stitch’s node descriptions, the current node and port definitions, value examples, and worked examples into the prompt. The generated graph-creation prompt snapshot was almost 14,000 lines. The wall we hit was at the boundary of prompting: we could give the model a huge map of the world, but without being trained on enough examples of finished, useful graphs, it still mostly learned how to obey the shape of the language without saying anything interesting.

At roughly 50 examples, simple graphs started to work. Math chains, a blue oval, a draggable rectangle, a small animation: those were believable. We learned that we could reliably generate JSON that followed our graph building language, but we could not yet generate graphs like the ones that users would actually want to build with the app.

Spring 2025: Building the Data Flywheel

Once the first fine-tuned models started to work, we realized we needed a better way to get more dataset examples. We built a pipeline in the app for uploading training data to Supabase:

• Raw prompt/action pairs
• Edited prompt/action pairs
• Added edit-before-submit so a generated response could be corrected before it became training data
• Rating and review flows so failures could be turned into example

As testers used StitchAI, they would be able to constantly generate new training data. Prompt the model, apply the generated steps, review the result, correct the JSON if needed, upload the prompt/action pair, fine-tune the next model, and ship that model back into the app.

That loop was the real project for a while. It was our practical answer to the data shortage: if the model needed hundreds or thousands of examples, the app had to help us create reviewed examples as part of normal use.

June 2025: The Data Wall

By June 2025, the dataset had 444 prompt/action examples. The Reinforcement Fine-Tuning split used 354 training examples and 89 validation examples. That was a lot for our internal workflow, and nowhere near enough to teach a model a visual programming environment.

The model became better at producing valid steps and staying inside the schema. But it was was not reliable on prompts that looked more like what a designer would actually want from StitchAI.

The gap showed up on prompts that required several interacting ideas at once:

• Animation
• Gestures
• Layout
• State

One example that we kept trying to get the model to reliably build was a prototype that mimicked the Tinder Swipe card UI: multiple layered cards, drag interactions, snap-back behavior, thresholds, and transitions. These types of prototype prompts required a deep understanding of how the tool itself worked and how a designer would actually go about building the graph in practice.

The bad results we’d get when trying to make this prototype was never malformed JSON - structured outputs and validation caught that class of failure. Bad output was well-formed and semantically wrong: the wrong port, a connection in the wrong direction, a node that looked plausible but did not belong, or a sequence that started wrong and made the rest of the graph useless.

We decided that continuing on fine-tuning alone was too expensive, both from a time and monetary cost.

Why We Tried Code

The original plan of building a graph directly failed because we were trying to get the model to learn a language of building graphs from a small dataset - not just its basic grammar, but its higher level concepts. Even with structured outputs, a generated schema, validation, a large system prompt, reviewed examples, supervised fine-tuning, and early RFT experiments, we still did not have a system where the app could reliably create complex graphs.

So the next bet was to move the model closer to a representation that we knew the models already understood: code. We decided to make the model generate SwiftUI code that we would translate into graph steps. Since every object on a Stitch graph is backed by SwiftUI, we would take the generated Swift code and translate it into steps that would construct the graph.

This work did clarify that the model should probably express intent in a language it already knew, while Stitch owned as much graph semantics as possible: parsing, validation, mapping, node layout, and graph application. But we still did not reach the consistency bar for complex prototypes. It’s important to remember that at the time of this work, the LLMs were just becoming great at writing passable Swift code (and today, they’re still not great at writing excellent Swift code).

Where the Experiment Landed

From the start, we could get good results from small, simple examples. However, we were never able to consistently get accurate results for interesting, complex graphs of the type that are users actually would want to make. We did not have enough examples to teach the model Stitch’s graph semantics directly, and the graph-action language was too narrow and private for the model to generalize from the data we had.

Around this time, many designers building interface prototypes were also starting to get farther by vibe coding than by using node-based interface tools. This explains why the code route felt worth trying — the model was better at code-shaped interface descriptions than at our private graph-action domain specific language..

This is one of four posts about my work at Stitch. See also: Building a 3D System Inside Stitch, Bringing AR to Stitch, and Computer Vision Nodes in Stitch.