Sandbar Audio App Prototype

In 2024, a portfolio company of mine, Sandbar, asked me for help to build a prototype of their voice-powered agentic app. The app was designed to interface with their then-upcoming hardware product, a voice input ring called Stream.

The biggest hurdle was getting real-time, bi-directional audio streams working between the iPhone app and the server-hosted agent.

To that end, I worked with them to build an iOS app that used Apple’s AVFAudio framework for handling audio capture and playback. Because low latency was of paramount importance, we encoded and decoded the audio streams using the Opus codec before sending the data to the server using the Starscream WebSockets library. When the server had an audio response, it sent it down to the client also encoded as Opus data, which we decoded and played back.

The flow of audio data looked like this:

%%{init: {"flowchart": {"nodeSpacing": 14, "rankSpacing": 24, "padding": 5, "subGraphTitleMargin": {"top": 4, "bottom": 12}}, "themeVariables": {"fontSize": "13px", "clusterBkg": "transparent"}}}%%
flowchart TD
    accTitle: Sandbar bidirectional audio pipeline
    accDescr: Microphone audio is converted to PCM, encoded with Opus, and sent over a WebSocket to a server that processes the audio, generates a response, and sends audio back. The response is decoded with Opus, played with AVAudioPlayerNode, and routed to the speaker or Bluetooth.
    microphone[Microphone] --> pcm[PCM conversion]
    pcm --> opusEncoding[Opus encoding]
    opusEncoding --> websocket[WebSocket]
    subgraph server[Server]
        direction TD
        processAudio[Process audio] --> generateResponse[Generate response]
        generateResponse --> sendAudio[Send audio]
    end
    websocket --> server
    server --> opusDecoding[Opus decoding]
    opusDecoding --> player[AVAudioPlayerNode]
    player --> output[Speaker / Bluetooth]

The overal behavior of the system is encapsulated in this state machine:

State diagram for Sandbar’s audio session: waitInput leads through recording, sending, waitResponse, and playResponse; playback returns to waitInput, while errors lead to quit.

The app allowed users to take photos and attach them as images to send to the backend agent, which enabled them to have a conversation with the agent about what they were seeing.

Sandbar iPhone prototype demonstrating voice input with an attached image.

At the conclusion of the project, Sandbar had an app that they could use for their first user research studies. I also helped on board their first full-time iOS engineer to the project and codebase.