If it isn't — if a cloud LLM is an acceptable part of your architecture — the ceiling doesn't apply to you at all. This post is about that path, because it's a completely legitimate one, not a lesser one, and it's exactly what this project ran on for its first real era.

Why the ceiling doesn't touch this path​

The CUDA 11.4 / Python 3.8 ceiling blocks local, on-device GPU inference — that's specifically what Ollama's GPU backends and modern local STT models need, and specifically what stock JetPack 5.1.1 can't provide. A cloud-based stack (GPT-4o or an equivalent hosted LLM, Deepgram or equivalent hosted STT/TTS) does none of that inference on the Jetson at all. The Orin NX's job in that architecture is orchestration — capture audio, make an API call, play back the response — not running a model. CUDA version becomes almost irrelevant to the core loop, because the GPU-heavy work is happening somewhere else entirely, on hardware you don't have to reflash, patch, or maintain.

This isn't a theoretical alternative. It's the actual architecture this project ran on in its earliest working era, on stock JetPack 5.1.1, with a real conversational loop functioning end-to-end on unmodified platform software.

What you get by staying cloud-based​

• Zero platform-version fighting. No CUDA compatibility chasing, no hand-compiled binaries, no reflash. Stock JetPack works as shipped.
• Access to frontier-quality models immediately. A hosted LLM is simply a stronger model than anything realistically runnable on-device on this hardware today. If response quality matters more than architecture purity, this is a real advantage, not a compromise.
• Lower up-front engineering cost. An API call is a fraction of the integration work a local inference stack requires.

What it actually costs you — the honest tradeoffs​

This isn't a free lunch, and this project has direct, painful evidence of where it bites:

Network dependency becomes a single point of failure for the entire conversational loop. This project's own demo-crash post-mortem is directly relevant here — even though that specific incident turned out to be a software crash rather than a network drop, the initial (wrong) assumption that it was a WiFi failure was completely reasonable, precisely because a cloud-dependent architecture makes network loss a totally plausible, totally severe failure mode. A local-inference stack doesn't have this exposure at all — if the model's running on-device, a flaky venue WiFi doesn't take down your ability to think, even if it might still affect other things.

Latency has a floor you don't control. Round-trip time to a cloud API, plus queueing on the provider's end, is added on top of whatever your own pipeline latency is — and unlike a local stack, you can't optimize it away by upgrading your own hardware.

Cost scales with usage, indefinitely. A local model is a one-time hardware and setup cost. A cloud API is a per-call cost for the life of the deployment — worth actually modeling out for your expected usage pattern, not just accepting as a rounding error.

Data leaves the device. For a project like this — audio captured from real conversations, in a public space, with children present — that's a real design consideration, not a footnote. Whether that's acceptable depends entirely on your specific deployment context and what commitments you're making to the people interacting with the robot.

The actual decision framework​

This isn't "cloud bad, local good" or the reverse — it's a genuine tradeoff, and the right answer depends on specifics:

• Reliability requirements matter more than raw model quality → lean local, accept the reflash, accept a smaller model
• Response quality matters more than guaranteed uptime, and your deployment can tolerate occasional network-dependent failure → cloud is a completely legitimate, lower-effort choice
• Data/privacy commitments are strict → local, regardless of the other factors
• You're prototyping and don't yet know which of the above actually matters for your specific project → start cloud, since it's the lower-effort path to a working end-to-end loop, and revisit the reflash decision once you know what you actually need

This project ultimately moved toward a local-first architecture — but that was a deliberate choice made after operating the cloud-based version and learning specifically where its tradeoffs bit, not a default assumption that local is inherently superior. If you're earlier in that same decision, it's worth actually running the cloud path first rather than reflashing on faith that local is the "correct" answer before you've felt what problem you're actually solving for.

Who this is for, and the core bet​

If your platform is on JetPack 5.1.1 (CUDA 11.4, Python 3.8), you already know the ceiling: modern local-LLM tooling and current STT/TTS libraries mostly assume newer CUDA and Python than that platform provides (see the hard-ceiling post for the full technical reasoning). You have two real options: reflash to a newer JetPack, or go cloud-first and sidestep the ceiling entirely (see the companion escape-hatch post for that tradeoff in full).

This guide covers the second path — a cloud-first stack that ran, in production, on stock JetPack 5.1.1, no reflash required. The core bet: a hosted STT engine, a hosted LLM, and a hosted TTS engine, orchestrated by a Jetson that does no heavy inference itself at all — just capture, routing, and playback.

The full pipeline​

Each stage is covered in full technical detail in the voice-to-action pipeline post — this guide focuses on the numbers and the decisions, not re-explaining each stage from scratch.

The real latency budget: ~2.5 seconds of work, ~1 second perceived​

This is the number worth internalizing before you start: the actual end-to-end pipeline takes roughly 2.5 seconds from wake word to spoken response. That's not a failure — cloud STT, a frontier LLM call, and cloud TTS synthesis genuinely take real time, and no amount of client-side optimization changes the physics of three sequential network round-trips to hosted services.

What actually matters is that the perceived latency is closer to 1 second, and that gap is closed by a specific, deliberate trick, not a speed improvement:

The mechanism: the instant a wake word fires, a pre-rendered opener WAV ("Oh, hello!") plays immediately, on a separate thread, in parallel with the real pipeline doing its actual work underneath. By the time that short opener finishes, the real response has usually arrived. The person experiences a robot that responded almost instantly — the actual 2.5-second pipeline never stopped running, it just wasn't the thing they were listening to.

This is worth being honest about: it's a genuine UX trick, not a technical optimization. If you're building something similar, budget your actual pipeline at 2-3 seconds realistically, and plan your latency-masking strategy as a first-class design decision, not an afterthought bolted on after the fact.

What actually worked, and why​

• Fuzzy wake-word matching, not exact string matching — real STT output reliably mishears wake words, and a difflib-based fuzzy match with an explicit list of known mishearings caught far more real attempts than exact matching ever would.
• The model choosing its own gestures and emotional tags, embedded directly in its own text output ([G:id], [E:tag]) — this is what made the interaction feel genuinely alive rather than scripted, and it's a capability worth taking seriously if you want a character that feels responsive rather than robotic.
• A cleanly layered audio-output stage — swapping the final output hardware (external speaker to onboard speaker) required touching nothing upstream of that one stage. If you're designing something similar, keep your output stage as replaceable as this one was; it's a good sign the rest of the architecture is properly decoupled.
• Session-aware word-count gating — different trigger thresholds inside vs. outside an active conversation session meant far less accidental triggering from ambient background speech, without needing a heavier, more complex intent-classification layer.

Where the hard edges are​

This stack has real, documented limits worth knowing before you commit to it:

• WebRTC as a gesture-control transport is genuinely fragile against vendor firmware updates, entirely outside your control — see the full WebRTC saga for exactly how badly this can go, and how long a fix can take even with an active, engaged community maintainer.
• Only one active WebRTC session at a time — the vendor app and your own client cannot both be connected; plan your operational workflow (close the app before connecting) around this constraint explicitly, not as a surprise you discover live.
• An LLM selecting its own physical actions is a real, physical-world risk surface, not just a conversational one — the Frankenstein incident is the concrete example: two commands the model fired close together put the robot into a genuine lockup. If you give a model this kind of autonomy, plan real safety architecture around it from day one — see the two-tier safety design for the shape that took here.
• A cloud-dependent architecture makes network loss a real, severe failure mode — see the demo-crash misdiagnosis for a case where this was initially (wrongly) assumed to be the cause of a failure that turned out to be something else — but the fact that it was a plausible wrong guess tells you how real a risk this actually is.

Decision framework, if you're starting fresh​

The honest tradeoffs, one more time​

This isn't the objectively correct architecture — it's the right one for a specific set of constraints: no reflash tolerance, acceptable network dependency, and a real budget for handling the physical-safety implications of model-driven autonomy. If your constraints are different — strict data-privacy requirements, zero tolerance for network-dependent failure, or a platform where the reflash is cheap and low-risk — a different starting point is genuinely more correct for you, not just a matter of preference.

What we'd tell anyone starting this exact path: budget the real 2.5-second pipeline honestly, design your latency-masking strategy deliberately rather than as an afterthought, and build your safety architecture before you give a model the ability to act in the physical world — not after the first incident makes it obvious you needed to.

The starting idea: motion capture​

The initial plan was straightforward: use xr_teleoperate, a motion-capture pipeline that maps a human operator's body pose (captured via VR headset) onto the robot's joint angles, to record real gestures by literally performing them.

The problem showed up immediately: xr_teleoperate's officially supported hardware list includes Meta Quest 3 and 3S — not the Quest 2 headsets actually on hand. Two paths presented themselves at that point: sink real time into getting unsupported hardware working against a pipeline that doesn't officially support it, or find another way entirely.

The pivot: the data already existed​

Rather than chase Quest 2 compatibility, the actual question worth asking was simpler: was live motion capture even necessary, given what was already sitting on disk? It was not. LAFAN1 G1-retargeted motion CSVs and the openhe/g1-retargeted-motions dataset — both already confirmed working in MuJoCo simulation — represented a substantial, ready-to-use library of real human motion data, already mapped to this exact robot's joint structure. No capture hardware required at all.

This is a pattern worth naming on its own: the instinct to build new capture infrastructure is often solving a problem that existing, already-validated data has already solved. Before investing in hardware or a live-capture pipeline, it's worth genuinely checking what usable data already exists — in this case, an entire viable gesture-authoring foundation was already sitting on disk, unused, simply because the original plan hadn't gone looking for it first.

The real idea: skip capture entirely, describe it in English instead​

The pivot led somewhere better than a simple substitution of one data source for another. If joint-angle data for gestures could come from existing datasets, why not go a step further — describe a new gesture in plain language, and have an LLM reason about the actual joint angles needed to produce it?

The design: a natural-language-to-joint-angle authoring tool. A person describes a gesture in plain English. The Claude API reasons about G1 joint angles anatomically, using a real joint reference table (indices 10–26, with documented safe min/max limits for each), and produces keyframes. Those keyframes get validated against joint limits and velocity thresholds, then exported as a ready-to-deploy Python script matching the existing rt/arm_sdk gesture-dispatch pattern exactly.

This was explicitly framed from the start as a small step toward a larger goal: the [G:tag] naming convention used for authored gestures is the same one that eventually lets an LLM running the live conversational pipeline autonomously trigger validated gesture scripts — this tool and the live dispatch system share the same underlying contract by design, not by coincidence.

Building it: three iterations, and an honest dead end​

The first attempts built this as an in-browser artifact — and hit real friction: sandbox API blocking, and a click-handler scoping bug that took a deliberate, minimal reproduction (a bare-bones click test confirming an IIFE-plus-addEventListener pattern actually worked in that environment) to properly isolate. Worth naming plainly: this wasn't solved on the first or even second attempt.

The actual fix wasn't a deeper artifact-environment workaround — it was recognizing the artifact sandbox was the wrong delivery format for this specific tool, and shipping it as a standalone local HTML file instead. That's a similar shape of decision to containerizing rather than fighting a missing CUDA wheel elsewhere in this project: when an environment's constraints keep fighting you, the fix is sometimes picking a different environment entirely, not eventually out-stubborning the original one.

What actually shipped​

A self-contained gesture_tool.html, structured as five stages: Describe → Keyframes → Validate → Export → Library. It handles its own Claude API key (saved locally, provided by whoever's running the tool), uses cosine-eased keyframe interpolation for natural-looking motion, validates against real joint limits and velocity thresholds, estimates center-of-mass to catch balance-affecting gestures before they ever reach hardware, and generates Python output matching the live pipeline's existing rt/arm_sdk format exactly — so an authored gesture and a live-dispatched one are structurally identical, not two different code shapes that happen to do similar things.

Why this whole detour mattered​

None of this would have happened if Quest 2 compatibility had just worked on the first attempt. The constraint — no supported capture hardware on hand — is what forced a genuine second look at what already existed, which surfaced better data than live capture likely would have produced anyway, and led to a more capable, more directly integrated tool than "record some gestures with a headset" was ever going to be. Worth remembering the next time a missing piece of hardware feels like a pure blocker: sometimes the workaround it forces is better than the original plan would have been.

One process, three threads, one render loop​

The whole face system runs as a single unified process — not a distributed set of services, deliberately. Three daemon threads handle the actual work underneath a Pygame render loop running at 30fps:

• A TTS worker thread, calling ElevenLabs
• An audio playback thread, via sounddevice
• An input worker thread, handling whatever's driving the conversation (text input, or the live pipeline output)

The render loop itself just reads current state and draws — it doesn't own any of the actual audio or network work, which keeps the visual side responsive regardless of what the other threads are doing. One nice practical consequence of this architecture: there are no pipeline restarts needed to iterate on expressions. Edit the face-rendering file directly, and the changes take effect immediately, without touching anything upstream.

FaceState: fifteen numbers, each moving at its own speed​

The core of the whole system is a FaceState dataclass — fifteen floating-point parameters (eye openness, brow position, mouth shape components, and more), each one lerping toward its target value at its own differentiated speed, not a single global animation rate.

This matters more than it might look like on paper. A single shared animation speed makes every expression change feel the same — uniformly snappy or uniformly sluggish, regardless of what's actually happening. Differentiated speeds let a blink snap shut fast while a brow raise settles in more slowly, which is a lot closer to how real facial movement actually reads: different muscles, different speeds, not one uniform "animation."

Blinking and micro-movement​

Blinks use cubic easing rather than linear interpolation — a blink that eases in and out reads as much more natural than one that moves at a constant rate, since real blinks aren't linear either. On top of that, small saccade micro-movements run continuously in the background — the tiny, near-constant eye movements real eyes make even when "still." Without this, a face that's technically animated correctly can still read as dead or uncanny, because real eyes are never actually motionless.

The mouth: Perlin noise, not a volume meter​

This is the detail worth dwelling on. A naive approach to mouth animation during speech is to drive mouth openness directly off the audio's volume envelope — louder means more open. That works, but it reads as mechanical, because real speech doesn't just vary in loudness, it has continuous, organic shape variation even at a constant volume.

The actual approach layers Perlin noise — a smooth, continuous pseudo-random function, commonly used in graphics for natural-looking terrain and motion — on top of the coarticulation/viseme interpolation driving the base mouth shape. The Perlin layer adds the small, continuous, non-repeating variation that makes the mouth look like it's actually shaping sounds, rather than mechanically opening and closing in sync with a volume meter.

Keeping lip sync accurate across a streaming pipeline​

The actual viseme-timing math — correcting for ElevenLabs' per-chunk relative timestamps and the sample-rate mismatch between ElevenLabs' output and the playback pipeline — is covered in full technical detail in the ElevenLabs viseme timing entry. The piece that connects to the face system specifically is the anchor pattern: a single T0 timestamp is recorded the instant sound.play() actually starts, and every subsequent viseme or expression event is scheduled as an offset from that one anchor (elapsed = time.time() - T0), rather than trying to track time independently in each subsystem. One shared anchor, many events scheduled relative to it — simple, and avoids any drift between what the mouth is doing and what's actually coming out of the speaker.

Why this design holds together​

Every piece of this system solves a version of the same problem: make discrete, computed state look like continuous, organic movement. Differentiated lerp speeds instead of one animation rate. Cubic easing instead of linear blinks. Saccades instead of a static gaze. Perlin noise instead of a volume meter. A single timing anchor instead of independently-drifting clocks. None of these tricks are individually complicated — the discipline is in applying the same underlying idea consistently across every part of the face, rather than solving expressiveness in only the most visible place (the mouth) and leaving everything else mechanical by comparison.

The ceiling, concretely​

The G1 EDU ships with JetPack 5.1.1: Python 3.8, CUDA 11.4. On paper, that's a perfectly capable embedded AI platform. In practice, it quietly locks you out of the current generation of local-AI tooling, for one specific, unglamorous reason: Ollama's GPU backends require a newer CUDA than 11.4 provides, full stop. Not slower. Not degraded. Incompatible.

That single fact cascades further than it looks like it should. Most current speech-to-text and text-to-speech libraries assume Python ≥3.10 as a baseline. Whisper's ecosystem, Pipecat, and a good chunk of the current voice-pipeline tooling generally, all lean on language and packaging features that Python 3.8 doesn't have. So the ceiling isn't just "no Ollama" — it's "no current local-LLM tooling, and increasingly awkward footing for current STT/TTS tooling too," all stemming from one version pin.

The workaround we actually tried, and why it was a dead end​

The instinct, reasonably, was: don't reflash the whole robot over a version number — compile around it. So that's what got built first: llama.cpp, compiled from source, targeting CUDA 11.4 and compute_87 explicitly, running local LLM inference in-process.

It worked. That's worth saying plainly — this wasn't a failed experiment, it was a functioning, hand-built inference path that ran real models on real hardware.

But "worked" and "worked for the long haul" are different claims. A hand-compiled binary, targeting an old CUDA version by hand, has no upstream maintainer keeping it aligned with anything. Every future dependency — a new STT library, a new pipeline framework version, a new model architecture — was a fresh compatibility check against a stack nobody but this project was responsible for keeping current. That's not a one-time cost. It's a permanent tax, paid again on every future addition to the stack, for as long as the platform stayed on JetPack 5.1.1.

This is the same shape of problem documented elsewhere on this site in containerizing rather than hand-compiling CUDA-dependent packages — except in this case, there was no container that could fix it, because the ceiling wasn't in a single package's missing wheel. It was the platform itself.

Why patching further wasn't the answer​

At some point, the honest question stopped being "what's the next workaround" and became "is this ceiling actually fixable in place, or is it structural." It's structural. CUDA 11.4 is a platform-level constraint tied to JetPack 5.1.1's driver stack — there's no user-space fix, no pip flag, no clever container trick that changes what CUDA version the underlying platform actually exposes. Every additional hand-compiled workaround was solving today's specific blocker while leaving the actual ceiling completely intact for the next one.

That's the moment worth naming explicitly for anyone else facing the same fork: if the blocker is a platform version, not a package version, workarounds are borrowed time, not a fix. The right question isn't "can I get this one thing working" — it's "how many more times am I going to ask that question before the answer is the same reflash I'm avoiding right now."

What the reflash actually bought​

JetPack 6.2 removes the ceiling entirely — Python 3.11, CUDA 12.6, and critically, Ollama's GPU backends work as intended, with zero hand-compiled anything. The full procedure — disassembly, NVMe flash, separate NX module firmware update, verified platform state afterward — is documented in detail in the reflash log entry if you're about to do this yourself.

The bigger shift wasn't really "newer versions" for their own sake. It's that the entire dependency stack going forward — Ollama, faster-whisper, Pipecat, all of it — collapsed into a single, normal, maintainable Python 3.11 environment instead of a hand-tuned exception. That's the actual payoff: not speed, not features, but no longer being the only maintainer of a compatibility layer that upstream projects were never going to support.

If you're deciding whether to reflash​

A few honest questions worth asking before you commit, based on what this project actually hit:

• Are you trying to run any current local-LLM tooling (Ollama, or anything GPU-accelerated) on stock JetPack 5.1.1? If yes, you will hit this exact wall, not a softer version of it.
• Are you hand-compiling anything against CUDA 11.4 right now to work around it? If yes, you're already paying the tax this post describes — the only question left is whether you keep paying it or reflash.
• Is your project's timeline measured in months, not days? If so, the reflash is worth doing sooner rather than later — every workaround built in the meantime is something you'll either maintain indefinitely or eventually throw away.

If none of that applies yet — if you're early, prototyping, not yet asking your platform to do anything CUDA-12-era — there's no reason to reflash preemptively. But the moment any of those three is true, the ceiling stops being theoretical.

The shape of the loop​

Nine real stages, each with its own specific gotcha worth knowing if you're replicating this.

1. Mic capture and echo cancellation​

A USB mic feeds into PulseAudio with module-echo-cancel (speex) active — ec_mic as the working capture device, not the raw hardware input. This matters because the robot's own speaker output would otherwise bleed back into the mic and get picked up as fresh input, which is exactly the kind of feedback loop that makes a voice pipeline unusable in a real room with real reflections and real ambient noise.

2. Streaming STT — and a specific trigger discipline​

Deepgram's nova-2 model, connected via streaming WebSocket, not batch. The callback that actually matters only fires on both is_final=True and speech_final=True — not just one or the other. That distinction is easy to get wrong: is_final alone can fire on interim segment boundaries mid-utterance, while speech_final specifically signals the speaker has actually stopped. Triggering on is_final alone risks firing the pipeline on a sentence fragment.

An automatic 2-second retry on Deepgram disconnect keeps the stream resilient without requiring a full pipeline restart on a transient network blip.

3. Wake word — fuzzy, not exact​

Exact-string wake word matching is fragile against real STT output, which reliably mishears things. The actual match list included the obvious target plus every plausible mishearing Deepgram had actually produced in testing — "rubbert," "bberg," "rupert," "brewer," "grouper" — plus a fallback rule matching any word starting with "ru" or the project's actual name (≥3 characters). A difflib fuzzy cutoff of 0.75 catches variants beyond even that explicit list.

4. Session and echo-gate logic​

Once a wake word fires, a session stays open for 60 seconds after the last interaction — not a global timeout, one that resets on every exchange. Word-count gating differs depending on whether you're inside or outside an active session: outside, a minimum 5 words are required to trigger (filtering out ambient noise and stray fragments); inside an active session, that drops to 3 words, since the system already has strong prior evidence someone is actively engaged. Barge-in is supported — saying the wake word again during playback interrupts the current response — and a small set of explicit stop words (like "stop," "quiet," "cancel," "enough") terminate playback immediately.

5. The LLM call, and the latency-masking trick​

GPT-4o, streaming, with a 6-message rolling history (three exchanges) — enough for real conversational continuity without an unbounded, ever-growing context.

The genuinely clever part: the moment a wake word fires, a pre-cached opener WAV ("Oh, hello!", "I'm listening!") plays immediately in parallel with the GPT-4o call, on a separate thread. By the time that short opener finishes playing, the real response has usually already arrived — so perceived latency drops well below the pipeline's actual end-to-end time, without any real speedup having happened at all. It's a genuine trick, not a technical fix, and it works because it's targeting perceived responsiveness specifically, not raw pipeline speed.

Opener WAVs themselves are pre-rendered once at startup via the TTS engine's REST API and cached locally at 16kHz mono 16-bit — no live synthesis cost paid for this specific latency-masking step.

6. Tags embedded directly in the model's own output​

This is the architectural core of the whole system: GPT-4o's system prompt instructs it to embed its own gesture and emotion selections directly into its text response — [G:N] for a numeric gesture ID, [E:tag] for an emotion/expression tag — interleaved with the actual spoken reply. A tag parser strips both before the text reaches TTS, and routes the extracted tags to their own dispatchers.

Worth being explicit about what this means: the model is choosing its own physical actions, inferred from its own read of the conversation, not selected by any separate rule-based layer. That's the capability — and, as covered further down, also the risk.

7. Text-to-speech​

Deepgram Aura, aura-orion-en, roughly 300–500ms per sentence for live generation — plus the pre-rendered WAV cache covering the fixed opener set, avoiding that latency entirely for the specific phrases used there.

8. Audio output — and the one thing that changed later​

Two different eras of this same pipeline used two different final output stages: earlier, audio went through sounddevice to an external Rokono speaker; later, it switched to AudioClient.PlayStream() direct to the robot's own onboard speaker. The genuinely notable thing about this swap: nothing else in the pipeline changed. Same mic, same GPT-4o, same TTS engine, same tag system — only the last few inches of the audio path were different. That's a good sign of a cleanly-layered architecture: a real change to the output hardware didn't ripple backward into anything upstream of it.

9. Gesture dispatch — the automatic path, as it ran on April 11​

[G:N] tags, once parsed, fired via api_id 7106 (built-in gestures) over the custom WebRTC connection — the same connection and topic covered in detail in the arm-topic-discovery entry. This is the part that made the April 11 demo feel genuinely alive rather than scripted: the robot's physical gestures were a direct, live consequence of what the model itself decided to say and how it decided to say it — not a fixed animation triggered by a human operator watching from the side.

What this bought: a real, working, autonomous loop​

Worth stating plainly, because it's the whole point: this is Human → Speech → LLM → Decision → Gesture → Physical response, running live, with no human in the loop selecting what the robot did or said in the moment. That's not a demo of individual components — it's a functioning autonomous interaction loop, and it's genuinely what was running on April 11.

Where autonomy went further than intended​

:::note A different day — not April 11 The incident below happened during a separate, private demo session — not the April 11 open house, which was a full success as covered in the WebRTC saga post. Keeping these clearly separate matters, since they're easy to conflate. :::

The same capability that made the April 11 demo feel alive — the model choosing its own gestures based on its own read of the conversation — went further than intended on a different occasion. GPT-4o interpreted a person's tone as sad, and, working entirely from its own instruction base and the local gesture-ID library available to it, inferred on its own that the appropriate response was to physically approach the person with arms extended — a gesture nobody had explicitly authored or instructed it to select for that situation. The walk itself was rough — a short, uncertain shuffle rather than a confident stride — but it was real locomotion, autonomously triggered.

The lockup was severe enough that only a full reboot cleared it — no software-level recovery, no reset command, nothing short of power-cycling the robot brought it back to a working state. During the lockup itself, communication errors surfaced in the Unitree Explore app, consistent with the robot's control layer having entered a state it genuinely couldn't resolve on its own.

The deeper issue: the robot's control layer at the time was only willing to accept one command context at a time. The locomotion attempt and the arm gesture, dispatched close together, put the system into a state it couldn't cleanly resolve — a real lockup, not a graceful failure. This is a different failure mode from the FSM-gating issue already documented — that one silently no-ops a command that returns success without ever moving; this one is a genuine conflict between two commands actively racing each other for the same control context, severe enough to require a full reboot rather than any in-session recovery.

What this incident actually revealed​

Notably, this is very plausibly why the eventual, more mature gesture architecture moved to rt/arm_sdk with an explicit weight-blending parameter — a design that lets arm commands layer smoothly on top of locomotion output (ramping control weight from 0 to 1 and back) rather than treating the two as separate, competing command contexts at all. The later architecture's own documentation states outright: locomotion and arm control run as genuinely parallel channels, specifically so that "the robot keeps walking and balancing while your gesture script controls only the arm joints" — a direct structural fix for the exact class of conflict this incident surfaced.

The honest tradeoff of an LLM calling its own physical actions​

This is the real lesson, not just an entertaining anecdote: giving a language model the ability to select and fire its own physical actions, based on its own inference about what's appropriate, is a genuinely powerful way to make a robot feel alive and responsive — and it comes with a real, physical-world risk surface that a purely conversational chatbot never has to reckon with. A bad text response is embarrassing. A bad physical-action inference, on real hardware, moving toward a real person, is a different category of risk entirely.

This incident is a big part of why later architecture put real structure around autonomous action-calling — the gesture arbitrator's strict drop-lock design, a human-curated gesture allowlist restricting which gestures a model can select from at all, and the two-tier SAFE_IDLE safety system all exist, in part, because of exactly this class of incident: an autonomous system doing something technically-successful-but-contextually-wrong, with real physical consequences if left unchecked.

Scoping note: the WebRTC break was the most visible failure to come out of firmware 1.5.1, but it wasn't the only one. The same update also killed the internal speaker, introduced new motor alarms, and surfaced a separate (unrelated) custom-gesture failure — four compounding problems off one release, not one. This post follows the WebRTC thread specifically; the full triage across all four is in the 1.5.1 systemic-regression entry.

What was actually working on April 11​

Worth being specific about this, because it's easy to undersell in hindsight once things broke: this wasn't a fragile demo held together with hope. The voice pipeline (USB mic → Deepgram → GPT-4o → Aura TTS → speaker) was confirmed working. Wake-word detection was reliable. Session management, echo-gating, all functioning. And gestures — real, physical robot motion, dispatched live based on GPT-4o's own output via a [G:id] tag system — were working over a custom WebRTC connection to the robot, with built-in gesture IDs (wave, various expressive motions) confirmed executing correctly in front of a real audience.

That's a genuinely complete, working system, not a proof of concept.

Six days later: firmware 1.5.1​

The robot's firmware was updated to 1.5.1 immediately after the demo. What followed was a cascading, multi-system regression:

• WebRTC connections broke outright. The /con_notify endpoint began returning a new, BLE-derived device-specific key (data2=3) as part of the handshake — undocumented at the time, and any client not deriving and presenting that key was rejected.
• The internal speaker died.
• Motor alarms appeared that hadn't been present before.
• Custom (taught) gestures were separately, silently broken — though this turned out to be a distinct, pre-existing problem tangled up in the same crisis: custom gestures dispatched via api_id 7112 were returning success codes with no physical motion at all, a finite-state-machine gating issue that had nothing to do with the firmware regression, but surfaced around the same time and initially looked like part of the same disaster.

The honest state of things at that point: a working demo robot had become, within a week, a non-operational one, with three or four separate problems compounding into a single confusing mess.

The community found what vendor support hadn't yet​

The break was reported to unitree_webrtc_connect's issue tracker on April 11 — issue #53 — by another affected user hitting the exact same data2=3 failure. legion1581, the library's maintainer, responded the same day: this was a new key-derivation method, device-specific, and would take real reverse-engineering effort to crack. By the next day, the mechanism was understood in more detail — the key is generated fresh on the G1 at every boot, obtained via BLE, and requires processing on Unitree's own cloud server.

That same issue thread is also where a good deal of this project's own early arm-service reverse-engineering got documented and shared back with the community — the undocumented rt/api/arm/request topic, the api_id 7112 custom-gesture-by-name endpoint, and several other G1-specific findings that weren't in Unitree's own docs at the time.

The fix that actually shipped, roughly three weeks later around May 1, was proper AES-128 key support built into the library itself — not the simpler pairing-based theory floated in the first days of investigation. The mechanism: a unitree-fetch-aes-key CLI tool pulls the device's own AES-128 key directly from Unitree's cloud (specifying region and device type), which then gets passed into the connection (aes_128_key=, device_type="G1") alongside a library upgrade (2.0.4 → 2.1.2).

This project confirmed it working end-to-end on May 20 — full handshake, data channel validation, heartbeat, and arm gestures firing cleanly over the data channel, on G1 firmware 1.5.2. From initial break to confirmed working fix: just over five weeks, and the real resolution came from a proper library-level implementation, not the earlier, simpler hypothesis about how the key exchange worked.

This is worth pausing on regardless of the exact mechanism: the fix came from a community member's own sustained investigation and implementation work, not a vendor patch. That's not a knock on Unitree's support process — it's a genuine case for staying plugged into the community around a platform like this, rather than just filing a ticket and waiting on an opaque timeline.

Worth being precise about one more thing, since it's easy to conflate: this AES-128 key fix is what actually restored WebRTC connectivity. A separate Unitree firmware update, 1.5.2, fixed a different problem — a conflict between WebRTC and the vendor's own onboard ASR service — documented separately in the WebRTC 1.5.1/1.5.2 entry. One was a community fix for a community-discovered break; the other was an unrelated vendor-side regression fix.

The actual decision: stop depending on WebRTC for gesture control​

This is the real turning point, and it's a decision made deliberately, not a fallback stumbled into. Once legion1581's method restored a working connection and the immediate crisis was survivable, the higher-priority question became: should gesture control keep depending on WebRTC at all, given what had just happened?

The answer was no. Direct rt/arm_sdk control — joint-level commands over DDS, bypassing WebRTC entirely — became the planned path forward for gesture dispatch, explicitly named as the highest-priority investigation once the immediate WebRTC crisis was stabilized.

The reasoning tracks with a pattern that shows up elsewhere in this project's decisions too: a dependency that can be silently broken by a vendor firmware update, with no advance notice and no control over the timeline, is a fragile foundation for anything meant to run reliably in front of real people. WebRTC wasn't abandoned because it was inherently bad — it worked, genuinely, on April 11. It was deprioritized because the April 17 regression demonstrated a real, uncontrolled failure mode that a DDS-based direct control path doesn't share to the same degree. This is the same category of judgment call as choosing to containerize rather than hand-compile CUDA dependencies — recognizing when a dependency's fragility outweighs its convenience, and deliberately routing around it rather than continuing to patch around the same weak point indefinitely.

Where this leaves things​

WebRTC didn't disappear from the stack entirely — it still has a role for vendor-adjacent audio and specific transport needs. But gesture control moved to a path this project actually controls end-to-end, rather than one that could be reshaped without warning by someone else's firmware release schedule. The FSM-gating investigation that got tangled up in this same crisis turned out to have its own independent fix, applicable regardless of which transport carries the command. And the whole episode is a big part of why this project treats "is this dependency something a firmware update outside my control could silently break" as a real design question now, not an afterthought.