Microphone capture, wake detection, transcription, speech playback, and the remaining room tests.
Published Updated 5 min read
Ainekio records speech through its microphone, sends it to MetaHuman OS for transcription and conversation, and plays the response through its speaker. The robot handles capture and playback; MetaHuman handles speech-to-text, conversation, and speech generation.
V1 has produced audible playback. Reliable repeated conversation in the room still needs testing, including wake detection, interruptions, and reconnects. In particular, no production Ainekio wake-word model has been accepted and tracked, so the initial configuration leaves wake detection unavailable.
MetaHuman now routes the final reply through Persona and the existing conversation speech system. Intent chooses whether a reply is needed; Task selects actions; Goal Review supplies assessment rather than another spoken response. Ordered Task speech steps have been removed. The systems report records deployment and the remaining model and playback qualification.
The listening and S3 implementation notes below retain their earlier evidence period. This response-owner change does not establish a newly trained wake-word model or a successful end-to-end room conversation.
Wake detection and voice activity detection do different jobs. A wake-word model would recognize a chosen phrase that starts listening. Voice activity detection (VAD) separates speech from silence and decides when the spoken segment, or utterance, has ended.
The S3 firmware captures microphone audio and applies those listening controls
when available. The gateway assembles an utterance into a WAV audio file and
sends it through MetaHuman’s Environment Bridge, the connection for external
devices. Whisper transcribes the recording into text, which enters the
conversation as an audio_utterance observation.
The VAD model uses a minimum recording window, a silence window to end the turn, and an absolute maximum duration. A fixed-size queue in PSRAM, temporary board memory, absorbs short network stalls without letting buffering grow without limit.
At the end of an utterance, the robot can take a still image with the same ID. The Bridge can join that image to the audio before submitting the turn. If the camera is unavailable, audio proceeds or fails under its own rules; an image is not invented to fill the gap.
The firmware includes microWakeWord and TensorFlow Lite Micro to run a small
wake detector locally. It loads an owner-trained model package from LittleFS,
the robot’s asset filesystem, and checks the package description, expected
input/output shapes, audio preprocessing, and model ID before reporting
wake_ready=true.
A pilot ran on the physical robot but triggered on sounds that were not the wake phrase. Its staged confidence threshold was much lower than the documented package default, and the test set had too few negative examples: ordinary sounds that should leave the detector silent. A threshold sweep provided tuning data, not an accepted production model.
The next evaluation needs hours of room audio containing robot speech, servo noise, television, fans, and similar-sounding words. It must record detection confidence and rearm behavior—whether the detector returns to a ready state for the next wake phrase after handling one turn.
MetaHuman uses Kokoro or the configured speech-synthesis service to turn a response into audio. The resulting file has size limits, a session identity, and an action ID. The Environment Bridge and Ainekio adapter send its PCM samples through the gateway to the robot’s speaker queue.
The gateway paces delivery rather than sending the whole file as fast as possible. The firmware accepts samples during an explicitly opened text-to-speech (TTS) burst and plays them through the I2S audio interface. It reports queue overflow, interruption, data outside a burst, and completion.
The adapter returns that playback result to the originating action. Successful
synthesis, or a sendText acknowledgement, does not establish that the speaker
played the response. September adapter work also keeps an uncertain outcome
when playback may have started but its final result is missing.
The September 8 live review found eight durable TTS deliveries claimed and acknowledged on their first attempt, with new outward responses in the conversation buffer. This establishes activity through the revised delivery path. It is not audible confirmation of every response, a wake-word acceptance test, or proof that an entire conversation completed.
Saved speech results let the parent request distinguish a delivered response, an interruption, and a failure. Speaking can be one step in a larger objective; its result returns to that request so unfinished work can be reviewed.
A limited prebuffer, which queues some audio before playback starts, and small blocks paced relative to send time produced clean audible V1 playback in one test. The run still recorded speaker underruns, where the playback queue empties before the next samples arrive, and separate microphone transport faults. Those records matter even when a particular response sounds good.
Full acoustic echo cancellation is absent. The microphone is deliberately suspended while the robot speaks, then resumes after a cooldown. This avoids listening to its own speaker during that interval, but also limits conversational turn-taking while speech is playing.
Small listening and speaking motion cues use the normal calibration, stop, Sleep, and fail-safe rules. Battery-voltage warnings do not establish whether the motor supply can handle combined motion and audio loads; that needs a physical power test.
The microphone produces 16 kHz, 16-bit mono PCM, meaning one channel of 16,000 samples per second, with two bytes per sample. Each 20 ms frame contains 640 bytes. The gateway packages a completed utterance as WAV for transcription.
A version-pinned neural VAD model applies the minimum, silence, and maximum windows. Its PSRAM queue has a fixed memory limit. The wake runtime validates the model manifest, tensor shape, audio frontend, and model ID before making wake-gated listening available.
MetaHuman’s September changes connect audio and images to conversation context, speech delivery, and saved workflow results. The September 8 delivery records show part of that path running; the later workflow repairs and repeated acoustic performance still need validation. P4 also needs its own microphone, speaker, camera, and transport implementation.
The next room test should repeat the whole exchange: wake detection, utterance capture and optional image, Whisper transcript, conversation, generated speech, gateway delivery, and robot playback result. Record wake rearming, dropped PCM, underruns, reconnects, and microphone resumption across multiple turns. The status report records the dated progress.