The software running on the body#

Firmware is the software installed on Ainekio’s body controller. On the working V1 robot, an ESP32-S3 controls eight joint motors, the camera, microphone, speaker, and face display. MetaHuman OS handles conversation and decisions elsewhere; the firmware turns its requests into physical actions.

For example, a “wave” request selects a prepared movement. The firmware checks that movement, applies joint calibration and limits, drives the servos, and reports the result. It also has to handle a stop or connection loss while camera and audio work compete for the same small computer’s resources.

This page describes the S3 implementation. October 8 V2 update: the owner reports that the separate twelve-servo robot can stand, walk and respond to commands. Its photographed LCD is working; the carapace is not attached and Active Operator has not been tested.

The face/media record covers dated P4 LCD/camera, speaker and wake-capture results. The responsive-command entry covers the MetaHuman/gateway split and distinguishes committed work from local refinements. The installed model/calibration for the walking result is not identified here. Earlier deployment and geometry records remain in the gait journal.

How commands become movement#

A shared body core, written in C, checks commands, tracks their progress, and applies motion rules. S3-specific platform code connects that core to the board’s motor outputs, network, storage, and media devices. This separation allows command rules to be tested without running the physical controller. The former simulator has been retired.

Eight hardware pulse channels drive the servos. Before playback starts, the firmware prepares the motion data and checks its size, timing, and joint ranges. Named movements and custom motion_plan_v1 plans use the same calibration, angle limits, stop behavior, and failure checks. A plan can contain at most 32 frames, each describing joint targets and timing.

The movement library includes stand, rest, a held sit, walking, turns, wave, bow, and dances including Macarena and salsa. The curious gesture looks right and left, then returns to its starting pose. Transitions ease between targets to avoid abrupt pose changes.

Named 45°, 90°, and 180° turns are open-loop estimates: the robot runs the sequence without measuring and correcting its final heading. They need measurement on the assembled body. V2 now has its own twelve-joint model, continuous gait planner and gesture library; its hardware qualification is tracked separately from these V1 clips.

Camera, listening, and speech#

The camera takes still images on request or after selected events. A limited operator preview starts disabled. Images can carry the ID of the movement or utterance that caused them, so MetaHuman can tell which result they describe. After a movement, the JPEG image is sent before the command’s final result. A failed snapshot is reported separately from a movement that already finished.

The microphone supplies digital audio samples. Voice activity detection (VAD) finds speech and ends an utterance after silence, subject to minimum and maximum duration limits. A queue in temporary memory absorbs short network stalls.

Wake-word detection is a separate step that would let a spoken phrase start listening. The runtime is present, but no accepted production Ainekio model weights are tracked. The initial configuration therefore leaves wake detection unavailable. Code being installed is not enough to make that feature ready.

Speech output arrives as PCM, the digital samples to play. The firmware accepts samples within an explicitly opened text-to-speech (TTS) burst, queues them, and sends them to the amplifier over the I2S audio interface. It reports queue overflow, interruption, samples arriving outside a burst, and completion. The camera and voice articles follow these paths through MetaHuman.

Setup, stored data, and recovery#

For initial setup, the owner joins the Ainekio-Setup WPA2 Wi-Fi network with a stable per-device key, then supplies Wi-Fi and gateway settings through a local form. New network settings are written and validated in an inactive storage slot before it is selected. If the write fails, the previous settings remain selected. Resetting network settings preserves calibration, poses, and battery-measurement settings.

LittleFS, an on-device filesystem, holds versioned motion, face, audio, and wake-model assets. A low-priority task writes limited rolling logs to microSD. Over-the-air (OTA) updates use two application slots so an unsuccessful new boot can roll back. The new image must reach gateway authentication before it is marked healthy for rollback validation.

Loss of the connection stops normal operation and begins limited reconnect attempts. Stale control traffic stops active motion while a healthy, authenticated idle session can remain connected.

Build and wiring reference#

The details below are for building or wiring V1. GPIO numbers identify board input/output pins. The S3 pin allocation must not be copied to the planned P4 without its own design and tests.

S3 firmware target, pin map, and media settings#

The entrypoint is Slave/firmware/esp32s3/main/app_main.c. It connects the C body core to ESP-IDF, Espressif’s device framework. Platform code owns FreeRTOS tasks, networking, nonvolatile settings (NVS), filesystems, GPIO, motor pulses (MCPWM), camera, display, and audio.

The target is ESP-IDF 5.5.4 on ESP32-S3 N16R8: 16 MB flash, 8 MB octal PSRAM at 80 MHz, two 3 MB OTA application slots, a large LittleFS asset partition, and size-optimized firmware settings. Flash retains code and data; PSRAM is working memory. Build reports track component size and memory use because camera buffers, audio queues, prepared plans, networking, storage, and update checks share limited resources.

The September 2 build reference and S3 pin map agree on these assignments:

Function	GPIOs
Eight servos	1, 2, 14, 21, 47, 48, 45, 46
OLED	SDA 0, SCL 43
Battery ADC	3
SD card	CMD 38, CLK 39, DAT0 40
I2S audio	Microphone 19, amplifier 20, BCLK 41, WS 42
Octal PSRAM	33–37 reserved

The OV3660 camera captures XGA stills; preview is limited to QVGA/VGA. Its fast profile uses short manual exposure for indoor 60 Hz lighting, moderate gain, denoise, neutral sharpening, and no auto-exposure settling delay. This replaced a higher-gain profile that produced cross-hatch artifacts.

Microphone capture is 16 kHz mono PCM. A version-pinned neural VAD model uses minimum-duration, silence, and maximum-duration limits. The microWakeWord / TensorFlow Lite Micro runtime validates a versioned local model package, including its manifest and tensor format, before reporting readiness.

What still needs physical testing#

Portable-core tests, protocol fixtures, firmware policy tests, ESP-IDF builds, flashes of the application and asset filesystem, and digest readback have checked code and installed images. A digest comparison checks that flashed bytes match; it does not establish how the assembled robot behaves.

Servo alignment, battery performance, microphone acoustics, speaker continuity, camera quality, and stability with motion and media together need physical checks. The dated status report records reported results, while the V2 roadmap describes the next controller.