When Ainekio takes pictures, how they reach MetaHuman, and how an image is used to check a robot objective.
Published Updated 6 min read
Body Control’s experimental depth source uses MoGe-2 and retained visual landmarks to estimate local motion for gait learning. This is a separate host measurement path from conversational image interpretation. It is activated, but metric accuracy is unverified and failed tracking intervals remain unavailable. The measured ground-plane method remains an alternative. Camera-origin motion becomes body-origin motion only when the mounting transform is supplied; neither source establishes physical foot contact.
Goal-directed locomotion distinguishes image-relative targets from metric distance. Ground-plane estimates require real camera calibration and an explicit target contact point; detection boxes alone do not supply depth. Visual-only endpoints can use available camera evidence without metric calibration or YOLO. Physical arrival accuracy remains unqualified.
The shared Vision category retains its selected provider and image input; unsupported image input no longer silently switches to Orchestrator. The systems report describes that category boundary and the current role assignments.
The P4 can now save its preview JPEGs as segmented SD video and download clips through Body Control. A dated hardware run verified concurrent preview, recording and download over 469 seconds without reconnecting. This short result does not establish endurance or visual-task completion.
Camera and image objectives, including mixed requests, route to the configured larger vision planner. The 0.8B Task specialist handles nonvisual planning and can request delegation. October 14 model qualification remains separate from physical vision acceptance.
The P4 camera now feeds a desktop YOLOE/ByteTrack pilot with exact-image overlays and search/select/release. A stationary chair track was observed, while camera recovery and the complete acquisition workflow remain unfinished. Read the dated tracking record. The guide below preserves the earlier snapshot-driven design.
Ainekio’s camera gives MetaHuman OS, its conversation and decision-making software, a view from the robot. The current system takes still images on request, after selected movements, or when a spoken utterance ends. It does not continuously send room video to the model.
Each picture carries an ID linking it to the event that requested it. That connection matters: an image taken before a movement cannot show the result of that movement, and an unrelated image should not be used to judge it.
The operator dashboard has a separate, low-rate preview for a person inspecting the camera. Preview starts off and does not replace the images tied to actions.
| Trigger | How the picture is matched |
|---|---|
| MetaHuman or dashboard requests a snapshot | The picture uses the request ID |
| A movement completes with a snapshot requested | The picture uses the command sequence ID |
| Voice activity detection ends an utterance | Audio and picture share the robot’s utterance ID |
Voice activity detection (VAD) finds the end of a spoken segment. Pairing an image with that segment lets MetaHuman receive what was said and what the camera saw as part of the same conversation turn.
For a motion snapshot, camera metadata and the JPEG are sent before the robot’s
done result. The gateway carries them from the robot; Ainekio’s adapter
translates them for MetaHuman’s Environment Bridge, its external-device
connection. The resulting context includes the frame ID, originating action or
utterance, supported features, and current body state.
MetaHuman’s saved workflow—the software steps handling one request—can then wait for that specific result and image.
Consider a future request to move into better light. A successful walk result would establish that the command finished. A fresh image would be needed to judge whether the lighting improved. This is an example of the intended review process, not a visual goal already demonstrated on the revised robot system.
MetaHuman records the required evidence as visual_observation for a visual
goal. A command-only goal can use action_result. Its Action Result step
interprets the returned feedback, then Goal Review decides whether the
objective is complete or needs another step. An inconclusive image may lead
to another validated action rather than a claim of success.
September workflow code handles results and images arriving in different orders. If an action is rejected, expires, or is cancelled, review can proceed without waiting for a picture from movement that never happened. Controlled software tests cover those cases. Actual camera quality and visual goal completion still need physical testing.
The September 8 live review found available images attached to later Autonomy Controller calls. A follow-up compared saved image-content hashes with the inputs to Action Result, the next Controller, and the Environment selector. That established that the captured bytes reached those model requests.
Source repairs also keep the originating instruction alongside the returned image, even when a standalone action has no saved broader goal. Controlled workflow tests use a returned image distinct from the starting image to check that the request and new evidence survive dispatch and resumption.
These checks establish delivery and continuity. They do not show that the model interpreted the scene correctly or that the robot achieved a visual objective. Those questions still require a new deployed run with an observable outcome.
The working V1 uses an OV3660 camera on its ESP32-S3 controller. It takes XGA (1024×768) JPEG stills. JPEG compression keeps images within the network payload limit while retaining useful detail. Preview uses the smaller QVGA or VGA formats at a capped requested rate.
One image buffer in PSRAM, the board’s temporary working memory, holds the capture. The camera remains idle between frames to limit memory and bandwidth use. The fast capture profile uses short manual exposure for common indoor 60 Hz lighting, moderate gain, denoise, neutral sharpening, and no delay for automatic exposure to settle.
An earlier high-gain, sharpened profile produced cross-hatch artifacts. Reducing gain and removing sharpening improved the capture. Lighting still needs real-room evaluation, and the planned P4 body needs its own camera and communication integration.
Ainekio does not currently advertise target-relative navigation or visual
approach: it cannot be assumed to locate an object and move to it just because
it can return a picture. A findObject concept elsewhere in MetaHuman does
not supply those abilities to this robot.
Earlier Active View research explored detection and temporal segmentation, which follows image regions across frames. It remains outside the current production path. A tracked detection alone does not establish the object’s identity or show that a requested goal has been met.
Repeat a visual objective while recording the accepted request, required evidence, action ID, robot acknowledgement, final command result, and matching fresh image. Each review should identify the actual frame used to judge the goal. If the condition is absent, further action should stay limited; completion requires an image that establishes it.
MetaHuman integration explains saved workflows. The dated status report separates reported software tests from physical results.