Ask / Conversation
Speech-driven questions with generation-owned requests and stale-result rejection.
Experimental multimodal system
A hybrid browser experiment combining motion, local object tracking, semantic labels, speech, and hosted visual analysis. Its central problem is evidence identity: the frame the user sees, the object a label names, and the window Ask analyzes can still be different.
Research prototype. Perception paths work end to end, while provenance, accuracy, browser breadth, and long-session behavior remain under evaluation.

One captured result from the working interface. It demonstrates the interaction, not general recognition accuracy.
Recorded product states


These are individual captured states. They show the product flow, not a general recognition accuracy result.
01 · Current implementation
Speech-driven questions with generation-owned requests and stale-result rejection.
Motion-triggered visual interpretation with bounded work and mode invalidation.
Automatic local object candidates, short-lived ByteTrack IDs, and bounded Florence attempts.
Camera start/stop, lifecycle cleanup, local floating labels, and bounded semantic attempts exist. AirScript, Spatial Lasso, and Spatial Ask are not mounted in the current default product. Ask is not grounded to a user-selected region, and app-wide local-only processing would be an inaccurate claim.
02 · Central argument
The user sees one scene. The implementation handles multiple time windows, coordinate systems, model boundaries, and lifecycles. Without a shared immutable identity, visual continuity becomes a presentation property rather than an evidence guarantee.
03 · Privacy and trust
04 · Runtime topology
Current integrated product. Ask, Watch, and Microscope share a surface but do not share one immutable evidence record.
05 · Orchestration
Model execution alone is not trustworthy product behavior. The scheduler, freshness policy, memory bounds, and final commit check are what keep slow inference from becoming stale UI.
06 · Temporal identity
Current evidence supports short-lived track IDs, but controlled occlusion, overlap, camera motion, disappearance, and re-entry performance remain unproven.
07 · Ask provenance
Visible continuity
A persistent label beside a live scene can make the later answer feel grounded to that same object.
Frames are captured after the transcript. No immutable frozen frame, exact crop, region ID, visible track ID, or Microscope best-frame record is attached to the answer. The answer can therefore describe a different evidence window from the label the user sees.
08 · Microscope
Automatic no-click operation, loopback inference, latest-frame backpressure, bounded raw-image memory, failure-stop behavior, visibility cleanup, and floating labels.
Broad recognition quality, calibrated confidence, selected-region magnification, pixel-level citation, or shared Ask evidence. Semantic analysis may take seconds and labels may be wrong or sticky.
09 · Governance failure
Deterministic hand, object, relation, lifecycle, and rendering contracts established an isolated evaluation vocabulary.
Historical preparationReadiness evidence must be commit-pinned to the active production graph.
10 · Evaluation architecture
Deterministic logic, schemas, geometry, policies, and normalizers
Current product integration
Several tests proved isolated contracts rather than the active product graph. Test count alone could not establish that a feature remained integrated.
11 · Qualified performance
Retained short Apple-hardware samples
Historical short samples; requires commit-pinned reproduction.Historical local-model evidence
Separate stage, not end-to-end response time.No idle baseline or controlled comparison
Not an energy-efficiency claim.The next valid benchmark is a commit-pinned stage budget covering capture, queueing, detector time, semantic time, dropped work, memory, and energy per completed task.
12 · Architecture conclusion
Implemented · parallel paths
Ask and Watch own a frame window. Microscope owns detector tracks and crops. Gesture artifacts belong to another lineage.
type VisualEvidence = {
frameId: string;
capturedAt: number;
source: { width: number; height: number };
transform: { mirrored: boolean; visibleWidth: number; visibleHeight: number };
trackId?: string;
region?: { x: number; y: number; width: number; height: number };
cropId?: string;
inferenceId?: string;
model?: { name: string; version: string };
processingBoundary: "browser" | "local-service" | "hosted-service";
};13 · Current status