← Selected work

Experimental multimodal system

Camera Harness

Building an evidence boundary for live visual interfaces

A hybrid browser experiment combining motion, local object tracking, semantic labels, speech, and hosted visual analysis. Its central problem is evidence identity: the frame the user sees, the object a label names, and the window Ask analyzes can still be different.

Research prototype. Perception paths work end to end, while provenance, accuracy, browser breadth, and long-session behavior remain under evaluation.

Camera Harness product interface showing Miguel raising a phone and a movement result awaiting confirmation.
Recorded state Raised phone

One captured result from the working interface. It demonstrates the interaction, not general recognition accuracy.

Recorded product states

Two movement results from the working interface.

Camera Harness product interface showing a live camera preview with a mug and a movement result awaiting confirmation.
Mug movement · awaiting confirmation
Camera Harness product interface showing a live camera preview, a raised peace sign, and a movement result awaiting confirmation.
Peace-sign movement · awaiting confirmation

These are individual captured states. They show the product flow, not a general recognition accuracy result.

01 · Current implementation

Observe → propose → track → question → confirm or reject

The latest integrated product contains three concepts: Ask / Conversation, Watch / Observing, and Microscope. They share an interface, but they do not yet share one evidence record.
Verified

Ask / Conversation

Speech-driven questions with generation-owned requests and stale-result rejection.

Verified

Watch / Observing

Motion-triggered visual interpretation with bounded work and mode invalidation.

Partial

Microscope

Automatic local object candidates, short-lived ByteTrack IDs, and bounded Florence attempts.

Camera start/stop, lifecycle cleanup, local floating labels, and bounded semantic attempts exist. AirScript, Spatial Lasso, and Spatial Ask are not mounted in the current default product. Ask is not grounded to a user-selected region, and app-wide local-only processing would be an inaccurate claim.

02 · Central argument

A visual interface can appear continuous even when its underlying evidence is not.

01Captured frame02Visible video frame03Motion event04Detector output05Detection box06Track ID07Best crop08Semantic label09Speech transcript10Ask frame window11Answer12User confirmation

The user sees one scene. The implementation handles multiple time windows, coordinate systems, model boundaries, and lifecycles. Without a shared immutable identity, visual continuity becomes a presentation property rather than an evidence guarantee.

03 · Privacy and trust

Local is a mode boundary, not a blanket promise

Microscope inference is designed to stay on the local machine, and raw image buffers are bounded and discarded. Ask and Watch may use hosted visual inference depending on configuration, and browser speech-processing locality depends on the browser.

Verified controls

  • Media tracks are explicitly stopped.
  • Requests are aborted and stale generations are rejected.
  • Collections and Microscope image buffers are bounded and cleared.
  • Local object processing uses loopback services.
  • Provider tokens remain server-side.
  • Test instrumentation is sanitized and bounded.

Required qualification

  • Ask and Watch may transmit JPEG frames to hosted inference.
  • Browser speech locality is browser-dependent and unknown.
  • No persistence does not mean no transmission.
  • Base64 images remain sensitive while in memory.
  • Confidence tiers are heuristic, not calibrated.
  • No broad security or browser-support guarantee is made.

04 · Runtime topology

One browser surface, several processing boundaries

The architecture spans browser signals, loopback services, optional hosted inference, persistent text settings, and transient media. Neural Field belongs to the historical record, not this current topology.
Current

Current integrated product. Ask, Watch, and Microscope share a surface but do not share one immutable evidence record.

05 · Orchestration

Cheap continuous signal → bounded expensive work → visible commit

Continuously running the deepest model would waste compute and increase stale work. This system favors freshness: the latest frame replaces queued work, semantic requests have budgets, and every asynchronous result must still belong to the active generation before it can alter the interface.
Cheap-to-deep processing path
  1. 01 Pixel differencing Detect possible visual change · Browser
  2. 02 YOLO-World Propose object candidates · Local service
  3. 03 ByteTrack Maintain short-lived identity · Local service
  4. 04 Florence-2 Bounded semantic refinement · Local service
  5. 05 Hosted / local VLM Interpret broader scene or movement · Hybrid
  6. 06 Generation check Reject stale work before visible commit · Browser

Model execution alone is not trustworthy product behavior. The scheduler, freshness policy, memory bounds, and final commit check are what keep slow inference from becoming stale UI.

06 · Temporal identity

Short-lived continuity, with explicit limits

YOLO-World proposes detections. ByteTrack assigns service-side IDs with an approximately two-second expiry policy. The browser scores candidates, keeps bounded best-frame crops, and rejects stale semantic results before updating floating labels.
DetectionTrack IDBest cropSemantic attemptFloating label
Current evidence supports short-lived track IDs, but controlled occlusion, overlap, camera motion, disappearance, and re-entry performance remain unproven.

07 · Ask provenance

Interface continuity is not evidence continuity

Ask owns its generation, scheduler, memory bounds, and speech. The missing piece is an immutable frame or crop that the user can inspect and that every downstream answer must cite.
Current Ask flow
  1. 01 Final transcript Speech intent
  2. 02 Generation-owned request Mode-scoped work
  3. 03 Fresh short frame window Captured after transcript
  4. 04 Hosted / local analysis Configuration-dependent
  5. 05 Schema normalization Bounded response
  6. 06 Stale-generation check Commit or reject

Visible continuity

One object. One answer.

A persistent label beside a live scene can make the later answer feel grounded to that same object.

  1. 01Visible label
  2. 02Spoken question
  3. 03Answer appears attached to the scene

Frames are captured after the transcript. No immutable frozen frame, exact crop, region ID, visible track ID, or Microscope best-frame record is attached to the answer. The answer can therefore describe a different evidence window from the label the user sees.

08 · Microscope

A local object-label pipeline, not a magnifying glass

Microscope starts or reuses the camera, opens a local perception session, captures a 416-pixel frame on a motion-adaptive cadence, runs YOLO-World and ByteTrack, scores candidate tracks, retains bounded best crops, and invokes Florence under a strict budget.

What is implemented

Automatic no-click operation, loopback inference, latest-frame backpressure, bounded raw-image memory, failure-stop behavior, visibility cleanup, and floating labels.

What is not established

Broad recognition quality, calibrated confidence, selected-region magnification, pixel-level citation, or shared Ask evidence. Semantic analysis may take seconds and labels may be wrong or sticky.

09 · Governance failure

When the test system outlived the product integration

The strongest lesson was not a model result. It was discovering how passing artifacts and screenshots could survive after the features they described left the active product graph.
01

Contracts and fixtures developed

Deterministic hand, object, relation, lifecycle, and rendering contracts established an isolated evaluation vocabulary.

Historical preparation
Readiness evidence must be commit-pinned to the active production graph.

10 · Evaluation architecture

Broad coverage did not prevent evidence drift

The repository contains contract tests, schemas, geometry policies, service tests, fake-media browser tests, visual regression, race scenarios, privacy scans, physical traces, and real-model samples. Each answers a different question.
Establishes

Deterministic logic, schemas, geometry, policies, and normalizers

Does not establish

Current product integration

Several tests proved isolated contracts rather than the active product graph. Test count alone could not establish that a feature remained integrated.

11 · Qualified performance

Performance without one misleading FPS number

Retained measurements describe different stages, commits, and environments. They are useful as historical observations, not current product guarantees.
Historical

YOLO local detector

Tens of milliseconds

Retained short Apple-hardware samples

Historical short samples; requires commit-pinned reproduction.
Historical

Florence semantic pass

Several seconds

Historical local-model evidence

Separate stage, not end-to-end response time.
Historical

Contextual power sample

Approximately 5.5 W

No idle baseline or controlled comparison

Not an energy-efficiency claim.

The next valid benchmark is a commit-pinned stage budget covering capture, queueing, detector time, semantic time, dropped work, memory, and energy per completed task.

12 · Architecture conclusion

Give every answer an evidence address

The current product has parallel camera paths. The proposed architecture introduces one immutable frame record and a shared track-and-region evidence store. It is a recommendation, not a shipped feature.

Implemented · parallel paths

Several camera consumers, no shared evidence address.

Ask and Watch own a frame window. Microscope owns detector tracks and crops. Gesture artifacts belong to another lineage.

  1. 01Ask / Watch frame window
  2. 02Microscope detector and track path
  3. 03Historical Neural Field artifacts
  4. 04No shared immutable evidence identity
type VisualEvidence = {
  frameId: string;
  capturedAt: number;
  source: { width: number; height: number };
  transform: { mirrored: boolean; visibleWidth: number; visibleHeight: number };
  trackId?: string;
  region?: { x: number; y: number; width: number; height: number };
  cropId?: string;
  inferenceId?: string;
  model?: { name: string; version: string };
  processingBoundary: "browser" | "local-service" | "hosted-service";
};

13 · Current status

What works, what remains open, and what is no longer current

The case study ends at the real boundary of the evidence. That boundary is more useful than promotional certainty.

Works in implementation

  • Explicit camera lifecycle ownership
  • Motion gating and speech-driven Ask orchestration
  • Automatic local object candidates and short-lived track IDs
  • Bounded semantic attempts and transient image memory
  • Stale-generation rejection

Partial or unproven

  • Recognition accuracy and occlusion robustness
  • Region-grounded Ask and shared Microscope evidence
  • Calibrated confidence and broad class coverage
  • Cross-browser behavior, accessibility conformance, and long-session resources
  • Controlled energy efficiency and latency distribution

Not current

  • Neural Field active UI
  • AirScript active UI
  • Spatial Lasso active UI
  • Spatial Ask in the default launcher
Next case study · Atlas →